Science Fiction: The Dervish House, by Ian McDonald
This was an insanely difficult choice, and that makes me happy. Both of my honourable mentions would have made worthy picks -- Anathem by Neal Stephenson is the sort of experience that'll stay with me for a long time, and Elizabeth Bear's Jacob's Ladder series (Dust, Chill, and Grail) is some of the best space opera I've ever read.
In the end, though, I'm going with The Dervish House by Ian McDonald (my review here). That's mostly because I think it's more accessible than Anathem. Either you've already read Anathem (or you're planning to), or you'll never consider it; I doubt my recommendation will change that. But I might just be able to convince you to give The Dervish House a look.
Honourable mentions: Anathem by Neal Stephenson, the Jacob's Ladder series by Elizabeth Bear.
Fantasy: Mechanique, by Genevieve Valentine
Calling Mechanique a fantasy novel feels like a gross oversimplification -- it's more like a horror-tinged post-apocalyptic science-fiction-ish fantasy. With a hint of steampunk. Or something. Whatever you want to call it, it's great.
The Mechanical Circus Tresaulti is a last bit of magic in a world ground down by endless war. But the magic has a sinister edge, and the Circus fights constantly to hold itself together, against the world and against a simmering internal conflict between two of its performers.
The characters here are rich and complex, but the writing is sparse and sharp-edged. The story is grim, but really compelling. It's Genevieve Valentine's first novel, and I can't really recommend it enough. I'm really hoping it pops up when awards season rolls around in 2012.
Honourable mentions: The Hundred Thousand Kingdoms by N. K. Jemisin, Zoo City by Lauren Beukes.
Other Thing: the Eclipse anthologies, edited by Jonathan Strahan
I didn't read enough literary fiction in 2011 to call it out separately, but I did read a bunch of comics and games and short story collections, so I've decided to go with Other Thing as my last category. I want to recommend to you the Eclipse anthologies, edited by Jonathan Strahan -- I read Eclipse One and the better part of Eclipse Three in 2011, and Eclipse Two a few years back.
They are a series of non-themed genre anthologies, and I think they're a really great way to expose yourself to a wide variety of unusual, interesting stuff. I've discovered a number of new favourite authors in the Eclipse anthologies, as well as read some really great stories. The absence of strict genre boundaries is really refreshing -- it makes starting each new story a small adventure. A great way to broaden your reading, I reckon.
Honourable mentions: Ghostopolis, a graphic novel by Doug TenNapel, and Do: Pilgrims of the Flying Temple, a storytelling game by Daniel Solis.
So that's it! Time to think about which books I'm going to nominate for the Hugo Awards in 2012...