I'm looking for techniques specific to a story with an unsympathetic main character. Confederacy of Dunces, for example. Why does that work, and why wouldn't that work with a likable hero?
Some great answers, but I disagree in some regards: interesting and compelling and understanding are not enough for someone to carry on reading about a main character. I can find a serial killer interesting and understand his motives, but if he's killing children, it's extremely unlikely I want to keep reading about him unless the author has tricked me into sympathising with him.
In any good book, the reader must sympathise, identify and empathise with the main character, and must want to be completely sucked into their shoes. Without these elements, your main character likely won't appeal, except perhaps as a villain, which is different (but avoid the "absolute evil" approach, as this simply makes him a caricature).
I don't think it's impossible to get the reader to sympathise with all manner of "anti-heroes", no matter how reprehensible. If your main character is so reprehensible that there is no sympathy at all for him from the reader (or even you, as the writer), then you probably should reconsider how your character is portrayed.
The reason for this is because the reader needs to follow this character every step of the way, and will him to succeed in his endeavours. If we have no sympathy for him, we will not want him to succeed. If we don't identify with him, we cannot relate. If we don't empathise, we don't care enough about him to want him to succeed. Worse, we don't care enough to want to even know whether he will or won't.
As an example, consider The Godfather. Right in the beginning, we are introduced to a man in court watching the men who raped and beat his daughter get off with a slap on the wrist. Justice has not been done. So he goes to Don Corleone who is at his own daughter's wedding, and asks the Don for his help. Don Corleone agrees to carry out justice on behalf of the man. We see the respect that Corleone and his family command from people.
Right there, in that instant, you as a reader have taken the side of Don Corleone. You sympathise with him, you can identify and empathise with him. Here is a mobster, head of a powerful crime family, responsible for the murder of many people, yet you sympathise with him, because he's a man of justice, a man with a code of honour that allows the reader to will him to succeed. You want to be drawn into his world.
A common way of invoking sympathy is by putting the character in danger: their life could be in danger, they could be under threat of losing their job, their wife - all manner of devices that immediately invoke sympathy from the reader.
Alternatively, you can portray the character in a way that everyone can relate to: perhaps he's experienced some hardship like divorce, a death in the family, some "wrong" has been committed against him that immediately gets the reader on their side.
Another trick is to use the character's reprehensible traits in a morally justifiable way. For example, consider the example I gave earlier about a serial killer. What if the serial killer only kills serial killers? That's exactly what happens in Dexter, and we are interested because, even though he's a serial killer, he kills only bad people. Suddenly, his actions take on a different moral light, and we actually will him to succeed.
I'm sure there are other ways. Pick up Confederacy of Dunces, and rethink how the characters are portrayed to induce sympathy in the reader. I haven't read it, but I'm sure you'll find what trick the writer used to get you onto the character's side.
I haven't read Confederacy of Dunces, but I think in general, it's not as important that your main character be sympathetic as that s/he be compelling. I can read a book about reprehensible characters and love it as long as I'm intrigued by the main character: Hannibal Lecter (Silence of the Lambs, etc.) is super-intelligent and creepily charming; Patrick Bateman (American Pyscho) is, well psycho, but also clever and just intriguing; Macbeth could have been so much more, etc. As long as they hold my INTEREST, they don't have to hold my affection.
Like any character, you need to be sure than an unsympathetic character isn't flat, and isn't a cliche. Can you introduce an element of humour, so that the character is wryly aware of his/her own shortcomings? Or can you show the character growing and changing through the story? It doesn't have to be a complete about-face, but just a little improvement can go a long way. Or could you give us enough background on the character so that we understand the nastiness, and manage to care about him/her anyway?
I think one important aspect of writing with unsympathetic characters is making it clear that the author KNOWS the character is unsympathetic. Nothing makes me throw a book across the room faster than finding a character that I can't stand but that the author seems to think is great. (I'm thinking of some of the alpha males in some romance novels, for example - if I'm yelling, "that's rape, you jackass!", the author shouldn't be going on about the character being a dashing rogue.)
As usual, Kate is spot-on. Also, you can make your unsympathetic main character more human and believable by going deep into his head, perhaps even using first-person point of view, to fully explore his motivations. Remember that even Hitler and Stalin did not consider themselves evil. They thought their actions were justified and considered their enemies the evil ones.
Both Kate and Kelly have some brilliant answers. To add my two-cents worth, I agree with both of them. Part of what makes a story so compelling to read (at least, for me) is the character development. Unsympathetic characters, as in your case, make a story interesting, as long as there is some insight into the character's motives and background. And yes, nothing is more frustrating than an author who doesn't know his characters well.
A good example that I have found is in George R. R. Martin's "A Song of Ice and Fire". The characters are beautifully presented. Not a single one is "perfect" and most of them border on downright ruthless. But what made it brilliant was the fact that you understood their motivations, even though you did not necessarily identify with it.
Another great thing is the slight changes in their personality, usually brought about by their interaction with particular scenarios or people. For example, an angry, bitter person "softens up" when he is in the company of the woman he loves. And he knows it. It is the introduction of these slight changes in personality, and what brings them about that allows us to really get to know a character.
I recently watched the movie "Greenberg" with Ben Stiller. His character is highly unlikable and at times I was really frustrated with the way he acted. But after a while he slowly opens up and you get to know what he is really about. It is a good movie with much attention given to the characters. Worth a watch.