Is distinctive voice always a good thing?

Obnoxious Distinctive Pinkie Pie

The other day, I talked about books that hooked me from page one. Why did they hook me? The same reason, every time: Voice. A well developed voice (be it author or narrator) is what grips the reader from the first paragraph and refuses to let go. Which of course is why authors crave it and why agents scream for it.

Voice is great. Distinctive voice, better. Really distinctive voice, best. Right?

From a reader's perspective I'd argue: not always.

Voice can be an instant turn-off for me. Particularly narrator voice. If I find a main character annoying, chances are I'm simply never going to like the book. In fact, I'm probably going to put it down right then and there. If I do keep going because I feel I have to, it's going to feel ... like I have to. It was that beautiful-much-sought-after-by-writers distinctive voice which suddenly made reading a chore. By developing an (overly?) distinctive voice for the main character, the author narrowed the audience for their book, excluding me. In that case, I'd much rather read a book with less voice, but killer characters, premise, and plot. But maybe I'm crazy?!?

What do you think about voice? Is distinctive voice always a good thing?
Can voice land you an agent, but limit your audience?