I think in some cases, it gets used for "someone saying stuff I'm scared of admitting to myself I agree with because it'll raise all kinds of questions about my worldview that I'm desperately trying to avoid having to deal with". Then it gets even more difficult to have a conversation. The accusation also comes up a lot when someone challenges, even mildly, the prevailing groupthink of a place. Especially when they're a bit odd, or awkward, or spiky to start with. But then, I've noticed that there's a very English attitude of "we'll accept you so long as you keep your head down, but the second you step out of line any distinguishing characteristic is fair game".

My own personal maxim is that if anyone calls me a troll, I'm gone. I try never to be anything other than genuine, but if I've been that bad at communicating (which is always possible!), it seems unlikely I'm going to find the magic words to heal the rift... and if, on the other hand, I haven't actually said anything to warrant the accusation, whatever issues the accuser is dealing with are both not my problem and beyond my reach. Either way, there's simply no possibility of a useful continuation of the debate.

(I suppose another way to put that is that I'm ridiculously easy to bully into silence...)