Dunland, home of the evil, hostile, barbaric men. Am I right? No, Dunlendings are misunderstood.
Rohirrim are good, right? They are fighting in the war against evil, side by side with Gondor. But invading a peaceful land already inhabited for generations by the Hill-folk doesn't seem overly heroic, even if it was granted to them by their friends..
Neutral with all but five
Most of the factions of Middle-earth are neutral to the primitive Dunlendings. The Dwarves of Ered Luin, the Hobbits, and Fangorn, along with the majority of the evil factions. The Haradrim and all current southern factions are neutral with them, too, but this is not surprising since there are hostilities between Dunland and Gondor. The Elves also don't attack the Dunlendings unless ordered to, and are also neutral.
Dunlendings are allied with the Uruk-Hai simply because they, too, hate the Horse-Lords. As attacking Mordor would enrage the friendly Uruks, and as the Dark Lord has offered help in re-conquering the grassy fields of Rohan, Dunland is neutral with Mordor. As a result, even Rangers attack them.
Durin's Folk have got it right, though, being allied with Dunland. Their history also has something to do with it, for the Dwarves of Erebor eventually came to dwell in Dunland after Smaug had taken the Lonely Mountain.
You can argue that Dunlendings attack the Rangers, causing them to hate them even more, but we are talking about a primitive, barely civilized tribal nation, isolated by the vast emptiness of Enedwaith and the immense wall of the Misty Mountains. We can understand if they are nervous of the outside world.