Doctor who cares wrote:
Ancestros of the Variags could have lived there, as the desert-roaming horse riding tribes lived in real life both in the north of the black sea (Khand) and in the mongolean desert (last desert). Perhaps a great mountain kingdon, perhaps older than friggin' gondor, could have lived in the mountains and steppes, wich were fertile. Then they refused to submit to sauron in the early second age. Sauron, with the help of some citizens of the realm loyal to him and melkor-worshipers sacked the mountain realm and sauron raized the steppes as punishmet, rendering then homeless and nomadic. Over time this people moved south (to Khand) to get closer to Sauron and to escape the wereworms (one of morgoth's creatons used by sauron to destroy the land). They became the Variags. Sauron released spirits (similar to the barrow-wights) in the mountains that still haunts then. By this day some remnants of that original people may still inhabbit the southern fields, with the mountain shielding then from sauron's corruption. The men of hildorien accepted then in their princedon and under instructions of the blue wizards the remnants and hildorien work together as a last stand in the east against Sauron, but with a deep hatred towards Khand.
Variags represent the real life Varangian guard ;) so middle-eastern vikings. However it is likely there would have been people living there before the Variags came of either Rhunic or Near Haradrim decent; or both. I will assume you mean these "Khandlings" or "Khandish Men", so yes, good theory.