Sins of a Solar Empire/WMG

The Vasari enemy is an army of rogue AI constructs
Between the Flotilla fleeing from the Geth in Mass Effect and the Colonial Fleet legging it from the Cylons in Battlestar Galactica, the Vasari story has some strong precedent resonance.
 * Considering the fact the Vasari make heavy use of nanotechnology, this sounds plausible at first, but when you really think about it long enough, it is pretty unlikely. Only a single Vasari warship ever survived contact with whatever this enemy is, with the ship's crew was driven made with fear. Think less Cylons/Geth and more along the lines of Cthulhu and the Elder Gods. Whatever the Vasari are running from ain't pretty and is far more dangerous than anything you or I could fathom.
 * Ah, so it's the Reapers, then.
 * Once again, probably not. The Vasari use nanotechnology all the time. Getting bit in the ass by it wouldn't drive anyone insane, though it might be slightly surprising at first. If we humans who haven't even begun to master nanotech can see its possible dangers, so can the Vasari. They've probably already figured out how to root out that nasty "destroy all life" bug. Anyway, the Reapers are basically a race of ancient sentient capital ships, little more. A reaction to them would be more along the lines of "oh shit, we're f***ed" as apposed to clawing one's eyes out. The Reapers barely manage a collective BOSD among the Citadel races, coming nowhere near mind rape territory. Yes Reapers are big, bad, and powerful, but they are still perfectly acceptable within a reality filled with space battleships. The Vasari are up against a big unknown, something that violates their views on reality so totally that it drives them mad. This troper stands by his original claim: the Vasari are running from something with powers beyond imagination; something having more in common with a black hole, supernova, or pulsar than a simple armada of warships.
 * The mad crew does fit with Reaper indoctrination though.
 * I'm not taking these particularly seriously, obviously, but as attempt number 3, I can think of a force of nature capable of driving its opponents into a frenzy with fear: the Oncoming Storm. Or the Doctor to his friends. While in recent episodes of Doctor Who he's viewed with almost supernatural fear by some of his enemies, it's implied by River Song that he'll grow still more terrifying to them as he continues to battle the evils of the universe.
 * How about the Zerg swarm. The Vasari might have trouble wrapping their minds around a huge swarm of alien locusts devouring everything in their path.
 * Or the Tyranids. Or the Necrons.
 * If we're doing 40K, Eye of Terror, anyone? Maybe the Vasari made the same mistake as the Eldar.
 * Or maybe the slave races rebelled and used similar weapon to the advent and drove that ship mad.

The Vasari enemy is a group of rogue Advent
Their minds have become so alien, they are Cosmic Horrors in all but appearance. They are also insanely powerful. For all we know, it could be a single ship. Now THAT would be terrifying.
 * Doesn't work. The Vasari were attacked 10,000 years ago, the Advent were only exiled a thousand years ago. Could be a similar set-up from within their own society though.
 * Actually, the intro stated they arrived ten years prior to the game.

the Vasari enemy is the Elder Things

 * the Elder things and Shoggi/ Shoggoths (what's the plural for those nasty gobs of protoplasm anyways?) are quite resilient. if they can survive unaided in space, I'm pretty sure they can royally fuck up the Vasari. besides, it would be pretty scary to watch 999 of your comrades ships being eaten by gobs of sentient tar from the ass-end of sanity and reason.

The Vasari are a lost splinter colony of the Ancients

 * Think about it. Subspace functions similarly to Phasespace (minus the wormholes), and the Vasari and Ancients both developed the means to travel through Phasespace and Subspace, respectively. Also, the Vasari had conquered and enslaved other less-advanced races just as the Ancients did. Perhaps the Vasari were refugees that had escaped the Shivan incursion and fled to the Inner Ring of the galaxy. They had developed Phasespace technology possibly because they had lost Subspace capabilities. The fall of their empire was probably because of a Shivan fleet that had tracked them to where they had fled, and it is the Shivans that have forced them into a nomadic state.

The Vasari enemy is a cosmic-scale version of Leliel
Think about it. Planets vanishing? Ships becoming Lost Forever? Survivors being driven batshit insane? It ticks all the boxes.
 * Its vague enough to be a lot of things