Not a single Mole Person is met in the game, nor is it ever addressed what happened to them. The storyline, however, points toward the idea that the Mole People, like the summoners, would have had eidolons, and that it would have been Garland who destroyed them.
An excerpt from the Eidolon Wall in Madain Sari appears to support this as well:
This information is yet to be confirmed, but there was an eyewitness account of a previously-undiscovered eidolon. It was witnessed in Esto Gaza. If there is an eidolon that can exist outside of our legends, our theory would no longer hold true. But maybe there are other tribes that have legends of their own.
Garland has sealed Mount Gulug off for an unknown reason. However, since it is well known Garland feared eidolons and saw them as hindrance to his plans, it can be speculated that Garland was the one who destroyed the Mole People's village, much the same way he destroyed Madain Sari in order to rid Gaia of the summoner tribe. Proof that the Mole People had eidolons is in the fact that Mount Gulug contains an extraction circle used for extracting eidolons from a person. Only one other such circle exists in the world, in the basement of Alexandria castle, where the summoner tribe used to live before moving to Madain Sari.