The villagers are residents of Skytown. They are birds, like the main character. Many are seemingly unwilling to leave, most likely due to the ghosts' presence.
The birds of Skytown originally lived in a city atop the mountaintop, in-game called The Curtain, where the sun always shone. The birds were then threatened by the ghosts to leave the mountaintop city. In response to this, the birds hung up a giant curtain, believing it would keep the ghosts out. However, this was only to the benefit of the ghosts, who pushed the birds and the sun away from the mountain, forcing then to live on a cliffside town now known as SkyTown.
Ruins of this once-prosperous city can be seen on several occasions: most notably the mountaintop and the Library.