There were 14 village alliances within the ritual sphere of the Yimin Temple in Fangliao during the Qing dynasty. One wonders why the ritual sphere excluded three other such village alliances, the 13 village alliances of Zhongli to the north, the Toufen Yimin sphere to the south, and the Hakka villages in Miaoli. In this study, I use data from the 1926 Survey of Ethnic-Han Taiwanese’s Ancestral Registry conducted by the Japanese government to investigate why the Fangliao ritual sphere was limited to the 14 village alliances. I will argue that reasons for the limitation lie in the different immigrant groups’ ancestral domiciles and their respective Hakka dialects. While Chaozhou (speaking in Raoping or Dapu dialects) and Huizhou Hakka (Hailu dialect) people dominated the Fangliao 14 village alliances, the Jiaying Hakka descents remained a minority within. The three village alliances not included in the Fangliao Yimin Temple’s ritual sphere are mainly of Jiaying descents.