In the background of China’s urbanrural dual structure, population flowed out of the traditional villages, leading to a lack of endogenous power in the conservation and development of rural areas. However, along with the weakening of urbanrural barriers, some outsiders who flowed out from their villages came back with new resources and dreams to rejuvenate traditional villages. Based on the theory of rural elites and the discussion of public, shared, and private zones in rural governance research, this paper, taking G Village in H Province as an example, conducts an in-depth analysis on the relationship between the outsiders and insiders of rural governance. Additionally, it illustrates that the rural elites coming back from the outside world would have a positive impact on the rejuvenation of the village, but they also met some difficulties in reality. Finally, based on the specific activities of traditional village protection and development, this paper puts forward some suggestions for solving the difficulties, including shape an elite culture, establish a forum for the elites, and corporate with the insiders, etc., which are of positive significance to innovate the model of village governance, revitalize the development and construction of traditional villages, expand the theoretical perspective of village planning, and so on.