Pre-Qin's Confucianism and Taoism basically lay down the foundation of Chinese traditional hermitism, which experiences its greatest change when it fused with Buddhism during Southern Dynasty. This can be well proved that nearly half of the hermits in the historical records of Southern Dynasty more or less displayed Buddhism in their thoughts. The purpose of this article is to analyze why the traditional hermits were so willing to accept Buddhism, which was considered to be an exotic culture at that time. By examining the historical records of Southern Dynasty, we can find that Hui-yuan is the key person who connects hermitism and Buddhism. He secluded himself on Mount Lu for thirty years, similar to a hermit, and revealed both hermitism and Buddhism in his thoughts. Therefore, many hermits and celebrated scholars came to follow Hui-yuan during the late Jin and early Song Dynasty. This article first presents the basis for the thesis by surveying the biography of the hermits of Southern Dynasty and attesting that Buddhism had affected the thoughts and deeds of the hermits. Secondly, it discusses the similarities and differences between Hui-yuan's belief in Amitabha and hermits' ideas of returning to paradise and seeking for long life. Next, it examines Huiyuan's perspectives on Sramana's disrespect to the king and hermits' insistence on practicing morality and refusing to yield to the nobility. Lastly, a conclusion is presented.