In the Cheng-Zhu School of Neo-Confucianism (朱子學), and even in Confucianism overall, discussion on ghosts and spirits, gui-shen (鬼神), involve two theses, qi (氣) and "ghosts and spirits using sacrifices to speak." However, Confucianists such as Zhu Xi advocated that "the two theses on spirits and ghosts are not two affairs" because the validity of the theses determines the structure of Confucian theory that man is an integral part of nature. But why aren’t the two theses on spirits and ghosts two affairs? First of all, in this paper I will expound my view on "the theory of spirits and ghosts in the rite of music" through the statement, "there are rites and music when it is bright, and spirits and ghosts appear when it is dark "(明則有禮樂,幽則有鬼神) from the Yueji 樂記 (Book of Music). Then I will discuss the theory that ghosts and spirits embody the logic of rites and music, thus integrating the Book of Rites (Liji 禮記) and the Yi zhuan 易傳 (Commentary on the Yijing) on the communication between heaven and humanity. Second, according to Zhu Xi, "the theory of spirits and ghosts is verily the theory of the heart;" from this I will indicate that the communion of humans and spirits, which is the purpose of seeking yin and yang during rites and sacrificing with a sincere heart, is not an artificial construct. Finally, within the realm of East Asian Confucianism, particularly among the Confucian scholars of Edo Period Japan, the thought that "the two theses of spirits and ghosts are not two affairs" was completely deconstructed. Scholars of Chinese Confucianism spared no effort in demonstrating that humanity is based on heaven, and that human relations are based on nature. Therefore, with a view to "reframing Shinto as a religion," they presented it as a marvelous creed in their annotations, claiming that it was the true way of rites and music, the way of the cosmos and of nature. By the same token, from the middle and later part of the Edo Period, Ogyū Sorai pioneered a more religious understanding based on the absolute faith and the integration of society.