The story of Xu Fu 徐福 has stimulated the imagination of both Chinese and Japanese artists and scholars for many centuries. The legend first appeared in Japan in the latter half of the Heian period (794–1186) and its popularity grew with increased circulations of associated texts in subsequent years. The boom reached its peak during the Tokugawa period (1603–1868) when scholars, writers, and artists were actively engaged in the discourse, showing the rise of national consciousness and the complicated feelings towards China among Tokugawa Japanese. Scholars of Xu Fu in both Japan and China have hitherto examined the Xu Fu story with an archaeological or philological emphasis, paying special attention to the issue whether Xu Fu had actually landed in Japan. This paper, however, looks into the legend from an intellectual and textual perspective. It studies the significance of the Xu Fu discourse in relation to the Tokugawa history and the Sino-Japanese relations. Xu Fu has become a metaphor in Sino-Japanese discourse, a subject matter that scholars take advantage of in their discussion of bilateral ties. Two major different accounts of Xu Fu are found in Tokugawa writings—Xu Fu as a transmitter of Chinese culture to Japan and Xu Fu as a refugee to Japan. To a certain extent, these two accounts represent diverse views that the Tokugawa intellectuals conceived of the Sino-Japanese cultural relationship. Through a textual analysis of Tokugawa writings, this paper aims to examine the way in which the Japanese scholars of the time overcame the dilemma of accepting Chinese culture without compromising their national or cultural identity.