Not everyone would prefer the life of a wandering fisherman, especially those trained in a academic circle which promoted scholars to the highest echelon of social hierarchy. Life of a fisherman involves a passive view of life and it certainly would not become a shared value and philosophy of life. However, when the external environment does not permit scholars to realize their ideals, it is understandable for them to pursue the life of a fisherman-recluse. When the Song imperial family fled from north to south, many scholar families in the process lost their original confidence in their own social status. In reality, the governmental posts were fewer, the new official class was on the rise, and the old scholars could only retire even though they did not want to do so. Returning became retiring; their mentality was just like that of a wandering fisherman. Fisherman has a different form of life from the literati-scholars. Why does fisherman's life become a subject in scholar's paintings? If it is a subject deliberately chosen by scholars, what significances do they have for the painters? Are the paintings of fisherman-recluse we can see today created under the same thought? Where does the tradition of fisherman painting come from? These are some issues this paper would like to explore through literary texts and relevant works of paintings.