Theories of interpretation of the Yijing (Book of Changes) can be summarized as two different approaches: the philosophical approach and the "xiangshu" (images and numbers) approach. This paper describes the differences between these two approaches, and, concentrating on the latter method, further defines ten categories of images and three categories of numbers of the Yijing. Through analyzing and comparing the examples within the commentaries of the Han philosophers, Wang Bi and the Song philosophers, this paper draws the following conclusion: Rather than being invented by the Han Confucians, the xiangshu theory came into existence when the Yijing was edited. The hexagram and line texts comprised different kinds of mystical numbers and images that were too effective and valuable, for ancient traditions of historians, astronomers, and calendricists, to be neglected in cultural history. The xiangshu approaches and the philosophical approaches are two parallel lines with no conjunction since both of them are based on some specific concept of world order. Wang Bi criticized the xiangshu theory of the Han philosophers, but he also limited himself unconsciously by their xiangshu theory so that his cosmological concern overcame his concern for humanistic values.