Based upon previous researches, the testing topics evaluated, problem solving tactics and steps involved are three major factors that influence the difficulty of the math tests. Combining these three factors, previous researches also indicated how to predict the difficulty of a test. We found that these predictions are quite helpful, but still there were about 10% of the test items, the predictions were less accurate. The main purpose of this research is to point out that the examinee’sfamiliarity to the content of the test items is another important factor. Adding this factor to the original three, we suggested a Factor-Indexed Model to increase the prediction accuracy. Furthermore, the calculation of solving steps for a multiple-choice test item was always a major obstacle to the prediction of difficulty. We also address this issue in this research. We picked the math exam items of the previous years (2002~2007), for which the examiner’s difficulty prediction and examinee’s response had large variance, and tested by our Factor-Indexed model and the calculation of solving steps, we found that our research would increase the accuracy of the prediction. Nevertheless, there is still prediction bias to some test items by our methods, and we hope that future researches will improve further so that one can have better control of the difficulty prediction.