In developing an automatic process to recover the relative orientation, tie points in each pair of stereo images should be identified. The least-square matching (LSM) has been widely used in the past few years but the fact that it tends to fail to match locally different images. Difficulties arise due to the influence of the image rotation and unequal exposures. The purpose of this study is to remedy the drawbacks of LSM. For identifying conjugate image points, we detect the line segment of building roof using the Hough transform and compute the intersections of the lines to obtain the building corners. The result will be invariant with respect to the size and rotation of the image. In order to solve the occlusion problem, a simple criterion is proposed to match conjugate building corners. In order to test the practicability of this method, some experiments on a pair of real stereo image are processed with this method. The results are compared with the numbers obtained by using LSM. The comparisions show that this method is more robust to the effect of local changes in the images. We therefore conclude that this method can be a backup of LSM in image matching.