Linear Discriminant Analysis (LDA) is a well-known technique for dimensionality reduction and classification, while the classical LDA formulation fails when the total scatter matrix is singular, encountered usually in undersampled problems. In this paper, regularized Least Squares LDA (RLS-LDA) based on the elastic net, is proposed to handle the problems, and the resulting models are robust and sparse. Firstly, the theories about linear regression and regularization are explored, and the equivalence relationship between the least squares formulation and LDA for multi-class classifications under a mild condition is summarized. Secondly, the construction of RLS-LDA is presented. Performance evaluations of these approaches are conducted on benchmark collection of text documents. Results demonstrate the effectiveness of the proposed RLS-LDA and it’s the RLS-LDA based on the elastic net that is better than others.