Binary classification has been broadly investigated in machine learning. In addition, binary classification can be easily extended to multi class problems. To successfully utilize machine learning methods for classification tasks, preprocessing and feature extraction steps are essential. These are important steps to improve their classification performances. In this paper, we propose a new learning method based on weighted least squares. In the weighted least squares, designing weights has a significant role. Due to this necessity, we also propose a new technique to obtain weights that can achieve feature transformation. Based on this weighting technique, we also propose a method to combine the learning and feature extraction processes together to perform both processes simultaneously in one step. The proposed method shows the promising performance on five UCI machine learning data sets.