【目的】通過實驗對比分析,比較不同停用詞表對于不同類型的文本數據的作用效果,對停用詞表的構建與使用提供參考意見。【方法】選取百度停用詞表、哈爾濱工業大學停用詞表以及四川大學機器智能實驗室停用詞表,基于三個不同語料庫運用漢語分詞技術、TF-IDF特征評估函數以及VSM模型進行文本處理,并且采用Java編寫的K-means算法進行聚類實驗,通過準確率P、召回率R和F1三個評價指標對不同聚類結果進行效果評估。【結果】不同停用詞表對于不同類型的文本數據作用效果差異明顯,詞表的長度、內容結構是影響作用效果的直接因素,其中兩字停用詞作用效果最為明顯。【局限】實驗文本類型及數量有限,同時對于不同停用詞表僅在詞語數量及內容上做了簡單的分析比較,未對停用詞按照類別分類進行實驗分析。【結論】停用詞表對于文本聚類準確度有很大的影響,構建或選取適宜的中文停用詞表極為重要。同時,過度增加停用詞的數量并不會一直改善聚類結果。

[Objective] This paper compares and analyzes the impacts of stopwords on textual data processing, aiming to improve the construction and use of stopwords. [Methods] We obtained stopword lists from Baidu Search Engine, Harbin Institute of Technology and the Machine Learning Laboratory of Sichuan University for this study. First, we processed text message with the stopword lists and Chinese word segmentation technique, the TF-IDF feature evaluation function and the VSM vector model. Secondly, we analysed the texts with the K-means algorithm to calculate the P, R and F1 values. [Results] Different stopword lists posed various effects to the text data processing tasks. The length of the list and the content structure of the texts directly influenced the clustering results. More importantly, the two-character stopwords was the biggest factor. [Limitations] The text types and quantity were limited. More research is needed to analyze the text with different types of stop words. [Conclusions] Stopword list poses significant impacts on text clustering, thus, it is extremely important to build or choose the appropriate Chinese stopword list. However, excessively increasing the number of stop words might not always improve the clustering results.