youpaishidifu/msgbot
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(kb): 搜索阶段过滤查询停用词，防止通用名词导致TF-IDF虚高

Browse Source 
问题：查询"服务器的公告"时，"服务器"的IDF=3.2虚高，
player.md的"运营方权利"段落含"服务器"多次→keyword_score=12.8，
压过index.md的"近期公告"（keyword_score=3.2）。

修复：在search()中使用AutoReply._STOPWORDS过滤查询token，
只保留"公告"等核心查询词参与评分。

效果："服务器的公告"现在正确匹配到index.md的"## 🔈近期公告"。
This commit is contained in:
youpaishidifu
2026-08-13 15:24:46 +08:00
parent 93c50ecef3
commit 8d648f37fb
1 changed files with 11 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
knowledge_base.py
+11 -3
View File
@@ -505,6 +505,14 @@ class KnowledgeBase:
if len(query_stripped) < 2 or len(tokens) == 0: if len(query_stripped) < 2 or len(tokens) == 0:
return [] return []
# ⭐ 过滤查询停用词：通用名词（服务器、玩家、游戏等）会导致TF-IDF虚高
# 修复"服务器的公告"被player.md"运营方权利"压过index.md"近期公告"的问题
from auto_reply import AutoReply
query_tokens_filtered = [t for t in tokens if t not in AutoReply._STOPWORDS]
if not query_tokens_filtered:
# 所有token都是停用词，回退到原始tokens
query_tokens_filtered = tokens
# 对每个扩展查询计算 TF-IDF，取最高分 # 对每个扩展查询计算 TF-IDF，取最高分
tfidf_scores = np.zeros(len(self.docs)) tfidf_scores = np.zeros(len(self.docs))
for eq in expanded_queries: for eq in expanded_queries:
@@ -512,11 +520,11 @@ class KnowledgeBase:
scores = cosine_similarity(query_vec, self._tfidf_matrix)[0] scores = cosine_similarity(query_vec, self._tfidf_matrix)[0]
tfidf_scores = np.maximum(tfidf_scores, scores) tfidf_scores = np.maximum(tfidf_scores, scores)
# 关键词命中 # 关键词命中：使用过滤后的token
expanded_tokens = set() expanded_tokens = set()
for eq in expanded_queries: for eq in expanded_queries:
expanded_tokens.update(t for t in jieba.cut(eq) if len(t) >= 2) expanded_tokens.update(t for t in jieba.cut(eq) if len(t) >= 2 and t not in AutoReply._STOPWORDS)
query_tokens_set = expanded_tokens query_tokens_set = expanded_tokens if expanded_tokens else set(query_tokens_filtered)
idf_dict = {} idf_dict = {}
if hasattr(self._vectorizer, 'idf_') and self._vectorizer.idf_ is not None: if hasattr(self._vectorizer, 'idf_') and self._vectorizer.idf_ is not None: