修复OCR残片检测:双策略匹配(最长公共子串+字符重叠率)

问题:
- 单用最长公共子串策略,"端0(基岩版)" vs "端口(基岩版)" 相似度仅60%,未达80%阈值
- 导致碎片通过检测入库

修复:
增加双策略匹配:
1. 最长公共子串≥60%且≥3字符(严格)
2. 字符重叠率≥50%且中文字符重叠≥2(宽松,防OCR噪声)

验证:
- 端0(基岩版) → fragment=True ✓ (字符重叠"基岩版"=3字)
- 端0(Java) → fragment=True ✓
- 关于我们 → fragment=False ✓
- 那服务器有哪些基本信息 → fragment=False ✓
This commit is contained in:
youpaishidifu
2026-08-19 16:23:49 +08:00
parent 416124ebc0
commit b02bb8c3eb
+16 -4
View File
@@ -8259,8 +8259,8 @@ class MessageMonitor:
]
# ⭐ 新增:bot回复碎片检测(在基线轮判断之前过滤,防止OCR残片入库)
# 如果文本与已回复内容高度重叠(相似度≥80%),直接丢弃
# 例:"端0(基岩版)" vs 已回复表格含"端口(基岩版) | 30679" → 相似度≥80% → 拦截
# 如果文本与已回复内容高度重叠(字符重叠率≥50%),直接丢弃
# 例:"端0(基岩版)" vs 已回复表格含"端口(基岩版) | 30679" → "基岩版"3字重叠 → 拦截
if self._replied_texts:
import difflib
filtered_new_items = []
@@ -8277,10 +8277,22 @@ class MessageMonitor:
r_norm = self._norm_frag(r)
if not t_norm or not r_norm:
continue
# 策略1:最长公共子串占比≥60%且≥3字符(严格匹配)
sm = difflib.SequenceMatcher(None, t_norm, r_norm)
lm = sm.find_longest_match(0, len(t_norm), 0, len(r_norm)).size
if lm >= 0.8 * len(t_norm) and lm >= 3:
logger.debug(f"[{platform}] OCR过滤: bot回复碎片(相似度≥80%) ts={ts!r}")
if lm >= 0.6 * len(t_norm) and lm >= 3:
logger.debug(f"[{platform}] OCR过滤: bot回复碎片(lm={lm},ratio={lm/len(t_norm):.0%}) ts={ts!r}")
is_fragment = True
break
# 策略2:字符级重叠率≥50%且核心词命中(宽松匹配,防OCR噪声)
# 计算t_norm中有多少字符出现在r_norm中
r_char_set = set(r_norm)
overlap = sum(1 for c in t_norm if c in r_char_set)
overlap_ratio = overlap / len(t_norm) if t_norm else 0
# 如果重叠率≥50%且至少有2个中文字符重叠 → 可能是碎片
cjk_overlap = sum(1 for c in t_norm if '\u4e00' <= c <= '\u9fff' and c in r_char_set)
if overlap_ratio >= 0.5 and cjk_overlap >= 2:
logger.debug(f"[{platform}] OCR过滤: bot回复碎片(char_overlap={overlap}/{len(t_norm)},cjk={cjk_overlap}) ts={ts!r}")
is_fragment = True
break
if not is_fragment: