修复OCR残片检测:双策略匹配(最长公共子串+字符重叠率)
问题: - 单用最长公共子串策略,"端0(基岩版)" vs "端口(基岩版)" 相似度仅60%,未达80%阈值 - 导致碎片通过检测入库 修复: 增加双策略匹配: 1. 最长公共子串≥60%且≥3字符(严格) 2. 字符重叠率≥50%且中文字符重叠≥2(宽松,防OCR噪声) 验证: - 端0(基岩版) → fragment=True ✓ (字符重叠"基岩版"=3字) - 端0(Java) → fragment=True ✓ - 关于我们 → fragment=False ✓ - 那服务器有哪些基本信息 → fragment=False ✓
This commit is contained in:
+16
-4
@@ -8259,8 +8259,8 @@ class MessageMonitor:
|
||||
]
|
||||
|
||||
# ⭐ 新增:bot回复碎片检测(在基线轮判断之前过滤,防止OCR残片入库)
|
||||
# 如果文本与已回复内容高度重叠(相似度≥80%),直接丢弃
|
||||
# 例:"端0(基岩版)" vs 已回复表格含"端口(基岩版) | 30679" → 相似度≥80% → 拦截
|
||||
# 如果文本与已回复内容高度重叠(字符重叠率≥50%),直接丢弃
|
||||
# 例:"端0(基岩版)" vs 已回复表格含"端口(基岩版) | 30679" → "基岩版"3字重叠 → 拦截
|
||||
if self._replied_texts:
|
||||
import difflib
|
||||
filtered_new_items = []
|
||||
@@ -8277,10 +8277,22 @@ class MessageMonitor:
|
||||
r_norm = self._norm_frag(r)
|
||||
if not t_norm or not r_norm:
|
||||
continue
|
||||
# 策略1:最长公共子串占比≥60%且≥3字符(严格匹配)
|
||||
sm = difflib.SequenceMatcher(None, t_norm, r_norm)
|
||||
lm = sm.find_longest_match(0, len(t_norm), 0, len(r_norm)).size
|
||||
if lm >= 0.8 * len(t_norm) and lm >= 3:
|
||||
logger.debug(f"[{platform}] OCR过滤: bot回复碎片(相似度≥80%) ts={ts!r}")
|
||||
if lm >= 0.6 * len(t_norm) and lm >= 3:
|
||||
logger.debug(f"[{platform}] OCR过滤: bot回复碎片(lm={lm},ratio={lm/len(t_norm):.0%}) ts={ts!r}")
|
||||
is_fragment = True
|
||||
break
|
||||
# 策略2:字符级重叠率≥50%且核心词命中(宽松匹配,防OCR噪声)
|
||||
# 计算t_norm中有多少字符出现在r_norm中
|
||||
r_char_set = set(r_norm)
|
||||
overlap = sum(1 for c in t_norm if c in r_char_set)
|
||||
overlap_ratio = overlap / len(t_norm) if t_norm else 0
|
||||
# 如果重叠率≥50%且至少有2个中文字符重叠 → 可能是碎片
|
||||
cjk_overlap = sum(1 for c in t_norm if '\u4e00' <= c <= '\u9fff' and c in r_char_set)
|
||||
if overlap_ratio >= 0.5 and cjk_overlap >= 2:
|
||||
logger.debug(f"[{platform}] OCR过滤: bot回复碎片(char_overlap={overlap}/{len(t_norm)},cjk={cjk_overlap}) ts={ts!r}")
|
||||
is_fragment = True
|
||||
break
|
||||
if not is_fragment:
|
||||
|
||||
Reference in New Issue
Block a user