文档审计引擎

name: audit-agent description: >- Structured document audit engine with a 6-stage LLM-powered pipeline (coherence check, claim extraction, assumption surfacing, stakeholder analysis, methodology review, bias detection), quantitative weighted scoring with pass/fail thresholds, severity distribution analysis, traffic-light visual dashboard, and multi-format reports (text, HTML, JSON). version: 1.0.0 tags: - audit - document-analysis - critical-thinking - bias-detection - methodology - quality-assurance trigger_phrases: - "audit this document" - "audit this paper" - "review document quality" - "check for bias" - "extract claims" - "identify assumptions" - "stakeholder analysis" - "methodology review" - "文档审计" - "审查文档"

Overview

AuditAgent performs a structured 6-stage audit of any text document. Each stage is an independent LLM-powered module that examines the document from a different angle. The stages run sequentially with context flowing from one to the next. A quantitative scorer aggregates findings into a weighted composite score (0-100) with pass/fail determination, severity distribution, red flag summaries, and traffic-light visualization. Reports are produced in structured text, self-contained HTML, and machine-readable JSON formats.

Capability Boundaries

What this tool does: - Run a 6-stage audit pipeline on any document:

• Quantitative scoring with:
• Per-stage scores (0-100) computed from finding severities.
• Weighted composite score with configurable pass/fail threshold (default: 70).
• Severity distribution histogram (critical, high, medium, low, info).
• Red flag extraction (critical + high findings across all stages).
• Traffic-light map (green >= 80, yellow 50-79, red < 50).
• Multi-format reporting:
• Structured text report with per-stage findings, severity levels, evidence quotes, locations, and recommendations.
• Executive summary (1-page, character-limited).
• Action items list sorted by priority.
• Self-contained HTML report with color-coded severity badges, traffic-light stage indicators, severity distribution bars, and finding cards with evidence blockquotes.
• Machine-readable JSON export.
• Customizable stage weights and pass threshold via AuditScorer configuration.

What this tool does NOT do: - It does NOT audit code, financial statements, or legal contracts with domain-specific rules. - It does NOT guarantee factual correctness of findings -- LLM output is probabilistic. - It does NOT modify or annotate the original document. - It does NOT run in real-time -- it is a batch pipeline.

Trigger Conditions

Use this skill when the user: - Provides a document (text, article, report, paper) and asks for an audit, review, or quality assessment. - Wants to identify logical inconsistencies, unstated assumptions, or biases in a text. - Needs stakeholder analysis or methodology evaluation for a policy document or research paper. - Asks to "audit this document", "review document quality", "check for bias", "extract claims", "identify assumptions", "文档审计", or "审查文档".

Usage

Python API

from audit_agent import AuditAgent

agent = AuditAgent(model="local")  # or model="qwen2.5:7b"
result = agent.audit("path/to/document.txt")

# Check results
print(result.executive_summary())

# Export reports
result.export_html("output/report.html")
result.export_json("output/report.json")
result.export_text("output/report.txt")

Configuration

The 6-stage pipeline is defined in audit_agent/stages.py and audit_agent/prompts.py. Each stage has: - A unique stage_name mapping to its prompt template. - An __init__ that accepts an LLM client. - A run(document, prior_stage_output, model_name) method that executes the stage.

Scoring Configuration

from audit_agent.scoring import AuditScorer

scorer = AuditScorer(
    weights={
        "coherence_check": 0.20,
        "claim_extraction": 0.15,
        "assumption_surfacing": 0.15,
        "stakeholder_analysis": 0.15,
        "methodology_review": 0.20,
        "bias_detection": 0.15,
    },
    pass_threshold=70.0,
)

Dependencies

• LLM backend: OpenAI-compatible endpoint (default: Ollama with qwen2.5:7b)
• PyYAML for rules configuration.

Output Example

Text Report Structure

=======================================================================
  AuditAgent -- 结构化文档审计报告
=======================================================================
  文档: My Document
  审计时间: 2026-07-06T10:00:00
  综合评分: 68.5/100
  审计结果: 未通过 (FAIL)
=======================================================================
-----------------------------------------------------------------------
  各阶段评分
-----------------------------------------------------------------------
  🟡 一致性检查 (coherence_check): 72/100 (加权贡献: 14.4)
  🟢 主张提取 (claim_extraction): 85/100 (加权贡献: 12.8)
  🔴 隐含假设揭示 (assumption_surfacing): 45/100 (加权贡献: 6.8)
  ...
-----------------------------------------------------------------------
  问题严重度分布
-----------------------------------------------------------------------
  严重 (Critical): 3  |  高危 (High): 7  |  中等 (Medium): 12  |  低危 (Low): 5  |  信息 (Info): 2

-----------------------------------------------------------------------
  一致性检查 — 检测文档内部逻辑矛盾与不一致性
-----------------------------------------------------------------------
  [CRITICAL] COH-001 (severity=90)
    Section 2 claims X while Section 5 implies not-X.
    位置: Paragraphs 12-15
    原文: "The policy achieved its stated goals..."
    建议: Clarify the contradiction between Section 2 and Section 5.
  ...
=======================================================================
  审计结束
=======================================================================

HTML Report Features

• Gradient header with document name and timestamp.
• Circular score display with PASS/FAIL badge.
• Severity distribution bar chart.
• Per-stage sections with traffic-light indicators, finding cards with severity badges, evidence blockquotes, and recommendations.

FAQ

Q: What model should I use? A: Default is Qwen2.5:7b via Ollama. Any OpenAI-compatible model works. Larger models may produce better quality audits, especially for complex documents.

Q: Can I audit documents in any language? A: Yes. The stage prompts are in Chinese and the system prompt instructs the LLM to work with the document's language. Multi-language documents are supported.

Q: What does a "pass" mean? A: A composite score of 70 or above (configurable) indicates the document passed the audit. This is a relative quality metric, not an absolute guarantee.

Q: How long does an audit take? A: Each of the 6 stages makes an LLM call. With a local Ollama model, expect 30-120 seconds total depending on document length and hardware.

Q: Can I skip certain stages? A: The pipeline expects all 6 stages. To skip a stage, you would need to modify the scoring weights to zero out unwanted stages and adjust the orchestrator.

这个 Skill 质量很好，定位清晰——专门用于审计文档质量而非通用聊天。6个审计维度覆盖全面，从逻辑一致性到偏见检测都有涉及。评分系统直观，能给出具体分数和红牌警告。多语言支持和多格式报告输出是加分项。主要缺点是缺少测试用例，普通用户难以验证输出准确性；另外对提示词质量的依赖较高，不同模型可能产生不一致结果。整体适合需要严肃评估文档质量的用户使用。