Policy-Based Text Redaction

8 steps · start to finish.

1. 1
   Step 1

   Environment Setup

   ▶

   # No external packages required — pure Python stdlib (json) is sufficient
   
   # Create output directory
   mkdir -p {{results_dir}}
   

   Verify {{input_file}} exists and is valid JSON before proceeding. If that path is absent, look in /app/assets/ — files uploaded through the API are stored as <run_id>.NN.json — and use the single JSON file found there (record the actual path in validation_report.json). Load it and confirm it parses as a list of records with the expected fields (id, domain, policy, policy_ids, text).

   ---

2. 2
   Step 2

   Parse Input

   ▶

   Load {{input_file}} and for each record extract: - id, domain — carry through unchanged to the output - policy — the full bulleted policy text; parse it into individual policy statements (each bullet…

3. 3
   Step 3

   Redact Each Record

   ▶

   For each record, produce redacted_text by replacing only the spans that violate a stated policy with a marker in the {{redaction_marker}} style (e.g. [REDACTED: PERSON_NAME], [REDACTED: SCHEDULE]…

4. 4
   Step 4

   Evaluate Against Rubric

   ▶

   For each record, score redacted_text against each criterion on a 1-5 scale, then average across all records to get the overall per-criterion and overall scores.

5. 5
   Step 5

   Iterate on Weak Criteria (max 3 rounds)

   ▶

   If the rubric score is below the pass threshold:

6. 6
   Step 6

   Write Executive Summary

   ▶

   Write {{results_dir}}/summary.md with the following structure:

7. 7
   Step 7

   Write Validation Report

   ▶

   Write {{results_dir}}/validation_report.json. Use exactly this shape — every entry in stages has the keys name (string), passed (JSON boolean, not a status word) and message (string); the dashboard…

8. 8
   Step 8

   Final Checklist (MANDATORY — do not skip)

   ▶

   echo "=== FINAL OUTPUT VERIFICATION ===" RESULTS_DIR="{{results_dir}}" for f in "$RESULTS_DIR/redacted_dataset.json" "$RESULTS_DIR/summary.md" "$RESULTS_DIR/validation_report.json"; do if [ ! -s "$f"…