Conference Abstract Scraper

8 steps · start to finish.

1. 1
   Step 1

   Environment Setup

   ▶

   # Install dependencies
   pip install beautifulsoup4 lxml playwright requests
   python -m playwright install chromium
   
   # Create output directories
   mkdir -p {{results_dir}}
   

   Verify Playwright's Chromium browser launches successfully, and that api.crossref.org is reachable, before proceeding (no external credentials are required for this runbook).

   ---

2. 2
   Step 2

   Discover Abstracts & Metadata via CrossRef

   ▶

   Query the CrossRef REST API directly (no browser needed for this step) to discover and order the target issue's abstracts:

3. 3
   Step 3

   Fetch Full Text Directly from Confex

   ▶

   CrossRef records carry metadata but not full abstract text. Open {{confex_program_url}} with Playwright — this is a client-rendered (Backbone.js) single-page app, so wait for the program's…

4. 4
   Step 4

   Evaluate Outputs

   ▶

   For each of the {{max_abstracts}} abstracts, assign an evaluation status:

5. 5
   Step 5

   Iterate on Errors (max 3 rounds)

   ▶

   If any abstracts received FAIL or PARTIAL status, or the completeness check failed:

6. 6
   Step 6

   Write Executive Summary

   ▶

   Write {{results_dir}}/summary.md with the following structure:

7. 7
   Step 7

   Write Validation Report

   ▶

   Write {{results_dir}}/validation_report.json:

8. 8
   Step 8

   Final Checklist (MANDATORY — do not skip)

   ▶

   echo "=== FINAL OUTPUT VERIFICATION ===" RESULTS_DIR="{{results_dir}}" for f in "$RESULTS_DIR/abstracts.json" "$RESULTS_DIR/summary.md" "$RESULTS_DIR/validation_report.json"; do if [ ! -s "$f" ]; then