← All runbooks
gooseworks-ai / composites-messaging-ab-tester

Messaging A/B Tester

Generate 3-5 distinct messaging variants for a value proposition, design a structured A/B test, and analy

agent codexmodel gpt-5.5snapshot python312-uveval programmatic7 stepsv1.0.0

Deploy Messaging A/B Tester to your jetty.io

One-click installs this runbook into a collection on your Jetty account. You can run it from the Spot dashboard, schedule it, or pipe inputs in via the API.

Deploy on jetty.io →View source
The shape of the run

7 steps · start to finish.

  1. 1
    Step 1

    Environment Setup

    ▶
    1. Create the results directory if it does not exist.
    2. Confirm the user supplied core_value_prop, test_goal, and icp.
    3. Set test_channel to linkedin, email, or both; default to both when the user has both channels available.
    4. Initialize validation_report.json with setup status and input completeness.
    mkdir -p /app/results
python - <<'PY'
import json, pathlib, datetime
report = {
  "version": "1.0.0",
  "run_date": datetime.datetime.utcnow().replace(microsecond=0).isoformat() + "Z",
  "stages": [{"name": "setup", "passed": True, "message": "Results directory ready"}],
  "overall_passed": True
}
pathlib.Path("/app/results/validation_report.json").write_text(json.dumps(report, indent=2) + "\n")
PY
  2. 2
    Step 2

    Intake

    ▶

    Collect the decision context before writing variants:

  3. 3
    Step 3

    Generate Messaging Variants

    ▶

    Create 3-5 variants that test different strategic angles, not superficial wording changes. Use these angle families where relevant:

  4. 4
    Step 4

    Design and Deploy Tests

    ▶

    Choose the test design that matches `test_channel`.

  5. 5
    Step 5

    Collect Results

    ▶

    Accept metrics in any usable form: pasted values, CSV export, screenshot transcription, or a concise manual summary. Normalize the data into tables before scoring.

  6. 6
    Step 6

    Analyze Results

    ▶

    Use the scoring framework below and explain confidence based on sample size.

  7. 7
    Step 7

    Iterate on Errors (max 3 rounds)

    ▶

    If required inputs, output files, or metrics are incomplete, perform up to max 3 rounds of targeted repair: