Astra and GPT-6 Complete Guide: Prompts, Proofs and Evals

Who this guide is for

This is the practical companion to our news brief (OpenAI Astra and GPT-6: what we know so far). Instead of headlines, you get a repeatable testing workflow for OpenAI Astra and GPT-6: prompts that survive model switches, a verification pattern for math/Code/facts, and a minimum viable eval suite. Skill level: anyone shipping real work with language models.

Direct answer: professionals test new models with a frozen golden set of 10–20 real tasks, scoring accuracy, format compliance, latency, and cost per task — switching only on repeated wins, never on demos.

Step 1 — Build the golden set

Collect 10–20 tasks from your actual work: three writing, three analysis, two formatting, the rest from your niche. For each, store the exact prompt, a reference answer, and the scoring rule. Scoring options in ascending rigor: exact match for structured outputs, rubric grades (0–2 per criterion) for prose, and blind human preference for taste-driven work. Version everything. A golden set is a contract with your future self during chaotic release weeks.

Step 2 — Write prompts as output contracts

Fragile prompt: "summarize this." Transferable prompt: "Summarize in 5 bullets, max 12 words each, one quoted source span per bullet, no new claims." Contracts specify deliverable, format, length, and citation rule. They transfer because they constrain the degrees of freedom where models differ most. Keep temperature fixed across comparisons, and change one variable per run — otherwise you cannot attribute improvements.

Step 3 — The proofs workflow (verify everything checkable)

Split every verifiable task into reasoning then verification. First pass: "show step-by-step working, then the final answer." Second pass with the working in context: "check each step above; name the first error, or confirm all steps." Then verify externally: execute code, open cited pages, recompute figures. Classify results as verified, partially verified, or unverified — and never ship unverified reasoning as fact. Copy-paste verification prompts:

• "List each factual claim above with its source. Mark any claim with no source."
• "Re-derive the result using a different method and compare."
• "What would falsify your answer? Name the check."

Step 4 — Minimum viable evals: four numbers

Per task, record: accuracy on the golden set, format-compliance rate, median latency, and cost per task. Then build a failure taxonomy with separate counts for wrong-format, hallucinated-source, refused-valid, and degraded-style failures. Each class has a different remedy — constraints for format, retrieval grounding for hallucinations, policy-aware reframing for refusals. Re-run weekly while providers ship; a spreadsheet beats a dashboard you never open.

Step 5 — Switch criteria (the decision rule)

Switch models when: (1) the challenger beats the incumbent on your golden set twice consecutively, (2) cost per task fits budget at your volume, and (3) failure-class analysis shows no new catastrophic category. Pilot on 10% of traffic first with rollback ready. Document the decision — "switched Oct 2026 on golden-set v3, +8pp accuracy, -12% cost" — so future-you can audit it.

Step 6 — Regression hygiene during release season

Pin production to exact model versions, log version + date on every eval screenshot, and keep a canary task that runs daily. When silent behavior shifts hit (and they do), the canary tells you within a day instead of at the angry-client stage.

Worked example: testing a summarizer switch

Task: 5 customer threads → 5 bullets each. Golden answers from last quarter. Run incumbent and Astra with identical contract prompts. Score: bullets matching reference facts (accuracy), bullets within length/format (compliance), seconds per thread (latency), tokens × price (cost). Suppose Astra wins accuracy 82→91 but costs 30% more: acceptable only if summary quality drives revenue, else keep incumbent for bulk and route premium threads to Astra. That is professional model management — portfolio thinking, not fandom.

Prompt contract library (five reusable shells)

Contracts name the deliverable, format, length, and citation rule. Adapt these shells to your tasks:

• Briefing: "Brief this document for a busy executive. Output: 3 bullets (situation, complication, recommendation), each under 15 words. No jargon."
• Extraction: "Extract all dates, amounts, and named owners. Output a table with columns Item, Value, Source-span. Mark uncertain cells UNCERTAIN instead of guessing."
• Drafting: "Draft a reply that approves the request with two conditions. Under 120 words, warm tone, end with the next step and owner."
• Critique: "Critique this plan against its own stated goal. Output the three strongest objections, each with the evidence that would change your mind."
• Rewrite: "Rewrite at half the length preserving every number and commitment. List anything you had to drop."

Canary tasks: your early-warning system

Pick three tasks so sensitive to model behavior that any silent provider-side change shows up immediately: one formatting-strict task, one refusal-boundary task, and one long-context recall task. Run them daily against pinned versions and alert on any score change. When the canary moves and you changed nothing, the provider changed something — check docs and re-run the golden set before assuming your prompts broke.

Failure taxonomy deep dive (fix the right problem)

Wrong-format failures respond to tighter contracts and few-shot examples, not bigger models. Hallucinated-source failures need retrieval grounding or explicit "say UNCERTAIN" instructions. Refused-valid failures need scope reframing — narrower, clearly-benign task decomposition. Degraded-style failures (correct but off-brand) need style examples in context. Log every failure into exactly one bucket before changing anything; mixed-bucket "it got worse" notes are unactionable.

Keep practicing with the community

Post your golden sets and results, borrow templates, and compare notes in the live thread: AI creators playbook: testing Astra and GPT-6 together. For headline context any day, the fast brief stays current: what we know so far.

Migration memo template (one page per switch)

Write this before moving traffic — it forces honesty and creates an audit trail:

• Decision: switch X% of workload Y from model A to model B.
• Evidence: golden-set versions, both scores, dates, cost-per-task math.
• Risk: new failure classes observed, mitigations in place.
• Rollback: exact revert command, owner, and verification query.
• Review date: when to re-evaluate (suggest 30 days during release season).

Glossary: eval terms worth using precisely

• Golden set: frozen tasks plus reference answers plus scoring rules.
• Regression: a previously passing task that now fails after a model change.
• Canary: a tiny daily task suite that detects silent provider-side shifts.
• Failure taxonomy: the fixed bucket list every miss is sorted into.
• Holdout: tasks never used during prompt tuning, reserved for final validation.

Frequently asked questions

How many tasks belong in a golden set?

Ten to twenty. Fewer is noise; more delays every run until you stop running it. Cover writing, analysis, formatting, and your niche.

What is a proofs workflow?

A two-pass pattern: generate step-by-step reasoning, then verify it with an independent check plus external execution (run code, open sources) before trusting the answer.

When should I switch models?

After two consecutive golden-set wins with acceptable cost per task and no new catastrophic failure class — piloted at 10% traffic with rollback ready.

Do benchmarks replace my own evals?

No. Public benchmarks measure generic capability; your golden set measures your revenue tasks. Use benchmarks for shortlisting, your evals for deciding.

How do I keep evals honest as prompts evolve?

Freeze a holdout subset you never tune against and score it only at decision time. If holdout and tuned-set scores diverge, your prompts are overfit — refresh the set with new real tasks before trusting the numbers.

Model details change fast — verify against official OpenAI docs before production moves. Then bring your scorecards to the community thread and help the next builder decide with evidence, not hype.