A complete agentic evaluation pipeline: multi-agent analysis, policy-driven routing, human-AI collaboration, and calibration — in one system. Every candidate's personal data is redacted locally before it reaches a model.
Astronauts have PII too. Before any agent — AI or human — sees a candidate's personal statement, psychological profile, specializations, or awards, those fields pass through a local redaction pass and come back de-identified. Only then does the application move through four agents in sequence, each with a single responsibility, each handing a typed artefact to the next.
Evidence object with key strengths, key concerns, and an overall narrative. Purely extractive — no scoring yet.The most architecturally significant design decision: the explicit, user-controlled weighting between quantitative and qualitative signals. A single quant_weight slider (0.40–0.80) shifts the balance — and changes who gets selected.
Flight hours, degree credentials. Objective and independently verifiable — cannot be gamed by good writing.
Fitness score (0–100), medical clearance status. Ground truth anchors that hold the qualitative signals accountable.
Specializations, prior missions, commendations. Assessed — captures operational context that numbers miss.
Personal statement, psychological profile. Assessed — surfaces judgment, resilience, and self-awareness.
This is the Initial Screening default. At Deep Space (0.72), verifiable data dominates. Watch Dr. Priya Sharma vs. Tom Briggs as you move it.
Approval thresholds, signal weights, review rates — none are hardcoded into the agents. They live in a ProgramPolicy dataclass. Swapping a preset changes the system's behaviour across every dimension simultaneously, without touching implementation code.
High volume, early-funnel
Generous approval threshold, balanced signal hierarchy, low auto-decision ceiling. Processes a large applicant pool efficiently, reserving human review for genuine borderline cases.
Specific crew slot
Elevated threshold, increased human review on borderlines. Calibrated for situations where every admit will actually be assigned to a mission. Operational readiness weighted higher.
Multi-year, high-risk mission
Most rigorous tier. Strict reviewer pool, highest threshold, quantitative signals dominate. Flight hours, fitness scores, and medical clearance are decisive. Zero margin for error.
After scoring, the Signal Evaluator computes a composite risk score — a measure of how much the system trusts its own assessment, not the score itself. High risk = high uncertainty = more human.
Confident assessment, score well clear of thresholds, dimensions agree. AI recommendation is final. A configurable spot-check rate still routes a fraction for calibration.
Human and AI evaluate independently and concurrently. Both outcomes recorded. Agreement tracked. This is where most of the interesting calibration data comes from.
Human-led, AI-advisory. The reviewer is given AI analysis as context but expected to exercise independent judgement. Override reasons are recorded when outcomes diverge.
Senior reviewer pool — strict and balanced archetypes only. Typical cases: conditional clearances, exceptional profiles with critical gaps, deep inter-dimension disagreement.
Every UI control directly modifies a field in ProgramPolicy. Each has a meaningful, observable effect on routing, scoring, and human review volume.
The mock candidates span the full score spectrum and are designed to exercise every part of the pipeline — including the most architecturally interesting cases.
| ID | Candidate | Flight Hrs | Fitness | Medical | Prior Missions | Tension |
|---|---|---|---|---|---|---|
| IEA-001 | Dr. Sarah Chen | 2,450 | 94.0 | Full | 1 | High confidence |
| IEA-002 | Marcus Webb | 820 | 78.5 | Full | 0 | Some gaps |
| IEA-003 | Dr. Elena Vasquez | 310 | 81.0 | Full | 0 | Low hrs, strong qual |
| IEA-004 | James Okafor | 1,650 | 88.0 | Full | 0 | Well-rounded |
| IEA-005 | Dr. Priya Sharma ⭐ | 180 | 72.0 | Conditional | 0 | Quant/qual tension |
| IEA-006 | Tom Briggs ⭐ | 3,280 | 96.5 | Full | 0 | High quant, low qual |
| IEA-007 | Yuki Tanaka | 590 | 83.0 | Full | 0 | Below threshold |
| IEA-008 | Carlos Mendez ⭐ | 1,920 | 91.0 | Full | 2 | Auto Decision anchor |
| IEA-009 | Dr. Aisha Nkosi | 145 | 76.0 | Full | 1 | Payload specialist |
| IEA-010 | Rachel Kim | 1,120 | 86.0 | Full | 0 | Strong profile |
⭐ Most instructive cases. Run these first. Switch presets and watch the routing tiers change.
The astronaut domain was chosen because it is obviously fictional, engaging, architecturally rich (genuine quant/qual tension), and memorable. Every component maps directly to any domain where AI assists in evaluating people at scale.
Inclusion/exclusion criteria are quantitative. Patient motivation and self-reported history are qualitative. The routing engine handles borderline clinical cases for physician review.
Traction metrics are quantitative. Founder narrative and team assessment are qualitative. Process hundreds of applications and route only genuinely ambiguous cases to partners.
Credit scores and income are quantitative. Business plans and extenuating circumstances are qualitative. Policy presets map directly to lending policy tiers and risk appetite.
After every evaluation run the system produces a calibration report. This is how the system earns — or loses — the right to make autonomous decisions over time.
Tracks AI recommendation vs. human outcome for every reviewed case. The metric that determines how much autonomy the system is granted.
Records the delta between AI total score and human-adjusted score across all reviewed cases. Surfaces systematic over- or under-scoring patterns.
Surfaces every case where AI and human disagreed, with the reviewer's override reason. The input to rubric and weighting updates.
Shows how cases distributed across Approved / Waitlisted / Declined — and whether that distribution matches programme intent.