Baseline
POST-TRAINING · RESEARCH ENGINEERING · PRODUCTION AI
AI Model Release Control Center
Train → Evaluate → Compare → Investigate → Gate → Ship → Monitor → Learn
Did the new model actually get better — and is it safe to ship? This project explores the engineering loop between post-training and production: model improvements become baseline/candidate experiments, regressions become research questions, release policy becomes CI enforcement, and production telemetry feeds the next experiment.
01
Why I built this
A model improvement is not automatically a product improvement.
The engineering question I care about is: How do we know that a post-training change actually improves the experience of real users? A candidate can improve code-generation quality while simultaneously regressing safety, reliability, latency, cost or previously solved tasks. The goal is therefore not a prettier benchmark dashboard; it is a reproducible and explainable control loop that supports continuous experimentation.
02
Post-Training Experiment Lab
Treat every training intervention as a multi-objective experiment.
Candidate
CodeModel-v2-sft
Training intervention
SFT
Dataset
10,000 coding tasks
Evaluation set
Frozen coding + safety regression suite
EvaluationBaselineCandidateΔSignal
Code execution pass rate71.4%76.9%+5.5 ptsIMPROVED
Unit-test pass rate68.2%74.6%+6.4 ptsIMPROVED
SWE-bench-style success31.7%36.4%+4.7 ptsIMPROVED
Instruction following88.0%91.2%+3.2 ptsIMPROVED
Hallucination rate7.8%6.1%-1.7 ptsIMPROVED
Safety96.1%94.8%-1.3 ptsREGRESSION
Latency P952.8 s3.1 s+10.7%REGRESSION
Cost / request$0.031$0.035+12.9%REGRESSION
Output tokens / solved task812861+6.0%REGRESSION
Release decision
INVESTIGATE
Code quality improves materially, but safety, P95 latency, cost and token efficiency regress beyond investigation thresholds. These values are deliberately illustrative; they demonstrate the research workflow rather than claiming measured performance for a deployed foundation model.
03
What would I investigate?
A regression number is an observation, not an explanation.
01
Training-data distribution shift
The new SFT corpus may overweight task completion relative to secure and defensive coding behavior.
02
Instruction-data contamination / conflicting supervision
Conflicting examples can improve average coding quality while weakening safety on specific slices.
03
Training-objective overspecialization
Completion-oriented optimization may trade conservative behavior for higher task success.
04
Longer generated trajectories
Longer outputs can explain latency, token-efficiency and cost regressions.
05
Serving / inference configuration
Batching, decoding, quantization or max-token settings can create apparent model regressions.
06
Reward-model / judge bias — only if a preference stage exists
Not a primary SFT hypothesis; relevant only when the candidate also passed through preference or reward-guided training.
Recommended experiments
- Re-run the candidate on the frozen regression set with identical inference configuration.
- Stratify safety failures by task category, language, vulnerability class and prompt length.
- Compare baseline and candidate outputs for every newly introduced failure.
- Inspect training examples nearest to regressed safety tasks for conflicting supervision or contamination.
- Measure judge/reward-model correlation against deterministic safety and correctness labels when applicable.
- Run an ablation without the newly added instruction subset to isolate its causal contribution.
- Repeat evaluation under matched decoding, batching, quantization and max-token configuration.
- Slice latency and cost by generated-token count to separate model behavior from serving overhead.
04
Research-to-production architecture
Evaluation is the interface between model research and product release.
POST-TRAINING EXPERIMENT
↓
Evaluation Data
↓
Baseline vs Candidate
↓
Quality · Safety · Reliability · Performance · Cost
↓
Regression Analysis
↓
Failure Investigation / Causal Hypotheses
↓
Release Policy
↙ ↓ ↘
SHIP INVESTIGATE HOLD
↓
Production
↓
Traces / User Feedback
↓
Next Experiment05
Production layer
Research reasoning continues through CI and operation.
Live provider evaluation
Server-side Hugging Face inference can compare real baseline and candidate outputs without putting provider tokens in the browser.
Deterministic safety authority
Security checks remain independently testable and can block release regardless of subjective judge quality.
Task-aware correctness
Objective benchmark-style checks complement subjective helpfulness scoring without executing untrusted code in-process.
CI release enforcement
Machine-readable SHIP / INVESTIGATE / HOLD decisions participate directly in software-delivery control.
Production trace ingestion
Schema-validated traces turn latency and error behavior into operational evidence for the next investigation cycle.
Evidence-linked promotion
candidate → approved → production transitions retain the evidence identity that authorized them.
Generated code is not executed inside the public Space process. A production executable-code evaluator should use an isolated sandbox or worker boundary with resource, network and filesystem controls.
06
What this demonstrates
A research-engineering story, not a dashboard story.
Evaluation methodology
Baseline/candidate experimentation
Regression detection
Failure analysis
Causal hypothesis formation
Release-policy design
Safety-quality trade-offs
Latency/cost-quality trade-offs
Reproducible evaluation
CI/CD release gates
Production feedback loops
Evidence-linked model lifecycle
The system complements — rather than replaces — post-training methods such as SFT, preference optimization, RFT/RLVR and other model-improvement techniques. The purpose is to connect those interventions to reproducible release evidence and operational learning.
07
Public proof chain
Experiment → evidence → policy → CI → production feedback.
GitHub Source
Source, post-training experiment logic, tests, release policy, CI, trace schema and registry semantics.
Open ↗Hugging Face Space
Interactive Post-Training Experiment Lab plus live evaluation, traces and promotion workflow.
Open ↗Versioned Dataset
Published benchmark versions and reproducible evaluation evidence.
Open ↗Methodology Card
Research-to-production evaluation methodology and public artifact index.
Open ↗Connection to Agentic AI Academy
Production AI requires both research judgment and release discipline.
This project extends the Academy’s evaluation, safety and production-engineering work by showing how a model-change hypothesis becomes an experiment, how regressions become follow-up research, and how the final evidence participates directly in CI and model promotion.