A transparent model-quality system
How I test a two-stage system — an interpreter that reads a photo and a natural-language edit into a measured AttributeSpec, and a generator that turns that spec into a reliable, safe, compact 3D color transform — and refuses everything outside its remit, honestly.
A model does not pass because its output looks plausible. It passes because the entire artifact survives a measurable chain of obligations.
The harness starts from the client-visible contract: read a request into an AttributeSpec, route it as grade, clarify, or refuse, produce a valid transform, preserve image integrity, and decline what a global LUT cannot solve.
Boundary, syntax, decode, color direction, target fidelity, safety, and style are separate layers. A failure says what broke, not merely that a score went down.
The SFT checkpoint must beat nulls, constants, deterministic recipes, prompt-only ablations, image ablations, and frontier prompting — otherwise the claim is narrowed.
Small slices and qualitative examples illuminate behavior. Only predeclared, adequately powered metrics can approve a release.
The split is unit-aware, so no near-duplicate LUT straddles train and test; every held-out row is scored, not a convenient sample; and a supported row counts only with exactly 64 code positions. Where the frozen decoder is off, honesty slices use decoder-free proxies rather than pretending to measure color.
The deterministic stack
Supported rows must clear every deterministic layer from L0 through L7. The L8 judge adds explanation and qualitative context, but it cannot rescue an invalid or unsafe transform. In v1 the frozen decoder stays disabled, so the decode-dependent layers (L2, L4–L7) run as decoder-free proxies or are owner-gated; the load-bearing spine is L0 boundary, L1 syntax, the routing metrics, and held-out token accuracy.
The interpreter's route is scored three ways. A grade must not be refused; a refuse passes only with the exact token <unsupported>, split into out_of_scope and out_of_gamut; clarify is its own class.
A supported output is exactly LUT-BOS, 64 valid code tokens, and LUT-EOS. Runtime constrained generation should make syntax failures implementation bugs.
The 64 codes decode through the frozen VQ decoder into a finite canonical 17×17×17 residual LUT and a valid pinned .cube export.
The frozen tokenizer must satisfy reconstruction quality across the mean, difficult tail, source families, and target classes before model scores are trusted.
Measured behavior_v2 must move in every requested direction — warmer, cooler, brighter, contrast toe and shoulder, matte, a global hue cast, per-tone-region hue, and per-hue saturation across seven sectors — within calibrated magnitude windows.
Depending on the row, the decoded LUT must match an exact target, one of several valid references, or a declared behavior window using image and chart ΔE00.
Clip rate, out-of-range values, smoothness, foldover, neutral drift, and skin-locus behavior are tested before a transform can pass.
Named styles must land inside recipe windows and remain discriminable from neighboring styles rather than collapsing into one generic warm filter.
A pinned judge records perceptual and instruction-alignment commentary for review. It is evidence, not authority.
METRIC = held-out token accuracy · direction max
The complete run, in motion
A release claim is the last frame, not the first. Watch the same frozen row move through generation, artifact recovery, deterministic measurement, statistical gating, and an auditable report.
The harness binds a source image, the request, its gold route and AttributeSpec, an acceptance mode, and leakage-safe, unit-aware provenance before the model is called.
Interactive row walkthrough
Choose a representative request and generation mode. The simulation shows how the row is classified, parsed, decoded, measured, and finally accepted or rejected.
This is an explanatory simulation of the real layer logic, not a live model call.
Frozen evaluation design
The aggregate pass rate answers “does it work?” The slices answer “where, for whom, and under what conditions does it stop working?”
The client-facing mix of common supported requests, reserved for the primary quality claim.
Identical text across different images where the correct LUTs must differ. Defends against a prompt-only shortcut.
Phone exports, screenshots, heavy JPEG compression, odd white balance, ICC variation, and size extremes.
Small but visible edits that expose a model that only knows exaggerated transformations.
Single-style rows with neighbor-exclusion tests, recipe windows, and confusion analysis.
Held-out PPR10K and FiveK expert identities absent from the SFT training set.
Held-out source, style, or LUT families that measure generalization beyond memorized packs.
The interpreter's own scores: attribute-F1 against the gold AttributeSpec, three-way route accuracy, and over-refusal on valid work.
Non-global edits versus global looks the frozen tokenizer cannot represent — each refusal family measured on its own.
Decoder-free honesty: paraphrases stay consistent, known styles map to real axes, and invented terms route to clarify or refuse instead of hallucinating.
Local edits, geometry, generation, reference transfer, mixed traps, and contrastive near-boundary pairs.
Predeclared release criteria
Every gated metric has a frozen slice, minimum sample size, direction of the bound, test family, and underpowered behavior. The current v1 headline floor is 350 rows because the reserve is approximately 382 eligible supported rows.
All deterministic obligations from boundary through style.
Valid 64-code sequence or exact refusal, measured over all rows.
Catch out-of-scope work without refusing legitimate global edits.
Includes traps that combine a supported lure with an impossible local task.
Contrastive pairs differ by the smallest wording that changes supportability.
The model cannot protect itself by saying “unsupported” to difficult valid work.
Absolute v1 gate covering clipping, foldover, neutral drift, and skin-locus risk.
Primary headline advantage on the same frozen rows under the v1 power design.
Out-of-scope and out-of-gamut refusals are gated separately, so one family cannot hide behind the other.
The interpreter's AttributeSpec must match the gold axes and pick grade, clarify, or refuse correctly.
The decoder-free training metric: exact 64-code accuracy on the unit-aware holdout steers the SFT loop.
min_N → evaluable, not silently pass
Results, measured honestly
Fine-tuning is not the default winner. This dashboard reports only what is measured today: the prompted-frontier pilot on a 5-row single-attribute grid, and this project's own measured proxies. Where an end-to-end result does not exist yet, it is marked pending — never filled with a number.
Prompted-frontier pilot, 5 single-attribute rows (warmer · cooler · more magenta · more green · brighter). Frontier failures are truncation or refusal, not wrong color.
Provider list prices; wall-clock per grid. Grand total ≈ $30.22 across all frontier configs.
A frontier model must emit every LUT row as text; Chroma emits 64 VQ codes a frozen decoder expands into the 4,913-row grid.
These are the real numbers we have today. They are component measurements and decoder-free proxies — not an end-to-end harness pass rate, which does not exist yet.
382-LUT dev holdout · CIEDE2000 (lower is better)
Qwen2.5-0.5B · holdout n=684 · ships as router only
P6 adapter · spec-conditioned · ruler ceiling ≈ 0.89
| Model | Id | $ in / out (1M) | Effort | Completion |
|---|---|---|---|---|
| Chroma (ours) | local · Qwen2.5-VL-3B + adapter | ~$0 · local | deterministic | 100% valid by construction |
| Opus 4.8 | claude-group/claude-opus-4-8 | $5 / $25 | medium | 4/5 |
| Sonnet 4.6 | claude-group/claude-sonnet-4-6 | $3 / $15 | medium | 3/5 |
| GPT-5.5 | openai-group/gpt-5.5 | $5 / $30 | xhigh / med | 2/5 · 0/5 |
| Gemini 3.1 Pro | gemini-group/gemini-3.1-pro | $2 / $12 | — | 0/5 (token cap) |
supported_prompt_to_lut_pass_rate exists — the decode-dependent layers (L2, L4–L7) are still decoder-free proxies in the v1 spine.Roadmap · not done yet
The dashboard above is honest about the gap: free-running greedy generation collapses to ≈0.16 behavioral fidelity (~94% collapse), while sample-and-rerank best-of-N already reaches ≈0.42. The next step is GRPO — on-policy RL to bake that covered-but-unselected quality into a single deterministic greedy pass. It is not run yet; it needs GPU time and may need knob tuning. Here is why the bet is worth making — and where it could fail.
◆ The strongest evidence — from our own data
t0.7 mean fidelity is only ~0.09, but the best of 32 is ~0.42. That heavy right tail gives each group of rollouts a wide reward spread — exactly what group-relative advantage exploits (it pushes toward the top of each group). Low mean + high max is the favorable signature, not a red flag.
oracle_at_n rule: ≥0.6 → distill · 0.3–0.6 → capped · ≤0.3 → escalate. At 0.42 we're in the capped band — best-of-N ships today, distillation alone is capped, so RL/GRPO is the escalation of record.
◇ The honest counter-case
BEST won't promote it.
lr 5e-6, μ=1 — a conservative REINFORCE+KL. It might need more than 500 steps, or knob tuning.
BEST only promotes a guard-clean fidelity improvement. Worst case, GRPO doesn't help and BEST stays at the ~0.16 init — you've spent GPU time, not shipped a regression. You cannot end up worse than P6.
Auditability
The harness produces enough structure to reproduce a run, inspect individual failures, and distinguish model defects from parser, tokenizer, data, or evaluation defects.
Rows, split units, interpreter and generator artifacts, the AttributeSpec version, tokenizer hashes, canonical domain, parser/FSM versions, safety thresholds, and config versions are recorded.
version_manifest.jsonconfig_resolved.yamlsplit_unit_idPer-row layer results preserve the facts behind the aggregate: direction deltas, image/chart ΔE00, safety flags, style margins, and refusal classifications.
metrics_by_row.parquetOverall results, paired deltas, seed summaries, adjusted gate outcomes, and qualitative artifacts are written together as the release packet.
overall.csv + gates.csvbaseline_delta.csvsummary.md + qualitative outputsIt is a system whose scope, artifact validity, color behavior, safety, generalization, and comparative advantage are legible before anyone depends on it.