Chroma best-of-N inference

Inference

Sample many. Keep the best.

The deployable win: sample N candidate LUTs, rerank by behavioral fidelity, ship the winner.

Read off greedily, the generator suffers exposure bias: it over-commits to a single dominant code and collapses to a low-fidelity LUT that mostly misses the request. The fix needs no retraining — sample a handful of candidates, decode and score each against the requested spec with a reranker that needs no ground truth, and return the best one. That roughly doubles free-running fidelity (0.159 → 0.307).

Why one shot fails

The collapse, and the fix

Coverage vs selection

How far the reranker climbs with N

oracle@k is the best fidelity reachable among the first k samples (a perfect reranker's ceiling); the deployable reranker captures nearly all of it. The dashed line is the free-running greedy baseline.

Interactive

Step through a best-of-N run