Skip to the project story
Chroma project overview

01 My question 0:00–0:25

Can a 3B model make
a usable color LUT?

The class assignment was to find a limitation of a frontier model and test whether a smaller, focused model could match it on a narrower task. I chose a file-generation problem in color grading.

Skip to the cached-output demo
Starting point

In my pilot, all 9 LUTs that finished moved color in the requested direction and stayed safe. The reliability problem was the file itself: a LUT has 4,913 rows, and the best frontier model finished only 4 of 5 basic prompts.

Street portrait with a strong golden tokenizer-decoded LUT applied
Original street portrait before color grading
original candidate LUT
One global rule

A LUT is a reusable function: the same input color maps to the same output color everywhere in the frame.

4,913rows in a 17³ LUT
14,739RGB channel values
64learned code tokens

About 230× fewer output decisions for the language model.

02 Prompted-frontier baseline 0:25–0:55

They understood the color. They could not finish the file.

I gave five frontier configurations the same five basic edits and asked each one to write a complete 17³ .cube LUT.

Five single-attribute prompts per configuration · 25 attempts total
configurationusable LUTscompletioncostwall timewhat failed
Opus 4.84 / 580%$9.7651 min1 truncated
Sonnet 4.63 / 560%$7.7170 min2 invalid / null
GPT-5.5 · xhigh2 / 540%$9.0516 min3 invalid
GPT-5.5 · medium0 / 50%$0.483 min4 refused; 1 invalid
Gemini 3.1 Pro0 / 50%$3.2229 minoutput route capped
9 / 25usable files overall
$30.22spent on the main grid

Pilot on synthetic fixtures; target fidelity was not scored. This tests raw dense artifact completion, not whether frontier models have worse color taste.

03 The representation 0:55–1:20

First, I taught the model a LUT vocabulary.

Measured labels describe each transform. The tokenizer then turns the 4,913-row file into the short sequence the generator will learn to predict.

01 / synthetic labels

Five phrasings, one measured spec

A teacher writes literal, metaphor, mood, concept, and slang versions for each of the 2,761 supported LUTs. Deterministic checks decide whether every claimed color axis is true. Another 272 refusal rows cover requests a global LUT cannot perform.

“a touch warmer”“late-afternoon glow”“soft and nostalgic”“faded film stock”“warm it up a bit”→one AttributeSpec

“Make it warmer” originally labeled both slight and strong LUTs. That made magnitude impossible to learn, so I added measured intensity cues and route genuinely vague strength requests to clarify.

02 / tokenize

Train once, then freeze

The VQ-VAE compresses a 17×17×17 LUT into a 4×4×4 latent grid: 64 IDs from a 256-entry codebook. Freezing it means a token never changes meaning during generator training.

17³color lattice
→
4³latent cells
→
64code tokens
1.31
mean ΔE00
37.03
mean PSNR
256 / 256
codes active

04 Full system 1:20–1:50

I trained the LUT vocabulary before the generator.

I train the LUT vocabulary once. At request time, two small models only have to choose the right 64-code sequence.

BUILD ONCE

learn the output language

canonical corpus17³ LUTs4,913 rows each
train tokenizerVQ-VAE17³ → 4³ latent grid
freeze together256-code vocabulary+ LUT decoder
the generator learns this fixed vocabulary
EACH REQUEST

choose 64 codes

inputimage + words“soft monochrome”
interpretLFM2.5-350Mgrade · clarify · refuse
typeAttributeSpeccool · muted · low contrast
generateQwen-VL 3Bimage + spec → 64 IDs
fixed vocabulary64 code tokens<lut_031> …
decodefrozen decoderportable .cube LUT

05 Data collection 1:50–2:15

How I built the training set.

I pulled transforms from several sources, converted them to one representation, removed duplicates, and kept related examples together during the train/eval split.

BUILD LOGdata_pipeline / source to active SFT rows

Counts from the built corpus

acquire7,760candidates
canonicalize7,74117³ residuals
quality gates4,620survivors
leakage units2,804split groups
active SFT set3,033training rows
01 / collect

Web scraping + expert data

I wrote bounded, resumable connectors for PPR10K, FiveK, FreshLUTs, G’MIC/RawTherapee, public packs, and procedural coverage. Every artifact keeps its source and content hash.

PPR10KFiveKFreshLUTsG’MICpublic packsprocedural
02 / measure

Canonicalize, dedupe, gate

Everything becomes the same sRGB 17³ residual format. I reject transforms that clip, fold, or fail representability checks, then split by source unit so near-duplicates cannot cross the evaluation boundary.

raw .cube / XMP / image pair → residual.npy → behavior_v2

06 Development notes 2:15–2:55

Four bugs, and the guardrails I added.

WHAT BROKE / WHAT I CHANGED

Recovered from my development logs with Nessie

scraping2,882 downloads → 95 unique

Cause The first FreshLUTs endpoint returned a rotating pool instead of the requested item.

What I changed

Paused the crawl, verified the endpoint, and required content hashes plus cross-source dedup before admitting a LUT.

evaluation48.5% of the first holdout leaked

Cause Derived siblings shared a source but had different row IDs.

What I changed

Rebuilt the split around split_unit_id, grouped siblings, and made a missing unit a hard corpus-build error.

tokenizerThe loss fought real color casts

Cause The neutral term rewarded identity while reconstruction and tail quality carried too little weight.

What I changed

Matched neutrality to target chroma and raised reconstruction plus tail loss. The frozen codebook now uses all 256 entries.

trainingA path typo could train on zero rows

Cause Colab treats /content/SLM and /content/slm as different paths.

What I changed

Pinned the lowercase artifact root, abort when rows trained equals zero, and check tokenizer identity in every adapter manifest.

07 Technical choices 2:55–3:20

Why I used this stack.

I needed the training and inference path to fit on one GPU, and I wanted the web layer to be easy to deploy and debug by myself.

01 / interpreter

LFM2.5-350M

A separate text model handles ambiguity, refusal, and text-to-spec routing before the image model runs.

02 / generator

Qwen2.5-VL-3B + 4-bit QLoRA

The base model can see the source image. QLoRA trains small adapters and keeps memory low enough for one T4.

image + attribute_spec_text → 64 codes
03 / codec

Frozen VQ-VAE

Learned compression replaces 14,739 RGB channel values with 64 discrete code tokens.

04 / service + deploy

FastAPI, vanilla JS, Modal

No frontend build step. Static pages stay on CPU; a T4 wakes only when the real generator is called.

browser→CPU web→T4 on generate

08 Results + limits 3:20–3:45

What worked, and what still fails.

The compact representation works. Free-running generation is still the weak point, so I measure it separately from teacher-forced token accuracy.

behavioral fidelity.159greedy→.307Best-of-32
router study.884Qwen 0.5B accuracy
codec1.31mean ΔE00
Why rerank?

Score complete candidates, not next tokens.

I decode each candidate and compare its measured color axes with the request. The web path uses N=4 for latency; the study measured up to N=32.

The remaining gap

Teacher forcing hid free-running collapse.

High next-token accuracy did not guarantee a stable 64-token sequence. Exposure bias remains the main generator problem.

Scope boundary

A LUT changes color, not objects.

“Make it warmer” is in scope. “Remove the person” needs semantic image editing, so the interpreter should refuse it.

09 BrainLift 3:45–4:05

What I learned, and where I used it.

The file below is the BrainLift I kept during the project. It has 7,970 words covering color science, LUT mechanics, VQ tokenization, model evaluation, and how comparison helps people judge visual changes.

MD

BRAINLIFT.md

notes, sources, and design decisions

757
lines
7,970
words
34
cited sources
7
spiky POVs

DOK 3 · notes used in the build

Four notes I kept coming back to

Global LUTs are spatially blind.→Route local edits to refuse.
Pretty but wrong-direction color is still wrong.→Use contract-first, lexicographic evaluation.
Derived LUT families create hidden leakage.→Split on source units and near-neighbors.
Comparison reveals structure better than one result.→Apply the same LUT across reference images.

The BrainLift also documents the counter-evidence: the current generator still collapses free-running, and tokenizer compression moves some failure into quantization error rather than eliminating it.

research sourcesDOK 4 positionsfalsifiable testsADRsimplementation

10 One SPOV in detail 4:05–4:30

Kids usually have the idea before the editor’s vocabulary.

My third SPOV: natural language is the starting point, but the interface still has to give the technical words back.

SPOV 03 · natural language

“Make it warmer,” “less heavy,” and “make it pop” come before black point, saturation, and curves.

make it moodier→darker shadows · lower saturation
less heavy→lifted shadows · softer contrast
make it sharper→clarify: more contrast, or actual detail?
01Say the intentstart in the child’s words
→
02Map or clarifyname the color concept—or the boundary
→
03Compare + undoinspect what actually moved
→
04Name the decision“I lifted the shadows”

The testable claim On a new image, can the child name the visible change? If not, the interface is still prompt magic.

11 Saved LUTs 4:30–5:00

Three cached LUT candidates.

I am not running the generator live because the GPU is off. These are real 64-code corpus decodes I cached during the reranking work. I picked the vivid alternatives for this comparison; they are not the prompt-matching winners.

GPU stays idleNo warmup or generation request is made.

cached corpus candidate

Warm olive

served from Modal
Portrait with a cached warm olive LUT candidate applied
Original image before the LUT
original candidate

The warm olive cast is easy to see in the window light, floor, and skin tones.

Reranker recordsource=corpus:alt3 | fidelity=.143 | entropy=.781 | dominant_share=.125
See why it lost