We just published the first run of the IMG.LY GenAI Benchmarks, and I wanted to introduce it here before it goes anywhere else.

What it is: the same prompt, run across every leading image model, scored on the things that decide whether an asset actually ships. Not "which model makes the prettiest picture." Typography, brand-color fidelity, transparency, composition, character consistency, cost and latency.

The pilot run: 15 models, 37 canonical prompts, 3 seeds each, 1,665 generations, about $104 of API spend. Identical prompt text, default parameters, fixed seeds, no per-model tuning. Every result ships with its seed, latency, cost, native resolution and the raw measurements behind the score.

https://img.ly/ai-benchmarks/

Four things the data says

Transparency is basically unsolved. 13 of 15 models scored a flat zero.
We asked for die-cut stickers, isolated objects and shirt graphics on transparent backgrounds, then measured the alpha channel of the file that came back. Only GPT Image 1.5 emits real alpha (4.22/5). Recraft V3 gets partial credit (2.00). Everything else, the entire FLUX line, the whole Nano Banana family, Seedream, Ideogram, Qwen, Luma, Gemini Flash Image, returns the subject painted onto a white or fake-checkerboard background. Field average: 0.41 out of 5, the lowest score of any criterion in the suite.

The images look like stickers, but the files are not stickers.

Nobody hits your hex.

We put exact hex requirements in the prompts and measured the returned pixels with CIEDE2000, the perceptual color-difference standard. Best in field is Nano Banana Pro at 3.67/5. Twelve of the fifteen models land below 3. Field average 2.61. The strict two-color duotone prompt is where it falls apart: models drift the hues, add gradient shading, or quietly introduce a third color.

Text rendering is mostly solved at the top, and the text specialist isn't leading anymore.

This is the one that got better. FLUX.2 [pro], GPT Image 1.5 and Nano Banana Pro all score a clean 5.00 on exact-string rendering. Field average is 4.55. For contrast, our 2022 baseline (SD 1.5) manages 1.17 and garbles basically every word, which is a decent snapshot of how far three years got us.

The surprise: Ideogram 3.0, the model with the reputation for text, comes in 12th at 4.71. The general flagships absorbed the lead that made text specialists special.

That said, 4.9 out of 5 is not the same as shippable. One wrong character kills a headline, and baked-in text can't be fixed without regenerating the whole image.

No model holds a character across scenes.

Three scenes, one precisely described character. The whole field clusters between 3.75 and 3.89, with Nano Banana 2 Lite and Nano Banana Pro tied at the top. Outfits change, faces shift, props vanish. There is no leader here, just a shared ceiling.

The price/latency picture is not what you'd guess

Nano Banana Pro costs $0.36 per image with a 23 second median latency and a quality mean of 4.21. FLUX.2 costs $0.013 with a 2 second median and a quality mean of 4.07. That is 28x the price for about a third of a point, and FLUX.2 wins more individual benchmarks outright than anything else in the run. GPT Image 1.5 has the best quality mean and the only working alpha channel, and takes 34 seconds per image.

There is no single right answer, which is the actual finding.

Why this matters if you're building on top of these models

The reason we ran this: our customers embed image generation into products where the output has to meet a spec. A print file, a brand template, a merch design, a marketplace listing. And every gap above is the same gap. Generation is probabilistic. Production requirements are not.

Three practical takeaways:

- If your product promises cut-outs, stickers or apparel graphics, plan for a background-removal step. The data says prompt engineering will not get you an alpha channel.

- If brand color matters, enforce the palette after generation on an editable canvas. Prompting with hex codes improves the odds and never guarantees the value.

- Keep headlines as real text layers. A wrong word becomes a two-second fix instead of a re-roll, and localization becomes a string swap.

That last mile is the whole argument for pairing generation with an editor instead of shipping raw model output to users. It is also why model choice is a routing decision, not a marriage: send transparency-critical jobs to the one model that does alpha, brand-heavy work to the top of the color table, and volume work to the cheap fast generalists.

Caveats, stated up front

This is the pilot suite (pilot-0). Resolution, latency, cost, alpha and color accuracy are measured directly from every generation and reproducible from the recorded originals. The quality criteria are currently scored by an automated vision judge and are labeled provisional until a blind expert panel (three raters per cell, model names hidden) lands with the frozen suite. Scores will restate once at that point, with a new suite version, and never silently after that.

Full methodology, every prompt, every generated image and every per-model breakdown is published:

- Findings: https://img.ly/ai-benchmarks/findings/

- Model rankings: https://img.ly/ai-benchmarks/models/

- Methodology: https://img.ly/ai-benchmarks/methodology/

When new models drop, the whole suite re-runs.

Happy to answer questions about the setup, and genuinely interested in which prompts you think are missing. The suite is versioned and append-only, so good suggestions can go into the next run.