Collected Notes

01The paper in brief

Text-to-image models now produce real graphic design — posters, ad creatives, UI mockups, typographic compositions — yet they're still trained and evaluated using preference data collected on photographic generation, where a single “which is better?” label captures most of what matters (anatomy looks right, the prompt is followed).

Design doesn't collapse so neatly. A layout can preserve spatial structure while missing the color intent; another can follow the brief's semantics while breaking typographic hierarchy. A single overall label averages across these partly-independent axes and loses the per-axis signal. TASTE's response is to decompose the judgment.

Concretely, the dataset records:

• Two disjoint cohorts of five professional designers each (an “Aesthetics” cohort and a “Descriptions” cohort), ten designers in total.
• Rankings of outputs from four current generators — FLUX.2 max, GPT Image 1.5, Nano Banana 2, and Seedream 5.0 Lite — shown blind under code-names to prevent brand bias.
• Nine criteria. Aesthetics covers overall preference plus mood/tone, visual hierarchy, color harmony, and typography. Descriptions covers overall preference plus color accuracy, spatial accuracy, and typography. (The two typography criteria differ: one rates typographic craft, the other rates whether the prompt's demanded text is rendered correctly.)
• Per-image hallucination flags (none / minor / major) on the two overall-preference cohorts. The flags are visible per design in the explorer — a brief where two of the four outputs drew “major” flags shows what earns one.

For each prompt a rater makes all (4 choose 2) = 6 pairwise comparisons, which are then collapsed into a strict 4-way ranking. The criteria themselves weren't invented by the authors — they surfaced from pilot interviews with the same pool of designers.

Alongside the data, the paper contributes two things: a signal-validation framework (does this data carry learnable preference signal at all, or is it noise?) and a benchmark showing that off-the-shelf VLM judges and T2I scorers all fall below 0.55 agreement with the designer majority, while a small MLP head trained directly on TASTE reaches 0.611 — approaching the 0.741 single-rater ceiling.

02The disjointness description and the released data

The paper describes the collection design the same way in three places. The data section says each sub-dimension was rated on a “separate, disjoint set of 80 prompts.” Table 1's caption says the cohorts use “disjoint 80-prompt batches per sub-dimension.” The limitations section (Section 8) repeats it: each criterion's disjoint batch “forecloses within-rater within-prompt cross-criterion analysis.” A blog summarizing the paper restated it as a limitation:

Each criterion used its own set of 80 prompts, so no design was ever rated on two criteria at once. That kept each rating clean, but it means we can't see how one designer weighs color against typography on the same design, because no one judged the same design on both.

The rationale is sensible. Ask someone to judge a design on one axis only and you get a purer read on that axis: a designer rating color isn't influenced by the typography score they just gave. The paper calls this isolating per-dimension signal, and states the tradeoff explicitly — the design comes “at the cost of preventing within-rater within-prompt cross-dimensional comparison”: same person, same design, two criteria was not supposed to occur.

I took that description at face value until checking it against the released parquet, which shows something different.

What the released data shows

The rankings file (rankings_with_images.parquet) has 721 distinct prompt_ids — about what nine separate 80-prompt batches would produce. But those 721 ids carry only 162 unique prompt texts: 82 in the Aesthetics cohort, 80 in Descriptions. True disjointness would need around 720. The prompt_id turns out to be a bookkeeping key for a prompt-times-sub-dimension pairing, not a distinct brief.

Within a cohort the sharing is complete. Every one of the Descriptions cohort's 80 prompts appears under all four of that cohort's criteria — same prompt, same four generated images, same five designers on each. In Aesthetics, 79 of 82 prompts appear under all five criteria, again with identical image sets and identical raters.

So the release does contain the comparison the paper describes as unavailable. One example: rater eval_002 judged Seedream's output for a calming mental-wellness notification and ranked it #1 on color harmony, #2 on overall preference, #3 on mood, #4 on hierarchy, and #4 on typography — one person, one design, five criteria. That design is brief 2 in the explorer; the axis links there show the same four images ranked on each axis of the cohort.

It also means the analyses the paper lists as foreclosed — whether a designer weighs color against typography on the same design, whether color and type verdicts correlate within a rater — can in fact be run on the released file.

What matches the paper

Everything else checks out. The Aesthetics cohort and the Descriptions cohort share zero images, zero prompt texts, and zero designers — “two disjoint cohorts” is exactly right. And the rater-sharing within a cohort is no discrepancy at all: the paper itself says “each evaluator within a cohort sees every prompt batch in that cohort.” The only element that differs from the text is the prompt (and image) disjointness across sub-dimensions. Each user intent appears twice, once per cohort: same intent, different spec, different generated images. A brief under both cohort views makes it concrete: the two cohorts picked different winners on 54 of the 80 paired intents.

How this could happen

The likeliest explanations are mundane. The prompt_id structure — one id per prompt-times-dimension — suggests each sub-dimension was run as its own annotation session, so from a rater's seat each dimension really was a separate batch, judged in isolation, never two criteria side by side. If that's what happened, the point of the protocol is intact and the “disjoint set of 80 prompts” wording describes the sessions rather than the underlying briefs. Reusing the same briefs and images across dimensions also keeps generation and annotation costs down, a pressure the paper's own limitations section mentions. It's also possible the released file was assembled differently from what the text describes, or that I'm misreading something — this is one reader's check of a public file, not an audit. What I can verify is narrow: the parquet shares prompts, images, and raters across each cohort's sub-dimensions, and the paper's wording says each sub-dimension had its own disjoint 80 prompts. The authors could resolve the gap quickly, and nothing else in the paper's analysis depends on it.

03Could low consensus mean a criterion is too vague?

A natural hypothesis: if color harmony has poor consensus among raters, maybe the criterion is too vague — and an expert could make it more specific (hue relationships, saturation spread, contrast, correspondence), turning a fuzzy abstraction into something more quantifiable and agreed-upon. Is that reasonable?

Partly yes. The instinct mirrors what the paper already does one level up: it decomposes the vague question “which design is better” into nine narrower axes. And the data supports the direction — the highest-agreement criteria are the most verifiable ones (spatial accuracy, and description-typography, i.e. “does the prompt's text render correctly”), while the lowest is color harmony, the most purely aesthetic axis. The more a criterion shades toward something checkable, the more designers converge.

But “low agreement” has three possible sources

Only the first is the one decomposition fixes:

1. Construct vagueness — raters are quietly answering slightly different questions because “harmony” isn't pinned down. Decomposing into crisper sub-questions genuinely helps here.
2. Real taste heterogeneity — even with a perfectly crisp definition, people just disagree about which palette is more pleasing. Color preference is highly subjective. Decomposition doesn't dissolve this; it relocates the disagreement to the sub-questions.
3. Weak stimulus signal — maybe the four models are simply close to each other on color, so there's little real difference to detect. That produces low agreement no matter how sharp the rubric.

What the cycle data hints

Color harmony's cycle rate is 0.138, below the random-rater baseline of 0.211 (see §4). The authors use that statistic to separate “weak shared preference plus noise” from “internally consistent factions with incompatible tastes.” A below-null cycle rate points to the former. That partially undercuts the most flattering version of the hypothesis — the version where crisp-but-different expert notions of harmony lurk underneath (camp A scoring by complementary schemes, camp B by analogous ones). If that were happening, you'd expect elevated cycles. Instead it looks more like everyone weakly pulling the same direction with a lot of individual noise on top.

Color specifically resists reduction

People have built formal quantitative color-harmony models (e.g. Ou & Luo). They exist — real expert artifacts of exactly the imagined kind — but their predictive power on actual human harmony ratings is modest and context-dependent. Compare typography-as-rendered-text, which reduces well (“is it spelled right, does it fit”) and scores high agreement. The feasibility of “just quantify it” isn't evenly distributed; color sits on the hard end.

Where this lands: an expert could write a more specific color-harmony rubric, and doing so would plausibly raise measured agreement somewhat — a reasonable, testable bet, in the spirit of the paper. But low consensus is not by itself evidence that the criterion is too vague; it's one of three explanations, the cycle data leans toward “weak shared signal plus noise” rather than hidden expert factions, and color is the criterion least likely to fully cash out into quantifiable parts without quietly changing the question. Treat it as a hypothesis worth an experiment, not a conclusion the low number already licenses.

04Condorcet cycles and the cycle rate

To get a group verdict for a prompt, you take the 5 designers' rankings of the 4 outputs (A, B, C, D) and go pair by pair, taking a majority vote: do more designers rank A above B, or B above A? Do that for every pair and you get a set of majority verdicts.

Usually those verdicts sort into a sensible order: if the group says A beats B and B beats C, you expect it to also say A beats C. When that holds, the group preference is transitive. But sometimes you get A beats B, B beats C, and yet C beats A — a loop with no top, rock-paper-scissors. That's a Condorcet cycle (an intransitive triple).

How a cycle arises — three voters

• Designer 1: A > B > C
• Designer 2: B > C > A
• Designer 3: C > A > B

Tally the majorities: A beats B (designers 1 & 3), B beats C (designers 1 & 2), C beats A (designers 2 & 3). So A → B → C → A. Each individual was perfectly coherent, yet the group has no coherent winner. That's what coherent but incompatible viewpoints produce.

The 0.211 baseline and what “noise” means

If all five designers ranked completely at random, cycles still appear sometimes by chance. At this shape (4 items, 5 raters) the paper computes — by Monte Carlo — that random rankings cycle on about 21.1% of prompts. That 0.211 is the “pure noise” benchmark.

“Noise” here is the idiosyncratic, unshared part of each designer's judgment — personal quirks, momentary impressions, the bits that don't line up with anyone else in a structured way. It is not the same as “everyone ranking randomly.” The picture is: a faint shared signal everyone feels a little, plus a big pile of individual scatter on top.

Two stories the cycle rate tells apart

• Weak shared preference plus noise. All five lean faintly toward the same ordering, each buried in personal noise. The common pull nudges the majority verdicts into agreement, making them more transitive than chance — so cycles drop below 0.211.
• Internally consistent factions. No shared signal, but coherent camps (warm-palette lovers vs cool-palette lovers, etc.), each internally consistent but pointing in incompatible directions — the three-voter example, scaled up. Clashing camps push cycles above 0.211.

Color harmony's 0.138 sits below 0.211, pointing toward the first story — and away from the “hidden expert factions” reading.

The explorer's most contested brief — a motivational social-media post where overall-preference agreement goes negative in both cohorts — shows what this kind of disagreement looks like in practice.

05A correction: “perfectly consistent”

The non-trivial content of the example is the other half: individually-coherent rankings can combine into a group-level cycle. That isn't forced by anything — it's a genuine property of how the five orderings interlock. So the accurate framing is: cycles are purely a group phenomenon here, manufactured by how individual orderings interlock, never by individual incoherence.

One wrinkle: designers actually made six pairwise comparisons that were then collapsed into a ranking via a Bradley–Terry fit, and the authors report no intransitive triples in the released data — so individuals were transitive in practice too. But even where they weren't, the aggregation would have forced a clean ranking anyway.

A related wording fix: a cycle rate above the null signals that disagreement is structured (organized rather than random), not that anything “objective” or “correct” is being tracked. Coherent camps are still taste — just clustered taste.

06Error bars and why you never see exactly 0.211

0.211 isn't a number you'll observe — it's the center of a cloud of possible outcomes. Each prompt is like flipping a biased coin where “heads” = this prompt happened to produce a cycle. The long-run probability of heads is 0.211, but in any particular batch of 80 you might get 14 heads, or 19, or 11, purely by luck. So the measured cycle rate is itself random and bounces around 0.211 from batch to batch.

The standard-deviation math

Each prompt is an independent yes/no trial — a biased coin landing “cycle” with probability p = 0.211. A count of successes over n independent trials follows a binomial distribution, whose standard deviation (the typical spread of the count around its average) is √(n · p · (1−p)):

• n = 80 — the trials (prompts)
• p = 0.211 — probability of a cycle per prompt
• (1−p) = 0.789 — probability of no cycle

So √(80 × 0.211 × 0.789) = √13.32 ≈ 3.65 cycles. To express that as a proportion instead of a count, divide by n: 3.65 / 80 ≈ 0.046 — or use the proportion form directly, √(p(1−p)/n) = √(0.211×0.789/80) ≈ 0.046. Same number, two units.

07The three signal tests

Before treating TASTE as training data, the authors test whether the ratings carry signal at all. They use three statistics, each catching a different failure mode, all measured against the same null (random raters). The shorthand cycle / τ / p_max just names these three.

Side by side: graded agreement asks “how correlated are the full rankings,” majority decisiveness asks “how often does a clear winner emerge on a given pair,” and the cycle rate asks “is the disagreement random or factional.” Different lenses on the same question.

08Cross-domain anchors and subsampling

To answer “is this agreement a lot or a little,” the paper builds three reference anchors at the identical shape (4 items, 5 raters) by subsampling existing datasets: Sushi (food), MovieLens (movies), and the HPSv2 test set (photo-style image quality).

What “subsampling” means — and what a “synthetic prompt” is

Here “prompt” is reused loosely to mean the grouping/unit, not a text input. In TASTE proper a prompt is a real design brief, but what the statistics actually consume is only its shape: a 5-rater × 4-item rank matrix. To compare apples to apples, the anchors manufacture synthetic “prompts” of that shape from the big datasets.

Judgment calls in the recipe

The datasets have different shapes and quirks, so the recipe needs judgment calls that affect the result: raters must actually overlap on the 4 drawn items (you can't rank a film someone never saw), so they restrict to popular items and high-overlap users; and for the HPSv2 photo anchor they exclude real (non-generated) images and pick a clean cluster of 4 comparable models, so one obviously-worse generator doesn't inflate agreement.

What the anchors show

On cycle rate the anchors come in at Sushi 0.107, MovieLens 0.146, HPSv2-test 0.060 — all below the 0.211 null. TASTE's criteria mostly land in that same 0.06–0.15 band. So the “signal” is concretely: designers agree about graphic design roughly as much as random people agree about sushi or movies — real and detectable, but modest. On graded agreement the same story holds: design preference sits between subjective taste (food, movies) and photo-style image quality, with photo-quality showing the strongest agreement of the four.

09Kendall’s τ and Krippendorff’s α

Kendall’s τ

(Greek letter tau — commonly pronounced to rhyme with “now” or “saw”; “taw” is most standard.) It measures how similar two rankings are by counting pairs. For every pair of items, ask whether the two raters put them in the same order (concordant) or opposite order (discordant). Then τ = (concordant − discordant) / total pairs.

Krippendorff’s α

(Greek alpha; “KRIP-en-dorf's alpha,” named after Klaus Krippendorff.) A general-purpose inter-rater reliability coefficient: how much annotators agree, corrected for how much they'd agree by chance. α = 1 − (observed disagreement / expected-by-chance disagreement). α = 1 is perfect agreement; α = 0 means they agree only as much as random labeling would; negative means systematic anti-agreement. It's popular because it handles any number of raters, missing data, and different data types (nominal, ordinal, …) in one framework. In TASTE it's used mainly on the hallucination flags and as a cross-check on the rankings.

10Why the α values are “low”

TASTE's α values are around 0.19 on the rankings, and 0.141 (Aesthetics) / 0.165 (Descriptions) on the hallucination flags. “Low” is relative to conventional reliability thresholds. The widely-cited rule of thumb (from Krippendorff's own methodology work): rely on data at α ≥ 0.80; treat 0.667–0.80 as only tentatively usable; consider below ~0.667 too unreliable to draw conclusions from. Those cutoffs are conventions chosen to be conservative for treating human labels as ground truth — not laws of nature.

11The iid-uniform null, in plain terms

This is the “pure noise” baseline, unpacked term by term:

• Null (as in null hypothesis) — the boring default you're trying to disprove: “there's no real preference here.”
• Uniform — every possible ranking of the 4 designs is equally likely; no design is favored.
• iid (“independent and identically distributed”) — each rater draws their ranking independently of the others (no shared opinion, no influence) and all raters follow the same uniform distribution.

Put together: the hypothetical world where 5 raters each pick a random ranking out of a hat, independently, with no shared taste. Every statistic is compared against what that world produces. Beating it means “raters are doing better than random” — a deliberately low bar.

12The “coarse scalar” critique

The cycle rate is a single coarse scalar — one number compressing a lot of structure into a coarse summary. The concern: two opposite forces move it in opposite directions. Weak shared consensus pushes cycles below 0.211; internally-consistent factions push them above. If both are present at once, they can partially cancel, landing you near 0.211 — the same value pure noise gives. So a near-null cycle rate is genuinely ambiguous: it could mean “random,” or “consensus and factionalism in a tug-of-war.”

Reading the pattern across statistics

Each statistic alone is ambiguous, but the three together pin down the story, because the explanations leave different fingerprints:

• Pure noise: τ near 0, p_max near chance, cycles near 0.211.
• Weak shared preference + noise: τ modestly positive, p_max elevated, cycles below 0.211.
• Internally consistent factions: τ can look moderate, but cycles go above 0.211.

So you read the joint signature, not any single number. When all nine TASTE criteria show “τ positive, p_max elevated, cycles at-or-below null,” that combination fits “weak real consensus” and is inconsistent with both “pure noise” and “warring factions.” The reliance on the pattern is simultaneously the right move and a tacit admission that no single statistic is trustworthy in isolation.

13Significance: what’s solid, what’s soft

Ten rater-pairs per prompt

With 5 raters, the number of distinct pairs is (5 choose 2) = (5×4)/2 = 10. For each prompt, Kendall's τ is computed between every pair of designers — A vs B, A vs C, … D vs E — and those 10 values are averaged into the single per-prompt T.

p < 10⁻¹⁰, Bonferroni, and what “rejection” means

• Rejecting the null = the data is so far from the pure-noise world that “it was just random raters” becomes untenable; you reject that hypothesis.
• p < 10⁻¹⁰ = the probability of agreement this strong if raters were random is below one in ten billion — astronomically unlikely.
• Bonferroni correction guards against multiple comparisons: run many tests and some look “significant” by luck, so you tighten the threshold by dividing by the number of tests. Here 2 statistics × 9 criteria = 18 tests, so a 0.05 cutoff becomes 0.05/18 ≈ 0.0028. The observed p-values clear even that by a huge margin.

Those rejections are robust: they clear significance by ten-plus orders of magnitude, survive multiple-comparison correction, hold in every one of the nine criteria, and are corroborated by two statistics. So “designers agree more than chance” — i.e. non-randomness — is solidly established. (Note the scope: this establishes non-randomness, not strong agreement.)

The color-harmony 1.6-sigma calculation, piece by piece

• 11 out of 80 — the reported rate is a proportion; 0.138 × 80 ≈ 11, so cycles appeared on ~11 of 80 prompts.
• Null expectation ≈ 17 — if raters were random, the expected count is 0.211 × 80 ≈ 16.9 (the center of the cloud in §6).
• SD ≈ 3.6 — the standard deviation of that count under the null, √(80 × 0.211 × 0.789) ≈ 3.65. This is the “error bar.”
• ~1.6 sigma — measure the gap in error-bar units: (16.9 − 11) / 3.65 ≈ 1.6. “Sigma” (σ) is the symbol for standard deviation, so this is “1.6 error-bars below center.”
• p ≈ 0.05, one-sided, weaker with continuity correction — a 1.6σ deviation is roughly a 5% chance that pure noise lands this far below center (one-sided = only the “fewer cycles” direction). That's right at the conventional threshold, not past it. A continuity correction adjusts for approximating a discrete count with a smooth curve; it nudges the distance smaller and the p-value larger (less significant).

Why τ and p_max are stronger than the cycle rate

It comes down to how much information each statistic extracts from the same 80 prompts. The cycle rate reduces each prompt to a single yes/no bit — cycle or not — then counts. Throwing that much away leaves one coarse proportion with a wide error bar, so even a real effect (color harmony's 1.6σ) barely clears significance. τ and p_max use far more per prompt: τ averages 10 graded pairwise correlations across all 80 prompts; p_max uses the full vote split on every pair. More information per prompt means a tighter handle and far more statistical power — which is why those tests land at p < 10⁻¹⁰ while a single criterion's cycle test sits near p ≈ 0.05. Same underlying agreement; the binary cycle test is just a blunter instrument.

That contrast ties the whole thread together: “real agreement exists” rests on the powerful τ/p_max rejections and is solid, while the faction-versus-noise discrimination the cycle rate is meant to supply is the weakest, most ambiguous leg — which is exactly why no single cycle number, color harmony's included, should be made to carry a strong conclusion on its own.

14Do frontier labs already do this?

Partly, but not in this exact form — and mostly we don't know. Labs do run quality controls on preference data: inter-annotator agreement checks, gold-standard test questions, filtering low-agreement annotators, calibration, sometimes modeling annotator disagreement explicitly. Some research lines (on annotator disagreement, “distributional” preference modeling) clearly recognize that a single label hides structure. So the spirit — validate that preference labels carry learnable signal before training — isn't foreign.

What's less standard, and what TASTE pushes, is (a) a specific published battery distinguishing noise from factions via Condorcet cycles against an exact null, and (b) doing it as a gating step you report before training. Much of what labs actually do here is internal and unpublished, so the safe statement is: they probably do something in this neighborhood, with details undisclosed.

Other benchmarks for these metrics

Kendall's τ has no universal good/bad thresholds — it's read relative to a null or to comparison datasets, which is exactly why the anchors exist. Krippendorff's α does have the widely-cited rules of thumb (§10). For inter-rater work generally, people also cite Cohen's/Fleiss' κ with the Landis–Koch labels (“substantial” agreement at 0.61–0.80, etc.). The cycle-rate-versus-null test, by contrast, is uncommon as a reported benchmark — part of why the paper supplies its own reference points.

15Reference tables

Cohort structure (Table 1)

Cross-domain anchors (Table 2)

Per-criterion statistics (Table 3)

Note the pattern: the description-fidelity dimensions (spatial accuracy, description-typography) carry the strongest graded agreement, while color harmony carries the weakest — the basis of the §3 discussion.

16Caveats and open questions

• The disjointness wording doesn't match the release. The paper says each sub-dimension got its own 80 prompts; the released parquet shares one prompt batch — same designs, same raters — across each cohort's sub-dimensions (§2). The likeliest explanations are mundane (per-dimension sessions described as disjoint batches, or a release assembled differently from the text); only the authors can say.
• Beating the null is a low bar. “Designers aren't literally rolling dice” is much weaker than “designers agree usefully.” The paper is honest that the agreement is significant but moderate.
• The cycle rate is a coarse scalar. Weak consensus and mild factionalism can partly cancel in it, so a near-null cycle rate doesn't cleanly certify the absence of factions (§12). Mitigated by reading the pattern across three statistics and nine criteria.
• Subsampling involves judgment calls. Tie handling, overlap requirements, and model-cluster selection in the anchors all affect the comparison — and the MovieLens tie-breaking mechanism specifically isn't documented in the text reviewed here (§8).
• It's a recent preprint. The June 2026 arXiv posting may not yet be peer-reviewed; the framework-as-best-practice is a fresh proposal whose reception hasn't accumulated. The underlying statistical tools, separately, are well-accepted.
• Reliability is low by convention. α ≈ 0.14–0.19 sits well below the usual 0.667/0.80 thresholds — real signal, but not “trust as gold labels” territory.

The simplest check on all of this is the data itself. Browse the 80 paired briefs in the explorer, most contested first, to see what these numbers look like in practice — τ ≈ +0.13 is five professionals agreeing weakly through a lot of individual noise.