5 to 8 minute read
Judging & Evaluation Craft
01
Blind and Double-Blind Review
The many names these programs go by, and the single shape they almost all share.
5–8 min read
Of all the structural choices covered in this book, this is the one most likely to get treated as optional when it shouldn’t be. Bias doesn’t announce itself, it operates quietly, through name recognition, company reputation, or a familiar writing style, and blind review is the single most direct tool available for interrupting it.
In plain terms: Blind review hides entrant identity from judges. Double-blind hides both entrant identity from judges and judge identity from entrants. Both exist to reduce the effect of reputation, name recognition, or personal connection on scoring.
- Blind review is the more common of the two, and it’s useful whenever entrant identity could unconsciously sway a score. A well-known company name, a familiar author, a recognizable brand, these carry reputational weight that has nothing to do with the quality of the specific entry in front of a judge. Blind review strips that signal out entirely, forcing evaluation of the work itself. This doesn’t require judges to be dishonest or biased on purpose, unconscious bias operates below the level of intent, which is exactly why a structural fix works better than asking judges to simply try harder to be fair.
- Double-blind adds a second layer, and it matters most in small or tight-knit fields. In a niche industry or a small professional community, judges and entrants often already know each other, sometimes well. Double-blind review protects against a different risk here, not just a judge favoring a known entrant, but a judge softening or hardening their language, or their standard, because they know who might eventually see the feedback. Hiding judge identity from entrants as well removes that pressure.
- Neither approach is free, and it’s worth being honest about the tradeoff. Blind review requires someone to actually strip identifying information from submissions before judges see them, company names, distinctive branding, sometimes even specific project details that would give away who submitted. This takes real process discipline. Skipping this step and calling a program “blind” without actually anonymizing submissions is worse than not attempting it at all, it creates a false sense of rigor.
Recap
- Blind hides entrant identity from judges; double-blind hides identity in both directions.
- Both aim to reduce reputation-based bias, which operates unconsciously, not through bad intent.
- Double-blind matters most in small, tight-knit fields where judges and entrants likely already know each other.
- Blind review only works if anonymization is actually done properly, a nominal claim of blindness without real anonymization is worse than not attempting it.
Quick check
Regardless of what a program is called, the design questions behind it are mostly the same:
1. What does double-blind review add on top of standard blind review?
2. Blind review is the less common of the two approaches.
3. Why is unconscious bias a good reason to use a structural fix rather than just asking judges to try harder?
02
Score Normalization
Four recurring motivations — and why the dominant one shapes every decision downstream.
5–8 min read
In plain terms: Score normalization adjusts raw scores from different judges onto a common scale, so one judge's tendency to score harshly or generously doesn't unfairly affect entrants who happened to draw that judge.
- Without normalization, an entrant’s outcome can depend more on which judge they were assigned than on the quality of their work. Every judge has a personal scoring tendency, some naturally cluster scores in the 6-to-8 range out of ten, others use the full range freely. An entrant reviewed by a naturally strict judge can be disadvantaged compared to an identical entry reviewed by a lenient one, purely due to judge assignment, not merit. This is one of the most common sources of perceived unfairness in multi-judge programs, and it’s rarely intentional on any single judge’s part.
- Normalization works by adjusting each judge’s scores relative to their own scoring pattern, not by changing the underlying judgment. A common approach converts each judge’s raw scores into a relative ranking or a standardized scale based on that judge’s own distribution of scores across all the entries they reviewed. This preserves each judge’s relative view of which entries were strongest, while correcting for how generously or harshly they scored overall.
- Normalization matters most when judges don’t review the same set of entries. If every judge scores every entry, raw score differences partly cancel out naturally, since everyone’s being compared against the same pool. But in larger programs, judges are often split across different subsets of entries, and that’s exactly when normalization becomes essential, without it, you’re comparing entries that were never actually evaluated by the same standard.
Recap
- Normalization corrects for judges scoring on different personal scales, not for differences in entry quality.
- Without it, an entrant’s outcome can depend more on judge assignment than on the quality of their work.
- Normalization adjusts scores relative to each judge’s own pattern, preserving their relative view while correcting for overall tendency.
- It matters most when judges are split across different subsets of entries rather than everyone reviewing everything.
Quick check
Regardless of what a program is called, the design questions behind it are mostly the same:
1. Without normalization, judge assignment alone can affect an entrants outcome
2. What does score normalization adjust for?
3. When does normalization matter most?
03
Building a Rubric That Reduces Bias
The eight core jobs every platform in this category is built around.
5–8 min read
In plain terms: A rubric breaks an overall impression into specific, weighted criteria, giving judges something concrete to score against instead of a vague gut feeling.
- Effective rubrics name each criterion explicitly and define what a high and low score actually looks like. A criterion called “quality” means something different to every judge who reads it. A criterion called “technical execution, scored on whether the core functionality works as described, with partial credit for functionality that works with minor issues” gives every judge the same target to aim at. The more concrete the definition at each score level, the less room there is for two reasonable judges to land on wildly different numbers for the same entry.
- Weight criteria according to what the program genuinely values, not evenly by default. The most common rubric-design mistake is listing four or five criteria and letting them implicitly count equally, without ever deciding on purpose whether that’s actually right. If originality matters twice as much as polish to your program’s actual goals, the rubric should say so numerically, not leave it to each judge’s private sense of priorities.
- Vague criteria invite inconsistent scoring in a way that specific criteria don’t. “Quality” or “overall impression” as a rubric line produces the widest score spread between judges of almost anything you could put on a rubric, because it’s really asking each judge to apply their own undefined internal standard. Specific criteria like “originality of approach” or “clarity of the proposed solution” narrow that gap considerably, because they point judges toward the same thing rather than leaving the target open to individual interpretation.
- A rubric is only as good as the calibration behind it. Even a well-written rubric with specific, defined criteria still leaves room for two judges to interpret the difference between a 7 and an 8 differently. This is a real limit of rubric design alone, and it’s exactly why calibration, covered in Section 5, exists as a companion practice, not a replacement for a good rubric, but a necessary complement to one.
Recap
- Good rubrics name specific criteria and define what different score levels actually look like, not just what’s being measured.
- Weight criteria deliberately based on what the program actually values, not by unstated default equal weighting.
- Vague criteria like “quality” produce the widest score variance between judges, specific criteria narrow that gap.
- Even a strong rubric benefits from calibration, since interpretation gaps at the margins persist regardless of how well the rubric is written.
Quick check
Regardless of what a program is called, the design questions behind it are mostly the same:
1. What is a common flaw in weak rubrics?
2. Rubric criteria should always be weighted equally by default.
3. What still matters even with a well-written rubric?
04
Conflicts of Interest
The eight core jobs every platform in this category is built around.
5–8 min read
In plain terms: A conflict of interest is any personal, professional, or financial connection between a judge and an entrant that could affect, or appear to affect, impartial judgment.
- The appearance of a conflict matters almost as much as an actual one. A program’s credibility depends on outside trust in the process, not just the internal reality of whether bias actually occurred in any specific case. A judge who scores a former colleague’s entry fairly, with zero actual bias, still creates a credibility problem if that relationship wasn’t disclosed, because from the outside, there’s no way to distinguish a fair outcome from a quietly biased one. Disclosure is what makes the difference visible.
- Declaring a conflict and recusing from that specific entry is normal, expected practice, not a mark against the judge. Some organizers worry that asking judges to disclose conflicts will make judges reluctant to volunteer, fearing it looks like an admission of wrongdoing. The opposite framing works better: judges who proactively disclose potential conflicts should be treated as demonstrating exactly the kind of integrity the program wants, not as a source of suspicion.
- Conflicts come in more forms than the obvious ones. Direct financial interest or close personal relationships are the clearest cases, but conflicts also include things like a judge who’s a direct competitor to an entrant, a judge who’s previously mentored or advised an entrant, or a judge with a documented history of public disagreement with an entrant. A conflict-of-interest disclosure process that only asks about employment and family relationships will miss a meaningful share of real conflicts.
Recap
- Conflicts should be declared even when a judge believes they can stay impartial, since appearance matters too.
- Recusal is normal, expected practice, and disclosing a conflict should be treated as a sign of integrity, not suspicion.
- Conflicts extend beyond direct financial or family ties, to competitive relationships, past mentorship, and documented history.
Quick check
Regardless of what a program is called, the design questions behind it are mostly the same:
1. Only actual bias matters, not the appearance of a possible conflict.
2. What is recusal, in this context?
3. What kind of relationship is often missed by disclosure forms that only ask about employment and family?
05
Calibration Sessions
The eight core jobs every platform in this category is built around.
5–8 min read
In plain terms: A calibration session is a meeting where judges review sample entries together before real scoring begins, to align on what different score levels actually mean in practice.
- Without calibration, two judges can read the exact same rubric and still score identical entries quite differently. Words like “strong,” “excellent,” or “meets expectations” are interpreted differently by different people, even when everyone genuinely believes they’re applying the same standard. A judge who considers themselves a tough grader and one who considers themselves generous will produce systematically different scores on the same entry, not because either is wrong, but because neither has a shared reference point for what a given score level actually represents.
- A short calibration session meaningfully improves consistency, and it doesn’t need to be long to work. Having judges independently score the same two or three sample entries, then briefly compare and discuss where their scores diverged and why, surfaces very different interpretations of the rubric before those differences affect a full round of real scoring. Even 20 to 30 minutes of this exercise produces a measurable improvement in cross-judge consistency, it’s a small time investment relative to the credibility it protects.
- Calibration works best with real or realistic sample entries, not abstract examples. Using actual past entries, anonymized if needed, or entries specifically constructed to represent different quality levels, gives judges something concrete to disagree about and discuss, which surfaces real interpretation gaps far more effectively than a purely theoretical conversation about what the rubric words mean.
Recap
- Calibration aligns judges’ interpretation of the rubric before real scoring starts, improving consistency.
- Without it, judges’ personal scoring tendencies (naturally tough or generous) produce systematic differences even on the same rubric.
- Even a short session, 20 to 30 minutes, meaningfully improves consistency across a judging panel.
- Calibration works best using real or realistic sample entries, not abstract discussion of what rubric words mean.
Quick check
Regardless of what a program is called, the design questions behind it are mostly the same:
1. What problem does calibration mainly solve?
2. A calibration session needs to be several hours long to be useful
3. What kind of sample entries work best for calibration?
06
Deliberation and Resolving Disagreement
The eight core jobs every platform in this category is built around.
5–8 min read
In plain terms: Deliberation is the discussion stage where judges compare notes and work toward a final decision, typically after individual scoring is done.
- Effective deliberation grounds disagreement in specific rubric criteria rather than general impressions. “I scored this lower on criterion two because the proposed timeline seemed unrealistic” gives the group something concrete to discuss and potentially reconsider. “I just didn’t like it as much” gives the group nothing to work with, it’s not actionable, and it tends to just restate a disagreement rather than move toward resolving it. Judges should be encouraged, and ideally trained, to frame disagreement in rubric terms even when their initial reaction was more instinctive.
- Programs should decide in advance how unresolved disagreement gets settled, rather than deciding in the moment. Common methods include simple averaging of individual scores, a discussion-based consensus process, or giving a lead judge or committee chair a deciding vote when judges can’t reach agreement. None of these is inherently better than the others, but whichever one a program uses should be decided and communicated before deliberation starts, not improvised on the spot when a real disagreement actually happens, which tends to produce a method that looks like it was chosen to favor a particular outcome.
- Deliberation should have a clear endpoint, not run indefinitely until everyone happens to agree. Open-ended discussion aimed at full consensus can drift into whoever argues longest or most persistently effectively winning, which isn’t the same as reaching the best decision. A structured deliberation process, with a defined time limit and a defined fallback method if consensus isn’t reached, produces both a faster and a fairer outcome than an unstructured conversation with no clear stopping point.
Recap
- Ground deliberation in rubric criteria, and decide your disagreement-resolution method before you need it.
- Vague disagreement (“I just didn’t like it”) isn’t actionable, rubric-grounded disagreement is.
- Deciding the resolution method after a disagreement happens tends to look like it favors a particular outcome, decide it in advance.
- Give deliberation a clear time limit and fallback method, open-ended consensus-seeking can reward persistence over the best decision.
Quick check
Regardless of what a program is called, the design questions behind it are mostly the same:
1. It is best to decide how disagreement gets resolved only if and when it actually happens
2. What makes deliberation more productive, according to this section?
3. What is a risk of open-ended deliberation with no time limit?
07
Confidentiality in Judging
The eight core jobs every platform in this category is built around.
5–8 min read
In plain terms: Confidentiality means scores, discussions, and results stay between judges and organizers until an official announcement, no exceptions for informal conversations.
- This extends further than most people initially assume. Mentioning a leaning or a score casually, even outside any official channel, at a networking event, in a text to a friend who happens to know an entrant, still breaks confidentiality. It doesn’t need to be a formal leak to undermine trust in the process, it just needs to get back to an entrant or the public in some form, and casual mentions travel further than people expect.
- Confidentiality protects judges as much as it protects the program. A judge whose informal comment about an entry gets relayed and misquoted can find themselves defending a position they never actually took, or facing pressure from an entrant who heard a partial or distorted version of their feedback. Strict confidentiality during the process protects judges from exactly this kind of situation, in addition to protecting the program’s credibility.
- Confidentiality expectations should be stated explicitly to judges, not assumed to be common sense. What counts as a breach isn’t always obvious, a judge might reasonably think that mentioning general trends (“the entries this year seemed weaker overall”) is harmless, when it can still signal information that shouldn’t be shared yet. Spelling out concrete examples of what’s covered, not just a general instruction to “keep it confidential,” reduces accidental breaches significantly.
Recap
- Confidentiality applies to informal conversations too, not just official reporting channels.
- It protects judges as well as the program, an informal comment that gets relayed or misquoted can put a judge in an unfair position.
- State confidentiality expectations with concrete examples, rather than assuming “keep it confidential” is self-explanatory.
Quick check
Regardless of what a program is called, the design questions behind it are mostly the same:
1. Does casually mentioning a score outside official channels count as breaking confidentiality?
2. Confidentiality only applies to written reports, not spoken conversation
3. Who does strict confidentiality protect, according to this section?
08
Eligibility Screening
The eight core jobs every platform in this category is built around.
5–8 min read
In plain terms: Eligibility screening checks that an entry meets the program's basic rules before it ever reaches a judge, so judging time is spent only on entries that could actually win.
- Keeping this step separate from judging protects judge time and judgment quality. It saves judges from reviewing entries that were never going to qualify regardless of quality, and it keeps the judging process itself focused purely on merit rather than administrative compliance. A judge who has to simultaneously evaluate whether an entry meets basic eligibility rules and how good it is produces less consistent scoring on both fronts than a judge who’s handed only entries that already passed screening.
- Eligibility screening should use clear, checkable criteria, not judgment calls. Good screening criteria are binary and verifiable, does the entry meet the word count, is it in an eligible category, was it submitted before the deadline, not subjective questions that belong in judging instead. If a screening criterion requires real judgment to apply, it’s probably actually a judging criterion that’s been mislabeled as an eligibility rule.
- Screening earlier catches problems when they’re cheaper to fix. An entry that’s disqualified during screening, before judges have invested time reviewing it, is a minor administrative event. The same disqualification discovered after judging, or worse, after a winner has been announced, is a much bigger problem, both operationally and for the program’s credibility. Front-loading screening as early as possible in the process minimizes the cost of catching an ineligible entry.
Recap
- Eligibility screening is a separate, earlier step from judging, protecting judge time and keeping judging focused on merit.
- Screening criteria should be clear and checkable, not judgment calls, if it requires judgment, it’s probably actually a judging criterion.
- Screening earlier catches problems when they’re cheapest to fix, before judge time is invested or a winner is announced.
Quick check
Regardless of what a program is called, the design questions behind it are mostly the same:
1. Eligibility screening should generally happen at the same time as judging, not before it.
2. What does eligibility screening protect, according to this section?
3. What kind of criteria belong in screening rather than judging?
09
Common Questions and Edge Cases
The eight core jobs every platform in this category is built around.
5–8 min read
In plain terms: Eligibility screening checks that an entry meets the program's basic rules before it ever reaches a judge, so judging time is spent only on entries that could actually win.
- How should judges handle a tie between two entries? Decide your tie-break method in advance, a secondary criterion the rubric already tracks, a lead judge’s deciding vote, or naming co-winners if your program structure allows it, so it isn’t improvised after the fact. Document whichever method you choose in your judging guidelines, an undocumented tie-break decision, even a reasonable one, can look arbitrary after the fact simply because there’s no record of it being decided in advance.
- Can a judge abstain from scoring a specific entry? Yes, especially with a possible conflict of interest or a genuine lack of relevant expertise for that particular entry. Abstaining is the responsible choice in these situations, not a failure or a gap in the judging process, a forced score from someone who shouldn’t be scoring that entry is worse than an honest abstention that gets covered by another qualified judge.
- Can a judge change a score after submitting it? Usually only within a window the program defines, and often with a stated reason required, rather than an open-ended ability to revise at any point. This balances two real needs: allowing for genuine second thoughts or corrections of a clear error, while preventing scores from drifting after a judge learns something about how others scored or reacted, which would undermine the independence of individual scoring.
- Is it normal for judges to know entrants personally in small fields or communities? Yes, this is common, and it’s often exactly why those judges are qualified in the first place, expertise and community overlap tend to go together in smaller or specialized fields. Disclosure is what matters here, not the fact of the connection itself, a disclosed relationship that’s appropriately handled (through recusal or simply flagged awareness) is a normal part of running programs in niche fields.
- What is the difference between a jury and a judging panel? In practice, little to none, the terms are generally used interchangeably across programs and industries. Neither term implies a specific structure, size, or process on its own, what actually matters is how a specific program defines and documents its panel’s role, not which word it uses to describe it.
- What should happen if a judge is found to have broken confidentiality? Programs should have a stated consequence in their judge agreement, decided in advance rather than negotiated after the fact once a specific breach has already happened and emotions or relationships are involved. Having this written into the judge agreement upfront, even briefly, makes handling a real breach considerably less fraught than working it out in the moment.
Recap
- Set tie-break rules, score-change windows, and confidentiality consequences before you need them, and document them, not after.
- Abstaining and disclosing personal connections are normal, responsible judge behaviors, not shortcomings, especially in smaller or specialized fields.
- “Jury” and “judging panel” are effectively interchangeable terms, what matters is how a program defines the role, not the label.
Quick check
Regardless of what a program is called, the design questions behind it are mostly the same:
1.When should a tie-break method be decided?
2.knowing an entrant personally always disqualifies a judge from a program in a small field
3.Why should score-change windows have a defined limit rather than being open-ended?





























