Scoring AI-Generated Ideas with Criteria and a Scorecard

AI in facilitationMethods

Fix criteria before ideation, run the AI scoring twice in shuffled order, cut the pool on the totals, and let the group decide with a method that fits.

•••
12 min read

Quick answer

Fix criteria and weights before anyone sees an idea. Let the AI score the pool twice, in a different order each time, with a reason for every score. Use the totals only to cut the pool to about a dozen, then let the group decide: in the method library, Dot Voting covers 2 to 100 people, the Impact & Effort Matrix 2 to 15 and the Decision Matrix 2 to 12. The AI ranking is an input.

Key takeaways

  • In a preprint by Si, Yang and Hashimoto, the best LLM evaluator told top-rated from bottom-rated ideas with 53.3% accuracy, below the 56.1% consistency between the human expert reviewers.
  • In the State of AI Facilitation 2026, a second AI rating agreed with the first in 65% of 40 scores, below the 70% threshold set in advance.
  • Shi and colleagues ran 15 LLM judges over more than 150,000 evaluations of benchmark solutions and conclude that position bias is not due to random chance, which is the reason to shuffle the idea list between scoring runs.
  • In the method library, Dot Voting is written for 2 to 100 people, the Impact & Effort Matrix for 2 to 15 and the Decision Matrix for 2 to 12.
  • The Dot Voting write-up warns against using it for a high-stakes or binding decision because it measures preference, and the Dotmocracy Handbook calls multi-voting good for up to about a dozen distinct ideas.

Fix the criteria before anyone sees an idea. Let the AI score the pool twice, in a different order each time, with a reason per score, and use the totals only to cut the pool. The group then decides with a method that fits its size; the AI ranking is an input.

In the two studies used here, selection is a weak step. A 2006 study in the Journal of Experimental Social Psychology by Rietzschel, Nijstad and Stroebe had groups generate ideas and choose among them, and "idea selection was not significantly better than chance". In Management Science (2010), Girotra, Terwiesch and Ulrich name "the ability of the group to discern the quality of the ideas" as one of four variables behind a group's best ideas. Both predate generative AI, and their abstracts give no sample sizes. See AI brainstorming workshop: diverge, narrow, converge for the round that produces the pool.

1. Fix the criteria before anyone sees an idea

Write down the criteria, the scale and the weights before the first idea exists, so nobody can redefine them around a favourite. The closest experiment is about hiring. In Uhlmann and Cohen (Psychological Science, 2005), 117 volunteers judged one written applicant for a police-chief job; among those who rated the hiring criteria before learning the applicant's gender, "the commitment intervention eliminated gender discrimination", a bias found among the male participants. No favourite idea was involved, so the step to a scorecard is an analogy.

Kahneman, Lovallo and Sibony propose the same order for strategic decisions (MIT Sloan Management Review, 2019): "Define the assessments in advance", score each attribute independently, judge overall last; their evidence is borrowed from structured hiring interviews, and of their own protocol they write that "Quantitative effects will take years to assess."

The library's Decision Matrix write-up aims for four to seven criteria and explains the early weights: "Fix the weights before scoring starts, otherwise people adjust them until their preferred option wins." A group can write the sheet and the prompt together as a group prompt writing activity.

AI built for workshop planning
Describe your goal and get an agenda draft built from facilitation methods. You stay in charge of every block.
7 days free. No credit card.
Try it free

2. Start from desirability, feasibility and viability

These three lenses are a workable default. IDEO names "three essential perspectives", weights them "depending on what we believe matters most", and gives each a question:

  • Desirability: "Are we solving a problem that matters to people?"
  • Viability: "Can the solution succeed and endure within real-world constraints?"
  • Feasibility: "Can we build it with today's technology and capabilities?"

Older scoring methods show what else a scorecard can carry:

  • A gate. Kepner-Tregoe's Decision Analysis, which the vendor explains with a hiring case, splits objectives "into two categories: MUSTs and WANTs": MUSTs are mandatory minimums, and only WANTs are weighted.
  • A confidence factor. RICE, Intercom's own scoring system, uses one to "curb enthusiasm for exciting but ill-defined ideas", divides by effort, and is not to "be used as a hard and fast rule".
  • A reference idea. Stuart Pugh's controlled convergence, dated to the 1980s by the Cambridge Institute for Manufacturing, "uses a simple matrix to compare concepts" against one datum, without weights.

Pugh's datum is also a weak point. A 2019 conference paper by Nicholson and Collopy, whose abstract describes an enumeration of all cases of attribute ranks, reports that for a typical application the chosen concept varies "with the arbitrary choice of datum or is simply incorrect in about one-third of the cases." The library's Decision Matrix is the weighted variant.

3. Ask for a reason with every score

A score without a reason cannot be checked, so the prompt asks for one sentence per score. NN/g's article on AI-enhanced workshops recommends a scorecard for this narrowing step, from its author's experience. Scored examples belong in the prompt too: in the peer-reviewed Creativity and Innovation Management (2025), Kranzle and Sharratt compared GPT-4 with expert evaluators and write that LLMs need a base framework and a few-shot prompt "to achieve moderate evaluation alignment with experts"; the abstract gives no sample size. Adapt this prompt:

Score these workshop ideas against criteria fixed before the ideas existed. Do not add, merge or rewrite ideas.

Gate: [hard constraint]. Mark every idea "pass" or "fail" with a reason and score only those that pass.

Criteria, each from 1 (low) to 10 (high): [criterion 1 with a one-line definition], [criterion 2], [criterion 3], [criterion 4]. Scored examples from the group: [a weak, a middling and a strong idea].

Return one row per idea: number, one score per criterion, total, one sentence of reasoning per score. Write "not enough information" where you cannot score.

Sort by total and mark every total within two points of the highest total as tied.

Ideas: [numbered list]

The gate is the MUST from step 2. The last two rules follow the library's Decision Matrix write-up, which warns that "guessed scores give a false sense of precision" and treats close totals as effectively tied.

4. Treat the AI ranking as a first pass

One scoring run is a weak instrument, whether the rater is a model or a person. In a preprint by Si, Yang and Hashimoto, 79 expert researchers blind-reviewed 49 ideas from each of three conditions, and the best LLM evaluator, a 2024 model, told top-rated from bottom-rated ideas with an accuracy of 53.3%, "lower than our inter-reviewer consistency of 56.1%", which the authors describe as relatively low agreement between the human reviewers.

Order changes scores too. A preprint by Wang and colleagues on LLMs judging chatbot answers says a ranking can be changed "by simply altering their order of appearance in the context". In a paper listed for AACL-IJCNLP 2025, Shi and colleagues ran 15 judge models over more than 150,000 evaluations of benchmark solutions and conclude that "position bias is not due to random chance". Neither judged ideas, but shuffling between runs is cheap.

Repeat runs differ as well. A paper accepted at EMNLP 2025 reports that "LLM judges have low intra-rater reliability in their assigned scores across different runs" (Haldar and Hockenmaier), and a preprint warns against "over-reliance on single-shot evaluations" (Schroeder and Wood-Doughty); both rated machine-generated text. In the State of AI Facilitation 2026, where an AI rater and no person scored 9 AI-drafted agendas (3 briefs × 3 assistants), a second rating agreed with the first in 65% of 40 scores on six of them, under the 70% set in advance.

Totals suit a coarse cut better than a fine ranking. Hirota and colleagues, listed for the ACL 2026 Industry Track, analysed about 3,000 expert scores on 300 patent-grounded product ideas and report "disagreement on fine-grained ordinal scores, while agreement is higher under coarse selection". That is a finding about people; it still supports the tie rule.

People also defer to systems. Buçinca and colleagues, in a 2021 arXiv paper on one experiment with 199 participants, report that explanations did "not appear to reduce the overreliance", while "cognitive forcing significantly reduced overreliance". Lee and colleagues (CHI 2025) surveyed 319 knowledge workers and report, from self-reports, that "higher confidence in GenAI is associated with less critical thinking". A JAMIA review of 74 studies on automation bias, centred on clinical decision support, pooled four of them in "a small, indicative meta-analysis" in which erroneous advice "increased the risk of an incorrect decision being made by 26%"; it lists user accountability among the mitigators. All three concern individuals, so a group choosing ideas is an extrapolation.

In the room, that becomes four moves. The third is this article's inference and untested: a 2011 journal review by Furnham and Boo documents anchoring across many domains and tasks, and its abstract does not mention AI scores.

  1. Run the prompt twice in separate chats, with the list in a different order.
  2. Cut to about a dozen on the totals and mark the ideas whose totals moved most.
  3. For those ideas, have people write their own score before any model score is shown.
  4. Read both rationales, set the score from the criterion wording, and name who owns the decision.

5. Let the group decide with a method that fits its size

Choose the method by group size and by what the decision commits people to. The library's bounds:

Method Group size in the library Duration in the library Use it for
Dot Voting 2-100 5-20 min Narrowing a long list to a shortlist
Impact & Effort Matrix 2-15 30-60 min Sorting a shortlist on two axes
Decision Matrix 2-12 30-60 min Three to seven comparable options

Above 12 people the Decision Matrix is out of range, and above 15 only Dot Voting remains. With dozens of ideas, the Decision Matrix write-up says to sort first "and bring only the shortlist into a matrix".

A dot vote carries preference and little else. The library advises against it for a high-stakes or binding decision because "dot voting measures preference, not feasibility or evidence", and below four people it prefers a short structured discussion. Jason Diceman's Dotmocracy Handbook (2010), a critique by the author of a competing method, calls multi-voting good for "up to about a dozen related, but distinct, ideas" and warns that similar ideas split the vote.

Sources differ on the vote budget. Gamestorming says "giving each participant five votes to cast works well"; the library gives one third of the options, so 12 options mean 4 dots.

No experiment on dot voting turned up for this article, so a pull from visible dots is untested here. The nearest evidence is an estimation task: in Lorenz and colleagues (PNAS, 2011), 144 participants revised answers to factual questions after seeing others' answers, which narrowed the answers "without improvements of its collective error" and raised confidence "despite lack of improved accuracy." With the library's note that "anonymous formats prevent follow-the-boss voting", that is reason enough to collect scores and dots silently.

Check the method when an assistant proposes it. In the State of AI Facilitation 2026 agenda test, 7 of 9 AI-drafted agendas used a method against its write-up's warning without saying so, five of them a dot vote shaping a binding decision with the leadership in the room. The five checks for an AI-generated agenda cover that review.

A worked scorecard

Picture eight people from one product team with 40 AI-generated ideas and one prototype slot. The team, the ideas and every score below are invented; the method figures come from the library.

Before ideation, the group fixed four equally weighted criteria, the three lenses plus "fits this quarter's goal", each scored from 1 to 10, so the maximum total is 4 × 10 = 40. The prompt ran twice with the list reshuffled. Six rows of the first pass:

Idea Desirability Feasibility Viability Goal fit Total, pass 1 Total, pass 2 Gap
A 8 7 8 7 30 31 1
B 9 5 8 7 29 24 5
C 7 8 6 7 28 28 0
D 6 9 6 7 28 29 1
E 8 6 6 6 26 31 5
F 5 8 6 6 25 25 0

By the tie rule, A, B, C and D (30, 29, 28 and 28) are within two points of the highest total and tied on the first pass; E (26) and F (25) are not. B falls from second to last of the six and E rises from fifth to joint first. Those two get read first, and the session runs like this:

  • Minutes 0-10: Two scoring runs; the 12 highest first-pass totals go up with both totals and the rationales, the other 28 (40 − 12) on a second list.
  • Minutes 10-20: Silent reading. People write their own scores for B and E, then the group sets them.
  • Minutes 20-38: Dot Voting on the 12, at the 18 minutes of the library's script (3 + 2 + 5 + 3 + 5). One third of 12 options is 4 dots each, so 8 × 4 = 32 dots, and the top five go forward.
  • Minutes 38-71: Decision Matrix on the five, which carries the binding choice while the dot vote only narrows. Of the script's 50 minutes, the 10 for collecting criteria and 7 for weighting were spent before ideation, which leaves 50 − 10 − 7 = 33, inside the method's 30 to 60.
  • Total: 10 + 10 + 18 + 33 = 71 minutes.

Conclusion

Before your next AI-assisted ideation round, write the criteria sheet and adapt the prompt. Run it twice with the list reshuffled, show both totals with rationales, and pick the method from the table in step 5.

References

Share:

Ready to plan your next workshop?

7 days free. No credit card. No setup fees. Cancel anytime.