How to Interpret Merge Scorecard Results for Your Team

Reading a Merge scorecard effectively means understanding the context behind each number and what it signals about a candidate’s day-to-day judgment as a code reviewer. This page walks through each dimension, explains how difficulty level shapes expectations, and gives you practical guidance for bringing scorecard results into your hiring conversations.

​

Calibrating by difficulty

Scorecard expectations are normalized to the difficulty level you set when configuring the assessment. A score of 4/5 on risk detection at the mid-level setting reflects a meaningfully different performance than a 4/5 at the senior setting — the senior pull request embeds subtler, higher-stakes issues that require broader experience to catch. Always compare candidates who were evaluated at the same difficulty level. Mixing results across difficulty tiers produces misleading comparisons and can work against candidates who were assessed on harder material. If you realize partway through a hiring cycle that your difficulty setting was misconfigured, re-invite affected candidates rather than adjusting your interpretation after the fact.

​

Reading code quality reasoning

Look at what issues the candidate flagged and, more importantly, how they framed them. Strong reviewers explain why something is a problem and suggest a concrete fix — they connect the observed code to a consequence, such as a performance bottleneck, an incorrect invariant, or a maintenance burden. Weak reviews tend to surface observations like “this looks off” or “consider renaming this” without grounding the comment in a real impact. Pay attention to the ratio of substantive comments to noise. A candidate who leaves twenty comments, half of which are style preferences and half of which are genuine defects, tells you something different from a candidate who leaves eight highly targeted, well-reasoned comments.

​

Reading risk detection

Check which bugs or vulnerabilities the candidate caught and which they missed. The scorecard surfaces both hits and misses so you can see the full picture, not just what the candidate chose to comment on. High-stakes roles — security engineering, infrastructure, fintech, healthcare — should weight this dimension heavily. For these roles, a candidate who writes polished comments but misses an injection vulnerability or an off-by-one error in a critical path is a meaningful signal worth discussing with your team before moving forward. For roles where security and correctness are less central, risk detection still matters but can be weighed against code quality reasoning and revision judgment based on your team’s priorities.

​

Reading revision judgment

This is the most differentiating dimension for senior roles and the one candidates are least prepared to game. After the AI revision lands, candidates must decide which of their original comments still apply, which were addressed, and whether the revision introduced any new problems. Ask yourself: Did the candidate re-read the diff after the revision, or did they treat it as a checkbox? Did they notice a regression that the revision introduced? Did they update or withdraw comments that the revision already addressed, or did they leave stale feedback in place? Candidates who engage carefully with the revision demonstrate the kind of sustained attention that separates strong senior engineers from those operating on autopilot. Low revision judgment scores in otherwise strong candidates can be a useful conversation starter in a follow-up interview — ask them to walk you through how they approach a second review pass.

​

Hiring recommendation

The hiring recommendation is a calibrated signal, not a binary pass/fail gate. It factors in the difficulty and specialization settings you configured, and it synthesizes performance across all three dimensions into a single summary output. Think of it as the assessment’s answer to “given the bar you set, how did this candidate do?” Use the recommendation as a conversation starter with your hiring team rather than a final verdict. A strong recommendation on a well-configured assessment is meaningful evidence; a borderline recommendation on an assessment that turned out to be poorly scoped for the role deserves more scrutiny before acting on it.

​

Discussing results as a team

Scorecards are most useful when they fuel structured conversation rather than replacing it. Here are a few ways to bring scorecard results into your hiring process effectively:

• Share the scorecard link with interviewers before the next round. Give them time to read it before the call, not during it.
• Use specific comment examples as interview prompts. For example: “We noticed you flagged the N+1 query in the data access layer — walk us through your reasoning there.” This lets you probe depth and verify that the written comment reflects genuine understanding.
• Align on which dimensions are most important for the role before you start comparing candidates. If your team hasn’t agreed on weights in advance, you’ll find that different interviewers read the same scorecard differently, which makes consensus harder to reach.