Interview Design

Structured interview design is the discipline of building hiring processes that produce consistent, defensible, and predictive hiring decisions. The core insight is that unstructured conversations are notoriously unreliable predictors of job performance - structured processes with explicit rubrics dramatically improve both accuracy and fairness. This skill covers the full lifecycle: scoping the interview loop, writing rubrics, building coding challenges, calibrating interviewers, and running debriefs that lead to confident decisions.

When to use this skill

Trigger this skill when the user:

• Needs to design an interview loop or process for a role
• Wants to create scoring rubrics or evaluation criteria
• Asks how to build a coding challenge or take-home assignment
• Needs help writing behavioral interview questions
• Wants to design a system design interview round
• Is trying to assess culture fit in a structured, defensible way
• Needs to run calibration sessions with a panel
• Asks how to run an effective debrief meeting

Do NOT trigger this skill for:

• Preparing as a candidate to pass interviews (different audience, different goal)
• Compensation benchmarking or offer negotiation (use a compensation skill instead)

Key principles

1. Structured beats unstructured - Consistent questions asked in the same order with pre-defined scoring criteria outperform free-form conversations every time. Interviewers who "go with their gut" introduce bias, not signal.

2. Score independently before debrief - Every interviewer must submit a written score and evidence summary before the panel debrief. Verbal-only debrief allows the first strong opinion to anchor everyone else. Written scores first.

3. Test for the actual job - Every interview exercise should map to a real task the candidate will perform in the role. If a backend engineer will never sort arrays on the job, don't test array sorting in isolation. Use job-relevant problems.

4. Rubrics prevent drift - Without a rubric, two interviewers evaluating the same candidate will produce wildly different scores. A rubric aligns on what "strong" and "weak" looks like before the first candidate walks in.

5. Debrief is where decisions happen - The debrief meeting is not a vote-counting exercise. It is a structured discussion to surface new evidence, resolve disagreements, and reach a confident collective judgment. The hiring manager owns the final call.

Core concepts

Interview types map to different evaluation needs. Coding interviews assess problem-solving and technical mechanics. System design interviews assess architectural thinking at scale. Behavioral interviews (using STAR) assess past behavior as a proxy for future behavior. Values/culture interviews assess alignment with how the team operates. Take-homes assess real-world execution and follow-through. Most loops include 3-5 rounds covering different dimensions so no single round carries all the weight.

Rubric design is the practice of defining expected performance at multiple levels (typically 1-4 or Strong No / No / Yes / Strong Yes) before interviews begin. A good rubric specifies concrete behaviors, not adjectives. "Breaks problem into subproblems, names variables clearly, asks clarifying questions before coding" is a rubric. "Good technical skills" is not. See references/rubric-templates.md for ready-to-use rubric templates.

Signal vs noise distinguishes real predictors of job performance from irrelevant factors. Signal: how a candidate structures ambiguity, responds to hints, explains trade-offs. Noise: how polished their communication style is, whether they went to a brand-name school, how quickly they reached the solution. Train interviewers to write down evidence (what the candidate said/did) rather than impressions ("seemed smart").

Calibration is the practice of running mock interviews with known candidates (or invented personas) so interviewers practice applying the rubric consistently before live interviews begin. A calibration session where two interviewers score the same response and then compare notes surfaces misalignment early.

Common tasks

Design a structured interview loop

Start by mapping the role's core competencies - typically 4-6 dimensions that predict success. Common dimensions for engineering roles:

Rules for a well-designed loop:

• Every dimension is covered by exactly one round (no redundancy)
• No interviewer covers more than one dimension (keeps each fresh)
• The loop can be completed in one business day on-site or two days virtual
• Assign a "bar raiser" - someone outside the immediate team with veto power

Create scoring rubrics - template

Use a 4-level rubric for each dimension. The key is defining the middle levels precisely - candidates cluster there, and those are the hard decisions.

Dimension: [Name, e.g., "Problem Decomposition"]
Weight: [High / Medium / Low]

4 - Strong Yes
  Candidate independently breaks problem into clean subproblems. Names
  intermediate data structures without prompting. Explains trade-offs of
  multiple approaches before choosing. Handles edge cases proactively.

3 - Yes
  Candidate breaks problem into subproblems with minor prompting. Solves
  the core problem correctly. Handles most edge cases when prompted.
  Explains the primary trade-off.

2 - No
  Candidate solves simple version but struggles to generalize. Requires
  significant prompting to identify subproblems. Misses important edge
  cases. Does not discuss trade-offs unless directly asked.

1 - Strong No
  Candidate cannot decompose the problem independently. Solution is
  incorrect or incomplete. Does not respond to hints. Cannot explain
  what their own code does.

See references/rubric-templates.md for complete rubrics for coding, system design, behavioral, and culture fit rounds.

Build a take-home coding challenge

Take-homes reveal real-world execution that 45-minute whiteboard problems cannot. Design one that:

• Scopes to 2-3 hours max - Respect candidate time. If it takes a senior engineer 2 hours, calibrate down. State the expected time in the instructions.
• Uses a realistic problem - "Build a rate limiter for our API" beats "implement a binary search tree." Domain-adjacent problems reveal how candidates think about the actual work.
• Provides a starter repo - Give candidates a repo with the scaffolding, CI, and test runner already wired. Evaluating candidates on setup skills is noise.
• Defines evaluation criteria upfront - Include a EVALUATION.md in the repo that lists exactly what reviewers will look for: correctness, test coverage, code clarity, README quality.
• Has a follow-up interview - Schedule a 30-minute code walkthrough. This prevents submitting work that isn't the candidate's own and surfaces how they think about their own decisions.

Evaluation checklist for reviewers:

• Does the solution solve the stated problem?
• Are edge cases handled?
• Is the code readable without explanation?
• Are there tests, and are they meaningful?
• Does the README explain design decisions?
• Are there obvious improvements the candidate noted themselves?

Design behavioral interview questions - STAR format

Behavioral questions follow the pattern: "Tell me about a time when..." The STAR framework (Situation, Task, Action, Result) gives candidates a structure and gives interviewers a rubric for what a complete answer looks like.

Writing strong behavioral questions:

• Anchor to a specific competency (e.g., "conflict resolution" or "driving alignment without authority")
• Phrase as past behavior, not hypothetical: "Tell me about a time you disagreed with your manager" not "What would you do if..."
• Prepare follow-up probes in advance

Scoring STAR responses:

• Situation/Task - Is the context clear and relevant to the role?
• Action - Did the candidate describe their specific actions (not "we")?
• Result - Is there a concrete, quantified outcome?
• Learning - Does the candidate show reflection and growth?

Design system design interviews

System design interviews assess whether a candidate can architect solutions for real-world scale and ambiguity. The structure matters as much as the content.

Interview structure (45-60 minutes):

1. Requirements clarification (5-10 min) - Candidate should ask scoping questions: scale, read/write ratio, latency requirements, consistency model. Award signal for good questions, not just correct answers.

2. High-level design (10-15 min) - Candidate draws the major components and data flows. Watch for separation of concerns and component boundaries.

3. Deep dive (15-20 min) - Interviewer picks one or two components to explore in depth: database schema, caching strategy, failure modes.

4. Trade-offs and bottlenecks (5-10 min) - Candidate explains what they would improve with more time, where the system might break, and why they made specific choices.

Rubric signals to watch:

• Does the candidate ask clarifying questions before whiteboarding?
• Can they estimate load and justify their component choices with numbers?
• Do they proactively identify single points of failure?
• Can they explain the trade-off between consistency and availability?
• Do they adjust the design when the interviewer changes a constraint?

Run calibration sessions

Calibration prevents rubric drift before it happens. Run one calibration session per new interviewer and one per quarter for existing panelists.

Calibration session format (60 minutes):

1. Distribute a transcript or video of a mock interview (use a fabricated candidate, never a real one without consent)
2. Each interviewer scores independently using the rubric - no discussion yet
3. Reveal all scores simultaneously (prevents anchoring)
4. Discuss every dimension where scores diverge by 2+ points
5. Reach consensus on the "correct" score and the reasoning
6. Document the calibrated examples as reference cases for future interviewers

Red flags indicating calibration is needed:

• Two interviewers gave the same candidate a 4 and a 1 on the same dimension
• An interviewer cannot cite specific evidence for their score
• Scores correlate with candidate demographics, not candidate performance
• The team has not hired anyone in 6 months despite many interviews

Conduct effective debriefs

The debrief is the most consequential 30-60 minutes in the hiring process. Run it badly and you amplify bias. Run it well and you surface the truth.

Before debrief:

• All interviewers submit written scorecards independently (hiring manager cannot see scores until all are submitted)
• Block 48 hours maximum between last interview and debrief

Debrief agenda:

1. Hiring manager reads all scorecards silently (5 min)
2. Each interviewer speaks to their dimension only - what evidence they saw, what level they scored, why (2 min per interviewer, no interruption)
3. Open discussion on dimensions with significant disagreement
4. Hiring manager asks: "Is there anything about this candidate we have not yet discussed that is relevant?"
5. Hiring manager states the decision and the primary evidence that drove it

Decision framework:

• Any Strong No from a bar raiser or domain expert is a block unless directly rebutted with evidence (not "they seemed nervous")
• "Probably yes" is a No - only hire on conviction
• Document the stated rationale in the ATS for every decision, hire or no-hire

Anti-patterns

Gotchas

1. Rubrics calibrated only on strong candidates produce grade inflation - If interviewers see 10 qualified candidates in a row, their mental model of "a 3" drifts upward over time. Run calibration sessions quarterly using the same reference examples to anchor scores. Grade inflation causes you to reject good candidates because they score "only" a 3 when the bar has silently moved to 3.5.

2. Take-home exercises without a time cap select for availability, not skill - Without an explicit time cap, candidates who are between jobs or have no outside commitments will spend 12 hours on a "2-3 hour" challenge, and their submissions will look objectively better. State "we expect this to take 2-3 hours" in the instructions and calibrate your evaluation rubric to what a strong engineer can produce in that time.

3. Allowing the hiring manager to see scores before all interviewers submit creates anchor bias - If one interviewer submits a Strong Yes early and the hiring manager shares it in Slack, every subsequent scorer is subconsciously anchored. Enforce blind independent scoring: no interviewer sees anyone else's score or written feedback until all scorecards are submitted.

4. Interview loops covering the same dimension twice produce inflated confidence - Two coding rounds both asking algorithm questions don't double the signal - they create redundant data while leaving system design, communication, or values completely unevaluated. Map each round to a unique dimension before the first candidate interviews, and stick to the map.

5. Debrief dominated by the most senior person in the room is not a debrief - If an engineering director speaks first with a strong opinion, junior interviewers defer and the decision reflects the director's prior, not the collective evidence. The facilitator must ask every interviewer to present their evidence before any discussion begins, starting with the most junior voice.

References

For detailed content on specific topics, read the relevant file from references/:

• references/rubric-templates.md - Ready-to-use scoring rubrics for coding, system design, behavioral, and culture fit rounds

Only load a references file if the current task requires deep detail on that topic.