Coding Challenge Patterns

Organizing by signal, not by topic

Don't organize your question bank by data structure ("graph questions", "tree questions"). Organize by the competency you want to assess. This lets you pick the right question for the role, not just the right difficulty.

---

Pattern: API Design

Signal: Can the candidate design clean interfaces, handle edge cases, and think about consumers of their code?

Example: Design a key-value store API

Level: Mid Time: 45 min Prompt: Implement a key-value store class with get, set, delete, and keys methods. Then extend it to support TTL (time-to-live) on keys.

Base case: Basic CRUD operations work correctly Standard: TTL works, expired keys are not returned Extension: Implement lazy vs active expiration, discuss trade-offs

Rubric anchors:

• Strong Hire: Clean API surface, handles edge cases (get expired key, delete nonexistent key), discusses memory implications of lazy expiration
• No Hire: API works but no thought given to edge cases or consumers

Example: Build a middleware pipeline

Level: Senior Time: 60 min Prompt: Implement a middleware system where functions can be chained and each can modify a request/response or short-circuit.

Base case: Sequential middleware execution Standard: Support for async middleware, error handling middleware Extension: Conditional middleware (route-based), middleware ordering

---

Pattern: Data Modeling

Signal: Can the candidate choose appropriate data structures and model relationships between entities?

Example: Design a permission system

Level: Mid-Senior Time: 45 min Prompt: Model a role-based access control system. Users belong to organizations, have roles, and roles grant permissions on resources.

Base case: User-role-permission model with basic check function Standard: Hierarchical roles (admin inherits editor permissions), resource scoping Extension: Attribute-based overrides, permission caching strategy

Rubric anchors:

• Strong Hire: Considers inheritance, discusses denormalization for performance, handles the "admin of org A should not see org B" case
• No Hire: Flat user-to-permission mapping, no consideration of scale

Example: Model an event calendar

Level: Mid Time: 45 min Prompt: Design data models for a calendar app supporting one-time events, recurring events, and event modifications (cancel one occurrence, change time of one occurrence).

Base case: One-time events with CRUD Standard: Recurring events with RRULE-style patterns Extension: Exceptions to recurring events (modify/cancel individual occurrences)

---

Pattern: Debugging & Code Reading

Signal: Can the candidate read unfamiliar code, identify issues, and reason about behavior?

Example: Fix the race condition

Level: Senior Time: 30 min Prompt: Present a 50-line function with a subtle race condition (e.g. check-then-act on a shared counter). Ask the candidate to identify the bug, explain the failure scenario, and fix it.

What to prepare:

• The buggy code (should look reasonable at first glance)
• 2-3 specific failure scenarios to discuss
• Multiple valid fixes (mutex, atomic operations, redesign)

Rubric anchors:

• Strong Hire: Identifies the race condition quickly, explains a concrete interleaving that causes failure, proposes multiple fixes with trade-offs
• Hire: Identifies the issue with some hints, proposes a working fix
• No Hire: Cannot identify the issue even with hints

Example: Review this pull request

Level: Mid-Senior Time: 30 min Prompt: Present a 100-line PR with 3-5 intentional issues of varying severity (one security issue, one logic bug, one style issue, one performance concern, one missing test).

Rubric anchors:

• Strong Hire: Catches security and logic issues, prioritizes feedback by severity
• Hire: Catches most issues, reasonable feedback quality
• No Hire: Only catches style issues, misses the security/logic bugs

---

Pattern: Concurrency & Async

Signal: Does the candidate understand parallel execution, synchronization, and async patterns?

Example: Implement a connection pool

Level: Senior Time: 60 min Prompt: Build a generic connection pool with max size, checkout/checkin, timeout on checkout, and health checking.

Base case: Fixed-size pool with blocking checkout Standard: Configurable max size, timeout, idle connection cleanup Extension: Health checking, connection recycling, metrics

Example: Build a rate-limited task queue

Level: Mid-Senior Time: 45 min Prompt: Implement a task queue that processes at most N tasks per second, with configurable concurrency.

Base case: Serial execution with rate limiting Standard: Concurrent execution within rate limit Extension: Priority levels, retry with backoff

---

General rubric template for coding challenges

Dimension	Strong Hire (4)	Hire (3)	No Hire (2)	Strong No Hire (1)
Problem solving	Breaks down problem systematically, identifies edge cases early	Reasonable approach, handles main cases	Struggles with approach, needs significant hints	Cannot make progress even with hints
Code quality	Clean, readable, well-named, idiomatic	Functional with minor style issues	Disorganized, hard to follow	Does not produce working code
Communication	Thinks aloud clearly, explains trade-offs, asks good questions	Communicates approach, responds to prompts	Mostly silent, unclear explanations	Cannot articulate their thinking
Testing mindset	Proactively discusses test cases, boundary conditions	Tests when prompted, covers main cases	No consideration of testing	Does not understand what testing means in context
Extension ability	Elegantly extends to harder variant, code structure supports change	Can extend with some refactoring	Extension requires rewrite	Cannot extend beyond base case

Rubric Calibration Guide

Why calibrate

Without calibration, interviewers develop personal definitions of "strong" and "weak." One interviewer's "Hire" is another's "Strong Hire." This inconsistency means your hiring bar depends on which interviewer the candidate gets - not on the candidate's actual ability. Calibration fixes this by aligning everyone on what each rubric level looks like in practice.

Target: Interviewers agree within 1 point on a 4-point scale at least 80% of the time after calibration.

---

Calibration session format

Frequency: Quarterly, or when onboarding 3+ new interviewers Duration: 90 minutes Participants: 4-8 interviewers who use the same question Facilitator: Interview program lead or senior interviewer

Pre-session preparation

1. Select 3-4 candidate responses to calibrate on:
   • 1 clearly strong response
   • 1 clearly weak response
   • 2 borderline responses (these generate the most useful discussion)
2. Anonymize all responses (remove candidate names, companies, schools)
3. Distribute the rubric being calibrated (not the responses) 1 week before
4. Prepare scoring sheets with the rubric criteria pre-filled

Session agenda

0:00 - 0:05  Context setting
  - State the goal: align on what each rubric level means
  - Remind: no "right" answers - we're calibrating the rubric, not the candidates

0:05 - 0:15  Review rubric together
  - Read each level's behavioral anchors aloud
  - Ask: "Any criteria that are unclear or ambiguous?"
  - Note ambiguous items (these are calibration opportunities)

0:15 - 0:30  Score Response #1 (clear strong)
  - Everyone reads silently and scores independently (5 min)
  - Facilitator collects scores simultaneously (raise fingers or digital poll)
  - Discuss: "Why did you give this score? Which criteria did you weight?"
  - Outcome: should be mostly aligned; if not, rubric needs clarification

0:30 - 0:50  Score Response #2 (borderline)
  - Same process: read, score independently, reveal simultaneously
  - This is where disagreements surface
  - For each disagreement: "Which specific rubric criterion led to different scores?"
  - Update rubric language where ambiguity caused divergence

0:50 - 1:10  Score Response #3 (borderline)
  - Same process
  - Focus on whether rubric updates from Response #2 help alignment

1:10 - 1:20  Score Response #4 (clear weak)
  - Quick alignment check
  - Discuss: "What separates No Hire from Strong No Hire?"

1:20 - 1:30  Wrap-up
  - Summarize rubric changes decided during session
  - Assign someone to update the canonical rubric document
  - Schedule next calibration session

---

Facilitator guidelines

Preventing anchoring

The biggest risk in calibration is anchoring - where one person's opinion influences everyone else.

Rules:

• Never ask "Who wants to go first?" - use simultaneous reveal
• If a senior person speaks first, explicitly ask others to share before responding
• Use anonymous digital polling if available (Slido, Google Forms)
• If someone says "I agree with [person]" - ask them to articulate their own reasoning independently

Handling persistent disagreement

If two interviewers consistently disagree by 2+ points:

1. Identify which rubric criteria they weight differently
2. Ask: "Which of these criteria is more predictive of on-the-job performance?"
3. If the group can't resolve it, add specificity to the rubric (more behavioral anchors, clearer must-haves vs nice-to-haves)
4. Document the disagreement and revisit with data after 1-2 hiring cycles

Common calibration discoveries

Discovery	Resolution
"Clean code" means different things to different people	Add specific examples: "functions under 20 lines, no nested callbacks deeper than 2 levels"
Some interviewers penalize for asking questions	Clarify: questions show maturity, not weakness. Add to rubric as positive signal
Interviewers give bonus points for using specific tech	Remove technology-specific criteria. Test concepts, not brand loyalty
"Communication" is scored inconsistently	Split into sub-criteria: explains approach before coding, responds to hints, asks clarifying questions
Strong candidates who are nervous score low	Add explicit note: evaluate peak demonstrated ability, not average. Nerves are not signal

---

New interviewer onboarding

Shadow program (2-4 weeks)

Week 1: Observe 2 interviews
  - Shadow an experienced interviewer
  - Take notes using the rubric
  - After each: compare your score with the interviewer's and discuss gaps

Week 2: Reverse shadow 2 interviews
  - You run the interview, experienced interviewer observes
  - Experienced interviewer gives feedback on:
    - Question delivery and pacing
    - Hint-giving calibration (too many? too few?)
    - Score accuracy vs rubric

Week 3-4: Solo with review
  - Run interviews independently
  - Submit scores with written justification
  - Interview lead reviews justifications for first 3-4 interviews
  - Graduate to independent when scores align within 1 point consistently

Common new interviewer mistakes

Mistake	Coaching point
Giving too many hints	Let candidate struggle for 2-3 minutes before hinting. Struggling is signal.
Not giving enough hints	If stuck for 5+ minutes, a targeted hint prevents wasting the whole session
Asking leading questions	"Would you use a hash map here?" is leading. "How would you optimize lookup?" is not
Scoring based on personality	Introverts who code well should score the same as extroverts who code well
Comparing to themselves	"I would have done X" is not a rubric criterion. Only score against the rubric
Writing vague feedback	"Seemed okay" is not useful. Write specific observations: "Solved base case in 15 min, needed 2 hints for extension, did not discuss edge cases"

---

Measuring calibration effectiveness

Track these metrics over time:

• Inter-rater reliability (IRR): Percentage of interviews where all interviewers agree within 1 point. Target: 80%+
• Score distribution: If one interviewer gives 90% "Hire" and another gives 50%, there is a calibration problem
• Offer-to-accept ratio by interviewer: If candidates who interview with specific interviewers accept less often, investigate the experience quality
• New hire performance correlation: Do interview scores predict 6-month performance ratings? If not, the rubric (not just calibration) needs work

System Design Question Library

Question selection principles

• Match question complexity to candidate level (mid vs senior vs staff)
• Choose domains the candidate likely understands as a user
• Prepare 4-6 follow-up dimensions per question
• Write expected discussion points before using in interviews

---

Mid-level questions (45 min)

Design a URL shortener

Initial constraints: 100M URLs created/month, 10:1 read-to-write ratio

Expected discussion:

• Hashing strategy (MD5 truncation, base62 encoding, counter-based)
• Storage: key-value store for fast lookups
• Redirect: 301 vs 302 and caching implications
• Analytics: click counting, geographic data

Follow-ups:

• Custom aliases and collision handling
• Expiration and cleanup
• Rate limiting creation

Strong signal: Candidate discusses trade-offs between hash collision rate and URL length without prompting.

Design a paste bin

Initial constraints: 5M pastes/day, 10:1 read-to-write, max 10MB per paste

Expected discussion:

• Object storage for paste content vs database
• Content-addressable storage for deduplication
• Expiration policies (TTL-based)
• Access control (public, unlisted, private)

Follow-ups:

• Syntax highlighting as a service
• Versioning / edit history
• Abuse prevention (spam, malware)

---

Senior questions (60 min)

Design a notification system

Initial constraints: 50M users, supports push, email, SMS, in-app

Expected discussion:

• Channel abstraction and routing logic
• Priority queue for urgent vs batched notifications
• User preference storage and opt-out handling
• Template engine for personalization
• Delivery tracking and retry logic

Follow-ups:

• Rate limiting per user (no notification spam)
• Cross-channel deduplication
• Real-time in-app with WebSocket fallback to polling
• Digest mode (batch low-priority into daily summary)

Strong signal: Candidate separates the ingestion pipeline from delivery pipeline and discusses backpressure handling.

Design a rate limiter

Initial constraints: API gateway processing 100K req/sec, per-user limits

Expected discussion:

• Token bucket vs sliding window vs fixed window
• Distributed counting (Redis, consistent hashing)
• Header communication (X-RateLimit-Remaining, Retry-After)
• Differentiated limits by endpoint and user tier

Follow-ups:

• Distributed rate limiting across multiple data centers
• Graceful degradation under extreme load
• Rate limit key design (IP, API key, user ID, composite)

Design a chat application

Initial constraints: 10M DAU, 1:1 and group chat, message persistence

Expected discussion:

• WebSocket connection management and reconnection
• Message ordering (per-conversation sequence numbers)
• Fan-out: write-time vs read-time for group messages
• Presence system (online/offline/typing indicators)
• Message storage and retrieval pagination

Follow-ups:

• End-to-end encryption implications on server architecture
• Media message handling (images, files)
• Search across message history
• Read receipts at scale

---

Staff+ questions (60 min)

Design a distributed task scheduler

Initial constraints: 10M scheduled tasks, at-least-once execution, sub-second precision

Expected discussion:

• Task storage and indexing by execution time
• Partition strategy for horizontal scaling
• Leader election or leaderless coordination
• At-least-once vs exactly-once semantics
• Dead letter queue for failed tasks
• Clock skew handling across nodes

Follow-ups:

• Multi-region deployment with region-affinity
• Task dependency graphs (DAG execution)
• Dynamic priority adjustment
• Observability: how do you know a task was dropped?

Strong signal: Candidate proactively discusses failure modes and recovery without being asked.

Design a collaborative document editor

Initial constraints: Google Docs-style, 100 concurrent editors per document

Expected discussion:

• Conflict resolution: OT vs CRDT trade-offs
• Operation log and transformation pipeline
• Cursor and selection synchronization
• Document storage (snapshots + operation log)
• Permission model (owner, editor, viewer, commenter)

Follow-ups:

• Offline editing and sync
• Version history and rollback
• Comments and suggestions as separate CRDT
• Performance with very large documents (100K+ characters)

---

Rubric template for system design

Competency	Strong Hire (4)	Hire (3)	No Hire (2)	Strong No Hire (1)
Requirements gathering	Asks clarifying questions, identifies core vs nice-to-have, states assumptions	Asks some questions, identifies main requirements	Jumps to solution without clarifying	Cannot articulate what the system should do
High-level design	Clear component diagram, explains data flow, justifies choices	Reasonable architecture with minor gaps	Vague or missing components, hand-wavy connections	No coherent design emerges
Deep-dive ability	Proactively dives into hardest component, discusses trade-offs in detail	Can deep-dive when prompted, shows reasonable depth	Stays surface-level even when prompted	Cannot explain any component in detail
Scalability	Identifies bottlenecks, proposes concrete solutions with numbers	Acknowledges scale challenges, proposes some solutions	Vague "just add more servers" without specifics	Does not consider scale
Trade-off reasoning	Articulates multiple options with pros/cons, makes justified choice	Sees some trade-offs when prompted	Binary thinking ("this is the right way")	Cannot articulate any trade-offs