Debugging Tools

Systematic debugging is a discipline, not a guessing game. This skill covers the principal tools used to diagnose bugs across the full stack - browser front-ends, Node.js servers, native binaries, and the network between them. The underlying mindset is consistent: form a hypothesis, isolate the variable, confirm or refute, then move inward. Tools are instruments; systematic thinking is the method.

---

When to use this skill

Trigger this skill when the user:

• Opens Chrome DevTools to investigate a performance, network, or memory problem
• Wants to set breakpoints, step through code, or inspect call stacks
• Needs to debug a Node.js process with --inspect or --inspect-brk
• Is tracing system calls with strace or ltrace on Linux/macOS
• Needs to find a memory leak using heap snapshots or the Memory tab
• Is capturing or replaying network traffic with curl, tcpdump, or Wireshark
• Is analyzing a core dump or a crash from a native application with lldb/gdb
• Wants to use conditional breakpoints or logpoints instead of console.log spam

Do NOT trigger this skill for:

• General code review or refactoring (use clean-code or refactoring-patterns)
• CI/CD pipeline failures that are config errors, not runtime bugs

---

Key principles

1. Reproduce before debugging - A bug you cannot reproduce reliably cannot be debugged reliably. Before touching any tool, find the minimal set of steps that trigger the problem every time. A flaky reproduction is a second bug to solve.

2. Binary search the problem space - Never start debugging from line 1. Bisect: is the bug in the frontend or backend? In the request or the response? In the query or the result processing? Each question cuts the search space in half. git bisect applies this directly to commit history.

3. Read the error message twice - The first read captures what you expect to see. The second read captures what it actually says. Most debugging time is lost chasing the wrong problem because the error message was skimmed. Copy the exact message. Look up exact error codes.

4. Check the obvious first - Before reaching for strace or heap profilers, verify: Is the service running? Are environment variables set? Is the right binary being executed? Is the config pointing to the right database? Exotic tools are for exotic problems.

5. Automate reproduction - Once you can reproduce a bug manually, write a script or test that reproduces it. This prevents regression, speeds up iteration, and becomes the fix's test case. A bug with an automated reproduction is already halfway fixed.

---

Core concepts

Breakpoints vs logging

console.log debugging is slow and noisy. Breakpoints pause execution at a precise point and let you inspect the entire state. Use logging when you need a history of state over time (e.g., a value changing across many requests). Use breakpoints when you need to inspect a single moment in detail.

Logpoints (Chrome DevTools, VS Code) are a middle ground: they log a value at a line without pausing execution and without modifying source code. Prefer logpoints over adding and removing console.log statements.

Call stacks

A call stack is a snapshot of how execution reached the current point. It reads bottom-to-top (oldest frame at bottom). When debugging, always read the full stack, not just the top frame. The top frame is where the error surfaced; the root cause is often several frames down, at the point where your code made an incorrect assumption.

Heap vs stack memory

The stack holds function call frames and local variables. It is fast, bounded, and automatically managed. Stack overflows (infinite recursion) are immediately fatal. The heap holds all dynamically allocated objects. Heap memory leaks are slow and insidious - the process grows until it crashes or becomes unresponsive. Heap profiling tools (DevTools Memory tab, valgrind, heaptrack) identify objects that accumulate without being freed.

Syscalls

Every interaction between a process and the OS kernel is a syscall: file reads, network connections, process creation, memory allocation. strace captures these calls with arguments and return values. When a program hangs or fails with a cryptic error, strace often shows exactly which syscall failed and why (e.g., ENOENT: no such file or directory on a missing config path).

Network layers

Network bugs live at different layers. HTTP-level bugs (wrong status codes, missing headers, bad JSON) are visible with curl -v or browser DevTools Network tab. TCP-level bugs (connections refused, timeouts, RST packets) require tcpdump or Wireshark. DNS bugs (resolving the wrong IP, NXDOMAIN) are diagnosed with dig and nslookup.

---

Common tasks

Profile a slow page with Chrome DevTools Performance tab

1. Open DevTools (F12) > Performance tab
2. Click Record, perform the slow action, click Stop
3. In the Flame Chart, find the widest bars - these are the most expensive calls
4. Look for Long Tasks (red corner flags, >50ms on the main thread)
5. Identify the function consuming the most self-time vs total-time

Self time  = time spent in the function itself
Total time = self time + time in all functions it called

Key areas to check:

• Scripting (yellow) - JS execution, event handlers
• Rendering (purple) - style recalc, layout (reflow)
• Painting (green) - compositing, rasterization

Rule: a layout thrash occurs when JS reads then writes DOM geometry in a loop. Fix by batching reads before writes, or using requestAnimationFrame.

Find memory leaks with the Memory tab

1. Open DevTools > Memory tab
2. Take a Heap Snapshot (baseline)
3. Perform the action suspected of leaking (e.g., open and close a modal 10x)
4. Force GC (trash can icon), then take a second snapshot
5. In the second snapshot, select Comparison view
6. Sort by # Delta descending - objects with a growing positive delta are leaking

Common leak sources:
- Event listeners added but never removed
- Closures capturing DOM nodes that were removed
- Global variables holding references to large objects
- setInterval / setTimeout callbacks referencing stale state

Debug Node.js with the inspector protocol

# Start with inspector (connects DevTools or VS Code)
node --inspect server.js

# Break immediately on start (useful when the bug is at startup)
node --inspect-brk server.js

# Attach to a running process by PID
kill -USR1 <pid>

Then open chrome://inspect in Chrome and click inspect under Remote Target. Full Chrome DevTools is now connected to the Node process. Set breakpoints in the Sources panel, use the Console to evaluate expressions in any stack frame.

For production processes, prefer --inspect=127.0.0.1:9229 to avoid exposing the debug port publicly.

Trace syscalls with strace / ltrace

# Trace all syscalls of a new process
strace ./myapp

# Attach to a running process
strace -p <pid>

# Filter to specific syscalls (file operations)
strace -e trace=openat,read,write,close ./myapp

# Timestamp each call and show duration
strace -T -tt ./myapp

# Write output to file (avoids mixing with stderr)
strace -o /tmp/trace.log ./myapp

# ltrace: trace library calls instead of syscalls
ltrace ./myapp

Reading strace output:

openat(AT_FDCWD, "/etc/app.conf", O_RDONLY) = -1 ENOENT (No such file or directory)

Format: syscall(args) = return_value [error]. A negative return value with an error name is a failure. This line shows the app tried to open a config file that does not exist.

Debug network issues with curl / tcpdump / Wireshark

# Verbose HTTP request - shows headers, TLS handshake info
curl -v https://api.example.com/users

# Show only HTTP response headers
curl -sI https://api.example.com/users

# Time each phase of the request
curl -w "@curl-format.txt" -o /dev/null -s https://api.example.com/users
# curl-format.txt: time_namelookup, time_connect, time_appconnect, time_total

# Capture all traffic on port 443 to a file for Wireshark
tcpdump -i eth0 -w capture.pcap port 443

# Capture HTTP traffic and print to stdout
tcpdump -i eth0 -A port 80

# DNS resolution chain
dig +trace api.example.com

For Wireshark analysis:

• Filter by http or http2 for application layer
• Use tcp.analysis.retransmission to find packet loss
• Use tcp.flags.reset == 1 to find unexpected connection resets

Debug crashes with core dumps

# Enable core dumps (Linux - set in /etc/security/limits.conf for persistence)
ulimit -c unlimited

# Run the crashing program
./myapp   # produces core or core.<pid>

# Open with lldb (macOS / modern Linux)
lldb ./myapp core

# Open with gdb (Linux)
gdb ./myapp core

# Inside lldb/gdb: key commands
(lldb) bt           # print backtrace (call stack at crash)
(lldb) frame 3      # switch to frame 3
(lldb) print ptr    # print value of variable 'ptr'
(lldb) info locals  # show all local variables in current frame
(lldb) list         # show source around current line

A crash in a null dereference will show the offending frame in bt. Navigate to the frame with frame select N, then inspect variables to find which pointer was null and why it was never initialized.

Use conditional breakpoints and logpoints

Conditional breakpoint - pauses only when an expression is true:

In Chrome DevTools: right-click a line number > Add conditional breakpoint

// Only pause when userId is the problematic one
userId === 'abc-123'

In VS Code launch.json:

{
  "condition": "i > 100 && items[i] === null"
}

Logpoint - logs a message without pausing (non-intrusive, no source changes):

In Chrome DevTools: right-click a line number > Add logpoint

User {userId} called checkout with {items.length} items

In VS Code: right-click breakpoint > Edit Breakpoint > select Log Message

Use conditional breakpoints when iterating over large collections and the bug only manifests for a specific element. Use logpoints when you need time-series data across many invocations.

---

Anti-patterns / common mistakes

Mistake	Why it's wrong	What to do instead
console.log driven development	Clutters output, requires code changes, leaves logs in production	Use logpoints or structured logging with debug levels
Debugging on production	Modifying production state to understand a bug risks data corruption and outages	Reproduce locally or in staging; use read-only observation tools (strace -p)
Fixing without understanding	Changing code until tests pass without knowing root cause leads to the same bug resurfacing in a different form	State the hypothesis in writing before making any change
Ignoring the call stack	Looking only at the top frame of an exception misses the call path that created the bad state	Always read the full stack; the root cause is usually 3-5 frames down
Heap snapshot without baseline	Comparing one snapshot gives no signal - you cannot tell what grew	Always take a baseline snapshot before the action under test
Running strace on production without -o	strace output mixed with the program's stderr and interleaved in logs	Always use strace -o /tmp/trace.log to isolate output

---

Gotchas

1. strace significantly slows the traced process - On production systems, attaching strace can reduce throughput by 50-80%. Use -e trace= filters to limit syscalls and -o file to write output off stderr. Never attach to a high-traffic process without understanding the performance impact.

2. Chrome DevTools heap snapshots don't capture the GC root path automatically - The Comparison view shows object counts but you must manually navigate to the Retainers pane to find which object is keeping a reference alive. Without checking retainers, you know something is leaking but not why.

3. node --inspect exposes the debug port on 0.0.0.0 by default in some environments - If running in a container or VM, the port is accessible from outside. Always use --inspect=127.0.0.1:9229 to bind only to localhost.

4. --inspect-brk breaks on the first line of Node.js internals, not your code - After attaching DevTools, you need to click "Resume" once to get past the internal breakpoint and land in your application code. Many people miss this and think the debugger is broken.

5. Core dumps require debug symbols to be useful - A stripped production binary produces a backtrace with ?? instead of function names. Build with -g or retain a separate debug symbol file (.dSYM on macOS, .debug on Linux) that lldb/gdb can load alongside the binary.

---

References

For detailed command references, read the relevant file from references/:

• references/tool-guide.md - Quick reference for each debugging tool with key commands

Only load the references file when you need the full command reference for a specific tool and the task at hand requires precise flag-level detail.