Performance Engineering

A systematic framework for diagnosing, measuring, and improving application performance. This skill covers the full performance lifecycle - from identifying bottlenecks with profilers and flame graphs, to eliminating memory leaks with heap snapshots, to validating improvements with rigorous benchmarks. It applies across the stack: Node.js backend, browser frontend, and database query layer. The guiding philosophy is always measure first, optimize second.

When to use this skill

Trigger this skill when the user:

• Observes high P95/P99 latency or slow response times in production
• Reports memory growing unboundedly or OOM crashes
• Wants to profile CPU usage or generate a flame graph
• Needs to benchmark two implementations to decide between them
• Is investigating event loop blocking or long tasks in the browser
• Wants to reduce JavaScript bundle size, TTI, or Core Web Vitals scores
• Is tuning garbage collection, heap limits, or worker thread pools
• Needs to set up continuous performance monitoring or performance budgets
• Is debugging N+1 queries, slow database queries, or connection pool exhaustion

Do NOT trigger this skill for:

• General code quality refactors with no performance goal (use clean-code skill)
• Capacity planning and infrastructure scaling decisions (use backend-engineering skill)

Key principles

1. Measure first, always - Never optimize based on intuition. Instrument the code, collect data, and let profiler output tell you where time actually goes. Assumptions about bottlenecks are wrong more often than not.

2. Optimize the bottleneck, not the code - Amdahl's Law: speeding up a component that is 5% of total runtime yields at most 5% improvement. Find the dominant cost, fix that, then re-measure to find the new dominant cost. Repeat.

3. Set performance budgets upfront - Define what "fast enough" means before writing a line. A target of "P99 < 200ms" or "bundle < 150KB" creates a measurable pass/fail criterion. Without a budget, optimization is endless.

4. Test under realistic load - A function that takes 1ms with 10 users may take 800ms with 1000 concurrent users due to lock contention, cache pressure, or connection pool exhaustion. Always load-test against production-like data volumes.

5. Premature optimization is the root of all evil - (Knuth) Write correct, readable code first. Profile in a realistic environment. Only then optimize the measured hot path. Code that sacrifices clarity for unmeasured performance gains is technical debt.

Core concepts

Latency vs throughput - Latency is how long one request takes. Throughput is how many requests complete per second. Optimizing one does not automatically improve the other. A batching strategy can dramatically increase throughput while increasing individual request latency.

Percentiles (P50/P95/P99) - Averages hide outliers. P99 latency is the experience of 1 in 100 users. In high-traffic systems, the P99 user matters. Never report only averages - always report P50, P95, and P99 together.

Flame graphs - A visualization of sampled call stacks where width represents time spent. Wide bars at the top of a flame are hot functions to optimize. Generated by 0x, clinic flame, or Chrome DevTools CPU profiler.

Heap snapshots - A point-in-time dump of all live objects in the JS heap. Compare two snapshots (before/after a suspected leak window) to find objects accumulating without being GC'd. Available in Chrome DevTools and Node.js v8.writeHeapSnapshot().

Profiler types - Sampling profilers (low overhead, statistical) vs instrumentation profilers (exact counts, higher overhead). Use sampling for production diagnosis, instrumentation for precise benchmark attribution.

Amdahl's Law - Max speedup = 1 / (1 - P + P/N) where P is the parallelizable fraction and N is the number of processors. A program that is 90% parallelizable has a theoretical max speedup of 10x regardless of how many cores you add.

Common tasks

Profile CPU usage

Use Node.js built-in profiler or 0x for flame graphs:

# Built-in V8 profiler - generates isolate-*.log
node --prof server.js
# Run your load, then process the log
node --prof-process isolate-*.log > profile.txt

# 0x - generates interactive flame graph HTML
npx 0x -- node server.js
# Then apply load; 0x auto-generates flamegraph.html

In TypeScript, mark hot sections explicitly for DevTools profiling:

// Wrap suspected hot paths to isolate them in profiles
function processItems(items: Item[]): Result[] {
  console.time('processItems');
  const result = items.map(transform);
  console.timeEnd('processItems');
  return result;
}

For browser CPU profiling, open Chrome DevTools > Performance tab > Record while reproducing the slow interaction. Look for long tasks (>50ms) in the flame chart.

Debug memory leaks

Capture two heap snapshots - one before and one after a suspected leak window - then compare retained objects:

import { writeHeapSnapshot } from 'v8';
import { setInterval } from 'timers';

// Snapshot 1: baseline
writeHeapSnapshot(); // writes Heap-<pid>-<seq>.heapsnapshot

// Simulate load / time passing
await runWorkload();

// Snapshot 2: after suspected leak
writeHeapSnapshot();
// Load both files in Chrome DevTools > Memory > Compare snapshots

Avoid closure-based leaks by using WeakRef and FinalizationRegistry for optional references that should not prevent GC:

class Cache {
  private store = new Map<string, WeakRef<object>>();
  private registry = new FinalizationRegistry((key: string) => {
    this.store.delete(key); // auto-cleanup when value is GC'd
  });

  set(key: string, value: object): void {
    this.store.set(key, new WeakRef(value));
    this.registry.register(value, key);
  }

  get(key: string): object | undefined {
    return this.store.get(key)?.deref();
  }
}

Common leak sources: event listeners never removed, global maps/sets that grow forever, closures capturing large objects, and timers/intervals not cleared.

Benchmark code

Proper microbenchmarking requires warmup to let V8 JIT compile, multiple iterations to reduce noise, and statistical comparison:

import Benchmark from 'benchmark';

const suite = new Benchmark.Suite();

suite
  .add('Array.from', () => {
    Array.from({ length: 1000 }, (_, i) => i * 2);
  })
  .add('for loop', () => {
    const arr: number[] = new Array(1000);
    for (let i = 0; i < 1000; i++) arr[i] = i * 2;
  })
  .on('cycle', (event: Benchmark.Event) => {
    console.log(String(event.target));
  })
  .on('complete', function (this: Benchmark.Suite) {
    console.log('Fastest: ' + this.filter('fastest').map('name'));
  })
  .run({ async: true });

Rules for valid microbenchmarks:

• Warmup at least 3 iterations before measuring
• Run for at least 1 second per case to smooth JIT variance
• Prevent dead-code elimination - consume the result
• Test with realistic input size and shape

Optimize Node.js event loop

Detect blocking with clinic bubbleprof or manual measurement:

import { performance, PerformanceObserver } from 'perf_hooks';

// Detect event loop lag
let lastCheck = Date.now();
setInterval(() => {
  const lag = Date.now() - lastCheck - 100; // expected 100ms
  if (lag > 50) console.warn(`Event loop lag: ${lag}ms`);
  lastCheck = Date.now();
}, 100).unref();

Move CPU-intensive work off the main thread with worker threads:

import { Worker, isMainThread, parentPort, workerData } from 'worker_threads';

// main-thread side
function runCPUTask(data: unknown): Promise<unknown> {
  return new Promise((resolve, reject) => {
    const worker = new Worker(__filename, { workerData: data });
    worker.on('message', resolve);
    worker.on('error', reject);
  });
}

// worker side
if (!isMainThread) {
  const result = heavyComputation(workerData);
  parentPort?.postMessage(result);
}

Reduce frontend bundle size

Audit bundle composition first, then fix the biggest wins:

# Visualize what's in your bundle
npx webpack-bundle-analyzer stats.json
# or for Vite:
npx vite-bundle-visualizer

Apply tree shaking with named imports:

// Bad - imports entire lodash (~70KB)
import _ from 'lodash';
const result = _.debounce(fn, 300);

// Good - imports only debounce (~2KB)
import debounce from 'lodash/debounce';
const result = debounce(fn, 300);

Use dynamic imports for code splitting at route boundaries:

// React lazy loading - splits route into separate chunk
import { lazy, Suspense } from 'react';

const Dashboard = lazy(() => import('./pages/Dashboard'));

function App() {
  return (
    <Suspense fallback={<Spinner />}>
      <Dashboard />
    </Suspense>
  );
}

Set up performance monitoring

Track Core Web Vitals with the web-vitals library:

import { onCLS, onINP, onLCP, onFCP, onTTFB } from 'web-vitals';

function sendToAnalytics(metric: { name: string; value: number; rating: string }) {
  navigator.sendBeacon('/analytics', JSON.stringify(metric));
}

onCLS(sendToAnalytics);   // Cumulative Layout Shift - target < 0.1
onINP(sendToAnalytics);   // Interaction to Next Paint - target < 200ms
onLCP(sendToAnalytics);   // Largest Contentful Paint - target < 2.5s
onFCP(sendToAnalytics);
onTTFB(sendToAnalytics);

Add custom server-side timing for API endpoints:

import { performance } from 'perf_hooks';

function withTiming<T>(name: string, fn: () => Promise<T>): Promise<T> {
  const start = performance.now();
  return fn().finally(() => {
    const duration = performance.now() - start;
    metrics.histogram(name, duration); // send to Datadog/Prometheus
  });
}

// Usage
const user = await withTiming('db.getUser', () => db.users.findById(id));

Optimize database query performance

Fix N+1 queries by batching with DataLoader:

import DataLoader from 'dataloader';

// Without DataLoader: 1 query per user = N+1
// With DataLoader: batches into 1 query per tick
const userLoader = new DataLoader(async (ids: readonly string[]) => {
  const users = await db.users.findMany({ where: { id: { in: [...ids] } } });
  const map = new Map(users.map((u) => [u.id, u]));
  return ids.map((id) => map.get(id) ?? null);
});

// Each call is automatically batched
const user = await userLoader.load(userId);

Use connection pooling and avoid pool exhaustion:

import { Pool } from 'pg';

const pool = new Pool({
  max: 20,           // max connections - tune to (2 * CPU cores + 1) as starting point
  idleTimeoutMillis: 30_000,
  connectionTimeoutMillis: 2_000,
});

// Always release connections - use try/finally
const client = await pool.connect();
try {
  const result = await client.query('SELECT ...', [params]);
  return result.rows;
} finally {
  client.release(); // critical - never omit
}

Anti-patterns / common mistakes

Gotchas

1. Microbenchmarks without preventing dead-code elimination produce meaningless results - V8 will optimize away computations whose results are never used. A benchmark that calls computeResult() without consuming the return value may be measuring near-zero work. Always store the result in a variable and use it (e.g., sum += result) so the compiler cannot eliminate the hot path.

2. Connection pool exhaustion masquerades as slow queries - If all DB connections are in use, new queries queue behind them and appear in traces as 500ms+ "database time" when the query itself takes 5ms. Check pool.totalCount, pool.idleCount, and pool.waitingCount before optimizing queries. Pool exhaustion often looks like slow DB, not like a pool problem.

3. Profiling in development produces unrepresentative results - V8 optimizes differently in development (no minification, source maps active, NODE_ENV=development guards enabled). Profiling a dev build and optimizing based on that output can be entirely misleading. Always profile against a production build with production environment variables and realistic data volume.

4. Heap snapshots taken during GC produce inflated retained sizes - If you trigger a heap snapshot during a GC cycle, the snapshot may show objects that are already queued for collection but not yet freed. Compare two snapshots taken at the same phase of your workload (e.g., both after processing 100 requests) to get valid comparisons.

5. Worker threads do not share memory by default - serialization overhead can exceed compute savings - Offloading a task to a worker thread requires serializing input data (via postMessage) and deserializing results back. For tasks involving large objects, this serialization cost can exceed the compute benefit. Use SharedArrayBuffer for large data payloads that need to cross the worker boundary frequently.

References

Load the relevant reference file only when the current task requires it:

• references/profiling-tools.md - Node.js profiler, Chrome DevTools, Lighthouse, clinic.js, 0x, and how to choose between them