LLM App Development

Building production LLM applications requires more than prompt engineering - it demands the same reliability, observability, and safety thinking applied to any critical system. This skill covers the full stack: architecture, guardrails, evaluation pipelines, RAG, function calling, streaming, and cost optimization. It emphasizes when patterns apply and what to do when they fail, not just happy-path implementation.

When to use this skill

Trigger this skill when the user:

• Designs the architecture for a new LLM-powered application or feature
• Implements content filtering, PII detection, or schema validation on model I/O
• Builds or improves an evaluation pipeline (automated evals, human review, A/B tests)
• Sets up a RAG pipeline (chunking, embedding, retrieval, reranking)
• Adds function calling or tool use to an agent or chat interface
• Streams LLM responses to a client (SSE, token-by-token rendering)
• Optimizes inference cost or latency (caching, model routing, prompt compression)
• Decides whether to fine-tune a model or improve prompting instead

Do NOT trigger this skill for:

• Pure ML research, model training from scratch, or academic benchmarking
• Questions about a specific AI framework API (use the framework's own skill, e.g., mastra)

Key principles

1. Evaluate before you ship - A feature without evals is a feature you cannot safely iterate on. Define success metrics and build automated checks before the first production deployment.

2. Guardrails are non-negotiable - Validate both input and output on every production request. Content filtering, PII scrubbing, and schema validation belong in your request path, not as optional post-processing.

3. Start with prompting before fine-tuning - Fine-tuning is expensive, slow to iterate, and hard to maintain. Exhaust systematic prompt engineering, few-shot examples, and RAG before considering fine-tuning.

4. Design for failure and fallback - LLM calls fail: timeouts, rate limits, malformed outputs, hallucinations. Every integration needs retry logic, output validation, and a fallback response.

5. Cost-optimize from day one - Track token usage per feature. Cache deterministic outputs. Route cheap queries to smaller models. Set hard budget limits.

Core concepts

LLM app stack

User input
    -> Input guardrails (safety, PII, token limits)
    -> Prompt construction (system prompt, context, few-shots, retrieved docs)
    -> Model call (streaming or batch)
    -> Output guardrails (schema validation, content check, hallucination detection)
    -> Post-processing (formatting, citations, structured extraction)
    -> Response to user

Every layer is an independent failure point and must be observable.

Embedding / vector DB architecture

Documents are chunked into overlapping segments, embedded into dense vectors, and stored in a vector database. At query time the user message is embedded, similar chunks are retrieved via ANN search, optionally reranked by a cross-encoder, and injected into the context window. Chunk quality determines retrieval quality more than model choice.

Caching strategies

Common tasks

Design LLM app architecture

Key decisions before writing code:

import OpenAI from 'openai'

const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY })

async function callLLM(systemPrompt: string, userMessage: string, model = 'gpt-4o-mini'): Promise<string> {
  const controller = new AbortController()
  const timeout = setTimeout(() => controller.abort(), 30_000)
  try {
    const res = await client.chat.completions.create(
      { model, max_tokens: 1024, messages: [{ role: 'system', content: systemPrompt }, { role: 'user', content: userMessage }] },
      { signal: controller.signal },
    )
    return res.choices[0].message.content ?? ''
  } finally {
    clearTimeout(timeout)
  }
}

Implement input/output guardrails

import { z } from 'zod'

const PII_PATTERNS = [
  /\b\d{3}-\d{2}-\d{4}\b/g,                              // SSN
  /\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b/gi,        // email
  /\b(?:\d{4}[ -]?){3}\d{4}\b/g,                         // credit card
]

function scrubPII(text: string): string {
  return PII_PATTERNS.reduce((t, re) => t.replace(re, '[REDACTED]'), text)
}

function validateInput(text: string): { ok: boolean; reason?: string } {
  if (text.split(/\s+/).length > 4000) return { ok: false, reason: 'Input too long' }
  return { ok: true }
}

const SummarySchema = z.object({
  summary: z.string().min(10).max(500),
  keyPoints: z.array(z.string()).min(1).max(10),
  confidence: z.number().min(0).max(1),
})

async function getSummaryWithGuardrails(text: string) {
  const v = validateInput(text)
  if (!v.ok) throw new Error(`Input rejected: ${v.reason}`)
  const raw = await callLLM('Respond only with valid JSON.', `Summarize as JSON: ${scrubPII(text)}`)
  return SummarySchema.parse(JSON.parse(raw))  // throws ZodError if schema invalid
}

Build an evaluation pipeline

interface EvalCase {
  id: string
  input: string
  expectedContains?: string[]
  expectedNotContains?: string[]
  scoreThreshold?: number  // 0-1 for LLM-as-judge
}

async function runEval(ec: EvalCase, modelFn: (input: string) => Promise<string>) {
  const output = await modelFn(ec.input)
  for (const s of ec.expectedContains ?? [])
    if (!output.includes(s)) return { id: ec.id, passed: false, details: `Missing: "${s}"` }
  for (const s of ec.expectedNotContains ?? [])
    if (output.includes(s)) return { id: ec.id, passed: false, details: `Forbidden: "${s}"` }
  if (ec.scoreThreshold !== undefined) {
    const score = await judgeOutput(ec.input, output)
    if (score < ec.scoreThreshold) return { id: ec.id, passed: false, details: `Score ${score} < ${ec.scoreThreshold}` }
  }
  return { id: ec.id, passed: true, details: 'OK' }
}

async function judgeOutput(input: string, output: string): Promise<number> {
  const score = await callLLM(
    'You are a strict evaluator. Reply with only a number from 0.0 to 1.0.',
    `Input: ${input}\n\nOutput: ${output}\n\nScore quality (0.0=poor, 1.0=excellent):`,
    'gpt-4o',
  )
  return Math.min(1, Math.max(0, parseFloat(score)))
}

Load references/evaluation-framework.md for metrics, benchmarks, and human-in-the-loop protocols.

Implement RAG with vector search

import OpenAI from 'openai'

const client = new OpenAI()

function chunkText(text: string, size = 512, overlap = 64): string[] {
  const words = text.split(/\s+/)
  const chunks: string[] = []
  for (let i = 0; i < words.length; i += size - overlap) {
    chunks.push(words.slice(i, i + size).join(' '))
    if (i + size >= words.length) break
  }
  return chunks
}

async function embedTexts(texts: string[]): Promise<number[][]> {
  const res = await client.embeddings.create({ model: 'text-embedding-3-small', input: texts })
  return res.data.map(d => d.embedding)
}

function cosine(a: number[], b: number[]): number {
  const dot = a.reduce((s, v, i) => s + v * b[i], 0)
  return dot / (Math.sqrt(a.reduce((s, v) => s + v * v, 0)) * Math.sqrt(b.reduce((s, v) => s + v * v, 0)))
}

interface DocChunk { text: string; embedding: number[] }

async function ragQuery(question: string, store: DocChunk[], topK = 5): Promise<string> {
  const [qEmbed] = await embedTexts([question])
  const context = store
    .map(c => ({ text: c.text, score: cosine(qEmbed, c.embedding) }))
    .sort((a, b) => b.score - a.score).slice(0, topK).map(r => r.text)
  return callLLM(
    'Answer using only the provided context. If not found, say "I don\'t know."',
    `Context:\n${context.join('\n---\n')}\n\nQuestion: ${question}`,
  )
}

Add function calling / tool use

import OpenAI from 'openai'

const client = new OpenAI()
type ToolHandlers = Record<string, (args: Record<string, unknown>) => Promise<string>>

const tools: OpenAI.ChatCompletionTool[] = [{
  type: 'function',
  function: {
    name: 'get_weather',
    description: 'Get current weather for a city.',
    parameters: {
      type: 'object',
      properties: { city: { type: 'string' }, units: { type: 'string', enum: ['celsius', 'fahrenheit'] } },
      required: ['city'],
    },
  },
}]

async function runWithTools(userMessage: string, handlers: ToolHandlers): Promise<string> {
  const messages: OpenAI.ChatCompletionMessageParam[] = [{ role: 'user', content: userMessage }]
  for (let step = 0; step < 5; step++) {  // cap tool-use loops to prevent infinite recursion
    const res = await client.chat.completions.create({ model: 'gpt-4o', tools, messages })
    const choice = res.choices[0]
    messages.push(choice.message)
    if (choice.finish_reason === 'stop') return choice.message.content ?? ''
    for (const tc of choice.message.tool_calls ?? []) {
      const fn = handlers[tc.function.name]
      if (!fn) throw new Error(`Unknown tool: ${tc.function.name}`)
      const result = await fn(JSON.parse(tc.function.arguments) as Record<string, unknown>)
      messages.push({ role: 'tool', tool_call_id: tc.id, content: result })
    }
  }
  throw new Error('Tool call loop exceeded max steps')
}

Implement streaming responses

import OpenAI from 'openai'
import type { Response } from 'express'

const client = new OpenAI()

async function streamToResponse(prompt: string, res: Response): Promise<void> {
  res.setHeader('Content-Type', 'text/event-stream')
  res.setHeader('Cache-Control', 'no-cache')
  res.setHeader('Connection', 'keep-alive')
  const stream = await client.chat.completions.create({
    model: 'gpt-4o-mini', stream: true,
    messages: [{ role: 'user', content: prompt }],
  })
  let fullText = ''
  for await (const chunk of stream) {
    const token = chunk.choices[0]?.delta?.content
    if (token) { fullText += token; res.write(`data: ${JSON.stringify({ token })}\n\n`) }
  }
  runOutputGuardrails(fullText)  // validate after stream completes
  res.write('data: [DONE]\n\n')
  res.end()
}

// Client-side consumption
function consumeStream(url: string, onToken: (t: string) => void): void {
  const es = new EventSource(url)
  es.onmessage = (e) => {
    if (e.data === '[DONE]') { es.close(); return }
    onToken((JSON.parse(e.data) as { token: string }).token)
  }
}

function runOutputGuardrails(_text: string): void { /* content policy / schema checks */ }

Optimize cost and latency

import crypto from 'crypto'

const cache = new Map<string, { value: string; expiresAt: number }>()

async function cachedLLMCall(prompt: string, model = 'gpt-4o-mini', ttlMs = 3_600_000): Promise<string> {
  const key = crypto.createHash('sha256').update(`${model}:${prompt}`).digest('hex')
  const cached = cache.get(key)
  if (cached && cached.expiresAt > Date.now()) return cached.value
  const result = await callLLM('', prompt, model)
  cache.set(key, { value: result, expiresAt: Date.now() + ttlMs })
  return result
}

// Route to cheaper model based on prompt complexity
function routeModel(prompt: string): string {
  const words = prompt.split(/\s+/).length
  if (words < 50) return 'gpt-4o-mini'
  if (words < 300) return 'gpt-4o-mini'
  return 'gpt-4o'
}

// Strip redundant whitespace to reduce token count
const compressPrompt = (p: string): string => p.replace(/\s{2,}/g, ' ').trim()

Anti-patterns / common mistakes

Gotchas

1. Streaming guardrails can only run post-completion - You cannot validate a streamed response mid-stream for content policy or schema compliance. The full text is only available after the last token. Run output guardrails after the stream ends, and design your client to handle a late rejection (e.g., replace streamed content with an error state) rather than assuming the stream is always valid.

2. JSON mode does not guarantee valid JSON on all providers - OpenAI's response_format: { type: "json_object" } reduces but does not eliminate parse errors, especially on long outputs that hit max_tokens. Always wrap JSON.parse() in a try/catch and treat a parse failure as a retriable error, not a crash.

3. RAG retrieval quality is dominated by chunk boundaries, not embedding models - Switching from text-embedding-3-small to text-embedding-3-large rarely fixes poor retrieval. Poor recall almost always traces to chunks that split mid-sentence or mid-concept. Fix chunking strategy (overlapping windows, semantic boundaries) before upgrading the embedding model.

4. Tool call loops can exceed maxSteps silently on some SDKs - If the model keeps calling tools without emitting a stop finish reason, some SDK wrappers will retry indefinitely. Always set an explicit maxSteps cap and treat a loop-exceeded condition as a hard error, not a retry.

5. Semantic caches can return stale or incorrect answers for slightly rephrased queries - A semantic cache that matches "What is the capital of France?" to "Tell me the capital of France" is fine. But caches with broad similarity thresholds can match unrelated questions with similar wording. Set cosine similarity thresholds conservatively (0.97+) for factual queries; use exact caching only for truly deterministic prompts.

References

For detailed content on specific sub-domains, load the relevant reference file:

• references/evaluation-framework.md - metrics, benchmarks, human eval protocols, automated testing, A/B testing, eval dataset design

Only load a reference file when the task specifically requires it - they are long and will consume significant context.