Production Debugging: Model Loading and Server Issues

Production deployment of LLM servers is filled with subtle failure modes. What seems like a simple model load can become a cascade of cryptic errors. This post documents the real debugging journey and the systematic approach to resolving server issues.

The Common Failure Patterns

Pattern #1: Multiple Server Processes

curl http://localhost:8000/v1/chat/completions
{"error":{"message":"Invalid response format","type":"internal_error"}}

# Multiple servers running simultaneously
ps aux | grep llama-server
tomwest   1234  0.0  45.2  24567890  12345   ?  S   Jan10 0:02 llama-server --port 8000 --model qwen3-32b
tomwest   5678  0.0  47.8  24789012  13456   ?  S   Jan11 0:03 llama-server --port 8000 --model nemotron-nano

# Step 1: Complete server cleanup
pkill -f llama-server
sleep 3

# Step 2: Verify clean state
ps aux | grep llama-server  # Should return nothing

# Step 3: Start server with proper logging
nohup /path/to/llama-server [arguments] \
  > /tmp/nemotron-server.log 2>&1 &

# Step 4: Monitor startup progress
tail -f /tmp/nemotron-server.log

Pattern #2: Model Loading Deception

# Health check passes
curl -s http://localhost:8000/health
{"status":"ok"}

# But inference fails
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"nemotron","messages":[{"role":"user","content":"Hello"}]}'
{"error":{"message":"Loading model","type":"unavailable_error","code":503}}

# Wait for this specific log message first
grep "llm_load_tensors" /tmp/nemotron-server.log
# Output: "llm_load_tensors: loaded all 32 layers in 15.73 seconds"

# Then test with actual inference request
curl -X POST http://localhost:8000/v1/chat/completions ...

Pattern #3: Memory Management Mysteries

Starting server with 48GB model on 48GB VRAM...
# No error message, no immediate failure
# Then requests just hang or return generic errors

# Monitor VRAM during model load
watch -n 1 'nvidia-smi --query-gpu=memory.used,memory.total --format=csv'

#发现: Model needs 31GB +KV cache buffer + overhead = 48GB+ need

The Debugging Toolkit

Essential Monitoring Commands

# Process monitoring
ps aux | grep llama-server    # Running processes
lsof -i :8000                  # Port usage

# Resource monitoring
nvidia-smi                      # GPU usage, memory
htop                           # CPU, memory usage
nvidia-smi dmon -s pucvmet -o -d 1  # Real-time GPU metrics

# Real-time log following
tail -f /tmp/nemotron-server.log

# Specific error patterns
grep -i error /tmp/nemotron-server.log
grep -i warn /tmp/nemotron-server.log

# Model loading progress
grep "llm_load_tensors" /tmp/nemotron-server.log
grep "ggml_cuda" /tmp/nemotron-server.log

Common Error Messages and Solutions

Symptom: {"error":{"message":"Loading model","type":"unavailable_error","code":503}}
Cause: Model still loading in background
Solution: Wait for "loaded all 32 layers" log message
Time to Resolution: 30-60 seconds

Symptom: "model files are too large to fit into cache"
Cause: Insufficient VRAM for model + KV cache
Solution: Reduce --n-gpu-layers or use smaller quantization
Check: nvidia-smi for available VRAM

Symptom: Backend error: Out of memory
Cause: VRAM exhausted during inference
Solution: Restart server with reduced GPU layers
Prevention: Monitor VRAM usage during startup

Pre-Deployment Checklist

Before Starting Server

1. Clean Environment: Kill existing processes, verify port availability
2. Verify Dependencies: Check CUDA version, GPU drivers, model file integrity
3. Resource Planning: Confirm VRAM capacity, disk space, network connectivity
4. Configuration Review: Validate command arguments, config file settings

During Model Load

1. Monitor Progress: Track GPU memory usage, watch log messages
2. Watch Timeouts: Model load shouldn't exceed 60-90 seconds
3. Resource Allocation: Verify balanced GPU utilization in multi-GPU setups
4. Early Failure Detection: Any immediate error indicates configuration issues

After Server Ready

1. Inference Testing: Send sample requests, verify response quality
2. Performance Benchmarking: Test typical workloads, measure latency
3. Memory Monitoring: Check for memory leaks or growing usage
4. API Compatibility: Validate endpoint formats with client applications

Troubleshooting Methodology

Systematic Approach

# What exactly is failing?
- Server won't start?
- Server starts but model won't load?
- Model loads but health check fails?
- Health passes but inference fails?

# System health verification
- GPU available and drivers working?
- Sufficient VRAM for model?
- Port not in conflict?
- Model file complete and accessible?

# Error message analysis
- What was the last successful action?
- When did the first error appear?
- Are there resource exhaustion signs?
- Any permission or path issues?

# Controlled testing
- Restart cleanly with monitoring
- Reproduce original error
- Test minimal working configuration
- Gradually increase complexity

The Debugging Workflow

# 1. Clean slate
pkill llama-server
ps aux | grep llama-server  # Verify clean

# 2. Start with monitoring
nohup llama-server > server.log 2>&1 &
tail -f server.log

# 3. System verification
nvidia-smi
curl -s localhost:8000/health

# 4. Test inference
curl -X POST /v1/chat/completions [test request]

# 5. Full deployment
# Only after all above pass

Production Monitoring

Health Check Automation

#!/bin/bash
# healthcheck.sh - Monitor LLM server health

HEALTH_URL="http://localhost:8000/health"
LOG_FILE="/tmp/llama-server.log"

# Check if server is running
if ! pgrep -f llama-server > /dev/null; then
    echo "ERROR: llama-server not running"
    exit 1
fi

# Check HTTP response
RESPONSE=$(curl -s -w "%{http_code}" "$HEALTH_URL")
if [[ "$RESPONSE" == *"503"* ]]; then
    echo "WARNING: Model still loading"
    exit 2
fi

# Verify model actually loaded
if ! grep -q "llm_load_tensors.*loaded all" "$LOG_FILE"; then
    echo "WARNING: Model not fully loaded"
    exit 3
fi

echo "OK: Server healthy and ready"
exit 0

# Add to crontab for every 5 minute checks
*/5 * * * * /path/to/healthcheck.sh >> /var/log/llama-health.log 2>&1

Log Rotation

# Prevent log files from growing too large
# Add to logrotate configuration
/tmp/nemotron-server.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 644 tomwest tomwest
}

Key Takeaways

Understanding Server States

LLM servers have multiple readiness states: HTTP ready ≠ Model ready. Always test actual inference, not just health endpoints.

Process Management

Clean startup procedures prevent most common issues. Always kill existing processes before starting new ones.

Resource Planning

VRAM needs exceed model size. Always account for KV cache, overhead, and fragmentation when planning memory usage.

Systematic Debugging

Isolate symptoms systematically. Most errors fall into predictable patterns with repeatable solutions.

Automation is Essential

Manual debugging works for one-time deployments, but production requires automated health checks and monitoring.