Monitoring & Alerting

Exposing a /metrics endpoint in Prometheus format, Grafana dashboard setup for heartbeat run counts, agent costs per hour, and error rate. PagerDuty/webhook alerting on anomalies.

Monitoring and Alerting

What you'll learn

Set up health checks, configure uptime monitoring, export metrics to Prometheus/Grafana, and define the three most important alerts.

Built-in Health Endpoints

# Liveness: is the process running?
GET /health
→ {"status": "ok"}

# Readiness: is the database connected?
GET /health/ready
→ {"status": "ok", "db": "connected", "latency_ms": 2}

# Heartbeat stats: last N runs
GET /health/heartbeat
→ {
    "last_run_at": "...",
    "last_run_status": "completed",
    "runs_last_hour": 12,
    "issues_resolved_last_hour": 47,
    "errors_last_hour": 0
  }

Uptime Monitoring

Point an uptime monitor (Better Uptime, UptimeRobot, Freshping) at /health/ready:

URL: https://your-domain.com/health/ready
Interval: 1 minute
Expected status: 200
Alert: email + Slack when 2 consecutive checks fail

Prometheus Metrics

Enable Prometheus export in .env:

PROMETHEUS_ENABLED=true
PROMETHEUS_PORT=9090

Available metrics:

paperclip_heartbeat_runs_total{status="completed"} 142
paperclip_heartbeat_runs_total{status="failed"} 3
paperclip_issues_resolved_total{agent="Researcher"} 89
paperclip_cost_usd_total{agent="Researcher", model="claude-opus-4-5"} 1.23
paperclip_agent_status{agent="Researcher"} 1   # 1=active, 0=paused

Grafana dashboard JSON is included in packages/server/monitoring/grafana-dashboard.json.

Docker Compose Health Checks

services:
  api:
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3100/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

The Three Most Important Alerts

Alert 1: Company Paused (Budget Exceeded)

-- Check every 5 minutes
SELECT name FROM companies WHERE paused = true;

Alert if any company is paused unexpectedly.

Alert 2: Heartbeat Failing

-- Check every 15 minutes
SELECT COUNT(*) FROM heartbeat_runs
WHERE created_at > now() - interval '15 minutes'
  AND status = 'failed';

Alert if >2 failures in the last 15 minutes.

Alert 3: Issue Backlog Growing

-- Check every hour
SELECT COUNT(*) FROM issues
WHERE status = 'open'
  AND created_at < now() - interval '1 hour';

Alert if >50 open issues are more than 1 hour old (heartbeat may be stuck).

Log Aggregation

In production, ship API logs to a log aggregator:

# docker-compose.yml
services:
  api:
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
    # Or send to Loki:
    # driver: loki
    # options:
    #   loki-url: "http://loki:3100/loki/api/v1/push"

Checkpoint ✓

[ ] You can curl the three health endpoints and interpret each response
[ ] You know the three critical alerts and the SQL query for each
[ ] You understand how to configure Prometheus metrics export