Debugging and logs

Find out what a GAIA process did, starting from its structured JSON logs.

Find the logs

Run mode	Where	Read with
`mise dev`	terminal and `apps/api/logs/structured-<date>.json`	`rg`
`nx worker api`	`apps/api/logs/worker/structured-<date>.json`	`rg`
`nx dev voice-agent`	`apps/voice-agent/logs/structured-<date>.json`	`rg`
`nx dev bot-<platform>`	`apps/bots/<platform>/logs/structured-<date>.json`	`rg`
`mise dev:vm` or Docker	container stdout	`docker logs -f gaia-backend`
GAIA Cloud (team)	Loki	Grafana Explore

Files rotate daily and are deleted after 30 days.

Follow one request

Every API response has an x-trace-id header. That id is on every log line the request produced, including background work it started.

rg '<TRACE_ID>' apps/api/logs/structured-*.json | jq .

To list every error:

rg '"level": *"ERROR"' apps/api/logs/structured-*.json

Wide events

GAIA writes one rich event per unit of work (an HTTP request, a worker job, a WebSocket connection) instead of many small lines. Add context to the current event with log.set:

from shared.py.wide_events import log

log.set(user={"id": user_id}, todo={"operation": "create"})
log.audit("subscription cancelled", actor=user_id)
log.error("upstream failed", error_type=type(e).__name__, error=str(e))

Use the namespaces in WideEventFields (libs/shared/py/wide_events.py) so queries work across endpoints.

Rules of thumb

Use log.set() over log.info(). Info lines never reach the wide event.
Pass structured kwargs, not f-strings, for anything you might filter on.
Call log.audit() on auth, payments, and PII writes.
Never swallow exceptions. Log or re-raise.

Event types and shared fields

Unit of work	Event
HTTP request	`http_request`, from `LoggingMiddleware`
ARQ job	`worker_task`, from `wide_task()`
Background asyncio task	`background_task`, from `log_context()` or `spawn_logged_task()`
Bot	`bot_event`

Every line carries time, level, env, service, commit, logger, and message. Every event adds task, trace_id, duration_ms, outcome, and errors[], warnings[], audit[] arrays. Python services and the TypeScript bots share these key names. The wide-event-conformance CI lane (scripts/ci/wide-event-conformance/run.py) fails if they drift.

Query Loki (GAIA Cloud)

The observability compose profile needs config files from the team’s private infra repo. From a public checkout, read the local JSON files instead.

Logs are searched in Grafana with LogQL. Only service, container, level, and logger_name are labels. Everything else lives in the JSON body, so add | json.

{service="gaia-backend"} | json | trace_id = "<TRACE_ID>"

More LogQL recipes

Server errors:

{service="gaia-backend"} | json | message = "http_request" | status_code >= 500

One user’s requests:

{service="gaia-backend"} | json | message = "http_request" | user_id = "<USER_ID>"

Slow chat requests:

{service="gaia-backend"} | json | path =~ "/api/v1/chat.*" | duration_ms > 5000

Failed units of work by name:

sum by (task) (count_over_time({service=~"arq_worker|gaia-backend|.*-bot"} | json | outcome = "failed" [5m]))

Audit trail:

{service="gaia-backend", level="AUDIT"}

Bare | json drops arrays, so errors != "[]" matches every line. Read into an array with | json first_error="errors[0].msg". When a parsed field collides with a label, the label wins and the parsed value gets an _extracted suffix.

Observability score in CI

python3 tools/evlog_map scores every API and voice entry point on wide-event instrumentation. node scripts/ci/checks.mjs evlog-map-bots scores the bots. The observability-* checks of ci.yml’s lint-mixed job fail if the score regresses. See tools/evlog_map/README.md.

Open a pull request

The checklist, the commit format and what happens in review.

Start here

Everyday work

Go deeper

Reference

Find the logs

Follow one request

Wide events

Query Loki (GAIA Cloud)

Next

Open a pull request

​Find the logs

​Follow one request

​Wide events

​Query Loki (GAIA Cloud)

​Next

Open a pull request

Find the logs

Follow one request

Wide events

Query Loki (GAIA Cloud)

Next