Observability

SurfSense instruments the backend with OpenTelemetry and exports traces, metrics, and logs over OTLP to a self-hosted Grafana LGTM stack (Loki, Grafana, Tempo, Prometheus). Agent spans follow the OpenTelemetry GenAI semantic conventions (gen_ai.*) and nest under the FastAPI server span, so one Tempo trace shows the HTTP request, its DB/Redis/LLM calls, and the agent steps together.

Enable Locally

The development compose file reads backend settings from surfsense_backend/.env. Add these values there:

SURFSENSE_ENABLE_OTEL=true
SURFSENSE_ENV=dev
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-lgtm:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_RESOURCE_ATTRIBUTES=service.namespace=surfsense
OTEL_METRIC_EXPORT_INTERVAL=300000

Then start the development stack with the bundled LGTM backend:

docker compose -f docker/docker-compose.dev.yml up --build

Grafana is exposed on http://localhost:3001 by default.

Enable in Production

The app exports OTLP directly to a self-hosted LGTM instance — there is no separate app-managed collector to configure. Point the backend at your own Grafana LGTM (or the grafana/otel-lgtm all-in-one, which embeds a collector) and set:

SURFSENSE_ENV=production
SURFSENSE_ENABLE_OTEL=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://<your-lgtm-host>:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_RESOURCE_ATTRIBUTES=service.namespace=surfsense
OTEL_METRIC_EXPORT_INTERVAL=300000

Telemetry is disabled unless an OTLP endpoint is set, so leaving OTEL_EXPORTER_OTLP_ENDPOINT unset is a safe no-op. SURFSENSE_DISABLE_OTEL / OTEL_SDK_DISABLED are emergency kill switches that override the endpoint.

Automatic Traces

When OpenTelemetry is enabled, the backend auto-instruments:

• FastAPI inbound requests.
• SQLAlchemy queries from the main async engine and the Celery task engine.
• Raw psycopg calls used by the LangGraph checkpointer.
• Redis commands.
• HTTPX outbound requests (URLs stripped of query strings).
• Celery producer and worker execution.

Manual Spans

Domain spans live under app.observability.domains.*, one module per concept. Names are low-cardinality; agent spans carry gen_ai.* attributes (and SpanKind.CLIENT for model calls):

• model.call, tool.call, subagent.invoke, compaction.run, permission.asked, interrupt.raised (domains.agent)
• chat.request (domains.chat)
• kb.search, kb.persist, kb.rerank (domains.kb)
• embedding.generate (domains.embedding)
• connector.sync (domains.indexing)
• etl.extract, etl.parse, etl.ocr, etl.picture.describe, etl.picture.ocr (domains.etl)

model.call is emitted at the LLM-client chokepoint (ChatLiteLLMRouter), so every LLM caller — not just the chat agent — is covered (title generation, vision/OCR, memory rewrite, podcast/video generation, ...). The agent middleware still owns the span when it wraps a call; the chokepoint defers to it so chat is never double-counted.

Never attach user content — prompts, document titles, file paths, user-specific URLs, secrets, or raw queries — as span attributes.

Metrics

The instrumentors provide HTTP, HTTPX, and Celery runtime metrics. SurfSense adds project metrics from app.observability.domains.* (generic timers from app.observability.signals.metrics):

• surfsense.model.call.duration, gen_ai.client.token.usage, surfsense.tool.call.duration, surfsense.tool.call.errors
• surfsense.chat.request.duration, surfsense.chat.request.outcome
• surfsense.kb.search.duration, surfsense.kb.rerank.duration
• surfsense.embedding.duration
• surfsense.media.render.duration, surfsense.media.render.outcome (podcast + video, keyed by media.kind)
• surfsense.compaction.runs, surfsense.permission.asks, surfsense.interrupt.raised
• surfsense.subagent.invoke.duration, surfsense.subagent.invoke.outcome
• surfsense.indexing.document.duration, surfsense.indexing.document.outcome
• surfsense.connector.sync.duration, surfsense.connector.sync.outcome
• surfsense.etl.extract.duration, surfsense.etl.extract.outcome
• surfsense.celery.heartbeat.refreshes, surfsense.celery.heartbeat.failures, surfsense.celery.queue.latency
• surfsense.auth.failures, surfsense.rate_limit.rejections
• surfsense.perf.elapsed_ms

Runtime gauges include process RSS, CPU utilization, threads, open file descriptors, asyncio tasks, and CPython GC counters.

Logs

Application logs are exported over OTLP to Loki. On init, the backend installs a LoggerProvider + OTLPLogExporter and attaches a LoggingHandler to the root logger; LoggingInstrumentor also stamps otelTraceID / otelSpanID onto each record so logs correlate with their trace even outside Loki. When OTel is disabled, logs stay on the normal container stderr path.

Verification

1. Hit a FastAPI endpoint and confirm an inbound server span appears in Grafana.
2. Run a chat request and confirm model.call and tool.call child spans with gen_ai.* attributes.
3. Run a knowledge-base search and confirm kb.search spans and SQL child spans.
4. Run connector indexing and confirm Celery producer/worker spans share a trace ID and connector sync metrics increment.
5. Confirm gen_ai.client.token.usage, durations, and runtime gauges appear within one export interval.
6. Confirm logs from a traced request appear in Loki with non-zero trace/span IDs.

Agent Tracing (LangSmith, dev-only)

LangSmith remains available for local agent debugging via LANGSMITH_TRACING. It is dev-only and off by default — production agent telemetry goes to the self-hosted LGTM stack over OTLP. A dedicated LLM-observability backend (Langfuse / Phoenix) is deferred; because agent spans are already gen_ai.* over OTLP, adding one later is a collector/exporter change, not re-instrumentation.

Product Analytics (PostHog)

Separate from OpenTelemetry, the backend emits server-side product events to PostHog. This is the authoritative source for outcome events (chats, document ingestion, connector indexing, billing, automations) because it captures traffic the browser never sees — MCP clients, personal-access-token scripts, and Celery background jobs. It is fully opt-in and mirrors the OTel contract: with POSTHOG_API_KEY unset, every capture is a silent no-op.

PostHog carries product analytics only. LLM call cost/latency lives in the OTel gen_ai.* spans and metrics (LGTM), not in PostHog.

Use the same project key as the frontend's NEXT_PUBLIC_POSTHOG_KEY so server events merge onto the same persons the web app identifies by user id. Add these to surfsense_backend/.env (local) or docker/.env (production); they reach the API, Celery worker, and beat services via env_file:

POSTHOG_API_KEY=phc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
POSTHOG_HOST=https://us.i.posthog.com

Every backend event is stamped source=backend, carries auth_method / client for surface attribution, and sends disable_geoip=true so the server IP never overwrites a person's real location.

Keep event properties low-cardinality. Never attach user content — workspace names, connector titles, document titles, prompts, or raw queries.

Out Of Scope

• Frontend/browser OpenTelemetry.
• Profiling.
• Durable production trace/metric storage (the all-in-one is ephemeral).
• A dedicated LLM-observability backend and collector fan-out (deferred until evals / prompt management are real needs).