0028 - Adopt OpenTelemetry Semantic Conventions for Structured Logging

Status

Proposed

Context

Our web application's logging is currently unstructured (plain text), making it difficult to search, filter, correlate with requests, or analyze at scale. This leads to manual grep-based debugging, inconsistent formats across services, and poor integration with observability tools. As we standardize other areas like HTTP error responses with RFC 9457, we seek a similar convention-driven approach for logs that aligns with our values: standardization, AI-assisted productivity, writing less custom code, and established conventions over bespoke solutions. OpenTelemetry (OTel) provides a vendor-neutral, open standard for observability signals (logs, traces, metrics). For logging, it defines a Log Record data model with core fields (body, timestamp, severity, attributes, trace_id, span_id) and semantic conventions-stable, agreed-upon attribute names and structures (e.g., http.method, service.name, db.statement). These conventions ensure logs are machine-readable, interoperable across tools, and correlated with traces/metrics without reinventing formats per team or service. Even from unstructured logs, OTel offers a low-disruption path: parse at the collector level, then incrementally structure application logs. This mirrors our RFC 9457 adoption-starting with a standard envelope while allowing domain-specific extensions.

Decision

We will adopt OpenTelemetry semantic conventions as the standard shape for all application logs across our web app services, processed through an OTel Collector pipeline. Concretely: * All logs will conform to the OTel Log Record model, enriched with semantic attributes for common domains (HTTP, database, service metadata). * New code will use OTel-style structured logging directly where practical; legacy logs may be transformed in the collector as a transitional step. * BetterFleet-specific or domain-specific fields may be included as extension attributes, but standard OTel names should be preferred whenever an equivalent exists. * The logging pipeline should preserve correlation fields such as trace/span IDs so logs align with distributed tracing and future metrics.

Consequences

Positive: * A consistent, machine-readable log structure across services. * Better compatibility with observability backends, dashboards, and automated tooling. * Easier log querying, correlation, alerting, and AI-assisted analysis. * Lower long-term cost than inventing and maintaining a bespoke logging schema.

Negative / Trade-offs: * Some initial migration effort will be required to refactor existing logs or introduce collector transforms. * Teams will need discipline to avoid ad hoc field naming that undermines the convention. * OTel conventions may feel verbose in simple cases, but the standardization benefits outweigh that overhead.