Logs recount events, metrics show trends, and traces help follow a request through the system. Observability is being able to investigate what you did not anticipate.

Introduction

An application can work locally and fail under real traffic, with unexpected data, or because of a slow dependency. Observability lets you investigate those behaviors with evidence.

It does not mean installing three dashboards or keeping every log forever. It means instrumenting the system to answer operational questions: What is failing? Who is affected? Since when? Where is time lost? And what changed?

1. The three main signals

Logs

Discrete events with context. In distributed systems use structured logs and correlation fields (trace_id, request_id, order_id).

{
  "level": "error",
  "message": "payment_provider_timeout",
  "service": "checkout-api",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "order_id": "ord_123"
}

Do not log passwords, tokens, secrets, or unnecessary personal data.

Metrics

Values aggregated over time: request rate, errors, latency, CPU, memory, pool connections. They support alerts and trends.

Useful models:

• RED (for request/response services): Rate, Errors, Duration.
• USE (for resources): Utilization, Saturation, Errors.

Avoid very high-cardinality labels (user_id, request_id) on metrics: they drive cost and memory.

Distributed tracing

A trace is the path of an operation; spans are units of work. A request may cross a gateway, API, database, and an external provider.

Trace: POST /checkout
 ├─ API handler          18 ms
 ├─ PostgreSQL query      7 ms
 └─ Payment provider    940 ms  ← main suspect

The value is seeing the path of one request, not accumulating spans without criteria.

How they complement each other

Question	Signal that usually answers first
Is p95 worse than yesterday?	Metrics
What failed on this specific order?	Logs (+ trace_id)
Where did time go on this request?	Trace
How many users does this error affect?	Metrics + aggregated logs

2. OpenTelemetry: instrumentation, not the full backend

OpenTelemetry (OTel) provides APIs, SDKs, conventions, and protocols (OTLP) to generate and transport telemetry. It is not by itself a log store, an alerting system, or a UI.

Instrumented application
  ├─ traces
  ├─ metrics
  └─ logs
       ↓ OTLP
OpenTelemetry Collector
       ↓
Observability backend(s)
       ↓
Dashboards, queries, and alerts

The Collector receives, processes, filters, and exports. That way you avoid coupling the code to a specific vendor.

3. Minimal example: a span in Node.js

Configure the SDK and exporter before loading the modules you want to instrument. Prefer automatic HTTP/DB instrumentation when it exists; add manual spans only for relevant business operations.

import { trace, SpanStatusCode } from "@opentelemetry/api";

const tracer = trace.getTracer("orders-service");

export async function createOrder(orderId: string) {
  return tracer.startActiveSpan("createOrder", async (span) => {
    try {
      span.setAttribute("order.id", orderId);
      // Validate and persist the order here.
      return { id: orderId, status: "created" };
    } catch (error) {
      span.recordException(error as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw error;
    } finally {
      span.end();
    }
  });
}

Too many spans increase cost and noise. Instrument what helps you decide.

4. Context propagation

To follow a request across services, context must travel in headers compatible with W3C Trace Context. If each service creates a new trace, you lose end-to-end visibility.

Propagation is also a trust boundary: do not blindly accept all context from untrusted clients (it can be used to poison or bias sampling).

5. Tools: what problem each one solves

Tool	Usual role	Considerations
OpenTelemetry	Instrumentation and transport	Not a query backend
Prometheus	Metrics	Watch cardinality
Grafana	Dashboards (and more OTel products)	Needs well-designed sources
Jaeger / Tempo	Traces	Scalability and retention
Loki / Elastic	Logs	Labels, index, and cost
Datadog / New Relic	Integrated platform	Convenience vs cost and lock-in

OpenTelemetry can feed several backends. They are not one-to-one substitutes.

6. Useful alerts and SLOs

An alert must be actionable: indicate a problem someone can investigate and mitigate. Define SLOs relevant to the user (availability or latency of a critical operation), not arbitrary CPU thresholds.

Conceptual latency example in Prometheus (adjust the metric name to your instrumentation):

histogram_quantile(
  0.95,
  sum(rate(http_server_duration_seconds_bucket[5m])) by (le)
)

Test alerts with controlled failures. An alert nobody understands is noise.

7. Common mistakes

• Logs without structure or trace_id.
• Logging secrets or PII.
• user_id as a metric label.
• Alerting on every individual error without measuring impact.
• Installing OTel and believing you already have observability.
• Sampling traces without knowing which critical transactions you lose.
• Not correlating logs ↔ metrics ↔ traces.
• Pretty dashboards nobody uses in an incident.

8. Decision matrix

Need	Initial approach
Investigate concrete errors	Structured logs + correlation
Detect degradations	Metrics + alerts on SLOs
Follow a request across services	Tracing + propagation
Avoid vendor lock-in	OpenTelemetry + OTLP
Control costs	Sampling, retention, filters, cardinality
Small team	One platform the team can operate

Conclusion

Observability starts with the questions you need to answer. OpenTelemetry helps you instrument; backends store and query; alerts connect signals to action.

You do not need to capture everything. You need enough context to find causes without exposing data or paying for telemetry nobody uses.

In this series

1. Authentication: sessions, JWT, and OAuth
2. Observability (this note)
3. SQL, query builders, and ORMs
4. How to test an API
5. Containers and Twelve-Factor

Related reading

• Benchmarks and architecture decisions
• How to test an API
• Containers and Twelve-Factor
• Tool: which backend fits you

References

• OpenTelemetry Documentation
• OpenTelemetry Collector
• W3C Trace Context
• Prometheus Documentation
• Google SRE — Service Level Objectives
• OWASP Logging Cheat Sheet