Practical guide
"We have observability" usually means a monitoring vendor is installed and a couple of dashboards exist somewhere. It rarely means that when something breaks at two in the morning, the on-call engineer can answer "what changed, who is affected, and where should I look now" within minutes. That gap — between having the tool and having the answer — is what we call observability theatre.
It happens for an understandable reason: dashboards are visible and easy to show off, so they get built first. Alert discipline and structured logging are invisible until the night you need them, so they get deferred. This is about closing that gap deliberately — without turning it into a half-year platform project.
TL;DR
Key takeaways
- A dashboard is only useful if someone opens it during an incident. If nobody can name the three dashboards they would check first, you have theatre — not observability.
- Structured logs (consistent fields, correlation IDs, machine-readable) beat a wall of console.log every time you need to trace a request across services.
- Alerts should fire on symptoms users feel (error rate, latency, failed checkouts) — not on every metric crossing an arbitrary threshold. Alert fatigue is worse than no alerting.
- Traces pay off most the moment you have more than one service talking to another. One well-instrumented request trace answers questions that hundreds of log lines cannot.
- The real cost of observability is instrumentation time, not the vendor bill. Budget engineering time for it explicitly — otherwise it never happens.
- Start with the three questions you would actually ask during an incident, and build backwards to the dashboards and alerts that answer them.
Why this matters more than uptime numbers
The price of poor observability does not show up as downtime — it shows up as time-to-diagnose. Two systems can have identical uptime and completely different operational cost: one where an incident is understood quickly and fixed fast, and another where the team spends a long time figuring out what is wrong before the fix itself takes five minutes. The second team looks fine on an uptime graph and burns out at every incident.
Observability theatre is expensive in another, less obvious way: false confidence. A team with forty dashboards and no alert discipline thinks it is monitored. It discovers the opposite at the first incident nobody saw coming — because the metric that mattered was not on any of those forty panels, or was there but buried among thirty-nine others nobody checks.
Decoration dashboards vs. actionable observability
Decoration dashboards
Built to demo well, rarely opened during a real incident.
- Dozens of panels covering every metric the platform can emit
- Built once at setup, rarely revisited when the system changes
- No clear owner of what "healthy" means on each panel
- Alerts (if any) fire on raw metric thresholds, not user-visible symptoms
- Logs are unstructured text, hard to search across services
- Nobody can name which three panels they would check first during an incident
Actionable observability
Built around the questions you actually ask when something is wrong.
- A small number of dashboards, each with a specific operational question
- Reviewed and trimmed when the system changes — dead panels are deleted
- Every alert has an owner and a documented expected response
- Alerts fire on symptoms (error rate, latency, failed transactions), not every threshold breach
- Logs are structured with correlation IDs so a request can be traced end-to-end
- The on-call engineer knows exactly where to look first — because it has been practised
What minimal, honest observability looks like
You do not need a full observability platform from day one. The floor is lower than most teams think: structured logs with request/trace ID, a handful of metrics that map directly to what users experience (error rate, p95 latency, queue depth, failed payment rate), and alerts that page a human only when one of those symptoms crosses a threshold that actually means something is broken for someone.
The next layer — distributed tracing — earns its cost the moment your system has more than one service, or a request touches more than one datastore. Without it, debugging a slow request across an API, a queue, and a database means correlating timestamps across three separate log streams by hand. With it, you see the whole request as one timeline — and the slow span is obvious.
Where teams overspend is buying an enterprise observability platform before they have earned the complexity it handles. A small Next.js app with Postgres and one background worker does not need tracing infrastructure sized for a hundred-microservice fleet. Match the tool to the actual topology, and grow it as the system does.
Build minimal, actionable observability
This is the sequence that gets you from nothing to something you can trust fastest — without overbuilding.
- 01
Write the three questions you would ask during an incident
Before you touch tooling: what would you actually want to know first when something breaks? It is typically a version of "what changed", "who is affected", and "is it getting worse". Build everything else to answer that.
- 02
Add structured logging with correlation ID
Every log line should carry request or trace ID, timestamp, severity, and consistent field names. That alone makes debugging across services significantly faster than free-text logs.
- 03
Instrument the few metrics that map to user experience
Error rate, latency percentiles, and the business-critical path (checkout completed, login succeeded, job completion rate) — not every metric your framework exposes by default.
- 04
Set up alerting on symptoms, not raw metrics
Alert when error rate crosses a threshold that means users are affected — not when CPU ticks up for thirty seconds. Give every alert a documented, specific response.
- 05
Add tracing when more than one service is involved
If a single request only touches one service and one database, tracing is nice-to-have. The moment a request branches across services, tracing stops being optional.
- 06
Run an incident drill and trim what nobody used
Simulate an incident (or review a real one) and check which dashboards and logs were actually used. Delete or rebuild those that did not help — this is the step most teams skip, and it is the step that keeps observability honest over time.
Questions we get again and again
What is minimum viable observability for a small production app?
Structured logs with request ID, error rate and latency metrics on your critical paths, and a small number of alerts tied to those metrics with a documented response. That is enough to handle most incidents competently. Add tracing and more granular dashboards as the system gains more services.
How many dashboards should we actually have?
Fewer than you think. A handful of purpose-built dashboards that answer specific questions ("is the checkout flow healthy", "is the background job queue backing up") beat dozens of generic ones. If a dashboard does not answer a question someone actually asks, it does not help.
Should we use a dedicated observability vendor, or can we build it ourselves?
Both work. Vendors (Datadog, Grafana Cloud, Better Stack, and others) save setup time and add polish; a self-hosted stack (Grafana, Loki, Prometheus) costs more engineering time but avoids vendor lock-in and per-seat scaling costs. The choice matters less than the discipline behind it — a bad alert strategy is bad on any platform.
How do we stop alert fatigue when it is already a problem?
Audit every alert that fired last month and ask what action it should have triggered. If the answer is "none, really", turn it into a dashboard panel instead of a page. Rebuild the pager list from the symptoms that genuinely need a human — and be ruthless about trimming.
Is tracing worth it for a monolith with one database?
Typically not yet. Tracing earns its cost when a request branches across multiple services or datastores. For a single-service monolith, good structured logging with request ID gives you most of the same debugging value at a fraction of the setup cost.
If you want an honest assessment of your current setup
We will review your dashboards and alerting — no obligation.
We will tell you which parts actually help during an incident, which are theatre, and what the smallest useful next step looks like.