The platform ran the site, and it ran blind. Console output scrolled past in a window on the server. The only latency figure anyone could quote was an impression. The way you found out a page was slow was that somebody walked over and told you.
That is survivable on a small tool. On one the whole floor depends on, it isn't — because the questions that matter can't be answered by looking. Which route is actually costing people time. Whether the worker pool is saturated or merely warm. Whether a background job failed or is still running. Whether the error someone reported was a fault or a permission check doing its job.
- Diagnosing a floor complaint meant remoting into the server to watch a console scroll
- Nothing separated a slow page from a slow database call underneath it
- A 404 and a 500 counted the same, so "error rate" meant nothing in particular
- Thread-pool size was visible; what the threads were doing was not
Instrumentation at four levels, each answering a different question. Every HTTP request is timed in an after-request hook and written to a per-request log with method, path, endpoint, status and duration. Every background function call carries a generated request identifier and records its module, duration and outcome. A snapshot thread rolls up active users, request rate, latency, error rate, live thread count and queue depth once a minute for the long view. Alongside those, a Prometheus exporter publishes active users, worker threads, queue depth and per-function timings and error counts, so external monitoring can scrape the platform without going through the dashboard.
On top of that sits one screen: six selectable time ranges from one hour to thirty days, fifteen panels and six charts across six sections, refreshing every fifteen seconds. It is organised by question rather than by data source — real-world performance, how people actually use the app, where to spend engineering effort, usage patterns over the day and week, error intelligence, and a collapsed section holding the infrastructure detail that only matters when something is wrong.
- Latency reported as p50 and p95 — overall, per route, and per time bucket
- Trend bucket width scales with the range, from five minutes up to a day
- Human page views separated from API and background traffic by method and path
- Client 4xx and server 5xx counted separately; the headline rate is 5xx only
- Standard output and error are wrapped at process start and tee'd to a rolling ten-thousand-line memory buffer and a size-rotated file, so a print statement anywhere in the codebase becomes a levelled log line the viewer can filter
- A console viewer with level filtering, two-second auto- refresh, download and buffer clear
- A health indicator driven by server error rate and worker saturation rather than uptime
Measuring without becoming the load
A dashboard that polls itself every fifteen seconds will, within a day, be the most popular page in its own report. Telemetry paths, static assets, images and the metrics endpoint are excluded twice — once at the point of capture and again in the SQL fragment shared by every aggregate query — specifically so the two definitions can't drift apart and quietly reintroduce the noise.
The capture path fails open. The after-request hook is wrapped so that any exception is swallowed and the response leaves unchanged, and the persistence call inside it carries its own separate guard. Records are handed to an in-process queue and drained by a dedicated writer thread that flushes on ten records or two seconds, whichever comes first. A failed flush discards the batch in a finally rather than retrying it, and an unexpected error in the writer loop itself sleeps five seconds before continuing, so a database outage can't become a hot loop. Losing a metric is acceptable. Slowing down someone scanning freight is not.
The read path fails the other way — toward still showing something. Aggregates are computed in the database over the selected range, but the process also keeps in-memory ring buffers of recent timings, process records and errors. If the database read throws, the endpoint rebuilds the same breakdown structures from those buffers and serves them, so the dashboard keeps working during exactly the outage you'd most want to look at it during.
- The percentile, bottleneck and latency-trend queries are each guarded individually, so one unsupported query degrades a single panel instead of the page
- Timestamps are explicitly marked as UTC, because a browser silently reading them as local time makes a dashboard lie rather than fail
- Logging errors are written to the real error stream, not back through the wrapper, so a failing logger can't recurse
- The metrics exporter is initialised at the bottom of the module, after every route is defined, because it instruments the view functions that exist at the moment it runs
Python won't let you kill a thread, and a thread pool keeps its workers alive once it has grown. After a burst the pool climbs to its ceiling and stays there for the rest of the process lifetime — which on a graph looks identical to a platform that is permanently busy.
So the pool is replaced rather than pruned: build a new executor, swap it in under a lock, shut the old one down without waiting. What makes that safe is the refusal to do it at the wrong moment. Queue depth, tracked requests and live task activity must all read zero, and the check is made twice — once by the endpoint, which returns a conflict and a state dump instead of acting, and again inside the swap itself while holding the lock. A maintenance loop runs the same routine every minute and only recycles after ten unbroken minutes of idleness.
Knowing which workers are actually working needed its own mechanism. Task submission is wrapped so each job registers its thread, the callable it is running and its start time in a lock-guarded table, and removes the entry in a finally block. The dashboard reads that table alongside live interpreter frames, filters the stack down to frames from the application's own source tree, keeps the last ten and shows the deepest as the thread's current location. Idle threads say idle. Busy threads say what they are busy with, where in the code, and for how long.
The questions that used to need a person now have a screen. Which routes cost users the most total waiting time — ranked by average duration multiplied by request count, so effort goes where the volume is rather than where the single worst case is. Which routes are usually fine and occasionally awful, which only a per- route 95th percentile exposes. Which parts of the platform people actually reach, how many distinct people reach them, and at what hour of the day.
The unglamorous half was deciding what not to count. Excluding the dashboard's own polling, separating a routine permission denial from a genuine fault, reporting a median beside a tail instead of an average that hides both. None of it is difficult, and all of it is the difference between a dashboard people act on and one they stop opening.
