Observability
Health, Prometheus metrics, OpenTelemetry traces and JSON logs — and where readings live when they have to outlive a restart.
What Keydra reports about itself#
- Health
-
/q/health, with/q/health/liveand/q/health/readyfor the two probes the manifests use. - Metrics
-
Prometheus at
/q/metrics. The runtime’s own meters — requests, JVM, event loops — plus the ones only Keydra can produce: targets up and down, how long a reading takes and how often one fails, how much this instance has asked of the targets it watches, schedule runs by outcome, alerts raised and delivered, migrations in flight, and whether this instance holds the leader’s work.Labels are ids and outcomes, never names. A target’s name is a fact about somebody’s estate and a series keeps its labels for its whole life; an id says the same thing to anybody entitled to resolve it, and does not add a series every time a target is renamed.
- Traces
-
OpenTelemetry, exporting nowhere until
KEYDRA_OTLP_ENDPOINTis set — and turned on by that endpoint being set rather than by a second flag somebody has to remember. A tracing SDK pointed at nothing produces a log line per export attempt, which is worse than no tracing at all. - Logs
-
The ordinary human-readable format, or JSON on the console where the deployment asks for it with
KEYDRA_JSON_LOGS. Every line carries the instance.
The About page says what is exported and where, because "is this build sending metrics anywhere" should not be answered by reading a deployment manifest.
Metric labels#
Meters are tagged by target id and never by name. A metric label is read by systems nobody
in this project controls and lives provided that the series does; production-payments is a
fact about somebody’s estate, and an id says the same thing to anyone entitled to resolve it.
Every meter carries the instance that made it, under keydra_instance rather than
instance: behind a load balancer the same question asked twice reaches two processes, and
a graph that mixes them says nothing — while a scraper puts its own instance label on what
it collects and renames anything that arrives already carrying one.
Keeping readings across restarts#
By default Keydra keeps the last hour of readings per target in memory, and they go when the process does. Being told that memory is at 91% raises two questions that cannot then be answered: since when, and was it like this last Tuesday.
A ClickHouse answers both, and is optional and off by default.
KEYDRA_CLICKHOUSE_ENABLED=true
KEYDRA_CLICKHOUSE_URL=http://clickhouse.internal:8123
KEYDRA_CLICKHOUSE_USER=<username>
KEYDRA_CLICKHOUSE_PASSWORD=<password>Retention is ClickHouse’s own TTL rather than a policy Keydra invents; the number of days is configured alongside.
On a target’s Monitoring page, choose a Window longer than an hour. The chart says From the store, {seconds}s buckets rather than From memory.
Restart Keydra and the chart still shows the period before it.
Writes to the store are fire-and-forget and never delay a reading. A sampler that waits on a store is a sampler that stops sampling when the store is slow, which is when the readings matter most.
ClickHouse was chosen for the shape of the data rather than for fashion: a wide, dull, append-only stream of numbers per target per few seconds, read back as ranges. PostgreSQL would do it, and would be carrying seventeen thousand rows a day per target that it has no use for.