RU Contact us
MON / 06 of 09 · Monitoring

Monitoring that reports the outage before the user does

We build observability from service objectives, not from a list of metrics. Every alert must call for an action.

When people call us
  • 01The on-call team has muted the alerts
  • 02Outages are reported by a user phone call
  • 03There are dashboards, but none answers “what broke”
  • 04Monitoring went down together with what it watched
+Monitoring: ink illustration
ALR / 412 alerts overnight, zero actions
DEP / monitoring runs on the cluster it watches
MON / 06Findings are illustrative
+
Day one

What we check first

  • Define what “working” means for the service and measure it
  • Review a month of alerts: how many required action
  • Check that monitoring is independent of what it observes
  • Check coverage of dependencies: DNS, certificates, storage
Tool

Error budget and alert thresholds

Multipliers 14.4 and 6 come from SRE practice: 2% of the budget burned in one hour, 5% in six.

+
Failure patterns

Where it usually breaks

A

Noise instead of signal

A threshold on every metric. Hundreds of alerts, people stop reacting, and the real outage is lost among them.

B

Green metrics, dead service

CPU and memory are fine, yet the user cannot sign in. Resources are measured, not outcomes.

C

The observer inside

Monitoring lives on the same cluster and storage. In an outage it is the first thing to go dark.

+
Outcome

What you keep

  • Service objectives and error budgets for key systems
  • Alerts that each map to an action
  • Dashboards for on-call staff and for management
← BKP

Backup

DC →

Data centers

+
Contact

Describe your task

Tell us what is happening and what has been tried. The engineer who will handle it replies.