Reliability & operations
Reliability is a product decision
A reliability target becomes useful when it describes something a customer needs to accomplish and changes how the service is designed, released and operated. Start with the promise, then decide how to measure and protect it.
Imagine a professional-training provider with an online assessment portal. Its servers are responding, the homepage loads and the infrastructure dashboard looks healthy. A learner uploads an assignment near the deadline, presses Submit and receives no confirmation. From that learner's perspective, the essential service has failed even if most requests are succeeding.
This hypothetical situation makes the reliability conversation specific. The product promise includes preserving the work, recording its submission and letting the learner establish what happened. Each of those expectations needs an owner, a measurement and a recovery route.
Name the journey before choosing the target
Identify the small set of actions whose failure would seriously undermine the product. In the assessment portal, those might be saving a draft, submitting an assignment and retrieving the submission receipt. A reporting dashboard can have a different urgency from the submission journey.
Google's SRE guidance recommends centring objectives on user actions. It distinguishes a service level indicator, or SLI, which measures service behaviour, from a service level objective, or SLO, which sets the target. A common SLI expresses good events as a proportion of eligible events. [1] The difficult product decision is defining what good means.
For submission, define whether success requires durable storage, a recorded submission state and a visible receipt within an agreed time. Decide where the measurement begins and which attempts are eligible. A valid request that cannot reach the submission service should not disappear from the calculation simply because one server never logged it. Compare the measured events with the customer journey and document gaps in coverage.
Give the budget a consequence
Choose the objective and measurement window with the people responsible for the product and its operation. Discuss what failure costs the learner and the provider, how quickly the problem must be corrected and what the service can sustain. Avoid borrowing a target because it looks impressive on another company's status page.
For a good-event ratio, the error budget is the difference between complete success and the SLO over the chosen period. [1] Its practical value comes from the policy attached to it. Google's published example uses budget consumption to redirect effort towards reliability and restrict releases, with defined exceptions. It is an example policy, rather than a rule every organisation must copy. [2]
The training provider could agree that rapidly worsening submission reliability triggers an incident response, while repeated smaller failures change the next delivery priority. Define who makes that decision and how urgent corrective changes can proceed. Keep deadline periods visible during reviews: an overall reporting window should not be the only lens through which a concentrated customer problem is understood.
Design the incomplete experience deliberately
Walk the submission journey through a dependency failure. If the email service is unavailable, the portal might still preserve the assignment, complete submission and show a retrievable receipt. Email can follow later. If the durable record cannot be written, presenting a successful submission would break the central promise.
AWS describes graceful degradation as maintaining a component's essential function when a dependency becomes unavailable, with predictable and recoverable states. Its guidance also says failure paths need testing and should remain simpler than the primary path. [3] Choosing which capability can degrade requires agreement about the product's essential behaviour.
For this portal, specify the wording for pending work, the route to recover a draft and the evidence support staff can use when a receipt is missing. Assign ownership for reconciling any retained but incomplete submission. Exercise those paths with controlled failures before relying on them. Recovery should account for the learner's work and recorded state as well as restoring a healthy process.
Make the alert lead to a useful action
An alert should help someone protect the journey. A submission alert needs enough context to establish the affected capability, find the relevant diagnostic view and choose an initial response. Infrastructure signals remain valuable for investigation, but they do not by themselves describe whether a learner can submit work.
Google's SRE workbook describes burn rate as the speed of error-budget consumption relative to the objective. It uses different windows and rates to distinguish problems needing an immediate page from those suited to a ticket, and discusses additional care for low-traffic services. [4] Those choices should fit the service's traffic and consequences; copying thresholds without that context can create misleading urgency.
Google's monitoring guidance also emphasises actionable pages and the distinction between symptoms and causes. [5] Review each page with the responder: what can they inspect, what can they safely change and whom do they involve? After an incident, check whether the measurement represented the learner's experience and whether the recovery preserved their work. Use that review to improve the promise, the instrumentation or the procedure.
Sources & further reading
The references behind this perspective. Reviewed . Product documentation can change; check the source before making an implementation decision.
- Google SRE Workbook — Implementing SLOs
Primary engineering guidance on user-centred objectives, SLI specifications and implementations, good-event ratios and error budgets; checked 21 October 2026.
- Google SRE Workbook — Example Error Budget Policy
Published example of using error budgets to prioritise reliability and restrict change with explicit exceptions; cited as an example rather than a universal release policy; checked 21 October 2026.
- AWS Well-Architected Reliability Pillar — Implement graceful degradation
Primary guidance on essential functionality, recoverable states and testing simpler dependency-failure paths. Portal behaviour in this article is hypothetical editorial application; checked 21 October 2026.
- Google SRE Workbook — Alerting on SLOs
Primary guidance on error-budget burn rates, multiple alert windows, page-versus-ticket decisions and low-traffic limitations; checked 21 October 2026.
- Google SRE — Monitoring Distributed Systems
Primary guidance on actionable paging and distinguishing user-visible symptoms from diagnostic causes; checked 21 October 2026.
