The Anatomy of an SLO: Turning Vague Promises Into Math

Most teams say they want “99.9% uptime” without knowing what that number costs, who it protects, or how to measure it.

Ask ten engineers what “reliable” means and you will get ten different answers. One will say “the site loads.” Another will say “the API returns in under 200ms.” A third will shrug and say “nobody complained today.” This ambiguity is not a communication problem — it is an architecture problem. If you cannot define reliability with a number, you cannot design for it, budget for it, or defend it in a roadmap meeting.

Service Level Objectives (SLOs) exist to close that gap. They are the mechanism that turns a vague promise — “we care about uptime” — into an engineering contract that can be measured, tested, and argued about with data instead of vibes.

The Three Letters You Need: SLI, SLO, SLA

Before building anything, it helps to separate three terms that get used interchangeably but mean very different things.

• SLI (Service Level Indicator): The actual measurement. This is a number derived from your monitoring system — for example, “the proportion of HTTP requests that returned a non-5xx status code in the last 5 minutes.”
• SLO (Service Level Objective): The internal target for that indicator. For example, “99.9% of requests will succeed over a rolling 30-day window.”
• SLA (Service Level Agreement): The external, often contractual, consequence of missing the SLO. For example, “if uptime falls below 99.5% in a billing month, the customer receives a service credit.”

The SLI is a fact. The SLO is a goal. The SLA is a promise with a penalty attached. Most internal reliability work should happen entirely in the SLI/SLO space — SLAs are a business instrument, not an engineering one, and conflating them leads teams to over-engineer for contractual penalties that may never apply to 95% of their traffic.

Step One: Pick Indicators That Represent User Experience, Not Server Health

The single most common mistake in early SLO design is measuring the thing that’s easy to measure instead of the thing that matters. CPU utilization is easy to graph. It is also almost meaningless to your user. A server can sit at 95% CPU and serve every request in 40ms, and it can sit at 10% CPU while every request times out because of a lock contention bug in a downstream dependency.

Good SLIs are almost always one of four shapes, borrowed from the “four golden signals” tradition:

• Availability: the fraction of valid requests served successfully.
• Latency: the fraction of requests served faster than a threshold (not the average — averages hide the tail).
• Quality: the fraction of responses that were “correct,” which matters enormously for search, recommendations, and any system where a 200 status code can still return garbage.
• Freshness: the fraction of data served within an acceptable staleness window, critical for caches, replicas, and event-driven pipelines.

Notice the pattern: every one of these is expressed as a ratio of good events to total valid events. That ratio is what makes an SLI aggregable, testable, and comparable across time windows. “Average latency was 120ms” tells you almost nothing about the experience of your unluckiest 1% of users. “99.5% of requests completed in under 300ms” tells you exactly how many people had a bad time, and lets you draw a line describing how many bad experiences are tolerable.

Step Two: Choose a Target You Can Defend With Math, Not Aspiration

Once you have an SLI, the temptation is to set the target to “as high as possible.” This is the second most common mistake. Five nines (99.999%) sounds impressive on a slide, but it means your system can be down for roughly 26 seconds per year — not per incident, per year, total, forever. Almost no consumer-facing web application needs that, and pursuing it means burning enormous engineering effort on marginal returns.

Instead, work backward from what actually breaks trust:

1. Look at historical data. What was your actual availability over the last two or three quarters? If you’ve been running at 99.4% and nobody outside the engineering org has noticed or complained, your target should probably start near there, not at 99.99%.
2. Understand the cost curve. Reliability follows a steep cost curve. Going from 99% to 99.9% might cost you a load balancer and some retries. Going from 99.9% to 99.99% might require multi-region active-active failover, consensus protocols, and a much larger on-call rotation. Going from 99.99% to 99.999% might require rearchitecting your entire data layer. Each additional nine is not linearly more expensive — it’s often an order of magnitude more expensive.
3. Understand what your dependencies allow. If you depend on a third-party payment processor with a published 99.95% SLA, you mathematically cannot promise your own users 99.99% for any flow that touches payments, unless you build extensive fallback and degradation logic around that dependency.

A target chosen this way is defensible in a way that “99.99% because that’s what our biggest competitor advertises” is not.

Step Three: The Error Budget Is the Whole Point

Once you have an SLO — say, 99.9% availability over a rolling 30-day window — you have implicitly created something more useful than the target itself: an error budget.

If your target is 99.9%, your budget is the remaining 0.1%. Over 30 days, that’s roughly 43 minutes of acceptable downtime or degraded service. That number is not a failure allowance to be ashamed of — it is a resource to be spent deliberately.

This reframing changes team behavior in a way that raw uptime dashboards never do:

• If the budget is healthy (you’ve used 10% of your allowed downtime this month), the team has explicit permission to take risks: ship the risky migration, roll out the new caching layer, experiment with a new deployment strategy.
• If the budget is exhausted (you’ve blown through 100% of allowed downtime), the team has an explicit, pre-agreed obligation to stop shipping features and spend all engineering time on stability work until the budget recovers.

This is the mechanism that ends the recurring argument between “ship faster” and “stop breaking things.” Instead of a political negotiation, it becomes a simple lookup: check the budget, act accordingly. The rule was agreed to when nobody was angry, which is exactly when good rules should be written.

Common Failure Modes When Introducing SLOs

Failure mode one: too many SLOs. A service with fifteen SLOs has zero SLOs, because nobody can hold fifteen numbers in their head during an incident, and the on-call engineer will not know which one to protect first. Pick one to three SLIs per user journey — often availability and latency are enough — and resist the urge to instrument everything just because you can.

Failure mode two: SLOs owned by nobody. An SLO without an owner decays silently. Someone needs to be accountable for reviewing the error budget monthly, deciding whether the target is still correct, and triggering the “stop and stabilize” conversation when the budget is blown. Without an owner, the dashboard becomes wallpaper.

Failure mode three: measuring at the wrong layer. Measuring “the load balancer returned a 200” is not the same as “the user successfully completed checkout.” Synthetic monitoring and real-user monitoring (RUM) exist specifically to close this gap — instrument as close to the actual user action as you can, not just the infrastructure component that’s easiest to scrape metrics from.

Failure mode four: treating the SLO as a performance review metric. The moment an SLO is used to punish an individual engineer or team in a performance review, people will start gaming the measurement instead of improving the system. SLOs work because they create psychological safety around calculated risk-taking. Attach consequences to individuals and that safety evaporates immediately.

A Worked Example

Imagine an e-commerce checkout API. The SLI is defined as: the proportion of checkout requests that return a successful response within 500ms, measured at the load balancer, excluding requests where the user’s cart was already empty (a client error, not a server failure).

The team reviews six months of data and finds they’ve historically hit about 99.85% under this definition. They set the SLO at 99.8% over a rolling 28-day window — slightly below their historical baseline, to leave room for the migration to a new inventory service planned for next quarter.

That gives them an error budget of roughly 0.2%, or about 80 minutes of acceptable failed/slow checkouts per 28 days. The team agrees, in writing, before any incident happens: if more than 50% of the budget is consumed in the first two weeks of the window, the migration is paused and the team pivots to stability work. If less than 20% is consumed by the midpoint, the team is cleared to accelerate the migration timeline.

Six weeks later, a subtle connection pool exhaustion bug burns 60% of the budget in four days. Because the rule was written in advance, there is no debate. The migration pauses. The team fixes the pool sizing, the budget recovers, and three weeks later the migration resumes — with a policy, not a personality, having made the call.

When Multiple SLOs Depend on the Same Upstream Service

A subtlety that trips up many teams the first time they scale SLOs beyond a single service: what happens when Service A’s SLO depends on Service B, which has its own, independently defined SLO? If Service B is allowed to burn its entire error budget in a single day, and Service A has no visibility into that, Service A’s on-call team can be blindsided by an outage whose root cause lives entirely outside their own dashboards.

The fix is not to merge every team’s SLOs into one giant shared number, since that destroys the local accountability that makes SLOs useful in the first place. Instead, mature organizations maintain a dependency map between SLOs, so that when an upstream service’s error budget burn rate spikes, every downstream team with a dependency gets automatic visibility into it, even before their own SLI shows measurable degradation. This turns SLO monitoring from a purely reactive signal into a partially predictive one: a dependency you rely on is burning budget fast, so you should expect downstream degradation soon and can prepare accordingly.

This dependency awareness also changes how postmortems get scoped. An incident that burns Service A’s error budget because of a Service B failure should be investigated primarily as a Service B incident, with Service A’s postmortem focused narrowly on whether its own fallback and degradation behavior worked as intended, rather than re-litigating a root cause that belongs to a different team’s system entirely.

Getting Started This Week

You do not need a full observability platform overhaul to begin. Start narrow:

1. Pick the single most important user journey in your product — login, checkout, search, whatever generates revenue or trust.
2. Define one availability SLI and one latency SLI for that journey, measured as close to the real user experience as your current tooling allows.
3. Look at 90 days of historical data (or start collecting it now if you don’t have it) and set an initial target slightly below your observed baseline.
4. Write down, in one paragraph, what happens when the error budget is exhausted. Get your team and your manager to agree to it before you need it.
5. Review the budget monthly. Adjust the target if it’s consistently too easy or too hard to hit.

The math is simple. The discipline of using it consistently, especially under pressure during an incident, is what separates teams that talk about reliability from teams that actually engineer for it.