Scaling cloud infrastructure without over-provisioning

There are two familiar ways to get cloud infrastructure wrong. The first is running permanently oversized instances so that peak traffic is never a problem, and paying for that headroom every hour of every day. The second is running lean and discovering during a campaign that the system cannot scale because of a decision made in the application layer eighteen months earlier.

Both are avoidable, and the fix is mostly architectural rather than operational.

Scaling starts in the application, not the console

Autoscaling can only add copies of your service. If a copy behaves differently from its siblings, adding copies breaks things. That makes statelessness the precondition for everything else.

• Sessions belong outside the process. In-memory sessions mean a user must return to the same instance; the moment you scale, they are logged out at random.
• Uploads belong in object storage. A file written to local disk exists on exactly one instance and disappears when it is replaced.
• Scheduled work needs a lock. A cron job inside the application runs once per instance — three replicas means three invoice runs.
• Caches must be shared or disposable. Per-instance caches produce inconsistent results depending on which replica answered.

The database is where scaling actually stops

Application instances are cheap to add. Databases are not, and they are where almost every scaling limit is eventually found.

• Connection limits. Every new instance opens a pool. Twenty replicas at twenty connections each will exhaust a mid-sized database. A connection pooler in front is not optional at scale.
• Missing indexes. A query that is comfortable at ten thousand rows can be pathological at ten million. Load testing against production-sized data — not a seeded development set — is what surfaces this.
• Read/write split. Most applications read far more than they write. Read replicas absorb that cheaply, provided the application tolerates slight replication lag.
• N+1 queries. The single most common cause of database load in application code, and invisible until concurrency rises.

Autoscaling policies that behave

Default autoscaling configurations are usually tuned for demonstrations rather than production. Three adjustments make the difference.

• Scale on the right signal. CPU is the default and often the wrong one. Request queue depth or latency usually reflects user experience more directly, particularly for I/O-bound services.
• Scale up fast, down slowly. Aggressive scale-down causes thrashing: capacity is removed, latency rises, capacity is added again. Asymmetric cooldowns fix it.
• Set a ceiling. An upper bound converts a runaway loop or traffic spike from an unbounded bill into a degraded service and an alert — the far better outcome.

Also account for warm-up. If an instance needs forty seconds before it can serve traffic, the policy must trigger before saturation, not at it.

Infrastructure you can rebuild

The real test of a cloud environment is not whether it runs today but whether you could recreate it from scratch tomorrow. If that depends on someone remembering which settings were changed in a console, the environment is a liability.

• Define infrastructure as code in version control, reviewed like application changes.
• Containerise services so they behave identically in every environment.
• Keep staging genuinely representative — a staging environment that differs structurally from production tests very little.
• Store secrets in a managed secret store, never in the repository or the image.

Controlling cost without hurting reliability

Cloud spend drifts upward by default: resources are provisioned for a project and never removed. A few habits keep it aligned with usage.

• Right-size from measured usage rather than the estimate made before launch.
• Commit to baseline capacity through reserved or committed-use pricing, and let autoscaling handle the peaks on demand.
• Set lifecycle policies on storage and log retention — both grow silently and indefinitely.
• Tag resources by environment and owner, and review the untagged ones monthly. They are usually forgotten.
• Put a budget alert in place before you need one.

You cannot scale what you cannot see

Scaling decisions are only as good as the data behind them. That means request latency percentiles rather than averages, error rates by endpoint, database query timings, and traces that show where a slow request actually spent its time.

Alert on symptoms users would notice — latency, error rate, queue depth — rather than on every resource metric. Alerts that fire routinely get ignored, and an ignored alert is worse than no alert.

Our cloud infrastructure service covers architecture, pipelines and monitoring for systems built this way.