More instances do not fix every bottleneck
Adding web servers helps when independent application work is the constraint. It can make a database connection bottleneck worse if every new instance opens another pool. Begin with the slow user journey and identify where its time goes.
Google's SRE monitoring chapter organizes monitoring around latency, traffic, errors and saturation. These signals are useful together: average CPU alone cannot tell you whether requests wait on a remote API, a database lock or a busy event loop.
Trace one concrete request
For an illustrative article page, divide the request into queue wait, database query, content rendering and response transfer. Record p50 and p95 latency alongside request rate. A stable median with a worsening tail can indicate contention or an occasional expensive path rather than universally slow code.
| Observation | Investigate next |
|---|---|
| CPU saturated, low dependency wait | Profiling and parallel capacity |
| Slow database span | Plan, indexes, locks and scanned rows |
| Growing worker backlog | Arrival rate versus service rate |
| Pool acquisition delay | Connection budget and long transactions |
| Cache hit rate dropping | Key churn, expiry and eviction |
These are diagnostic hypotheses. Confirm them with traces or measurements before acting.
More application servers meet the same narrow database path
Suppose a search endpoint spends most of its time waiting for one database query. Doubling application instances can increase the number of concurrent queries without increasing database capacity. The queue grows, connections remain occupied longer and requests time out. CPU on each application server may look comfortable, creating the false impression that even more servers are needed. The bottleneck is the shared path that all instances still use.
Trace a request through queue wait, connection acquisition, query execution, external calls and serialization. Capture the distribution rather than only an average: a small fraction of very slow requests can consume a disproportionate share of resources. Compare arrival rate with completed work and observe where the backlog accumulates. A controlled change should target that stage, such as reducing scanned rows, caching a safe lookup, limiting concurrency or moving a long task off the request path. Treat autoscaling as a way to expand the resources it actually controls, not as a substitute for understanding a dependency's capacity.
Match the intervention to the limiting resource
A CPU-heavy image transform may benefit from more workers until another resource becomes limiting. A database query that reads too much data often needs a better access path or narrower result. Lock contention needs attention to conflicting transactions and their duration; additional connections can worsen it. A third-party quota cannot be expanded by adding your own instances, so the product needs scheduling, admission control or a degraded mode.
Write a hypothesis before each capacity change and include the signal that would disprove it. If adding workers raises throughput but also doubles database wait, the improvement may be temporary. If an index helps one query while increasing write cost, evaluate the entire workload rather than declaring a universal win. Preserve realistic data skew in experiments. One unusually active tenant, a large file or a hot key can produce contention that uniform synthetic traffic never reveals. Explain the tested request mix and resource limits so that results can guide a decision without being presented as a promise for every future feature.
Reproduce the load shape
Use a safe environment with realistic data size, request mix and cache state. Separate cold-start, warmed and sustained-load runs. A test that repeatedly requests one cached page cannot represent search queries or write-heavy traffic. Increase concurrency gradually and record the first point where latency, errors or queues rise sharply.
Use PostgreSQL EXPLAIN to understand database access plans. Running EXPLAIN ANALYZE executes the statement, so use appropriate fixtures and avoid casually analyzing destructive SQL against production. Compare estimated and actual rows when you have a safe execution environment.
Change one constraint at a time
If a query scans an unnecessarily large set, narrow it or add the justified index, then repeat the same workload. If slow external calls hold database connections, shorten the transaction boundary. If one media task monopolizes the web process, move that workload behind an independently budgeted worker.
Do not report a performance gain from different data, cache conditions or machines. Keep a run manifest with commit, configuration, workload, warmup, duration and hardware. A negative result is useful evidence too.
Capacity includes failure headroom
Choose an operating point below saturation so a failed instance or traffic burst does not immediately collapse the system. Define overload behavior: bounded queues, timeouts and explicit rejection are preferable to unbounded memory growth.
Read caching strategies when repeated reads dominate, and practice the architecture in AiFolder interviews.
Keep overload from turning into a retry storm
Overload is a feedback problem when clients automatically retry timed-out work. A request can still be consuming resources after its caller has given up, while the retry starts a second copy. Bound queue length, cancel work where the downstream API permits it, and apply retry budgets with backoff. For operations with side effects, pair those controls with stable idempotency identities so that retries do not create a second business effect.
Define the minimum useful service during saturation. It may be better to reject optional expensive requests promptly than to let every essential request wait indefinitely. Monitor accepted throughput, rejected traffic, queue age and completion latency together. A rising rejection count can indicate that protection is working, provided essential operations remain available and the user sees a clear response. Rehearse the overload boundary with a capped experiment and a stop condition. The aim is to discover where capacity stops producing useful work and to preserve recovery, rather than to chase the largest request number while leaving the system unable to drain its backlog.
