An online recommendation endpoint autoscales on a target requests-per-second metric. Each replica is configured with a high maximum-concurrent-requests setting to maximise throughput per replica. During a traffic spike, the throughput numbers look healthy and few new replicas are added, yet the 95th-percentile latency climbs far above the agreed service objective. The team wants to bring tail latency back under control during spikes without permanently over-provisioning idle replicas. Which change best addresses the latency regression?
- ALower each replica's maximum-concurrent-requests setting so replicas saturate sooner, triggering the autoscaler to add capacity earlier when requests start queuing. Correct
- BRaise the autoscaler's target requests-per-second threshold so replicas are added later, keeping each replica fully utilised and reducing overall cost during spikes.
- CRemove the autoscaler and run a fixed large pool of replicas sized for peak traffic to guarantee headroom at all times.
- DMove feature lookups to a larger cache so each request spends less time fetching features, eliminating the queuing that drives tail latency during spikes.
Why A is correct: A high concurrency limit lets one replica queue many in-flight requests while throughput still looks healthy, hiding the backlog; lowering it makes replicas saturate sooner so the autoscaler adds capacity before queuing inflates tail latency.
Why B is wrong: Higher thresholds do reduce replica count and cost, which sounds efficient, but they let each replica absorb even more load before scaling, deepening the per-request queuing that is already inflating tail latency.
Why C is wrong: A fixed peak-sized pool would hold latency down, making it tempting, but it permanently over-provisions idle replicas during normal traffic, which the team explicitly wants to avoid.
Why D is wrong: Faster feature lookups can reduce per-request work and are a real serving optimisation, but the bottleneck here is requests piling up behind a high concurrency limit, not slow feature reads, so caching does not address the queuing.