Traffic patterns in a month nobody planned for
March rearranged our traffic. What broke was not capacity — it was every assumption we had encoded about when load arrives.
March was not a normal month for anyone. From an infrastructure position it was interesting in a way we would rather it had not been, and there are a few things worth writing down.
The shape changed, not just the size
Total volume rose about forty per cent, which capacity handled without drama. The disruptive part was the shape.
Our traffic used to have a clear daily rhythm: a European morning ramp, a midday plateau, an American afternoon peak, and quiet nights. Weekends were roughly a third of weekdays. Every piece of automation we had was tuned to that rhythm — scaling schedules, maintenance windows, the batch jobs that assumed a quiet period existed.
In March the peaks flattened and spread. The quiet night got shorter and the weekend stopped being quiet. None of this broke anything on its own; it broke the things that had assumed otherwise.
What actually broke
A nightly rollup that had four hours of headroom for years suddenly had ninety minutes, because the traffic it competed with no longer went away. It did not fail, it just started finishing later each night until someone noticed the dashboards were stale by breakfast.
Two scheduled maintenance windows were cancelled because the window no longer existed. We have stopped defining maintenance windows by clock time and now define them by measured load, which is what we should have been doing anyway.
What did not break
Capacity, autoscaling, and the providers. We had braced for a provider to fall over under sustained load and none did.
The one customer-visible incident was ours: a rate limit tuned for the old shape started rejecting legitimate traffic from a customer whose usage had moved, not grown. We raised it within the hour and then went looking for every other limit set against an assumption about time of day.
The general lesson
Capacity planning is mostly about magnitude, and magnitude was the easy part. What cost us was every place where a schedule had been quietly encoded as a fact about the world.
We have removed the ones we could find. There are certainly more.