What Log4Shell week looked like from the gateway
We are not a Java shop, which turned out to be almost irrelevant. Ten days of dependency archaeology and what we changed afterwards.
CVE-2021-44228 was published on 10 December. This is what the following ten days involved, written down while it is still fresh.
The first question was the wrong one
Our first reaction was relief: the gateway is not Java. That relief lasted about an hour, which is how long it took to remember that "our code" and "our attack surface" are different things.
We run a log aggregator with a Java component. Two providers behind APIs on the platform are Java shops. Our CI system had a Java plugin nobody had thought about in two years. None of that is our code and all of it was our problem.
What we actually did
Day one was inventory, and it was slower than it should have been because we did not have a current dependency list for everything we run. We had one for our applications. We did not have one for the things we had installed and forgotten.
Day two was patching what we controlled and blocking the JNDI lookup patterns at the edge as a stopgap. The stopgap mattered — several of the affected components could not be patched until upstream shipped, which for one of them took four days.
Days three to ten were providers. We contacted every provider on the platform, asked a specific question rather than a general one, and published the answers on the status page as they arrived. Two were affected and both had patched before we asked. One took six days to reply, which told us something useful about them.
The uncomfortable finding
We had no reliable way to answer "what runs in our infrastructure" without a person going and looking. Every organisation says it has an inventory. Ours was a wiki page eleven months out of date.
What changed
We generate a software bill of materials for every deployed artefact now, automatically, and we keep them queryable. The next time a CVE lands in something ubiquitous, the inventory question should take minutes rather than a day.
We also added a provider security contact as a listing requirement, with a response time expectation attached. The provider who took six days is aware of why that requirement now exists.
For customers
No customer data was accessed and no gateway component was vulnerable. We said that on day two and it has not changed. The full timeline is on the status page, including the four days we spent waiting on an upstream patch, because leaving that out would have made us look better than we were.