War Stories - The breach from within
Responding to security incidents can take you to completely unexpected places.
“There’s a weird error on the admin page”
One of the lead engineers for a company I was helping walked over to my desk shortly before the day ended. He had been randomly checking on errors and he saw something suspicious - a series of errors on our internal admin pages for users.
We had never seen the error before - did we accidentally break internal tools relied on by our support team?
Thus started a saga that involved sprinklings that wouldn’t be out of place in a thriller: security breaches, detective work, and sudden leadership exits.
It all started with an error
While I don’t recall the details of the error, what I remember was that it was unusual - it wasn’t the kind of error we normally saw, and it was in a location that rarely broke.
The system was usually noisy, but predictable. Hundreds of the same kinds of error would occur like clockwork- some blocking, some weren’t, but they were all expected to various degrees as the day’s activity ebbed and flowed.
New errors, though - we were alway concerned with new errors. We treated new ones as change failures, and since this was a new error, we started an incident response.
We checked deploys - we hadn’t deployed anything recently to break things. We checked feature flag toggles - nope, nothing.
So, we dug deeper. As the lead and I investigated, we started becoming more and more puzzled.
We saw logs that showed access to page numbers in the triple digits. Our first reaction was very engineer: why was someone on support accessing the user list in the internal admin in such an inefficient way? Most people just searched for who they needed.
As we dug deeper, we started seeing other things that raised our alarm bells - sequential ids were being accessed, the user account didn’t match the name of anyone we knew, even some indirect object reference.
This wasn’t normal user behavior. This was a security incident.
Houston, we have a problem
We immediately started our incident response.
First - we wanted to identify and contain the threat, which was ongoing. After reporting it to the company leadership, we got to work.
The activity turned out to be a scraper that was scraping our internal admin pages that listed user accounts page by page by page. We saw logs with indirect object references, contiguous pagination coming from multiple sources.
We quickly got a list of IP addresses and accounts that were performing this activity. We went over account names, matching them with people we knew on the support team. We narrowed on a list of people we didn’t know or recognize.
We shut down the accounts and blocked access. After we were confident the immediate threat was contained, we moved to expand further to collect evidence and understand the extent of the attack.
We started the work of identifying who had created the accounts, what they had accessed, and when they had gained access. We poured over logs, audit trails, error alerts - anything that would give us information. We jumped back and forth from being glad we had logging in certain places to cursing our past selves for not implementing more robust observability.
Eventually, we found the smoking gun: the IP address of the creator of the accounts. We did a broader trace of activity and found something shocking.
The call is coming from inside the house
What we found was the biggest surprise: the accounts were created by IP addresses associated with one of our executives.
What? Did we accidentally copy-paste the wrong log? Was it a public internet? We double checked, then triple checked, then quadruple checked. Nope, it was accurate.
We cross-compared against everything we could think of. We checked IP addresses from Chat logins, email messages, account logins from when we knew they were in the office, going back months and even years. The IP was consistent. We even geolocated it to their house on days they were in office vs. days they weren’t.
The admin accounts used in the attack were created by the executive’s IP address many days prior, and they were created from an IP address that was associated with private, legitimate traffic. In short - it wasn’t a case of a hacked admin account.
The immediate question on our minds was - why? What was the motive?
It dawned on us - shortly before all this happened, the executive had declared they were stepping down for a new company. It’s possible the information would’ve been useful.
Reporting to the C-Suite
As we packaged up our findings, we notified the C-suite with the latest conclusions and gave them the evidence. The suspicion was that the executive hired contractors to write and run the scraper against our internal pages, and granted access to them to do so.
We documented everything - logs, screenshots, data fields that were accessed.
Afterwards, we secured a forensics team to dig deeper and prepare evidence. We spent a few sessions explaining what we saw, how we arrived at it, what confidence we had in the information.
After that, the rest was out of our hands.
The effects
After this event, I never saw that executive in the office again. They were just there one day and never came back.
I don’t know what specifically happened afterwards. I imagine there was yelling, threats of lawsuits, and even serious conversations of legal exposure.
Whatever happened, it seems that everyone involved found a way forward, likely interested in not making this “a thing”. The executive moved on to a new company, and the other company continued as if nothing had happened.
I heard rumors here and there afterwards that it was done “with permission from leadership” - at least, that was the formal messaging behind why it didn’t seem to be pursued further. I can’t say whether that’s true or not, but in my experience things done with permission don’t often need to be hidden and usually aren’t a surprise to the person who gave permission.
The irony
The ironic thing is - it could’ve been untraceable had any number of things happened.
If the engineer had headed out that evening just a minute or two earlier, we wouldn’t have noticed. The error would’ve been lost in a sea of other random errors as a result of poor error curation.
If the executive had just asked their team for a backup copy, they probably could’ve gotten the full data without any of the scraping that led to them. It wouldn’t have been out of the ordinary to give them copies of certain tables to analyze.
The whole incident was caught due to a bit of luck and overcomplicated execution.
The lessons
This story highlights a lot of things:
Observability and traceability are critically important, and you don’t want to wait until you need them to invest in them.
Automate anomaly detection as much as possible - don’t leave it to chance.
We only had “just enough” logging to figure out what was happening - that was pure luck, and this could’ve gone completely undetected.
Governance controls are important oversight layers especially in high trust environments - that extends to the leadership team.
Evidence and tight chain of custody is required for proof in response to security incidents - a good lawyer can argue anything.
When something strange happens, it’s valuable to really dig into it - we could’ve just chalked it up to a quirk and gone home.
Get written permission if you’re going to do something that looks remotely suspicious - arguing after the fact can get messy.
Integrity and trust are key at all layers of an organization - this includes leadership.
Rational behavior can’t always be expected when the stakes are high
Sometimes you get lucky detecting things - but that shouldn’t be a strategy you rely on