“There’s a weird error on the admin page”

One of the lead engineers for a company I was helping walked over to my desk shortly before the day ended. He had been randomly checking on errors and he saw something suspicious - a series of errors on our internal admin pages for users.

We had never seen the error before - did we accidentally break internal tools relied on by our support team?

Thus started a saga that involved sprinklings that wouldn’t be out of place in a thriller: security breaches, detective work, and sudden leadership exits.

It all started with an error

While I don’t recall the details of the error, what I remember was that it was unusual - it wasn’t the kind of error we normally saw, and it was in a location that rarely broke.

The system was usually noisy, but predictable. Hundreds of the same kinds of error would occur like clockwork- some blocking, some weren’t, but they were all expected to various degrees as the day’s activity ebbed and flowed.

New errors, though - we were alway concerned with new errors. We treated new ones as change failures, and since this was a new error, we started an incident response.

We checked deploys - we hadn’t deployed anything recently to break things. We checked feature flag toggles - nope, nothing.

So, we dug deeper. As the lead and I investigated, we started becoming more and more puzzled.

We saw logs that showed access to page numbers in the triple digits. Our first reaction was very engineer: why was someone on support accessing the user list in the internal admin in such an inefficient way? Most people just searched for who they needed.

As we dug deeper, we started seeing other things that raised our alarm bells - sequential ids were being accessed, the user account didn’t match the name of anyone we knew, even some indirect object reference.

This wasn’t normal user behavior. This was a security incident.

Houston, we have a problem

We immediately started our incident response.

First - we wanted to identify and contain the threat, which was ongoing. After reporting it to the company leadership, we got to work.

The activity turned out to be a scraper that was scraping our internal admin pages that listed user accounts page by page by page. We saw logs with indirect object references, contiguous pagination coming from multiple sources.

We quickly got a list of IP addresses and accounts that were performing this activity. We went over account names, matching them with people we knew on the support team. We narrowed on a list of people we didn’t know or recognize.

We shut down the accounts and blocked access. After we were confident the immediate threat was contained, we moved to expand further to collect evidence and understand the extent of the attack.

We started the work of identifying who had created the accounts, what they had accessed, and when they had gained access. We poured over logs, audit trails, error alerts - anything that would give us information. We jumped back and forth from being glad we had logging in certain places to cursing our past selves for not implementing more robust observability.

Eventually, we found the smoking gun: the IP address of the creator of the accounts. We did a broader trace of activity and found something shocking.

The call is coming from inside the house

What we found was the biggest surprise: the accounts were created by IP addresses associated with one of our executives.

What? Did we accidentally copy-paste the wrong log? Was it a public internet? We double checked, then triple checked, then quadruple checked. Nope, it was accurate.

We cross-compared against everything we could think of. We checked IP addresses from Chat logins, email messages, account logins from when we knew they were in the office, going back months and even years. The IP was consistent. We even geolocated it to their house on days they were in office vs. days they weren’t.

The admin accounts used in the attack were created by the executive’s IP address many days prior, and they were created from an IP address that was associated with private, legitimate traffic. In short - it wasn’t a case of a hacked admin account.

The immediate question on our minds was - why? What was the motive?

It dawned on us - shortly before all this happened, the executive had declared they were stepping down for a new company. It’s possible the information would’ve been useful.

Reporting to the C-Suite

As we packaged up our findings, we notified the C-suite with the latest conclusions and gave them the evidence. The suspicion was that the executive hired contractors to write and run the scraper against our internal pages, and granted access to them to do so.

We documented everything - logs, screenshots, data fields that were accessed.

Afterwards, we secured a forensics team to dig deeper and prepare evidence. We spent a few sessions explaining what we saw, how we arrived at it, what confidence we had in the information.

After that, the rest was out of our hands.

The effects

After this event, I never saw that executive in the office again. They were just there one day and never came back.

I don’t know what specifically happened afterwards. I imagine there was yelling, threats of lawsuits, and even serious conversations of legal exposure.

Whatever happened, it seems that everyone involved found a way forward, likely interested in not making this “a thing”. The executive moved on to a new company, and the other company continued as if nothing had happened.

I heard rumors here and there afterwards that it was done “with permission from leadership” - at least, that was the formal messaging behind why it didn’t seem to be pursued further. I can’t say whether that’s true or not, but in my experience things done with permission don’t often need to be hidden and usually aren’t a surprise to the person who gave permission.

The irony

The ironic thing is - it could’ve been untraceable had any number of things happened.

If the engineer had headed out that evening just a minute or two earlier, we wouldn’t have noticed. The error would’ve been lost in a sea of other random errors as a result of poor error curation.

If the executive had just asked their team for a backup copy, they probably could’ve gotten the full data without any of the scraping that led to them. It wouldn’t have been out of the ordinary to give them copies of certain tables to analyze.

The whole incident was caught due to a bit of luck and overcomplicated execution.

The lessons

This story highlights a lot of things:

• Observability and traceability are critically important, and you don’t want to wait until you need them to invest in them.

• Automate anomaly detection as much as possible - don’t leave it to chance.

• We only had “just enough” logging to figure out what was happening - that was pure luck, and this could’ve gone completely undetected.

• Governance controls are important oversight layers especially in high trust environments - that extends to the leadership team.

• Evidence and tight chain of custody is required for proof in response to security incidents - a good lawyer can argue anything.

• When something strange happens, it’s valuable to really dig into it - we could’ve just chalked it up to a quirk and gone home.

• Get written permission if you’re going to do something that looks remotely suspicious - arguing after the fact can get messy.

• Integrity and trust are key at all layers of an organization - this includes leadership.

• Rational behavior can’t always be expected when the stakes are high

• Sometimes you get lucky detecting things - but that shouldn’t be a strategy you rely on

A while ago I was given the responsibility of improving the Product Management organization at a startup. It was a Series B company with about 200 people, of which ~50 people were in the development organization. The group of ~10 Product Managers had their leadership recently depart, and they needed someone to step in to help guide them through that transition.

I knew some of the challenges and opportunity areas from the engineering perspective first-hand and through constant feedback from the engineering team.

• There was never enough “ready” work to do. The Product Management organization had a difficult time defining and prioritizing work. As a result, engineers went underutilized for weeks at a time, working on tiny things they found themselves here and there with little value.

• The requirements that made it to engineering were ill-defined. The work hadn’t been done to define the boundaries of the problem space or even what the end-state was supposed to be, leaving engineers to fill in massive gaps in requirements.

• The “Why?” wasn’t defined or answered, let alone asked. The value, the problem it was solving, and how that item mapped to a larger vision or picture wasn’t defined for the development teams.

• Cross-feature interaction analysis and product capability management was non-existent. Work items were done in silos without consideration to how they interacted with existing features, which led to duplicative functionality, or different functionality for the same capability. This lead to large numbers of gaps and quirks in the product overall that reduced quality and led to difficult defects - not due to implementation errors, but due to thought not being put into the intended change itself.

• Explicit functional boundaries prevented gap filling. The organization was afraid to step on toes or interfere with another function’s work, so boundaries formed, leading to decreased collaboration. Of course, because the work that did get handed off had issues, it meant an ever-increasing burden on Engineering to fill in all the gaps as the last function in the value stream capable of fixing issues.

These challenges were at best slowing down delivery of everything, and at worst completely stalling efforts to deliver value.

Identifying the issues

When I came on board to the Product organization, I had my own hypothesis and opinions of what the issues were, but I did my due diligence:

• I read through chat histories, meeting notes, and other documents to understand how decisions were being made

• I had 1:1s with every product manager and adjacent functions like Marketing, Privacy, Customer Success, and Solutions that interacted with Product to better understand the issues and challenges from their perspectives.

• I mapped out the relationships of all of the issues and their causes, influencers, and effects to better understand the first, second, and third effects.

• I conducted surveys and other analysis of sentiment.

From that, I saw some clear root issues that were creating a complex web of spiraling - a vicious cycle of negative feedback that reinforced itself the longer it went on.

• The Product Managers didn’t have a true understanding of the product or user.

  • Without understanding how the product truly worked or what the user truly needed, the Product Managers were afraid to make decisions because they couldn’t figure out the effect of it. When they did make a decision, it often broke some user workflow or had some issue they didn’t consider because of their lack of comprehensive, true understanding of the user, their journey, and their work. It was like the blind leading the blind.

  • There wasn’t any formalized mechanisms for the organization to get and leverage feedback. Certain members occasionally did interviews, or looked at a competitor, or read an article, but very little was baked into any formal feedback loop. Whatever knowledge we did obtain was silo’d and lost in random slack messages instead of becoming a concrete learning that could be used in the future to make decisions.

• The Product Managers wanted to be 100% correct, all the time.

  • Being wrong was viewed as a terrible thing. In fact, most discussions revolved around how to not be wrong, which naturally led to doing further analysis, measurement, and thinking to avoid being wrong. The quantification of the negative impact of being wrong wasn’t even being considered. Being wrong was avoided 100%, even for the smallest negative impact. Some of this was due to individual Product Manager personalities, and others were due to fear of punitive measures if something went wrong.

• The Product Managers over-indexed on working on the highest value item.

  • Not just value, but specifically the highest value. This meant a lot of discussion, argument, and analysis to intimately understand the exact value of all items being considered to eventually be able to decide which item specifically had the most value. This, of course, increased decision-delay.

• The Product Managers wanted everyone to agree before they decided anything.

  • The Product Management culture was one of consensus - full alignment and agreement was required before anything moved forward. Because nobody wanted to be the one that was wrong, even the smallest disagreement or concern raised derailed a plan to move forward. Few people made decisions, instead opting to have another meeting or touchpoint to get further alignment and clarity. This meeting, of course, was scheduled for a week or two after, which added wait time to even the smallest discussions.

  • If disagreement persisted, even by one person and even if it was an otherwise minor addressable element, it meant that the item was disregarded. While this was fine if it got replaced by another valuable item, the issue was it got replaced with nothing, which further exasperated the lack of delivery of anything.

• The Product Managers cared too much about optics.

  • There was a massive pressure to concretely measure the specific impact on a particular metric with exacting precision. The go-to method was the A/B test. Product Managers were pushing for every single change to be A/B tested. Despite this, features lacked basic instrumentation because Product Managers generally handed off those responsibilities to the Data Science team.

  • This of course, meant not only the setup of the A/B test adding time to development efforts, but also monitoring and running and interpreting results. Because the Product Managers as a group weren’t particularly organized or familiar with the data side, the A/B test interpretation fell almost entirely onto the data science function. Because a proper A/B test requires a stable environment, this also meant follow-up iterations were effectively blocked until the A/B test was completed.

In short - analysis paralysis ruled the day. All of these specific issues fed into each other like a vicious cycle. Because of a lack of understanding, product managers made mistakes. This made them more scared to make future decisions, so they tried to analyze more and measure more to prove they did a good job. Because they tried to analyze more before they made any decision, they slowed down their decision making. This increased pressure on the product managers to make sure whatever decision they did make had high value, which further led to more analysis and delays. This decreased confidence so they tried to get all of their stakeholders and peers to agree on every aspect of a change before moving forward, which meant valuable items with minor concerns got disregarded, further leading to slower decisions.

I knew I needed to break this cycle.

The Gordion Knot of Analysis Paralysis

The Gordion Knot is a legend about a complicated knot.

The belief was that whoever untied that knot would go on to rule all of Asia. Alexander the Great came along to untie it. Instead of painstakingly and slowly untying the knot as expected, he just cut it in into pieces with his sword.

It’s a metaphor for intractable problems and the solution spaces that are possible if you don’t make yourself beholden to perceived constraints.

I thought hard about the situation and the culture. I spent several weeks diving deep into the problem space that the organization found itself in.

Fear was the root cause of a lot of these issues. Fear of being wrong. Fear of looking bad. Fear of having others disagree. Fear of being uncertain. Fear of making mistakes. Fear of punishment. The only way to remove the fear is to remove the disincentives.

I created a bold and, frankly, insane, 120-day plan. One that I kicked off at the next Product Management all-hands.

Setting the tone

That Monday, I said clearly and in no uncertain terms to every Product Manager in the company:

I don’t care about the value. The value doesn’t matter. Stop considering it. Stop thinking about it.

I could see the wheels turning in all the product manager’s heads. Their whole job is oriented around delivering value. If value doesn’t matter, what does that mean for them and their careers?

I can see the wheels turning in your heads, too. I can hear the cries of “blasphemy” from all of you readers who are product managers that have been trained on the belief that the highest value is the only thing that matters.

But the fact is, it’s not.

The whole point of a Product Management is to deliver value. The key first word in that is to be able to deliver. It’s not enough to talk about value, or compare one value to another, or know what’s valuable in your head. All of the purpose of the role is predicated on the first word - that value is being delivered.

The team’s problem was that in their pursuit of the value, they forgot how to deliver. The muscle of intentionally and sufficiently defining a change, crafting a narrative, influencing others, working with a development team, and putting it out into the world, and then seeing what the effect was had atrophied. Habits of fear and talking replaced it. Under that fog of fear, they went through the motions of product management without fulfilling its spirit. They got slower because they didn’t know how to act without full knowledge. They got slower because getting full knowledge required delivering, which they didn’t know how to do.

It was lik trying to turn a car into tho right direction without moving either forward or backwards. We couldn’t even get out of the parking lot to know whether we were going in the right direction.

We needed to build up the delivery muscle, and to do that, we needed to go through the reps of delivering something - regardless of the value.

Breaking the desire for full confidence

Some of the Product Managers still weren’t sure. They were still fearful of changes they made. It might be fine to ignore value, but what if it has a negative impact?

My response was simple : who cares? If it has a negative impact, we can see it on the basic instrumentation and roll it back. Changes were reversible. Software is “soft”ware for a reason.

Some weren’t convinced.

“Well, that’s not precise enough”. They wanted to A/B test everything to be absolutely sure it wouldn’t harm a company outcome.

I told them that with change comes risk, and with risk comes the possibility of failure. Trying to measure every single step you take to see if it’s taking you in the right direction isn’t due diligence, it’s fear. There’s no place for fearful product management in a startup.

The fact is, at our company, A/B testing had become a crutch to hide a lack of decision-making.

So, I decided to remove that crutch.

I banned A/B tests.

The right tool for the wrong job

That’s not to say A/B tests aren’t useful. They are, in the right context. The situation our company in was absolutely the wrong context. The desire of the product managers to test everything was getting in the way of making progress.

There were proposals to A/B test even the smallest of changes - changing the text of a navigation bar link, adding a new filter to a dashboard list, or adding instructional text to a search bar. There was even a proposal to A/B test the error message of a donation!

Little thought was put into what the risk profile of these changes actually looked like. Nobody even considered whether that profile warranted needing the precision confidence of an A/B test to make a decision on outcome. That there were other cheaper, perfectly appropriate mechanisms to obtain outcome confidence didn’t seem to resonate.

The proponents of “A/B test everything” also didn’t put much weight into the fact we wanted to iterate rapidly on our product, and an A/B test would prevent rapid follow-up iterations to the exact areas we wanted to change as a company because we would need to wait weeks and months for an appropriately large sample size.

I stuck to my stance.

Drawing decision boundaries

The third thing I did was to ensure that the Product Managers didn’t get interfered with. One of the issues was that they got a lot of feedback from others in the company - folks that were there longer or whom were viewed as “subject matter experts” of a particular area or system that argued about the smallest detail or approach. Many arguments ended in the same way - the Product Manager seeing all of the possible problems, challenges, and issues and simply concluding that it wasn’t worth it to continue.

Every single thing we do has reasons for not doing them. Some are very good reasons. Some are bad reasons. They are all there, all the time. If we overpay attention to reasons we shouldn’t do something, we’ll just get stuck not doing anything at all. Instead, we should look for good reasons to do something, and treat all the reasons not to do something as the addressable, definable, and sometimes ignorable risks they are.

I needed to break the product managers free of the well-intentioned influence from others.

To do that, I took a page out of my autocratic dictator playbook. I told them that the only person who’s opinion mattered was my own, and that they should ignore any disagreements from anyone. If I said it was to be done, that was all that mattered. People who argued we hadn’t validated it, that it had risks, that it wasn’t aligned: none of that mattered except my opinion for them to do it.

Listening to the users

If the Product Managers weren’t doing analysis and discovery, then what would they work on? What would get prioritized? How would we determine what we worked on?

Simple: the users.

For years we had multiple mechanisms for collecting user feedback - annual surveys, in-product ratings, support tickets. While some analysis was done and an occasional item or two pulled from them, these sources of user feedback were almost entirely ignored.

It had gotten to the point that our users giving us feedback were answering our questions with statements like “why do we even bother answering this if you never change the things we bring up”. They were tired of making the same complaints, year over year over year and seeing nothing done.

I didn’t want the Product Organization to waste all their time trying to identify possible new problem spaces. With the surveys, we had problem spaces in front of us we could immediately address. Sure, it might not be the most valuable, but I didn’t care about the value. What use was identifying the most valuable if we had not built up the ability to act on it afterwards? That would be like hearing without listening.

I went through the thousands of answers on the survey and, with the product managers, manually created a list of feedback areas that I called “Pain Points” - the problem areas of our product that people complained about repeatedly.

Many of these were explicitly self-evident solutions, while some contained more ambiguity and nuance:

• Alphabetize the dashboard record list

• Add a new filter for date

• Add <X> information to the record

• Save notes properly

• Calculating totals

• Ensure we get paid for our work

• Improve account security

On that list of pain points, I put a “reason for doing” column. I chose 10 pain points - problems I felt had simple solutions with outcomes that would be meaningful to someone and little risk of being harmful to anyone. Next to those 10, I put under “reason for doing” - “Joseph”.

I told the team if an item had my name on it, it was a “do it”, regardless of any objections they may encounter from anyone. If anyone complained, argued, or pushed back, they should redirect that person to me.

My goal was simple: I wanted to see a completely different list of problems the next time the survey was run.

Being talked out of delivering value

There were still extensive discussions happening in Teams and tickets as to why the product managers shouldn’t do the items on my list:

• Did we understand the problem space?

• What’s the real problem being solved?

• Why can’t they just use the information or features we already have?

• Are we going to A/B test it?

• What if it takes too long?

• Why aren’t we going to user-test it?

These were happening for even the smallest changes. Product Managers were being talked out of doing work based on even the most insignificant complaints. For example - adding a new label would decrease the screen real estate by 16 pixels, or that adding a specific filter in the product was duplicative because a user could just go and export their data and filter their spreadsheet in Excel.

Reasons for not moving forward. Reasons for not acting. All reasons without risk-adjusted consideration.

I needed to create a culture where people were OK with delivering something that didn’t solve the problem fully, if at all. I needed to create a culture of build-measure-learn, which mean being OK with getting it wrong the first few times. I needed people to understand that sometimes, you just have to do something to see the effects of it.

Very few of these complaints were one-way doors: that is, items that had negative consequences that were irreversible or massively damaging. Most issues were easily adjusted or minor. Minor issues shouldn’t stop progress.

This was symptomatic of the exact problem we were having before. We would spend weeks talking about a half dozen possibilities to figure out which one to do. In that time, we could’ve successfully delivered all of them and then figured out the right one based on real results. It was all talk, no walk.

I re-iterated that they should ignore any objection and to proceed with solving the problem at hand. I explicitly told them to ignore every single person complaining or proposing alternatives.

Did that reduce autonomy and collaboration from the organization? Yes, it did. But the team had shown that they couldn’t move forward with the level of autonomy that they had, and that the level of collaboration was no longer productive.

It’s perfectly fine to reduce positive elements to appropriate levels if they start having negative impacts.

Supporting rapid solutions

With these adjustments, the Product Managers were able to bypass the “discovery” and “prioritization” steps that had been plaguing their slow speed and finally moved towards “defining” and “scoping” the solution space.

It immediately became clear solution definition skillsets had weakened greatly. The initial scope for many of the proposed requirements for solutions to even the simplest pain points contained weeks or months of development effort that required extensive analysis, discovery, and feedback. The Product Manager were putting the cart before the horse - they had been trying to define massive solutions to complex problems before they had even learned how to define and solve the smallest of problems.

I told the team to think about not just solving the problem in full, but to constrain their answers to different time-boxes.

• The 1 hour solution

• The 1 day solution

• The 1 week solution

• The 1 month solution

• The 1 quarter solution

These were solutions that could be implemented if they had a time limit to implement that solution.

I prohibited them from pursuing the 1 week solution unless they had first delivered to production the 1-hour solution.

This constraint was hard for them to think about first. They just couldn’t visualize how to solve a problem partially. I had to guide them to think about low-hanging things, and to think about solutions in stages of progressing difficulty and investment and comprehensiveness in solving the problem:

• Informing the user or making it clearer

  • eg. adding instructional text, clarifying terminology, writing an out-of-product FAQ

• Doing it manually internally

  • eg. taking on a request via email and having an employee do it ad-hoc, creating a meeting to do it once a week for all requestors, etc.

• Providing a way for the user to do it manually

  • eg. adding a basic form to collect information, or giving them an action or button

• Doing it automatically

  • eg. doing something in bulk, on schedule, or removing all the steps

• Removing the need for it completely

  • eg. removing the business process from having to exist

Sure, the best solution might be to remove the need for a user to do something completely, but sometimes just telling the user how to do it correctly solves 80% of the problem.

Did we solve the pain?

The important part about all these changes was that without having a tight feedback loop with our users, we would never be able to tell what effect we were having. It would backfire if we did a bunch of stuff that did not do anything for our users or the company.

We still needed feedback on the decisions we were making and the actions we were taking. We just wanted to not be beholden to it for every step we took.

To do that, I worked to operationalize feedback - to bake in leveraging quantitative and qualitative insights as part of the entire organization’s work.

I worked with the Product Management group to set up:

• Insights groups - groups of users we could talk to every week to get feedback, ask questions, and learn about their days

• Raised awareness for feedback - I had in-product feedback sent directly to a Slack channel, allowing us to discuss in real-time as comments came in and also personally amplified feedback we received by sharing it broadly with others and following up on it

• Mandatory relationships - I instructed all the Product Managers to directly email specific users and build feedback relationships with them

• User interviews - I placed more emphasis on talking to the users, incuding interviewing them, and sharing notes and learnings in a regular, organized manner

• Feedback intent - I created cycles of intentional times where we discussed feedback from users specifically as part of our day-to-day

• Instrumentation - I emphasized having product analytics and instrumentation implemented, and knowing what questions we wanted to be able to answer

• Relationships with support - I emphasized the need to work closely with support and meet with them regularly, monitor tickets they received, and leverage their knowledge as their front-line experience with our users was valuable

With these, we created firehoses of feedback into the Product organization, and set the expectation that we acted on it in a timely manner.

I especially emphasized qualitative and in some cases anecdotal feedback to ensure we moved past the paralysis caused by concerns of having the maximum quantitative impact.

The changes have massive effects

It was a tremendous 180-degree turnaround.

The Product Management team had gone from taking months and quarters to do a 1-day project because they were so busy thinking about value, to being able to respond to user feedback and deliver something within that same day. We solved dozens of identified user pain points in a short time, some of which led to deeper learnings about their causes and unlocked new valuable areas to explore.

The speed of decisions, the scoping of solutions, the responsiveness to user feedback, and the intentionality of instrumentation and monitoring all worked together to give us more opportunities “at bat” - more chances to get it wrong so we can get feedback and act to get it right.

The Product Managers had successfully done the reps to build up their delivery muscle, and it was showing at the rate we were delivering working, quality solutions to our users.

We had gone from delivering nothing to becoming a fairly efficient feature factory that delivered many things, some of which had value, but with the ability to course-correct based on feedback.

The effect was that we delivered more in a single month than we had in two quarters the prior year. Some of the delivered items were massive wins that saw immediate adoption and usage. Others flopped and saw no usage, but had no negative effects. All led to learnings and better understanding of the mental models and user journeys our product fit into. All were cheap bets that cost little investment, and were delivered incrementally and iteratively.

We would not have delivered on even half the wins had we been operating under the prior rules.

Knowing when to break the rules

Although unintuitive, ignoring value didn’t harm us given our context. Instead, it was the key to establishing a virtuous cycle and getting out of our death spiral. It broke through the fear of failure and opened up the possibility of learning from mistakes.

Focusing on rapid delivery meant we got better at getting real feedback and insights earlier. Rapid follow-through meant we incorporated that better into our future decisions. An operations process that tightly monitored for and fed new information into decision-making processes made it a virtuous cycle to make mistakes, get things in front of users, and learn from them. These built up capabilities and competencies we were lacking in the organization.

It resulted in faster validation of risks, and richer information than we could’ve gotten from all the interviews and analysis in the world. You can’t predict with 100% precision how users will interact and behave. Some problems you can’t analyze your way out of - not all problems are complicated: some exist in complex-adaptive systems where action is required.

The speed meant having more opportunities to have an effect. This ended up meaning we experienced more wins than if we had tried to avoid the failures.

If we made 25 attempts and failed 20 of those times, we would still be better off than if we had tried only 3 times and had all 3 win. More “at bats” meant more chances to win and learn and adjust course.

It also meant we got better at defining and shepherding solutions and delivering. There’s a study floating around about how making a thousand pots as fast as you can without regard to quality leads to you getting better at making pots than if you had focused on making high quality pots. The exact thing happened here - doing the reps let the Product Managers get better at the work.

That didn’t mean we ignored outcomes completely. We still built a culture of instrumentation and measuring and making sure we could observe the effects. We still built a culture of listening to users, building relationships and feedback loops, and sharing knowledge. We still built a culture of defining and validating risks.

We just decided that value was not to be a driving factor in our decisions for a certain time frame. It worked.

Re-orienting towards value

Ignoring value wasn’t where we wanted to be in an ideal world, but it was great progress relative to doing nothing. We don’t need a team of 50 to do nothing.

Now that we had the means to actually act on and learn from our actions, we could then shift the organization towards increasing diligence around value without losing our delivery effectiveness.

It was time to take the journey towards Value once again - this time without the baggage of analysis paralysis.

As I let go of the steering wheel, I started orienting Product Managers to think more deeply and strategically, giving back some of the autonomy I had taken away.

How I did that, however, is a War Story for another time.

A sudden flurry of Slack notifications appear on the top right of my screen: three new, separate DMs, surprisingly from the President, CEO, CTO, and CMO. I knew something blew up.

Given the absence of other blaring alarms, I knew the system wasn’t down.

What was?

An unexpected crisis

All four messages were about the exact same thing.

One of my reports, a newly minted engineering manager, had posted a message on his team channel that had suddenly ruffled a lot of executive feathers.

The message itself was simple: “We have unexpectedly received new goals from leadership and will need to reprioritize.”

It was viewed as extremely passive aggressive given the context of the company. 

What was the context?

My report had previously communicated feedback up of the difficulty caused by frequently changing goals. 

The leadership team had listened, and had been working feverishly on implementing a new, fair, and more consistent goaling system. It would take time because such a change is always fraught with everything from board dynamics to HR considerations to projection challenges and everything in-between. It was a heavy topic with many vigorous debates behind the scenes, but it was something actively being worked on.

The problem was that my report had also communicated feedback down, around, and across — pretty much to anyone an everyone who would listen. He wore his heart on his sleeve, and he oozed frustration in every conversation. With coaching it had decreased some, but old habits die hard.

The demands

Aligned by this particular problem, the entire C-Suite had the same demand: that I immediately correct the engineering manager for this passive aggressive message.

I had one problem: I can’t just tell my report he’s the new target of the C-Suite. It would change how he viewed his relationships with others and key partners, and make him afraid of posting and communicating. It would’ve made him less likely to trust others, when it’s such a large component of his effectiveness in the role. It would also make him less likely to give feedback upwards from a team that I needed to hear it from. 

In short — it would have created a cycle of destructive fear.

What did I do?

I took the blame.

I told the C-Suite: hey, I’m working with the report on communication techniques. It’s an area that we expect to improve over time and I’ve seen good progress — we’ll need to expect unintended mistakes happen as we refine and hone this skill and I take responsibility for the miscommunication here.

To my report — I coached him. as if I came up with the message and that it was me who had issue with it. I essentially gave that C-suite feedback as if it was my own.

I had a quick chat with the report:

“I read that message and I can see it being interpreted by others in a way that you may not intended. I recommend adjusting it to be more neutral or matter-of-fact that the goals changed — ie. without any particular charge, to avoid alienating any stakeholders. We want to cultivate a culture where change is welcome, and embraced as an opportunity to improve.”

I didn’t even mention the C-Suite.

My report took the feedback immediately as he typically did — “we will be reprioritizing our efforts to align with our new goals.” The C-Suite was satisfied — the newer message was perceived as less passive aggressive.

The crisis subsided. The C-Suite moved on to other issues and I was able to give my report some useful advice that he implemented moving forward.

3 months later

My report gave me a negative mark on the upward performance review, citing that I had “inappropriately micromanaged communication”.

My boss gave me negative remark on my own performance review, citing that I “needed to work more closely with reports on effective messaging.”

Sometimes you just have to take the “L” to do what’s best for the company. 

I thanked them for the feedback and carried on.

You don’t always know what you’re getting yourself into when you accept a job.

This is the story of my first two months at a startup I worked with — let’s call it Transaction Software, Inc. In short, it was a brutal mess.

I’ve written my recollections here. While it’s now been some time since the events occurred, much of the story here is based on notes and logs I had kept as the situation was unfolding.

I’ve changed the names to protect everyone involved. As a disclaimer, everything I write here is from my perspective, my understanding, and entirely my opinion only. To be completely fair, I may be missing pieces of the puzzle that could change my perspective.

Ready for my nightmare? Let’s begin…

The beginning of the strange journey

Elliot, the CEO of Transaction Software, Inc., was looking for a senior front-end developer. He somehow found me on a job board and reached out to me.

During my interview with the CTO, Bob. I dug into the company and its history. I found out about the team , the technology, the problem space, etc.

The engineering team consisted of 2 junior front-end developers and Bob. There were multiple projects in the works, but Bob and the engineering team were currently focused on an effort to upgrade an existing transaction platform written in Ruby on Rails by a previous team and give it a facelift. The projects were already late and running up against some upcoming final deadlines.

The new front-end was written in Ionic 2 / Angular 2, and my first task would be to wire it to the Ruby on Rails system, which I was told had fully-functioning JSON APIs. I would be leading the front-end development efforts as a senior front-end / full-stack developer, guiding the junior developers.

I believed that I had found a good place to work: the technologies were recent, the domain was interesting, and I believed Bob would be someone who could mentor me.

I was wrong.

The truth hits like a freight train

From the very first day it was clear that the amount of progress had been…misrepresented. There was still quite a long ways to go. The Ionic 2 / Angular 2 application I was hired to wire up to the backend turned out to be almost entirely a mock. Much of the data was entirely hard-coded and flows through the pages scripted in a smoke-and-mirrors way. I wasn’t just going to be wiring up the apps, I was essentially going to have to redesign and develop most of the pages and flow from the ground up.

The code quality of the front-ends themselves was very poor. There were 3000 line SCSS files copy-pasted throughout the app. There were repeated functions everywhere. The naming was inconsistent. There were no patterns or libraries. It was messy and unclear.

Questioning authority

The products we were building were intended to be used almost exclusively on the desktop, but for some reason we were building them using mobile technologies. I’m not one to pass up an opportunity to learn and get a “wide lens” of a project, so I asked Bob what the business reasoning behind using Ionic 2 was. Bob replied that he was running an “Ionic 2 / Angular 2 shop”, and that he didn’t want the other engineers confused when they also had to learn how to use Angular 2.

I pressed further —Bob’s reason didn’t make sense to me. Having to learn an entirely new mobile framework on top of an entirely new JS framework is a lot harder than having to just learn just Angular 2. Even the junior developers agreed — they had been struggling with Ionic 2 for quite some time and often spent significant amounts of time trying to do basic things with the framework.

Bob stated that we were using Angular 2 / Ionic 2 was there because the users would be using it on their phones. By creating an app we could provide offline queuing capabilities for the system in case connectivity was lost and that as a hybrid app it would prevent us from having to create both an iOS and an Android version.

Aha! A legitimate business reason…or so it seemed. I thought about the audience and the kind of system we were building — the business case of making it mobile just didn’t add up.

Our system needed always-on online connectivity to verify the validity of transactions and to provide immediate feedback on the result. Ours was not the kind of transaction we could queue and wait several hours to complete. The user had to know at the time of the transaction what the result was. There was no valid business case I could think of for offline queuing capabilities.

Our audience was also not primarily mobile app users. They were users at their desks on large-screen always-online computers, infrequent users, or users visiting the site for the first time and never visiting it again after completing their transaction. A vast majority of them would not be taking the time to download a mobile app to use our system.

So why were we making a mobile app?

The mobile focus leads to technical issues

Bob’s focus on creating a mobile app also influenced the architecture. The app was designed with the intention to load every transaction record on sign-in and store it in memory for offline support — searching, querying, reporting, calculations, etc. This data, along with all the other data in the app, was stored in a singleton reference that was then scattered throughout the entire codebase.

The problem —there‘s several, actually!

Problem 1 —the design was a giant ball of spaghetti

The first problem was the whole design was the question: how should we persist the transactions in an offline-queue capable way like Bob wanted? If we did pursue this direction, we would need some sort of mechanism to sync it sensibly — to keep track of changes, updates, successful and unsuccessful requests, errors, etc. To do so while dealing with a singleton modified and read from in every single file would be a nightmare.

I asked Bob what the plan was to sync and persist the data in a sensible way. He responded with the names of technologies like NoSQL and PouchDB in response without explaining anything beyond the name. I was confused — we were using Ruby on Rails and MySQL and had no intention to move away from a relational database — how did NoSQL or PouchDB factor into this at all?

Problem 2 — incredibly slow performance built into the design

The second problem was one of performance. Many users would easily generate over 70 megabytes of data a day! It was far too much to load even a week’s worth! Even with only a single day’s worth of test data, it took a good 10 seconds to sign-in! No sane user would wait that long.

I asked Bob about the performance issue and he responded that it “shouldn’t be a problem”. A reply like that was shocking — users nowadays barely wait 2 seconds to use a website, why would they wait 2 minutes? I even presented him with the math and actual numbers involved — he remained unconvinced. It was only later that he admitted that this was a problem, but he noted that he would solve this using Angular Universal. I couldn’t see how that was relevant — the data would still have to reach the client-side somehow.

The realization

It dawned on me that there was no business reason for using the technologies Bob chose. Even worse, our technology choice was actually going to become an issue down the line in accomplishing our business objectives — users simply wouldn’t use our system in the manner we were building it. We were designing and building a mobile app for something that was rarely, if ever, going to be used as a mobile app.

If the mobile side doesn’t work, at least the web side should, right? Wrong.

As the web side of the hybrid application, the picture was even worse. Ironically, for a project that took “mobile-first” quite literally, the web side was not responsive.

That wasn’t even the worst offender. One of our core functionality requirements was the ability for users to share their various pages across social media. However, in the SPAs this just wasn’t possible — any URL change would lead to a completely blank slate and the front-end would lose all state.

At the time, Ionic 2 didn’t support URL routing, and the Angular 2 router was undergoing frequent core changes. As a result, URL navigation didn’t route correctly. The browser navigation buttons (Back, Forward, etc.) weren’t supported — they would reload the application, as would any URL change.

Looking deeper into the codebase, I learned that all navigation was being done by programmatically and without history support. A link click would trigger a function that would crawl the DOM and look for a reference to a master Tabs element. Once found, the it would look up the current page’s tab index and add or subtract the number of tabs to move from the current tab depending on what page the link was supposed to navigate to.

In short, it was brittle, highly unclear, and not a good solution by any means.

No solutions — again

I asked Bob about a potential solution or design direction for this issue. Once again, he threw out the name of a technology without going into detail, claiming that he would use NGINX to solve it.

I didn’t understand how that would solve the client-side routing problem — having a manifest of the possible pages in NGINX was not a viable solution, and even if you made NGINX interpret the URL appropriately, the SPA would still needed to interpret the URL on the client-side and statefully keep track of it.

I asked for clarification, but Bob didn’t articulate his design approach.

He later produced a “prototype”, stating it had taken him a mere 15 minutes of development work to figure out:

It solved the issue of being unable to load a specific page on first load, but the client-side routing and history support remained unaddressed. To use Bob’s approach would have required us to write a custom client-side routing solution from the ground up. The resulting system would have been brittle, fragile, and highly unmaintainable — it would be re-inventing the wheel in the worst way possible. Since we were already months behind schedule, I believed it would’ve been a waste of the company’s time to do so, and certainly not a viable solution for any long-term use.

Those were just some of the technical issues that were plaguing the company. I wish I could say that the problems the company faced were purely technical, but I’d be lying if I did.

Toxicity

Bob had a way of managing and communicating that I would describe as passive-aggressive and controlling at its best. His management style didn’t go unnoticed — when he walked into the office (a less than 200 square foot closet), I noticed the junior developers would clam up and go quiet.

I spoke to both of the junior developers in private (they were very friendly and personable) and learned their true feelings towards the situation. Needless to say, it had to change quickly — everyone seemed to walk on eggshells whenever Bob was around.

I’m a stickler for effective team communication, so I set up Slack as a tool to give the team a communication outlet. This way, communication was still encouraged even if the atmosphere in the room itself proved too tense for the junior engineers to speak their minds.

Power games

Elliot gave me access to the Github repository for the backend to better acquaint myself with the product. Bob, who was in the office at the time, immediately demanded I be removed. When Elliot asked why, Bob stated that I hadn’t proven my velocity on the front-end code to be given access to the back-end code. Elliot reluctantly removed me, later asking if that reaction from Bob was a “power thing”.

No other engineer had access to any of the back-end code. All work was done by pointing our local front-ends to the URL of a shared staging environment. Bob’s reasoning was that he didn’t want “people’s PCs cluttered and confused with all the brittle legacy Rails back-end stuff”. He told me to base my attempts to communicate with the back-end on the front-end schema of the singleton object he had created (which, as I wrote previously, turned out to be wildly inaccurate on multiple levels).

This was a very strange, ineffective setup — even Elliot realized this, and he’s not even a technical person! Not having any documentation or visibility on how the APIs worked would make my job of actually integrating the front-ends and the back-end very difficult.

Making progress any way I can

Since I didn’t have much to work with, I decided working with the junior engineers would be mutually beneficial. I’d be able to help them understand better practices and the technology when I could, and they would be able to help me understand the various quirks and the general problem domain.

I was new to Angular 2, but the weekend I spent ramping up was more than enough to start contributing. Pairing helped us both, and we were even able to get past some issues they were stuck on.

During the time I was mentoring the juniors, I implemented a code review process through pull requests as a way to share knowledge and improve quality. One of my tenets of code reviews is to help guide people and teach them to arrive at the solution instead of giving the solution outright — it helps them grow as developers and ultimately leads to better decision making and improved code quality.

Bob didn’t seem to share my sentiments.

Threatened

Later that day I received an email from Bob stating that while my mentorship was valuable, I was hired to code and that my continuing employment would be based on output from my desk. Bob stated that Transaction Software, Inc. was “still in the process of evaluating [my] performance as a senior-level software developer”. Read in his voice, I found it vaguely threatening.

I’ve never been threatened before with losing my job in my first three days of working. Concerned, I asked Elliot the CEO (who was CC’d on the email) about this, and he assured me my job was safe. I asked how I could possibly stay at a company if the CTO had already determined in the first few days that I’m not productive enough? Elliot finally gave me the rundown.

The org-chart is a mess

It turns out Bob was actually a contractor, and not the CTO as he had told me! Even more surprising: Transaction Software, Inc. didn’t actually have an engineering team at all! Elliot had selected the junior developers to hire, but Bob had convinced Elliot to hire them under Bob’s own consulting company, for “insurance purposes”.

Elliot paid Bob’s consulting firm extra to pay for the junior developers. In effect, Bob and the junior engineers were a consulting company working for Transaction Software. It was confusing — I was technically Transaction Software, Inc.’s first engineering hire.

Bob was previously attempting to negotiate a CTO position or an acquisition of his consulting firm by Transaction Software, Inc. I assume keeping everything in-house and under one’s control puts one in a better negotiating position for a buyout or acquisition, especially if a company relies on your firm exclusively for their technology needs.

Additional meetings with Elliot gave me a clearer understanding of what was going on. Elliot had started wondering why the projects were being delayed repeatedly. One key project in particular was already months in the making, and Bob had told Elliot time and time again that production delivery was “only two weeks away”. The front-ends had been delayed time and time again, and the clients for that project were already highly agitated and losing faith.

Elliot was paying Bob tremendous amounts of money and felt he was getting very little in return. I was hired, in part, to shine a light on what was going on — were all these problems a normal part of software development?

At the time I still felt the best course of action was to work things out with Bob to make positive changes gradually. Bob hadn’t demonstrated to me that he was experienced at leading or working with teams. He hadn’t yet shown me that he was particularly knowledgable about the principles of developing large web applications.

As time went on I continued what I was doing, replying politely and professionally to Bob’s various remarks and tried to set a good example for the team. However, I believed blindly following Bob would lead to the failure of the projects and the organization in a very short amount of time. I couldn’t do that — I had a responsibility to the company.

Turning the ship around

I spoke a lot with Elliot, explaining from my perspective the various technical, team, and cultural issues I was discovering. I proposed a new approach that would get the product up and running on a much faster and more aggressive timeline, in the hopes of meeting the deadline for a critical stakeholder demo.

I designed and diagrammed a workable architecture for the various front-ends. Instead of the giant singleton ball of spaghetti, I created a linear hierarchy of pages and components. I created a schedule that we could realistically achieve.

Elliot and I agreed to deal with Bob with “kid gloves”, tolerating any outbursts and passive-aggressive behavior until we were in a better position to make a decision regarding him.

A New Hope

I presented my plans to Bob and Elliot on Monday, explaining that with some focused engineering effort, the system could be done in less than two weeks of development effort in Angular 1 if the endpoints were in the state Bob said they were in. I also presented alternative approaches incorporating Angular 2 if needed, although I clarified that this would greatly extend the actual timeline.

Bob was visibly upset. We had a very long debate, with him making points that writing the code in Angular 1 would be generating untold amounts of technical debt because it was a deprecated technology. I agreed that Angular 1 was on its way out, but the Angular 2 / Ionic 2 combo he had built the product with just wasn’t ready for our use.

I argued that Angular 1 was stable, the problems were well-known and the solutions readily available. I greatly preferred the “devil you know” approach to risk management. I reasoned that with months of development time already wasted on the Angular 2 / Ionic 2 solution with no real progress, with more problems arising every day, and with deadlines far missed, it just made more sense to start from scratch and build it in Angular 1 in 2 weeks.

Bob argued that he had the solutions to the problems we were encountering. I begged him to tell me what the solutions were, or at the very least to point me in the right direction so that I at least knew he had a viable way across the river, metaphorically speaking. If there was a way forward, I would pursue it to the end. He didn’t articulate his direction and vision, only mentioning the names of various technologies.

Elliot stepped up and made an executive decision to go with my 2-week plan of writing the front-end in Angular 1. If we didn’t succeed, at worst we would waste 2 weeks, and what’s 2 weeks wasted when we were already months behind?

At this point Bob’s gloves were off — he was adamant I wouldn’t be able to do it and said I was living in the “land of rainbows and unicorns and fairies” and not the real world, where people had to deal with legacy code. I told him that I had dealt with monstrous legacy systems before, and that I understood the challenges and risks associated with them.

Bob said that he “washed his hands” of the project, and that it was entirely on me and that he would not be held responsible at all if anything went wrong. It wasn’t the best team-oriented attitude, but it was enough for the company to start succeeding.

I laid out the plan in detail — for the next week, we would follow my Angular 1 plan and wire it up to the endpoints needed for it. I made a list of endpoints I needed from Bob (less than 7), and we set to work.

Ace up my sleeve

I have to admit, I had a secret weapon coming in to that meeting — I had already built the entire front-end in Angular 1 over the weekend. It wasn’t smoke and mirrors, either. It had full data flow to JSON endpoint mocks and full user-interaction. Routing was fully working. The code was architected in a maintainable and extensible way. The styles were just like the design minus a few font-weights and bolds here and there. It was mobile responsive. The deployment and production scripts were built and working. Quality-wise, it was a production-ready front-end. All it needed was the URLs for the endpoints and some minor field changes in case there happened to be field naming changes from the anticipated schema.

The light reveals some dust bunnies

It turns out endpoints weren’t in the state we were led to believe. The front-end schema we were given to follow was wildly inaccurate — different field names, missing fields, different accepted schemas, and even different datatypes. I had to go back and forth many times with Bob to get the endpoints to function properly.

Some of the issues were normal synchronization issues that are a normal part of integrating with APIs. However, some were indicative of poor testing and quality assurance practices. One issue was opened and closed 5 times as various bugs kept coming up — it led me to believe that the code changes weren’t being tested thoroughly before being deployed to staging (where my UI was pointing).

Despite the difficulties we encountered, the Angular 1 plan actually worked! We managed to meet our deadline (minus some promised features due to missing / non-functional endpoints) and we presented the product to our stakeholders, who were very satisfied with the progress. They had almost written us off by this point, but the new momentum had renewed their faith in us.

Fixing the cultural problems

The sprint gave me hope that Bob would still be able to be a productive member of the team.

Although some of his actions and communication during the sprint could be interpreted as passive-aggressive (such as changing the names of our Angular 1 repositories and adding the word “temporary”), they weren’t outright antagonistic and the edge seemed to have dulled somewhat. The office didn’t feel as tense when we were working on this effort (although there were still moments), and things were actually getting done for the most part. There was actually a nice buzz of activity as we all worked together towards a common goal. It was a glimmer of hope for how the engineering team could be.

I played with the idea that Bob could be redirected and he could work exclusively on tiger projects or other lone initiatives without negatively impacting the rest of the team.

Unfortunately, it quickly became evident that this would not work out:

That was the straw that broke the camel’s back. Bob’s apparent lack of care and his refusal to cooperate or play nice with the team led me to recommended to Elliot to end our business relationship with him.

The Bus Factor

Unfortunately, this relationship was somewhat difficult to end: Bob had the keys to the kingdom. He was the only with any knowledge of the backend or access to the various production servers, the passwords, the critical technical accounts, etc. He had admin access to most of the systems, and in some cases was the only person with admin access.

During this time, Elliot had requested multiple times for Bob to sit down with me and train/provide deployment instructions, passwords, etc. Every time, Bob refused and responded that anyone who “knew what they were doing” would be able to figure it out in a matter of minutes.

It was likely a jab at how unskilled Bob thought I was. Multiple attempts to get him to sit down with us proved futile — there was always some reason that prevented the meeting— time, money, schedule, etc. We weren’t getting anything.

Even with the bus factor of one, we decided the best thing to do was to end it.

Bob is gone

After we ended our business relationship with Bob, we were basically starting from zero. No passwords, no instructions, no diagrams, or deployment instructions. We didn’t even have the latest deployed backend source code — the earliest version we had was from a month before I joined the company. We hadn’t even received any code or deliverables from the critical, constantly delayed project Bob repeatedly said was only “weeks away”.

Luckily, I was friends with the team at a consulting firm that was highly skilled in DevOps and system administration. We set up a meet, I explained the situation, and Elliot decided to contract their services. I worked closely with the consulting firm’s lead, Charlie, to figure out our server issues and get a handle on where we were as a company.

Starting with a crawl

Our first move was to retain access to our servers. We decided to start by resetting the staging server’s root password so we could do some exploration on how everything was set up. It was the least risky course of action in our minds —we didn’t want to touch production directly because we didn’t know if we could bring it up if it when down.

Good thing we didn’t touch production! The staging server’s application and its dependencies didn’t start up after we brought it back up. We had to dig around for a while in the command history, search through various installed applications, and dig through files until we figured out which commands to run on which files and in what order. We eventually figured out the commands we needed to bring the application back up properly.

The fires start up

As Charlie and I researched the environments and learned our deployment procedures, we received panicked messages from our biggest client, saying that records were showing up twice in the production environments, messing up key reporting. We traced it back to the staging server’s job runner.

Apparently, the staging server’s job runner was being pointed to the production database, which caused it to create invalid records. We quickly stopped the staging server and simply decided to bring the whole system down, unsure of what other traps were lying in wait.

Over the next couple of weeks, we utilized Charlie and his team heavily, obtaining control of our servers and figuring out various account and network issues. We set up a secondary deployment procedure because we simply couldn’t risk changing the initial production server’s deployment. There was a month of untested code in the latest branch, the dependencies were old and hard to find — some weren’t even available online anymore.

A terrible discovery

At the same time, I was resetting the passwords for the various accounts that were being used by Bob for Transaction Software, Inc. — email accounts, 3rd-party vendor accounts, etc. We didn’t have a list of them, but there were easily several dozen.

During this process we discovered that Bob had actually deleted Transaction Software, Inc.’s Bitbucket repository! The deployment setup was partially relying on it and the latest version of the code was on it! Worse yet, this had been done this AFTER we had decided to terminate our business relationship with him!

We lost everything in that Bitbucket repository, including any wiki pages, documentation, past / existing issues, and code versions. No passwords, no recent codebase, no roadmap. The latest version of the code we had available to us on Github was from before I joined the company.

I had never heard of a contractor destroying the assets of the company he was contracting for. I guess there is a first time for everything.

In a pure stroke of luck, I had discovered that the staging server had the latest source code on it. I tarballed it, downloaded it, and then pushed it up to a branch on the company’s Github. The work was saved — or at least some of it.

Like a bad penny

I wish we could say that was the last we saw of Bob, but unfortunately it wasn’t. Bob sprang up again some time later, this time with outlandish IP claims.

He claimed that all of the code and software he wrote was his intellectual property, despite multiple references in his past invoices that it was work for hire. He demanded payment of his final invoices, despite the fact that some of the projects he was supposed to have been working on were never delivered to us at that time.

Several back and forth and multiple attempts to resolve the issue without getting lawyers involved proved futile. Both sides eventually got lawyers.

When we did finally receive the code for the missing projects, it became clear just how much we overpaid. The critical project that was “2 weeks away” for months? It consisted entirely of an automatically generated rails scaffold with a few files taken straight from easily discoverable online code samples. None of it was usable for any real world, production use. It would’ve taken anyone, even someone who didn’t know how to code, mere minutes to recreate.

This was too much. Elliot wanted to sue him into the ground, no matter how much it cost. I recommended we move on. We had already incurred significant costs trying to fix Bob’s mistakes and errors. Wasting more time, energy, and money would only serve as a distraction. It was simply easier to stop wasting time on Bob. We had a company to build, and we wanted to focus on that instead of fighting silly legal battles or dealing with Bob anymore. If Bob used or further laid claim to our IP, we would defend it. Otherwise, we would ignore him. He wasn’t worth our time or effort.

We decided to fully pay Bob’s final invoices and left it at that — a painful lesson learned.

This was a story about my first two months at Transaction Software, Inc. It was absolutely brutal, and if I had known what I was getting into I wouldn’t have shown up to the interview.

However, this story has a happy ending. Starting that low meant you could only go up from there. After the initial rocky start, we planned and successfully executed a turn-around for the company, working hard and smartly at removing the old baggage and creating a new future. I’m proud to report that the company culture turned around completely, and it actually became a fun place to work for everyone!

You don’t always know what you’re getting yourself into when you accept a job!

The Beginning of the End?

Engineers were leaving by the droves. We lost seven engineers in months due to a single project. Our turnover had hit a high of a 38%.

It was the beginning of an exodus and an organizational collapse was on the horizon.

What was going on?

We were supposed to be on a solid path. Product ran two-week sprints and performed all of the scrum ceremonies to the T. We had 1:1s with all of the engineers. By any organizational standard, we should have been doing well.

Our problem was that, while on paper we were agile, in spirit we were far from it. We had highly centralized, top-down command-and-control, driven by product management that told the developers exactly what to do and when to do it.

Product’s initiatives weren’t at all aligned with the business. The developers ended up creating feature after feature that the business never used or even asked for.

The project that broke the camel’s back was a multi-month waterfall technology project that provided zero value, was started with little to no planning, and touched every core part of our software systems and business processes. Despite multiple warnings raised by engineers, they were overruled and the project proceeded.

It was an intense and focused effort by the organization to get it done, with the promise from Product that it would solve all of the business’ problems by the end. It was an empty promise but we labored on.

On the one hand, it was great to see that level of organizational alignment. On the other, developers hate death marches. Sure, we ran it in two-week sprints, but a two-week waterfall is still a waterfall. By the end of it, the developers were burnt out.

Enough was enough. The rate of departure was unsustainable and not yet over. As an engineering manager close to team, I knew that more than a few developers were actively looking for other jobs.

We had to make a change.

Who Are We? Product or Engineering?

The organizational structure at the company hadn’t changed much from its startup roots. While the company had scaled by nearly 600%, its processes and structures hadn’t grown to match. The structure was very much the same as its startup roots.

There was no “Engineering” department. It was simply “Product,” which included all of the product managers, designers, and engineers under a single organizational entity. Engineering didn’t have a separate budget, OKRs, KPIs, or even roadmap.

As a result, Product was the sole authority on everything.

This led to a few challenges

It was difficult to prioritize or advocate for Engineering requirements. Things like addressing technical debt would always lose against “the next big product idea” that would (in an ideal world) generate millions of more dollars in revenue.

As a result, the system accrued a significant amount of technical debt over the years, making the development of even basic things like changing the color of a button across all of the pages a monumental effort. We had a joke for it — we called it “load-bearing CSS” because we knew somehow, somewhere, there was critical business logic depending on the color of that button.

It made resourcing and planning difficult. We shared a fungible budget with Product, which meant when an engineer left, that allocation often disappeared, going to either Design or Product, or disappearing outright.

It made it hard to give people raises. Our developers had significant pay disparities as a result of ineffective hiring practices during the company’s hyper-growth phase. Many were nearly 30–50% under market, some were 50% above market. In some cases, people who shared the same level had as much as a 60% salary difference!

Finally, it made engineering difficult. We were subservient to the Product organization, which meant we always had to do what the product manager said, without exceptions. We ended up with a team of ticket takers building tiny things the product manager specified, instead of leveraging our collective problem-solving skills to find a great solution to a problem.

Lacking voice

As far as the rest of the company was concerned, Product was the technology team and, as a result, the product managers were the voice of the technology, design, and engineering.

Unfortunately, the product managers simply didn’t know the technology as well as they thought they did. They had a lot of assumptions about how it was built and how it functioned — these assumptions were incorrect.

Part of the issue was their skill level. Novice product managers ended up drip-feeding information or projects to engineers with no cohesive end vision. As a result, things were built as separate items or as minor add-ons to existing systems. When it was time to make the pièce de résistance, tying them together into a synergistic whole, it was impossible because they weren’t built to be integrated or separated from other parts of the system.

Part of the challenge was the team structure. We had teams that operated at the layer level — front end, back end, and mobile teams. They weren’t cross-functional and each initiative required significant cross-over and synchronization. The result? Unnecessary hand-offs, gatekeeping, and delays on things that could have been executed rapidly with a single cross-functional team.

This happened repeatedly. When product managers communicated their features to the rest of the company, they were assumed to have wildly different capabilities than what was actually built. However, because product managers were viewed as the voice of the technology, their inaccurate statements were taken as gospel by other parts of the organization, leading to significant confusion.

Engineering needed a distinct voice.

How did we get there?

Like any major organizational transformation, it took a lot of conversations and a few lucky breaks.

The first issue to solve was the layered teams. This was relatively straightforward to fix — have engineers work in full-stack teams. This meant mixing engineers of varying skills into each team, removing the layers.

Once that was proven to work better, it was time to illustrate the need for separate departments. I talked about the confusion and development issues it caused to have them combined. I talked about how it was still possible to maintain collaboration and organizational cohesion even with the departments separated, citing the success of the newly formed full-stack teams as an example.

I emphasized the need for a leadership structure that represented engineering needs and could manage engineering-related personnel and structural issues.

I came prepared. I had everything from desired org charts to career progression ladders to budgets. I talked about team structure, development processes, with anyone and everyone who would listen.

I worked directly with executive stakeholders as well as middle managers and ICs throughout the company to share this message and understand the problems it would cause or solve, and then summarized them the essential details for the executive team.

We eventually got buy-in from executive stakeholders and thus formed the PED team — Product, Engineering, and Design as separate departments working collaboratively. I became the Director of Engineering responsible for the newly formed Engineering department and all of its developers.

This meant we had our voice, our autonomy, and our budget.

First Stop: Fair Compensation

As previously stated, developer compensation was all over the place and completely divorced from actual performance, by as much as 60%.

It’s not a great strategy to have that significant a pay disparity for the same quality of work. Pay disparities demotivate high performers who wonder why they should work harder to earn less and it causes low performers to become complacent. After all, if high performance isn’t necessary to achieve their desired salaries, that’s one less thing motivating them to improve and perform well.

I put a stop to that as soon as I could. One of the first things the separation enabled me to do was level pay bands across the board. Working with our finance team, I brought Engineers who were significantly under market up to market. One of the biggest gripes engineers had was about their compensation and I ensured that hygiene factors were met so we could focus on more important issues.

I provided clarity and reset expectations for the engineers who were significantly over the market level. For some, I made it known that their salary expectations were simply unrealistic given our company and their skill level. For others, it was about focusing them and providing them training and professional development so that they could perform at the level that matched what they were getting paid.

In some cases, the conversations turned into performance improvement plans. Some employees improved and thrived. Others didn’t and parted ways with us. I ensured a portion of departing employee salaries was re-invested back into the current team, enabling me to level up the employees that were performing well or improving.

Names Give Power — With Power Comes Responsibility

Because we were now the engineering department, we had some responsibilities to the rest of the company. They needed visibility of how our organization was performing and whether this change was worth it.

Numbers for everyone

We were in a position where the metrics we were looking at weren’t particularly useful. We had product metrics like velocity for team-level capacity, but not much else. We couldn’t use them to see where we were on our journey of continuous improvement.

We experienced lucky timing. The State of DevOps report had come out, which provided an easy-to-adopt set of metrics we could use as the foundation for our culture of measurement. As an avid fan of the book “Accelerate,” it was a great reminder of the existence of a set of principles for engineering success.

We took the four basic ones:

• Cycle time.

• Mean-time-to-restore.

• Deployment frequency.

• Change failure rate.

We hadn’t measured any of these in the past. I looked at some tooling to retrieve these, but they were complex, not quite what we needed, or were pricey. We simply didn’t have the resources to devote to exploring them. Besides, it wasn’t the greatest idea to suddenly increase costs by hundreds or thousands of dollars, right when the engineering department started. We first needed to prove ourselves.

We took the DIY approach. I immediately wrote a bash script that took our Git commit data and calculated the deployment frequency from it. I also wrote a second script that took the same Git data and estimated a change failure rate, looking for words like “hotfix” and comparing it to the deployment frequency.

I also implemented a process to manually document every single incident or outage — including timelines, when it started, when it was detected, when it was resolved, what the root cause was, and what actions were taken subsequently. It was documented in a Github issue, which was fully available to all of the engineers. Through this, we had our mean-time-to-restore — at least manually.

I couldn’t figure out how to get cycle time from what we were already tracking, so I took a proxy measure of lead time. I created a spreadsheet that took our JIRA issues and estimated lead time based on the time an issue was created and when it was resolved. It wasn’t precise or accurate, but it was a good enough ballpark to get a picture of where we were.

The results weren’t pretty

Looking back at the data as a benchmark the result were disappointing, to say the least.

Deployment Frequency was at 0.25 / day. That means we deployed, on average, once every four days. Digging deeper into the numbers, it was often small tweaks and bug fixes — feature deploys were happening a lot more rarely.

Our Change Failure Rate was 25%, which meant one out of every four deploys required a hotfix. Looking into data even earlier, our change failure rate at one point had hit 400%, meaning every deploy led to four hotfixes.

Our mean-time-to-restore was roughly eight hours. This was driven primarily by a database outage that we simply weren’t prepared for.

Our Lead Time was 14 days — coincidentally, the same length as a sprint. I guess that work does always fill the allotted time!

Getting Lean

There’s a lot of overlap with DevOps and Lean principles. Post-agile foundations just don’t change. Reducing batch size, creating and driving feedback loops, iteration — these are all things we could do.

However, the technology didn’t enable it. If I were to tell our team “we should deploy every day,” it would be impossible to do without greatly negatively affecting our output.

Our deployment took nearly half an hour. We didn’t have a staging environment. Our tests failed regularly for random reasons.

It was a chicken and egg problem — how do we improve our process, to improve our technology, to improve our process, to improve our technology…?

Starting With the Principles First — Small Batch Sizes

One of the principle practices of Lean and DevOps is risk reduction via small batch size. By reducing the risk of every deployment, we accelerate our ability to deliver by improving confidence and stability in our delivery process.

Our technology platform was not conducive to delivering anything without a big-bang release. We would have long-lived feature branches where everything would go (via the git-flow approach, which I introduced when I first started). These branches may have had several thousand lines of code be deployed all at once.

It’s because we didn’t have the capability to deliver small parts without impacting users.

The solution? Create the capability.

Feature Flagging to the Rescue

I set out to create a feature flagging infrastructure within the platform to enable us to toggle features off and on in a consistent way. My hope was that this would encourage the deployment of incomplete features but good code, which meant it would reduce batch sizes/deployment sizes.

This wasn’t as easy as it sounded. Over the years, multiple pseudo-flagging techniques were littered throughout the system. Some things were using ENV variables, others were using hard-coded conditional checks, and more. We even had columns that represented feature flags for a given table!

Managing and integrating all these to work consistently was a challenge that took more than a few phases. It was made more difficult by the way our system was designed — we had to approach system-level feature flags and context-level feature flags in separate phases due to legacy code.

We stuck to the basics. We moved all our feature checks behind an interface, built out a service class to encapsulate all of the various different kinds of flag checks we used. Once we transitioned everything to use the new service class, we created a new data model to hold the feature flags and ran both systems in parallel. After the system was proven to work, we sunset the old data model, then finally deprecated the old code. We also performed a developer training to ensure all the developers knew about the new feature flagging system and how to use it.

Once we had the service layer and data model controlling all feature flags in our platform, extending its functionality with even more power was easy. We added A/B testing, flag options, and more!

Because the interface was built in a generic way, we could easily swap it out with a more robust flagging library down the line without a problem.

Deployment != release

Feature flagging led to the separation of a deploy from a release — a terminology difference that people previously didn’t even consider within the realm of possibility. A deploy was a code integration and delivery into a production environment, whereas a release was a feature becoming used by one or more users.

An Unexpected Surprise

Surprisingly, the introduction of feature flags actually improved interactions between Engineering, Product, and Design.

There was always a tension between building things right, building the right thing, and building things right away.

Feature flagging made engineers happy because it helped de-risk large projects by providing a way to integrate and get things out quickly. Engineers started to deploy code earlier in the development process, which prevented code from piling up into massive, risky deployments.

It made product managers happy because they could test the business impact of an idea with less risk but still control when it went out. They could release a feature when they wanted, or test it on smaller audiences to manage risk and get feedback early without drastically impacting the business. If the feature didn’t work as intended or caused issues, they could easily turn it off with the click of a button. Features could be iterated multiple times to get the kinks out before a wider release.

This, in turn, made designers happy because by deploying things, but not releasing widely, they could be iterated on to eventually match design fidelity. It helped us to ensure we didn’t deliver subpar interactions and experiences to customers.

It was an unexpected but welcome effect on team collaboration. I won’t say feature flagging was the silver bullet to our collaboration problems, but it sure helped us realize that we all had the same goal in mind and get over the smaller execution quibbles.

Small Batch Sizes = More Batches = More Overhead

The feature flags helped reduce risk and batch sizes somewhat but we were still deploying releases with nearly a thousand lines of code.

Why?

Our deployments were painful. It took 30 to 45 minutes of a developer’s full attention, merging PRs, writing release notes, and performing smoke tests.

As a result, most developers hated to deploy. A few dedicated engineers did most of the deploys, while the rest cheered them on.

The frequency of deploys was low under this model. It only happened a few times week and each of the deploys was still relatively large and risky.

We needed to reduce the overhead of producing the batch/deploy.

Automation to the rescue

I wrote a tool in bash to automate this tedious process, at least for some of our deployments. The tool, which we called Patchy, allowed any engineer to create a release out of a single PR, and deploy it straight to production with a single command.

Running ./patchy.sh 6486 would deploy PR #6486, for example.

It handled everything related to the deploy that was previously taking up the developer’s attention: release notes, incrementing version numbers, checking the test results, and even notifying people with the notes.

Patchy reduced the time it took a developer to deploy from 30 minutes to just one minute. While the deploy itself took roughly 15 minutes, the developer simply triggered it in a “fire and forget” manner.

The results were instantaneous. Deployment Frequency went from 0.25/day, to an average of 6.5/day. More developers were involved in the deployment process. Gone were the 15–20 PR queues. We even had some days where we hit zero PRs open without a reduction in our productivity or output.

Increase deployment batch size

One issue we encountered was that, while our deployment frequency was high, the smaller deployment batch sizes led to an increased overhead when deploying multiple items.

We had “deploy trains” where engineers would take turns “Patchying” their PRs. These deploy trains would sometimes run for hours — an engineer would run Patchy for their PR, sending a deploy automatically, and once it was done, another engineer would run Patchy for their PR, and so on and so forth. This led to a stream of changes that, while good on the surface, were indicative of inefficiency.

I rewrote Patchy into a Ruby script, adding the capability to specify multiple PRs to deploy.

By enabling Patchy to deploy multiple items in an automated way, we were able to reduce the cost of deploying multiple things by slightly increasing the deployment batch size.

The result: a drop in deployment frequency to about 3/day without an accompanying drop in the change volume. The number of items actually being released didn’t go down at all.

I carefully monitored the change failure rate and didn’t see any increase. People were still careful to keep their releases low-risk: Trusting your engineers pays off!

Patchy V2 simply removed the overhead of deploying multiple things.

Iterating on Metrics

Around this time, I was able to update our tracking capabilities. JIRA had released new project types (thank you Atlassian!), which allowed me to automatically set timestamps and other items based on when items moved into certain statuses. This allowed me to automatically calculate cycle time, distinguishing it from lead time.

I also implemented automated uptime monitoring, using a combination of Pingdom and Statuspage. This allowed us to have basic server checks that ran once a minute and sent us an alert if our site returned any issues.

In the interest of transparency, we also shared the status page with the rest of the company, providing operations, sales, and other departments an inside look at our uptime to help hold us accountable.

It’s Not Just About the Technology

A lot of people equate DevOps with technologies. When they hear “DevOps”, they think of Docker, Kubernetes, etc. Some think of practices — containerization, orchestration, etc.

For me, these are just tools or methods to accomplish a cultural shift. DevOps begins in the mind and its primary focus is shifting the mindset and culture.

One of the biggest blockers to shifting towards a DevOps model on our organization was our own mindset towards change, ownership, and responsibility.

It had to change.

Ambiguity and the Unknown

“Ugh, the requirements changed.”

“Change” had become a bad word.

The team was not comfortable with ambiguity, to say the least. Previously, the highly centralized work was expected to be finely planned, intricately detailed, and perfectly accurate before being handed off to engineers. This was a result of a building mistrust between Product and engineers.

Any changes or missing information led to an outcry of “incomplete requirements!!!” and many complaints and grumblings from engineers. Perfect requirements are an impossibility, of course, but that never really occurred to the team.

It was a huge shift to get engineers comfortable with being adaptable.

“Change” had become a bad word. If we wanted to succeed and truly be agile, we had to embrace change. I immediately started a counter-culture that was tolerant and accepting of change, shifting the perspective to a new mindset where change was an opportunity to gain more insights into what reality represented.

Making the problem visible

Sometimes, there’s no better way of showing there’s a problem than to visually show it.

I made the lead time metric visible to everyone by publishing the graph on the shared TV in our office area to force the developers to visually see what the impact of demanding perfect planning and detail was. I shared it in 1:1s and meetings.

It was a drastic metric. Some requests waited over sixty days in planning before they even started getting worked on. From a customer’s perspective, that was nearly two months before anything happened — no wonder people in our organization felt Product was a black hole for requests!

Solving the problem

Now that the problem was visible, I could explicitly address it.

I encouraged limits to the length of time we planned items as a forcing function to force things to not be fully fleshed out. We encountered real analysis paralysis and encouraging short lead times improved the team’s adaptability.

Short lead times, however, led to more ambiguity in the requirements. I did significant 1:1 training with the engineers, leads, and product managers to overcome the reluctance to work with less detail. I taught them the kinds of questions to ask and the problems they should look for, and the solutions they could find on their own.

I gave them strategies and tactics for dealing with ambiguity, such as making the “70% decision,” or “guessing right.” I taught them architecture and patterns to help make their code more flexible and less dependent on policy details that could change at any time.

These foundational techniques helped them to get more comfortable with the loss of explicit detail, and also pushed them to leverage more of their own problem-solving skills.

Sure, some project requirements were rushed in the short-term, but the long-term positive impact on culture was invaluable. It might have been an uncomfortable experience, but it greatly improved the team’s capability to adapt and operate with ambiguous information.

Planning for the Future vs. Planning for the Never

While we had made progress in reducing lead times, Product direction still operated in a long-term manner that wasn’t adaptive to changing business needs. Roadmaps would be planned three months to a year out.

Startups change drastically in short time frames. The company’s focus might be one thing in one month, then completely different in the next. In an environment where change is the only constant, why attempt to create a 12-month roadmap?

Forecasts of the long time horizons would go out of date quickly. Worse, a month into the roadmap and we’d know better about our environment and problem, making our roadmap moot. Why stick with it when we know it won’t last? How do we change that mindset?

In an environment where change is the only constant, why attempt to create a 12-month roadmap?

Failure is an option and a darn good one

All of the changes we made in our culture and technology had a massive benefit — it made it cheap to fail.

Reducing the cost of failure allowed us to pursue opportunities that might have previously been incredibly risky or needed to be highly planned in the past. Getting it wrong doesn’t matter when the failure is easy to reverse, or the scope and impact are limited.

I formed a team with a couple of engineers with the sole intent of focusing on the opportunities the business wanted to pursue — those that had a high likelihood of failure, or that weren’t thought out that well. These were often the interruptive tasks that previously annoyed teams and took their attention away from the roadmapped items.

To execute with this team, we practiced Kanban and adopted lean principles. Other teams were running Scrum with all of its ceremony and pomp, but I felt it wasn’t the right approach for this team.

We wanted to be a quick reaction force — a nimble team capable of adapting to the whims of the business. We couldn’t have a quarterly roadmap and do that at the same time. As a result, I ran the team with no roadmap, keeping a close eye on our queue size and ensuring we never planned more than the next week’s worth of work.

Some of the product managers viewed this approach as chaotic and unaligned with the roadmap. They argued it was taking valuable manpower from the planned work we had.

I disagreed. I viewed it as a supportive effort for the product roadmap. By taking on all of the interruptive projects and demands of the business, it freed up the other teams to focus on the main efforts of their planned roadmaps exclusively.

You know what? It worked. The business threw problem after problem at us and the lean team adapted immediately and executed on it. We were able to protect the rest of the teams from these projects, which if we hadn’t done would’ve had to be done anyways, except on a much tighter schedule. While the other teams churned away on a predefined quarterly roadmap, our team was able to handle the business’s urgent intake requests.

The business was happy. Previously they had no ability to change or influence the direction of the product development once it was set. Now, we had a collaborative relationship that allowed us as an entire product development organization to make progress on roadmapped items while still remaining adaptable to business needs.

Other teams later adopted this lean execution model, although not to the extent of the quick reaction force I set up. By limiting their forecast and roadmapped projects to a month or two, they avoided doing extra work on items that would never get executed. They would also leave their roadmaps open to responding to the feedback they got from customers about their release. This meant we started listening to the customer more, which led to better feedback loops and more iterative execution.

By adhering to lean principles, such as work-in-progress limits and managing queue sizes, teams were able to build a backlog of work that was useful for executing and communicating, but adaptable and still responsive to changes found in feedback and business direction.

Promoting Decentralized Control

We were moving faster and delivering more and more, and the speed was uncomfortable for some. The speed, combined with all of these changes and the increased ambiguity, led some people to try and involve themselves in every decision and meeting in an attempt to keep abreast of all of the things that were happening. This led to slowdowns as their gatekeeping became bottlenecks in execution and delivery.

This wasn’t sustainable or effective and the lead engineers had to change their mindsets.

The question becomes, “how do you encourage lead engineers that were previously very centrally controlled to get out of their comfort zone and start making decisions?” When people have operated for a long time under a central command model, it becomes second nature for them to ask for permission and just do what they’re told.

“Learned helplessness” is contagious. It’s a culture shock to be thrust into a situation where you have full freedom with no warning.

We started small

We started small. We provided decision-making capabilities to people who had proven their competency. We limited the scope enough to ensure we didn’t go astray and that we could easily recover from bad decisions.

I personally coached lead engineers and other promising junior leadership on how to flip the mental model of leading. I talked to them about being tolerant of failure, leading with intent, asking not telling, and other foundational practices intending to build a culture of initiative. I taught them that by letting go of direct control and oversight, they would get teams able to make local decisions in a timely manner.

By providing intense clarity and alignment, you can indirectly promote autonomous decision-making that’s beneficial to the business.

The culture wasn’t for everyone

When you define a culture, some people are going to opt-in, and some people are going to opt-out.

It was a long journey. Unfortunately, not everyone followed us on that journey.

Some people just don’t operate well with ambiguity, or explicitly do not want that kind of working environment. That’s OK.

That’s the downside of defining a culture. When you define a culture, some people are going to opt-in, and some people are going to opt-out. It’s a choice every person makes.

I wish those that opted-out the best of luck!

Was It Worth It?

Did all of these efforts work? Were they useful at all? The answer is a resounding yes!

Organizational culture

The company used a tool called 15Five to do a quick weekly survey of the team, asking how their week went. I added additional questions from the Accelerate Westrum Organizational Culture survey in the 15Five to measure the results on the engineering team.

The idea is straightforward: If the biggest predictor of effective organizations is job satisfaction, how do you measure the factors that lead to increased job satisfaction? Westrum created a model for the factors correlated with an organization’s health. We took those factors and surveyed our team regularly over the year.

As we transitioned through our cultural shift, results went from an average of 3.1 to 4.3. It was a massive leap that signaled that we were on the right track.

Retention

Our retention numbers improved significantly as well. That year, only three developers quit — two for personal reasons and one for performance reasons.

People weren’t leaving because of our company anymore.

Delivery

Our delivery improved significantly in both speed and quality.

We were operating by every metric three to five times better than we ever were before.

Our cycle time had hit a low of three days. Our change failure rate went and stayed below 2%. Our deployment frequency hovered around 3.5 deploys / day after the introduction of batch deployment tooling. We introduced change volume as a counter metric and consistently released roughly 120–180 pull requests a month.

We released a multitude of changes that drove significant increases to retention, acquistion, and revenue. The business was happy — they were seeing tangible progress and results on a consistent basis.

Technology

On the technology front, we greatly improved the stability, performance, scalability, and security of our systems. We developed subsystems that helped us develop even faster — design systems, modular component libraries, isolated subsystems, etc.

The delivery of business value didn’t sacrifice the technical quality and the technical quality didn’t sacrifice the delivery of business value. It worked harmoniously.

Improvements

In just one short year, we had drastically transformed how the development organization operated. The changes we made proved to be effective. We were now releasing multiple times a day and had multiple experiments and tests running at any given time.

What’s more, the way we operate had led to better operations across the entire company. The company collaborated better and was more aligned than we had ever been.