Picture a map of a city. If it’s zoomed in enough, you’ll have fine-grained visibility of street names, shops, traffic, and restaurants.
But as you zoom out, the more challenging it becomes to find helpful information.
As an engineering leader, the same thing happens when your organization grows.
When your engineering team is small, you have visibility into all the nitty-gritty details of your software delivery pipeline. You know which tickets are moving, you know the CI runs, and you probably even know which PRs are pending.
But, as your business and team grow, the challenges you face evolve. And you’re pulled further and further away from those valuable details.
The more you zoom out on the map, the less visibility you have. Fortunately, you can always zoom in when needed.
The trick is to know where to zoom in.
Today we will delve into the challenges you will face as your organization scales from 5 to 250 engineers by focusing on three dimensions of your software development pipeline:
We will also cover:
👉 The engineering metrics you should look at within these three dimensions
👉 Important considerations before adopting engineering metrics tools
✅ Check out the complete Scaling Your Engineering Org Checklist!
Visibility is essential for organizations of all sizes.
But how do you go from that high-level visibility of a larger team, back to the details you had when your org. was smaller?
If there’s a challenge or a bottleneck, would you know where to zoom in to find the information you need?
Maybe it’s the CI system that’s slowing you down. Or maybe it’s the bug backlog that you’re struggling to work through. It could even be that your code base is slowing you down, and you need to pay down some tech debt.
You need to know where your challenges are, so you know what you have to do to fix them.
As engineering leaders, we have to zoom in and zoom out all the time.
If you have access to the right data, you’ll know exactly where to zoom in.
🏎 Velocity: How fast are we moving? Where are the bottlenecks in the team and process?
⭐️ Quality: What’s the quality of the software you deliver? How do you respond to quality issues? When we’re talking about “quality”, we’re referring specifically to quality that impacts the end user – think of bugs, incidents, downtime, etc.
🎯 Outcome: What are we shipping? Where are we investing our efforts when it comes to building new features, paying down tech debt, fixing bugs, maintenance, and R&D? How are we actually dividing our efforts? Most importantly, how is the impacting our customers?
Let’s look at some of the patterns organizations face as they scale from 5 to 250 engineers from the lens of velocity, quality, and outcome.
Many engineering leaders believe that when you double your team, you will double the output. But the truth is that velocity - or the speed of shipping code - doesn’t increase linearly as a team scales.
If you’re growing fast, you’ll have a constant onboarding challenge, which takes a toll on speed. At the same time, your newly promoted leaders might lack the needed experience to support your organization's growth.
Here’s what actually happens to your velocity as you scale:
At this stage, companies are trying to get to Product-market fit (PMF), which puts a lot of pressure on delivery. This “move/scale fast, break things” mentality can lead to companies keeping a steady speed but accumulating a lot of tech debt.
When the team is small, we must build a proper foundation.
When you start zooming out on the map, you lose visibility of the details.
Managers start having managers, and the new people you bring in will slow you down because it takes time to grasp the context of your product and organization.
If you didn’t set the right foundation you can lose 25-50% of effectiveness when your team gets this big.
You'll also begin to have misalignments, especially as your lines of communication increase:
As your company scales, your velocity may stagger due to a lack of visibility on bottlenecks and constant onboarding.
You'll continuously reshape the organization but lack clarity and visibility on where to act.
You are looking at the city 10.000 km in the air, which means you can’t see where bottlenecks are. This will slow you down.
Most of us fall into “the build trap”.
As we grow, we focus on delivering new features, eventually reaching a point where tech debt is too big for us to tame, which leads to a product rewrite.
If you’re growing fast, you’ll have to constantly deal with tech debt. Granted, most engineers will complain about tech debt, but they will never bring the consequential data to the table. This makes it hard to know where you are.
When your organization's just starting out, there are not enough customers that a bug or outage could impact. So you focus on speed instead of quality. This is essential as you race to PMF.
However, as your customer base increases, you need to start digging into the quality of the product.
PMF is evident and the size/value of your customer base increases - which means you’ll rapidly increase your nº of bugs and critical incidents, making the teams feel underwater.
If you lose focus, the product you built might no longer work for this market size.
All the challenges you faced when scaling from 20 to 100 engineers will continue to grow in line with your organization.
If you didn’t address issues as they came, some of your bugs will now come from the accumulation of tech debt, negatively impacting your MTTR.
In addition, teams will start pushing bugs to one another if you don't put good teams and architectural structures in place (Conway's law).
“Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.” — Melvin E. Conway
As engineering leaders, we’re constantly making trade-offs.
Seasoned leaders are masters at deciding where to invest efforts, and, as your organization grows, becoming more opinionated on the trade-offs between velocity, quality, and customer impact, is key.
If you’re growing fast, you’ll have to constantly deal with the features vs. quality tradeoff, which takes a toll on the capacity to deliver on the product. Again, being intentional in making these decisions will play a critical role in your success.
Shipping is king at this stage.
Your roadmap is directly impacted by the conversations you’re having with your customers, so you’ll invest most of your efforts in new features.
However, the technical decisions made when you were <5 engineers will start to slow you down.
You now have a live product with a mature client base. It's time to move your efforts from primarily creating new features to honing in on internal processes.
Start being pragmatic about where you invest your time.
Remember that business value is everything that creates value for the business. So you will focus more and more on fixing bugs and cleaning tech debt, and less on new features.
Your visibility will continue to decrease at this stage, so trust is critical.
You might realize one quarter later, that the team wasn’t focused on delivering the features you were expecting. Teams will start to invest more time on bugs, and you'll get frustrated when the push for new features slows down.
Investment is the mastermind behind outcomes. Of course, you'll need to commit to internal processes and work quality, but you need to be decisive on time allocation and push a lot of these decisions to the teams.
Now that we’ve seen the challenges of scaling your engineering org, let’s dive into what engineering leaders can do to prevent these scenarios.
We’ve seen what happens to your velocity, quality, and outcome as your team grows. What can you do to keep these things in check?
The key is to understand how different parts of your software delivery pipeline function to identify the levers you can pull for smooth sailing as you scale.
Modern software engineering organizations focus on delivering value through continuous delivery, integration, and improvement.
To make sure that this is happening, you need to keep a close eye on some key metrics.
Velocity metrics help you access the agility and leanness of your team so you can understand how to improve the speed and direction of developing software across all the stages of the software delivery pipeline (WIP → Review → Merge → Release → Deploy).
Lead Time tells you how long it takes for your team to go from a ticket being in progress to a released PR, allowing you to detect early bottlenecks and resolve them on the spot.
Release Frequency tells you how often you release value to your customers. Together with Lead Time, Release Frequency measures software delivery performance tempo.
The number of pull requests released during a period is an important metric to consider in combination with your PR Cycle Time. A decreasing Lead Time might be linked to a decrease in the number of released PRs.
The elapsed time between the 1st commit of a PR and the code being used in production. Compared to the lead time, the PR cycle time focuses on just the code pipeline section.
Quality metrics should be as close as possible to the end-user experience of your software.
Quality needs to be proxied by metrics that give an indication of the quality of the software delivered, such as no. of bugs by priority and bug fixing ratio.
Without quality metrics, it’s easy to find yourself going at a faster speed, or increasing volume, for the sake of your end-user experience.
The Bugs Fixing Ratio is the ratio of bugs fixed to resolved during the time period selected.
Bugs Raised by Priority helps you get more granular information into the bugs raised by severity level. Did the team raise a lot of bugs last month? Were they all critical issues? Correlating this with other metrics, might help you identify the root-cause of the issue.
As an engineering leader, you want to make sure that your team is fixing customer-impacting issues the fastest.
Outcome links engineering work to business goals, and it helps align the whole organization.
When building software, it's easy to lose sight of the business objectives we impact.
For instance, refactoring a major component without changing its behavior - how does that relate to our end user?
Other times we’re delivering a new feature in an application, such as the ability to buy a book online with one click instead of four - which can be mapped to the business objectives.
A throughput by work type can be looked at from the number of tickets fixed or the number of PRs released. It helps you understand if you’re truly aligning your engineering org with business goals.
There are two kinds of engineering leaders: the ones that have continuous improvement processes in place (like OKRs), and the ones that are starting on this journey.
Whether you have these processes in place or not, to continuously improve you have to have clear goals and visibility. You also need to know where you will invest your time.
If you already have a clear Northstar in place, we suggest you:
If you don't use OKRs or other goal-setting methods, we recommend a more expansive look into these steps.
Engineering Leaders have two responsibilities:
👉 Improve the developer experience
👉 Deliver impact to the end-user
These two are infinitely linked, and, when done successfully, will help you create a culture of continuous improvement. We believe this can be done with the right metrics and data.
If you’re a new engineering leader, ask for help by reaching out to more experienced leaders. You’ll find that the great ones are those who are constantly questioning their own methods and looking for ways to improve.
Finally, don’t be afraid of the challenges ahead. You can feel lost when the map is zoomed out, but with the right tools, the right people, and the right mindset, you can take your product and company to new heights, and have a happy team while doing so.