If you’re reading this, then you probably already know that you need the right tools to help your software engineering teams perform to the best of their abilities.
If not, then make sure to read our guide to the true value of software engineering metrics.
In this article, we’ll walk you through the key criteria to look for when choosing the best engineering metrics platform for your organization. We’ll explore what to look for, and what to avoid.
As a bonus, we also put together a list of questions to consider.
To put it simply, an Engineering Metrics Platform is where you, as an engineering leader, will get the visibility to understand how your teams are shipping software. And by “how” we mean: how fast, the quality of the code, the allocation of resources, and so on.
All of this information is important for you to make informed decisions to improve your productivity and your developers' experience, plus showcase your business value to leadership.
The best engineering metrics platforms will offer a bird’s eye view of your entire software pipeline, with the ability to drill down to view the data that matters.
Think of when you zoom in on a picture: from far, it shows you the whole, from close you can see all the little details, some that line up, some that don't. And you can only acknowledge it because you looked close enough.
Having an overview and a drill-down ability is not enough. It is extremely important to understand how the metrics are computed.
For you to have end-to-end visibility you need data to be computed from the ticket creation to deployment to production.
The lack of such will result in inaccuracy which can bear a negative impact on your decision-making.
Let's address the elephant in the room: we know DORA metrics are a good start on measuring software delivery, but they're not enough. You need to be able to go deeper, slice and dice according to your context and needs. You must know your context and understand if the tool measures PR cycle time, code reviews, CI run times, and other metrics that matter to you.
In a growing organization, you'll have different teams and there are situations where it doesn't make sense to see the data altogether. Instead, filtering by teams is the best way to see each's performance.
What's important to look for is the ability to compare the data across teams and organizations to identify best practices, opportunities for improvement, and learnings.
Being able to also compare by timeframe, repositories, components, Jira epics, and more is important to get the piece of data that matters most to your scenario.
Let's say you have 3 main teams working on 2 different repositories. Team 1 works on both, Team 2 only on one, and Team 3 only on the third. This week your cycle time went up by 3 hours overall. When you filter the data, you'll see that Team 2 is responsible for this increase. If you had only looked at the overall data you could've instructed all 3 teams to work on cycle time unnecessarily.
The way you structure your organization is unique, so your tool of choice must be compatible with your setup.
For example, if you have mono repositories, the platform must be able to break down monorepos and get metrics & insights by components, services or folders, and not only if your org has separate repos to work on.
Also, the ability to customize your release workflow per repository is key to being able to understand your metrics. You might use different release workflows in your organization and some of them might be complex.
You need a tool that is customizable and able to match all your different workflows.
To have an honest overview of your organization, you should not look at individual metrics. That's because they do not translate what your organization is delivering. Hence, engineering metrics platforms that focus on individual metrics will not help you improve as an organization.
Look for a tool that shows you what your teams are delivering.
Speaking of teams, your structure must be translated in the tool configuration. It must be able to automatically configure teams and also give you the ability to manually change them in the tool to give you the freedom to adapt as you grow.
If you're embarking on this journey for the first time or not, you should know that metrics are not enough. As a leader, you must make decisions, but a dashboard alone won't tell you what to do or where to focus.
Knowing how to do so according to your context is key in driving changes and achieving your goals.
Your tool of choice must be able to guide you through the whole process: from implementation to making decisions based on metrics.
It's what we call Engineering Success: a team of former engineering leaders to support your journey. And this is what draws a line between just having a dashboard of metrics and making actual improvements.
Here’s how engineering success can provide guidance and help make sense of the data as your engineering org scales.
You don't want to invest in something and later realize it doesn't work with your setup.
Make sure your platform of choice can integrate with your repositories, and with whichever other tools you use so you can get the most out of it.
It's also important to be able to plug and play to start using the tool right away. Setting up in minutes is really important to start leveraging metrics.
Finally, data doesn't have to be complicated.
A good metrics platform will be easy-to-use, giving you a clear overview of everything you need to do on a simple, uncluttered dashboard, with the ability to filter the data according to your needs.
We encourage you to evaluate if the UX is user-friendly and that you can easily navigate through the platform.
To help you even further in this process, we've put together a list of questions that can guide you in your process.
We hope this guide helps you in finding the best solutions for your engineering organization. If you want to learn more about how Athenian can help you, let's chat.