How To Choose The Right Engineering Metrics Platform

If you’re reading this, then you probably already know that you need the right tools to help your software engineering teams perform to the best of their abilities.

If not, then make sure to read our guide to the true value of software engineering metrics.

In this article, we’ll walk you through the key criteria to look for when choosing the best engineering metrics platform for your organization. We’ll explore what to look for, and what to avoid.

As a bonus, we also put together a list of questions to consider.

A reminder: What is an Engineering Metrics Platform and why do you need one

To put it simply, an Engineering Metrics Platform is where you, as an engineering leader, will get the visibility to understand how your teams are shipping software. And by “how” we mean: how fast, the quality of the code, the allocation of resources, and so on.

All of this information is important for you to make informed decisions to improve your productivity and your developers' experience, plus showcase your business value to leadership.

Key criteria to look for the best engineering metrics platforms

1) Have end-to-end visibility into software delivery

The best engineering metrics platforms will offer a bird’s eye view of your entire software pipeline, with the ability to drill down to view the data that matters. 

Think of when you zoom in on a picture: from far, it shows you the whole, from close you can see all the little details, some that line up, some that don't. And you can only acknowledge it because you looked close enough.

Having an overview and a drill-down ability is not enough. It is extremely important to understand how the metrics are computed. 

For you to have end-to-end visibility you need data to be computed from the ticket creation to deployment to production.

The lack of such will result in inaccuracy which can bear a negative impact on your decision-making.

Let's address the elephant in the room: we know DORA metrics are a good start on measuring software delivery, but they're not enough. You need to be able to go deeper, slice and dice according to your context and needs. You must know your context and understand if the tool measures PR cycle time, code reviews, CI run times, and other metrics that matter to you.

2) Ability to compare data

In a growing organization, you'll have different teams and there are situations where it doesn't make sense to see the data altogether. Instead, filtering by teams is the best way to see each's performance.

What's important to look for is the ability to compare the data across teams and organizations to identify best practices, opportunities for improvement, and learnings.

Being able to also compare by timeframe, repositories, components, Jira epics, and more is important to get the piece of data that matters most to your scenario.

Let's say you have 3 main teams working on 2 different repositories. Team 1 works on both, Team 2 only on one, and Team 3 only on the third. This week your cycle time went up by 3 hours overall. When you filter the data, you'll see that Team 2 is responsible for this increase. If you had only looked at the overall data you could've instructed all 3 teams to work on cycle time unnecessarily. 

3) Compatibility with your architecture and workflows

The way you structure your organization is unique, so your tool of choice must be compatible with your setup. 

For example, if you have mono repositories, the platform must be able to break down monorepos and get metrics & insights by components, services or folders, and not only if your org has separate repos to work on.

Also, the ability to customize your release workflow per repository is key to being able to understand your metrics. You might use different release workflows in your organization and some of them might be complex. 

You need a tool that is customizable and able to match all your different workflows.

4) Focus on teams, not individuals

To have an honest overview of your organization, you should not look at individual metrics. That's because they do not translate what your organization is delivering. Hence, engineering metrics platforms that focus on individual metrics will not help you improve as an organization. 

Look for a tool that shows you what your teams are delivering.

Speaking of teams, your structure must be translated in the tool configuration. It must be able to automatically configure teams and also give you the ability to manually change them in the tool to give you the freedom to adapt as you grow.

5) Support: Engineering Success

If you're embarking on this journey for the first time or not, you should know that metrics are not enough. As a leader, you must make decisions, but a dashboard alone won't tell you what to do or where to focus.

Knowing how to do so according to your context is key in driving changes and achieving your goals.

Your tool of choice must be able to guide you through the whole process: from implementation to making decisions based on metrics. 

It's what we call Engineering Success: a team of former engineering leaders to support your journey. And this is what draws a line between just having a dashboard of metrics and making actual improvements. 

‍Here’s how engineering success can provide guidance and help make sense of the data as your engineering org scales.

6) Integrations 

You don't want to invest in something and later realize it doesn't work with your setup. 

Make sure your platform of choice can integrate with your repositories, and with whichever other tools you use so you can get the most out of it. 

It's also important to be able to plug and play to start using the tool right away. Setting up in minutes is really important to start leveraging metrics.

7) Usability 

Finally, data doesn't have to be complicated. 

A good metrics platform will be easy-to-use, giving you a clear overview of everything you need to do on a simple, uncluttered dashboard, with the ability to filter the data according to your needs.

We encourage you to evaluate if the UX is user-friendly and that you can easily navigate through the platform.

Questions to consider

To help you even further in this process, we've put together a list of questions that can guide you in your process.

End-to-end visibility

1. Can I break down my software delivery pipeline from ticket creation to deployment in production?
2. Are DORA metrics supported?
3. Are the following metrics supported? Here are some examples:

• Lead Time
• PR Cycle Time
• Coding Time
• Review Time
• Merge Time
• Release Time
• Deploy Time
• Release Frequency
• Released PRs
• Deploy Frequency per Environment
• MTTR
• Lead Time for Changes
• Change failure rate
• Work in Progress
• Bug Fixing
• Work Allocation

Actionability

1. Can I compare and filter metrics by teams, timeframe and repositories?
2. Can I break down metrics according to components, services or folders?
3. Can I compare data according to Jira epics?
4. Can teams set and track their own goals?

Accuracy

1. How does the tool provide data for monorepos? 
2. Can I configure my releases according to specific tags?
3. Can I configure my releases according to specific events?
4. Can I configure my releases according to specific branch merges?

Support

1. Is there any support during the implementation phase?
2. Is there any support after the implementation?
3. Is the support comprised of experienced engineering leaders that are incentivized to help our org improve?
4. Is the support channel easy for our team? (ex: Slack)
5. Are there support SLAs?

Integrations

1. Does it integrate with your VCS (Github, Gitlab, Bitbucket etc.) done?
2. Does it integrate with our ITS (Jira, Linear, Shortcut etc.) done?
3. Does it integrate with the rest of our stack?

Usability

1. Is it easy to get started and configured?
2. Is the learning curve easy for my team?
3. Can I pull my organizational structure and teams automatically?
4. Can I easily manage my teams as the org scales?
5. Does it automatically keep teams in sync with other tools?
6. Can I easily navigate through the platform?
7. Can I share dashboards with my reports?

Security

1. Can it work without storing source code?
2. Is the company following security & regulatory compliance (SOC2, GDPR etc.)?
3. Is there encryption in transit and at rest?
4. Is customer data confidential and isolated?
5. Is the company subject to regular pentests? 

We hope this guide helps you in finding the best solutions for your engineering organization. If you want to learn more about how Athenian can help you, let's chat.