This article will guide you through the key criteria to consider when selecting the best engineering metrics platform for your organization. We’ll explore what to look for, and what to avoid.
We'll also provide a full list of questions to help you make the most informed decision in selecting the right engineering metrics solution.
A reminder: What is an Engineering Metrics Platform, and why do you need one
An Engineering Metrics Platform is where you, as an engineering leader, will get the visibility to understand how your teams are shipping software. And by “how” we mean: how fast, the quality of the code, the allocation of resources, and so on.
All of this information is important for you to make informed decisions to improve your productivity and developers' experience and showcase your business value to leadership.
Key criteria to look for the best engineering metrics platforms
1) Have end-to-end visibility into software delivery
The best engineering metrics platforms will offer a high-level Engineering Metrics Dashboard of your entire software pipeline, with the ability to slice and dice the data to drill down into the data that matters to your context.
Consider the analogy of zooming in on a photograph: you can see the entire image from a distance, while a closer view reveals intricate details. Only by examining the picture closely can you truly appreciate its intricacies. Likewise, having both an overview and the ability to drill down into your engineering metrics is essential. Just surface-level data like DORA metrics will not be enough. You must know your context and understand if the tool measures PR cycle time, code reviews, CI run times, and other metrics that matter to you.
It is also important to understand how the metrics are computed and ensure that the engineering metrics platform truly maps your metrics from ticket creation to deployment in production. A lot of engineering metrics tools will only track Github data. But work doesn't start at the first commit. You need to have the complete visibility to get accurate and actionable engineering metrics. Without full end-to-end visibility you won't be able to uncover insights, drive improvements and measure your impact autonomously.
2) Ability to compare data
As your organization grows, you will have multiple teams working on different aspects of your projects. In such cases, it's not always practical to view all the data together. Instead, filtering data by teams is the most effective way to assess each team's performance.
The ability to compare the engineering metrics across teams and organizations to identify best practices, opportunities for improvement, and learnings is critical.
It's crucial to find a solution that enables you to compare data across teams and organizations to identify best practices, opportunities for improvement, and valuable insights. Additionally, having the ability to compare metrics based on timeframes, repositories, components, Jira epics, and more helps you obtain the most relevant data for your specific context.
3) Compatibility with your architecture and workflows
The way you structure your organization is unique, so your tool of choice must be compatible with your setup.
For example, if you have mono repositories, the platform must be able to break down monorepos and get engineering metrics & insights by components, services, or folders.
Also, the ability to customize your release workflow per repository is key to being able to understand your metrics. You might use different release workflows in your organization and some of them might be complex.
You need a tool that is customizable and able to match all your different workflows.
4) Focus on teams, not individuals
You should not look at individual metrics and only focus on measuring how teams are performing. Stack-ranking individual developers will lead to metrics gamification, pushback from developers and you won't be able to leverage the solution across the org.
Look for a tool that shows you what your teams are delivering not individual developers.
Speaking of teams, your structure must be translated into the tool configuration. It must be able to configure teams automatically and also give you the ability to manually change them in the tool to give you the freedom to adapt as you grow.
5) Support: Engineering Success
Whether you're new to using engineering metrics or have experience with them, it's important to recognize that metrics alone are insufficient for driving meaningful change. As a leader, you'll need to make informed decisions, but a standalone engineering metrics dashboard won't provide all the answers or direct your focus. Understanding how to interpret and act on metrics within the context of your organization is vital for driving progress and achieving your goals.
Your tool of choice must be able to guide you through the whole process: from implementation to making decisions based on metrics.
The ideal tool should guide you through the entire process - from implementation to making data-driven decisions. This comprehensive approach, which we refer to as "Engineering Success," is supported by a team of seasoned engineering leaders who help you navigate your journey. It distinguishes between merely having an engineering metrics dashboard and actually driving tangible improvements. By offering guidance and insights tailored to your organization's needs, Engineering Success helps make sense of the data as your engineering organization scales.
Learn more about our approach to Engineering Success and how it helps our customers improve.
6) Integrations
It's essential to choose a platform that seamlessly integrates with your existing setup to avoid investing time and resources in a solution that ultimately doesn't meet your needs.
Ensure that the platform you select is compatible with your Git-hosting service and other tools in your stack, allowing you to maximize its benefits.
Additionally, opt for a solution that offers a plug-and-play approach, enabling you to begin using it immediately. A quick and hassle-free setup process is crucial for promptly harnessing the power of metrics in your organization.
7) Usability
Finally, engineering metrics don't have to be complicated.
A good metrics platform will be easy-to-use, giving you a clear overview of everything you need to do on a simple, uncluttered dashboard, with the ability to filter the data according to your needs.
We encourage you to evaluate if the UX is user-friendly and that you can easily navigate through the platform.
Questions to consider for the Engineering Metrics Platform Buyer
To help you further in this process, we've compiled a list of questions that can guide you in selecting the best engineering metrics platform.
End-to-end visibility
- Can I link tickets with PRs and measure my software delivery from ticket creation to deployment in production?
- Can I break down every step of my software delivery?
- Can I measure all four DORA metrics?
- Are the following metrics supported?
- Lead Time
- PR Cycle Time
- Work-in-progress or Coding Time
- Review Time
- Merge Time
- Release Time
- Deploy Time
- Release Frequency
- Released PRs
- Deploy Frequency per Environment
- Lead Time for Changes
- Change failure rate
- Pull request sizes
- Cycle Time per Pull Request Size
- Review Time per Pull Request Size
- Mean Time To Restore - MTTR by priority
- Mean Time To Acknowledge - MTTA
- Mean Time To Repair
- CI run times
- CI success ratio per repository
- CI flakiness per repository
- Bugs raised and fixed per priority
- Issues and Pull Request throughput per Work Type based on issue types or labels. Broken down by teams
- Tickets resolved by ticket labels
- Tickets resolved per epic
Accuracy & Relevancy
- Are the metrics provided focusing on teams and not individual developers to ensure great developer experience and avoid a Big Brother culture?
- Does the solution go beyond surface-level data and can I drill down into leading indicators of high-level metrics to understand where my bottlenecks are?
- Is the data using a P95 model to ensure outliers do not impact averages?
- Can I compare and filter metrics by teams, timeframe, and repositories?
- Can I break down metrics by monorepos, components, services, or folders?
- Can I easily configure the release and deployment settings to match my workflow? Can I use tags, events, or branch merges?
- Can I compare metrics based on Epics?
- Can I compare metrics between different teams?
Actionability
- Can I create custom dashboards for specific teams or groups?
- Can teams set and track their own goals?
- Are goals templates available?
- Can you set percentage targets for goals?
Integrations
- Does it integrate with my VCS (Github, Gitlab, Bitbucket etc.)?
- Does it integrate with our ITS (Jira, Linear, Shortcut etc.)?
- Does it integrate with the rest of our stack?
Usability
- Are engineering metrics dashboards coming out of the box?
- Is it easy to get started and configured?
- Is the learning curve easy for my team?
- Can I pull my organizational structure and teams automatically?
- Can I easily manage my teams as the org scales?
- Does it automatically keep teams in sync with other tools?
- Can I easily navigate through the platform?
- Can I share dashboards with my reports?
Support
- Is there any support during the implementation phase?
- Is there any support after the implementation?
- Is the support comprised of experienced engineering leaders that are incentivized to help our engineering org improve?
- Is the support channel easy for our team? (ex: Slack)
- Are there support SLAs?
Security
- Can it work without storing source code?
- Is the company following security & regulatory compliance (SOC2, GDPR etc.)?
- Is there encryption in transit and at rest?
- Is customer data confidential and isolated?
- Is the company subject to regular pentests?
Selecting the right engineering metrics platform for your organization is critical for driving improvements and showcasing your team's value. Use the criteria and questions provided in this guide to make an informed decision, and consider exploring how Athenian can help you achieve your goals.In today's competitive software development landscape, having the right tools is crucial for your engineering teams to perform at their best. One such essential tool is an engineering metrics platform.
