We all know that engineering metrics are essential in guiding our decisions.
As engineering leaders, we're responsible for product development and the engineering team's experience. Athenian provides dozens of relevant data points for engineering leaders, but figuring out where to start can be challenging.
These are the top 9 metrics we recommend looking at, so you can pick 2 or 3 to start monitoring and discussing with your engineering teams.
Once you've picked your metrics, here's how you can use them to set your engineering org. up for success.
But before that, let's dive in!
The time between a code change and it being run in production.
Athenian calls this metric PR Cycle Time. The data is obtained from GitHub and customers' CI/CD system to understand how work is flowing. This data is then enriched with ticketing information.
Athenian mainly looks at how work flows, not how developers report it in the ticketing system. This gives customers full visibility of the entire development pipeline: from the first commit to code deployed in production without disrupting development teams.
Athenian brings visibility of customer PR Cycle Time and provides deep visibility into the stages the code goes through so teams can identify bottlenecks.
Improve delivery speed by identifying bottlenecks.
The number of deployments to production.
By having a customer deployment system calling the Athenian API, we can obtain the deployment frequency to production or any relevant environment for the engineering teams (e.g., staging environment).
Improve agility by identifying and increasing deployment throughput.
The percentage of failed deployments to production.
Customers can notify the Athenian API every time they perform a deployment, allowing Athenian to report the success ratio of deployment activities. Athenian presents the success ratio instead of the failure rate.
Increase deployment quality by monitoring the percentage of deployments that are not successful.
The mean time to resolve an incident in production.
Athenian counts the time it takes for customer bugs to be acknowledged by teams (MTTA) and sums it with the time it takes the teams to solve issues (MTTRepair).
These two measurements allow customers to have clear visibility of their SLOs from the moment issues are reported into the ticketing tool until they are solved and deployed to production.
To accurately do this Athenian combines information from the ticketing system, GitHub, and the customer CI/CD system.
Increase agility by decreasing response time to resolve issues.
Average time to run test suit.
All GitHub actions are tracked to let customers know the average run time of their tests and other CI activities. You can drill down this information to identify improvement points in the CI system. For example, investigate the build run time to optimize it.
Accelerate delivery by identifying CI bottlenecks.
The flakiness and success ratio of test suit.
Observe the results of test runs based on GitHub actions to understand the effectiveness of the running tests. Increase customer release confidence by understanding how reliable tests are (flakiness) and how many problems are caught before a release (failing checks).
Speed up releases by monitoring quality of code.
The percentage of bugs solved versus bugs identified.
Athenian tracks all bugs submitted into the customer issue tracking system over time. This way, teams can understand how the bug backlog has been evolving, allowing them to properly manage the quality of the product.
Ensure product quality by keeping bug ratio under control.
The average size of the changes made to code.
Athenian measures the average PR size to help customers identify patterns of complex code changes, which typically increase deployment risk. Code complexity analysis is done by obtaining data from GitHub to understand, per team, what were the biggest changes made on the customer code.
Reduce deployment risk by identifying large changes in the code base.
The distribution of time invested by teams.
Athenian tracks all work reported in the issue tracking system or through PRs, to help customers understand where their teams are investing more effort.
Customers can understand where teams are investing more effort by defining customizable work categories to increase the visibility of how different activities are moving forward.
In addition, Athenian allows you to categorize the same items across different views to increase visibility. For example, obtain all investment on bugs while identifying the distribution of customer-facing bugs and bugs discovered internally.
Improve decisions by having clear visibility of team investment levels.
Save it, share it, print it, stick in on your office wall (or fridge, if you're WFH).