How to measure impact of Copilots using Hivel?
While AI Copilots like GitHub Copilot don't provide direct data on their usage (e.g., suggestions and acceptances), you can evaluate their impact through Hivel by focusing on key productivity metrics.
Here’s a step-by-step guide:
1. Establish a dedicated Copilot team
Identify a team of developers who will adopt Copilot for their daily workflows.
Ensure the team is diverse in roles and experience levels to gain balanced insights.
Create the team in Hivel.
2. Define key metrics to measure impact
Consider certain metrics around delivery speed, throughput, and quality whose change will indicate the effectivenes of the Copilot on developer's productivity.
These are the metrics that we recommend to be tracked:
Core Metrics
Cycle Time: The time it takes from first commit of a PR till the PR is opened.
Deployment Frequency: How often code is merged to release branch.
Active Days: The number of days developers actively contribute to the codebase.
Commits: The number of commits per developer or team.
Open PRs: The number of PRs opened.
Completed Story Points: Total story points completed within the time period.
Optional Metrics
Review Time: Measure whether Copilot affects the time it takes to review and approve PRs.
Rework and Change Failure Rate: Track any correlation between Copilot usage and code quality.
3. Create a dashboard to track the metrics:
In the custom dashboards, create a private or public dashboard with the metrics and save a filter from when Copilot was introduced.
4. Analyze Metrics After Copilot Adoption
Monitor the Copilot team’s metrics post-adoption to see if the metrics have improved or not.
Subsequently, check the same for other teams to understand if the impact is generally across the org or only for the Copilot team.
Utilize the tabular view to get a quick snapshot of the same.
Key Questions to Ask:
Has cycle time improved?
Is deployment frequency higher?
Are there changes in the number of active days or contributions (e.g., commits, PRs)?
Are story points completed more consistently or at a faster rate?
5. Interpret the Results
Look for trends or significant changes in the metrics:
Positive Indicators: Faster cycle times, higher deployment frequency, and increased completed story points suggest that Copilot is helping developers ship faster and more effectively.
A meaningful performance gap favoring the Copilot team over the control group strengthens the case for its impact.
By following these steps, you can derive actionable insights into how Copilot contributes to developer productivity, even without direct usage data. For assistance with setting up metrics dashboards, reach out to us.
Last updated
