search

Code Telemetry FAQs

How to Understand AI Written Code, With the Right Expectations

hashtag
Why Code Telemetry exists

AI assisted coding is now part of everyday development, but most teams struggle to answer one core question with confidence:

How much of the code that actually ships is written by AI, and what does that mean for delivery?

Code Telemetry exists to give teams a clear and practical signal to reason about AI usage over time. It is not designed to prove authorship or establish facts. It is designed to help teams see patterns and trends in shipped code.

hashtag
What Code Telemetry shows, and what it intentionally does not

Code Telemetry provides a directional estimate of:

  • Approximately how much merged code is AI written

  • Approximately how much is human written

This analysis is done on pull request diffs of code that is merged and shipped, not on editor activity or AI suggestions.

Just as important is what Code Telemetry does not claim.

It is not:

  • A factual determination of who wrote a specific line of code

  • A developer monitoring or enforcement tool

  • A benchmarking metric across teams or organizations

  • A one to one causal measure

AI models themselves are probabilistic and opaque. Any system analyzing their output must work on signals, not certainty. Code Telemetry is designed with that reality upfront, not hidden behind precise looking numbers.

hashtag
Confidence Score and what it really means

Alongside AI and human percentages, Code Telemetry surfaces a confidence score.

This score reflects how confident our algorithm is in the classification it has made for a given PR diff.

  • Higher confidence means stronger and clearer authorship signals were detected

  • Lower confidence means signals were weaker, mixed, or ambiguous

Even at very high confidence, the result represents our algorithm’s best judgement, not an absolute truth. We show confidence explicitly so teams can understand reliability instead of assuming certainty.

hashtag
How Code Telemetry differs from Cursor or Copilot metrics

Many tools like Cursor or Copilot provide AI usage metrics, but they answer a different question.

IDE level metrics:

  • Track AI usage inside a specific tool

  • Measure suggestions generated or accepted

  • Only capture activity within that environment

  • Do not account for copy paste or code generated elsewhere

Code Telemetry is different by design:

  • It is tool agnostic

  • It does not depend on IDE or assistant telemetry, and it does not assess which LLM or model is used to generate code

  • It analyzes the code that actually reaches production

  • It captures AI usage regardless of how or where the code was generated

In simple terms, IDE metrics show AI usage in the editor, while Code Telemetry shows AI impact in shipped code.

Over time, we plan to incorporate available IDE level signals, such as Cursor’s own metrics, to further improve accuracy. Code Telemetry will always remain independent and focused on production reality.

hashtag
How to read Code Telemetry correctly

The right way to use Code Telemetry is to look at trends over time, not isolated numbers.

It helps answer questions like:

  • Is AI usage increasing or stabilizing across teams?

  • How does increased AI usage align with delivery metrics?

  • Is AI adoption translating into faster or smoother shipping?

Example:

A team moves from:

  • 7 day cycle time with ~20 percent AI written code

To:

  • 5 day cycle time with ~40 percent AI written code over a few months

This does not prove causation. But it is a credible directional signal that increased AI usage may be contributing to faster delivery.

That is the level of insight Code Telemetry is designed to support.

hashtag
How not to use Code Telemetry

To avoid misinterpretation, Code Telemetry should not be used to:

  • Compare teams against benchmarks

  • Rank individuals or teams

  • Draw direct cause and effect conclusions

  • Make decisions based on single data points

Its value comes from patterns, not precision.

hashtag
Code Telemetry and AI Impact

Code Telemetry forms the foundation for the AI Impact feature.

Once we understand how much AI written code is actually shipped, AI Impact analyzes how that usage aligns with downstream engineering metrics such as:

  • Cycle time

  • PR throughput

  • Review velocity

  • Commit frequency

AI Impact does not assume AI is good or bad. It simply shows how AI usage in shipped code corresponds with changes in engineering outcomes.

hashtag
In summary

  • Code Telemetry is a directional signal, not a factual claim

  • It works on shipped code, not IDE activity

  • Confidence scores add transparency, not certainty

  • It is best used to understand trends over time

  • Its real value comes from pairing it with delivery metrics

With the right expectations, Code Telemetry helps teams reason clearly about AI adoption without overclaiming accuracy or intent.

PreviousHivel AI Impact Screen: Overview & InsightsNextBest Practices

Last updated