Measuring Developer Productivity via Humans

What is a qualitative metric?

We define a qualitative metric as a measurement comprised of data provided by humans. This is a practical definition – we haven’t found a singular definition within the social sciences, and the alternative definitions we’ve seen have flaws that we discuss later in this section.

Figure 1: Qualitative metrics are measurements derived from humans

The definition of the word “metric” is unambiguous. The term “qualitative,” however, has no authoritative definition as noted in the 2019 journal paper What is Qualitative in Qualitative Research:

There are many definitions of qualitative research, but if we look for a definition that addresses its distinctive feature of being “qualitative,” the literature across the broad field of social science is meager. The main reason behind this article lies in the paradox, which, to put it bluntly, is that researchers act as if they know what it is, but they cannot formulate a coherent definition.

An alternate definition we’ve heard is that qualitative metrics measure quality, while quantitative metrics measure quantity. We’ve found this definition problematic for two reasons: first, the term “qualitative metric” includes the term metric, which implies that the output is a quantity (i.e., a measurement). Second, quality is typically measured through ordinal scales that are translated into numerical values and scores – which again, contradicts the definition.

Another argument we have heard is that the output of sentiment analysis is quantitative because the analysis results in numbers. While we agree that the data resulting from sentiment analysis is quantitative, based on our original definition this is still a qualitative metric (i.e., a quantity produced qualitatively) unless one were to take the position that “qualitative metric” is altogether an oxymoron.

Aside from the problem of defining what a qualitative metric is, we’ve also encountered problematic colloquialisms. One example is the term “soft metric”. We caution against this phrase because it harmfully and incorrectly implies that data collected from humans is weaker than “hard metrics” collected from systems. We also discourage the term “subjective metrics” because it misconstrues the fact that data collected from humans can be either objective or subjective – as we discuss in the next section.

Later in this article we provide guidance on how to collect and use these measurements, but first we’ll provide a real-world example of this approach put to practice

Peloton is an American technology company whose developer productivity measurement strategy centers around qualitative metrics. To collect qualitative metrics, their organization runs a semi-annual developer experience survey led by their Tech Enablement & Developer Experience team, which is part of their Product Operations organization.

Thansha Sadacharam, head of tech learning and insights, explains: “I very strongly believe, and I think a lot of our engineers also really appreciate this, that engineers aren't robots, they're humans. And just looking at basic numbers doesn't drive the whole story. So for us, having a really comprehensive survey that helped us understand that entire developer experience was really important.”

Each survey is sent to a random sample of roughly half of their developers. With this approach, individual developers only need to participate in one survey per year, minimizing the overall time spent on filling out surveys while still providing a statistically significant representative set of data results. The Tech Enablement & Developer Experience team is also responsible for analyzing and sharing the findings from their surveys with leaders across the organization.

For more on Peloton’s developer experience survey, listen to this interview with Thansha Sadacharam.

Advocating for qualitative metrics

Executives are often skeptical about the reliability or usefulness of qualitative metrics. Even highly scientific organizations like Google have had to overcome these biases. Engineering leaders are inclined toward system metrics since they are accustomed to working with telemetry data for inspecting systems. However, we cannot rely on this same approach for measuring people.

We’ve seen some organizations get into an internal “battle of the metrics” which is not a good use of time or energy. Our advice for champions is to avoid pitting qualitative and quantitative metrics against each other as an either/or. It’s better to make the argument that they are complementary tools – as we cover at the end of this article.

We’ve found that the underlying cause of opposition to qualitative data are misconceptions which we address below. Later in this article, we outline the distinct benefits of self-reported data such as its ability to measure intangibles and surface critical context.

The two types of qualitative metrics

There are two key types of qualitative metrics:

1. Attitudinal metrics capture subjective feelings, opinions, or attitudes toward a specific subject. An example of an attitudinal measure would be the numeric value captured in response to the question: “How satisfied are you with your IDE, on a scale of 1-10?”.
2. Behavioral metrics capture objective facts or events pertaining to an individuals’ work experiences. An example of a behavioral measure would be the quantity captured in response to the question: “How long does it take for you to deploy a change to production?”

We’ve found that most tech practitioners overlook behavioral measures when thinking about qualitative metrics. This occurs despite the prevalence of qualitative behavioral measures in software research, such as the Google’s DORA program mentioned earlier.

DORA publishes annual benchmarks for metrics such as lead time for changes, deployment frequency, and change fail rate. Unbeknownst to many, DORA’s benchmarks are captured using qualitative methods with the survey items shown below:

We’ve found that the ability to collect attitudinal and behavioral data at the same time is a powerful benefit of qualitative measurement.

For example, behavioral data might show you that your release process is fast and efficient. But only attitudinal data could tell you whether it is smooth and painless, which has important implications for developer burnout and retention.

To use a non-tech analogy: imagine you are feeling sick and visit a doctor. The doctor takes your blood pressure, your temperature, your heart rate, and they say “Well, it looks like you’re all good. There’s nothing wrong with you.” You would be taken aback! You'd say, “Wait, I’m telling you that something feels wrong.”

The benefits of qualitative metrics

One argument for qualitative metrics is that they avoid subjecting developers to the feeling of “being measured” by management. While we’ve found this to be true – especially when compared to metrics derived from developers’ Git or Jira data – it doesn’t address the main objective benefits that qualitative approaches can provide.

There are three main benefits of qualitative metrics when it comes to measuring developer productivity:

How to capture qualitative metrics

Many tech practitioners don’t realize how difficult it is to write good survey questions and design good survey instruments. In fact, there are whole fields of study related to this, such as psychometrics and industrial psychology. It is important to bring or build expertise here when possible.

Below are few good rules for writing surveys to avoid the most common mistakes we see organizations make:

• Survey items need to be carefully worded and every question should only ask one thing.
• If you want to compare results between surveys, be careful about changing the wording of questions such that you’re measuring something different.
• If you change any wording, you must do rigorous statistical tests.

In survey parlance, ”good surveys” means “valid and reliable” or “demonstrating good psychometric properties.” Validity is the degree to which a survey item actually measures the construct you desire to measure. Reliability is the degree to which a survey item produces consistent results from your population and over time.

One way of thinking about survey design that we’ve found helpful to tech practitioners: think of the survey response process as an algorithm that takes place in the human mind.

When an individual is presented a survey question, a series of mental steps take place in order to arrive at a response. The model below is from the seminal 2012 book, The Psychology of Survey Response:

Decomposing the survey response process and inspecting each step can help us refine our inputs to produce more accurate survey results. Developing good survey items requires rigorous design, testing, and analysis – just like the process of designing software!

But good survey design is just one aspect of running successful surveys. Additional challenges include participation rates, data analysis, and knowing how to act on data. Below are some of the best practices we’ve learned.

Using qualitative and quantitative metrics together

Qualitative metrics and quantitative metrics are complementary approaches to measuring developer productivity. Qualitative metrics, derived from surveys, provide a holistic view of productivity that includes both subjective and objective measurements. Quantitative metrics, on the other hand, provide distinct advantages as well:

• Precision. Humans can tell you whether their CI/CD builds are generally fast or slow (i.e., whether durations are closer to a minute or an hour), but they cannot report on build times down to millisecond precision. Quantitative metrics are needed when a high degree of precision is needed in our measurements.
• Continuity. Typically, the frequency at which an organization can survey their developers is at most once or twice per quarter. In order to collect more frequent or continuous metrics, organizations must gather data systematically.

Ultimately, it is through the combination of qualitative and quantitative metrics – a mixed-methods approach – that organizations can gain maximum visibility into the productivity and experience of developers. So how do you use qualitative and quantitative metrics together?

We’ve seen organizations find success when they start with qualitative metrics to establish baselines and determine where to focus. Then, follow with quantitative metrics to help drill in deeper into specific areas.

Engineering leaders find this approach to be effective because qualitative metrics provide a holistic view and context, providing wide understanding of potential opportunities. Quantitative metrics, on the other hand, are typically only available for a narrower set of the software delivery process.

Google similarly advises its engineering leaders to go to survey data first before looking at logs data for this reason. Google engineering researcher Ciera Jaspan explains: “We encourage leaders to go to the survey data first, because if you only look at logs data it doesn't really tell you whether something is good or bad. For example, we have a metric that tracks the time to make a change, but that number is useless by itself. You don't know, is this a good thing? Is it a bad thing? Do we have a problem?”.

A mixed methods approach allows us to take advantage of the benefits of both qualitative and quantitative metrics while getting a full understand of developer productivity:

1. Start with qualitative data to identify your top opportunities
2. Once you know what you want to improve, use quantitative metrics to drill-in further
3. Track your progress using both qualitative and quantitative metrics

It is only by combining as much data as possible – both qualitative and quantitative – that organizations can begin to build a full understanding of developer productivity.

In the end, however, it’s important to remember: organizations spend a lot on highly qualified humans that can observe and detect problems that log-based metrics can’t. By tapping into the minds and voices of developers, organizations can unlock insights previously seen as impossible.