Maximizing Developer Effectiveness

Day in the life in a highly effective environment

The developer:

• checks the team project management tool and then attends standup where she is clear about what she has to work on.
• notes that the development environment has been automatically updated with libraries matching development and production, and the CI/CD pipelines are green.
• pulls down the latest code, makes an incremental code change that is quickly validated by deploying to a local environment and by running unit tests.
• depends on another team’s business capabilities for her feature. She is able to find documentation and the API spec through a developer portal. She still has some queries, so she jumps into the team’s Slack room and quickly gets some help from another developer who is doing support.
• focuses on her task for a few hours without any interruptions.
• takes a break, gets coffee, takes a walk, plays some ping pong with colleagues.
• commits the code change, which then passes through a number of automated checks before being deployed to production. Releases the change gradually to users in production, while monitoring business and operational metrics.

The developer is able to make incremental progress in a day, and goes home happy.

Day in the life in a low effective environment

The developer:

• starts the day having to deal immediately with a number of alerts for problems in production.
• checks a number of logging and monitoring systems to find the error report as there are no aggregated logs across systems.
• works with operations on the phone and determines that the alerts are false positives.
• has to wait for a response from architecture, security and governance groups for a previous feature she had completed.
• has a day broken up with many meetings, many of which are status meetings
• notes that a previous feature has been approved by reviewers, she moves it into another branch that kicks off a long nightly E2E test suite that is almost always red, managed by a siloed QA team.
• depends on another team's API, but she cannot find current documentation. So instead she talks to a project manager on the other team, trying to get a query. The ticket to find an answer will take a few days, so this is blocking her current task.

We could go on. But ultimately the developer doesn’t achieve much, leaves frustrated and unmotivated

Developer effectiveness

What does being effective mean? As a developer, it is delivering the maximum value to your customers. It is being able to apply your energy and innovation in the best ways towards the company’s goals. An effective environment makes it easy to put useful, high-quality software into production; and to operate it so that the developers do not have to deal with unnecessary complexities, frivolous churn or long delays — freeing them to concentrate on value-adding tasks.

In the example illustrating a low effective environment, everything takes longer than it should. As a developer, your day is made up of endless blockers and bureaucracy. It is not just one thing; it is many. This is akin to death by a 1,000 cuts. Slowly, productivity is destroyed by small inefficiencies, which have compounding effects. The feeling of inefficiency spreads throughout the organization beyond just engineering. Engineers end up feeling helpless; they are unproductive. And worse they accept it, the way of working becomes an accepted routine defining how development is done. The developers experience a learned helplessness.

Whereas in the organization that provides a highly effective environment, there is a feeling of momentum; everything is easy and efficient, and developers encounter little friction. They spend more time creating value. It is this frictionless environment, and the culture that supports it by fostering the desire and ability to constantly improve, that is the hardest thing for companies to create when they are doing a digital transformation.

Organizations look for ways to measure developer productivity. The common anti-pattern is to look at lines of code, feature output or to put too much focus on trying to spot the underperforming developers. It is better to turn the conversation around to focus on how the organization is providing an effective engineering environment. Being productive motivates developers. Without the friction, they have time to think creatively and apply themselves. If organizations do not do this, then in my experience the best engineers will leave. There is no reason for a developer to work in an ineffective environment when lots of great innovative digital companies are looking to hire strong technical talent.

Let's look at an example of a company that has optimized developer effectiveness.

Case Study: Spotify

Spotify conducted user research among their engineers to better understand developer effectiveness. Through this research, they uncovered two key findings:

1. Fragmentation in the internal tooling. Spotify’s internal infrastructure and tooling was built as small isolated “islands” leading to context switching and cognitive load for engineers.
2. Poor discoverability. Spotify had no central place to find technical information. As information was spread all over, engineers did not even know where to start looking for information.

Spotify's developer experience team describes these problems as a negative flywheel; a vicious cycle where developers are presented with too many unknowns, forcing them to make many decisions in isolation, which in turn compounds fragmentation and duplication of efforts, and ultimately erodes the end-to-end delivery time of products.

Figure 1: Spotify's negative flywheel

To mitigate these complexities, they developed Backstage, an Open Source developer portal with a plugin architecture to help expose all infrastructure products in one place, offering a coherent developer experience and a starting point for engineers to find the information they need.

How to get started?

What I am describing in the highly effective environment is what it feels like to work in a company that has fully embraced a DevOps culture, continuous delivery and product thinking. Very sensibly, most companies are on a journey towards achieving this environment. They have read Accelerate and the State of DevOps report. They know what type of organization they are striving to build. The four key metrics (lead time, deployment frequency, MTTR and change fail percentage) are great measures of DevOps performance.

One way to look at the DevOps measures is that they are lagging indicators. They are useful measurements to understand where you are, and to indicate when there is work to be done to figure out what tangible things the company should do to get better. Ideally, we want to identify leading lower level metrics of effectiveness that are more actionable. There is a correlation to the higher level metrics. It will ladder up. This should also be combined with other sources of research such as surveys on developer satisfaction.

There is an overwhelming amount of good advice, practices, tools, and processes that you should use to improve. It is very hard to know what to do. My research has shown that there are a number of key developer feedback loops. I recommend focusing on optimizing these loops, making them fast and simple. Measure the length of the feedback loop, the constraints, and the resulting outcome. When new tools and techniques are introduced, these metrics can clearly show the degree to which developer effectiveness is improved or at least isn't worse.

Feedback Loops

The key loops I have identified are:

The metrics are based on what I have observed is possible. Not every company needs every feedback loop to be in the high effectiveness bucket, but they provide concrete goals to guide decision-making. Engineering organizations should conduct research within their specific context to figure out what cycles and metrics are important technology strategy.

It is useful to look at what techniques have been applied to optimize the feedback loops and the journey that companies have taken to get there. Those case studies can provide many ideas to apply in your own organization.

Figure 2: Feedback Loops during feature development

The diagram above shows a simplified representation of how developers use feedback loops during development. You can see that the developer validates their work is meeting the specifications and expected standards at multiple points along the way. The key observations to note are:

• Developers will run the feedback loops more often if they are shorter.
• Developers will run more often and take action on the result, if they are seen as valuable to the developer and not purely bureaucratic overhead.
• Getting validation earlier and more often reduces the rework later on.
• Feedback loops that are simple to interpret results, reduce back and forth communications and cognitive overhead.

When organizations fail to achieve these results, the problems are quickly compounded. There is a great deal of wasted effort for the developers. Embodied in the time spent waiting, searching, or trying to understand results. It adds up, causing significant delays in product development, which will manifest as lower scores in the four key metrics (particularly deployment frequency and lead time).

Introducing micro feedback loops

From what I have observed, you have to nail the basics, the things that developers do 10, 100 or 200 times a day. I call them micro-feedback loops. This could be running a unit test while fixing a bug. It could be seeing a code change reflected in your local environment or development environments. It could be refreshing data in your environment. Developers, if empowered, will naturally optimize, but often I find the micro-feedback loops have been neglected. These loops are intentionally short, so you end up dealing with some very small time increments.

Figure 3: Micro-feedback loops compound to affect larger feedback loops.

It is hard to explain to management why we have to focus on such small problems. Why do we have to invest time to optimize a compile stage with a two minute runtime to instead take only 15 seconds? This might be a lot of work, perhaps requiring a system to be decoupled into independent components. It is much easier to understand optimizing something that is taking two days as something worth taking on.

Those two minutes can add up quickly, and could top 100 minutes a day. These small pauses are opportunities to lose context and focus. They are long enough for a developer to get distracted, decide to open an email or go and get a coffee, so that now they are distracted and out of their state of flow, there is research that indicates it can take up to 23 minutes to get back into the state of flow and return to high productivity. I am not suggesting that engineers should not take breaks and clear their head occasionally! But they should do that intentionally, not enforced by the environment.

In reality, developers will compensate by filling these moments of inactivity with useful things. They might have two tasks going on and toggle between them. They might slow their compile frequency by batching up changes. In my research both of these will lead to a delay in integration of code, and development time.

How far do you take optimizing? When is enough? Imagine that now we have that change down to 15 seconds, but we think we can get it to three seconds. Is that worth the investment? It depends on how difficult it is to make that change and the impact it will bring. If you can develop a tool or capability that will speed up 10 teams then it might be worth it. This is where platform thinking, rather than optimizing for individual teams, comes into play.

Distributed systems are a particular challenge. There are many valid reasons for splitting systems into different deployable units (usually microservices). However, distributed systems also make many things difficult (see Microservice Prerequisites), including developer effectiveness. Sometimes teams might optimize for team autonomy or for runtime performance, but they sacrifice developer effectiveness because they do not invest in maintaining fast feedback loops. This is a very common situation my company runs into.

Organizational Effectiveness

Highly effective organizations have designed their engineering organization to optimize for effectiveness and feedback loops. Leadership over time creates a culture that leads to empowering developers to make incremental improvements to these feedback loops.

It starts with a recognition by leadership that technology — and removing friction from development teams — is vital to the business. This is manifested in a number of ways.

Technical leaders continually measure and re-examine effectiveness. Highly effective organizations have created a framework to make data-driven decisions by tracking the four key metrics and other data points important to their context. This culture starts at the executive level and is communicated to the rest of the organization.

In addition to the metrics, they create an open forum to listen to the individual contributors that work in the environment day to day. Developers will know the problems they face and will have many ideas on how to solve them.

Based on this information, engineering managers can decide on priorities for investments. Large problems may require correspondingly large programs of modernization to address a poor developer experience. But often it is more about empowering teams to make continuous improvement.

A key principle is to embrace the developer experience. It is common to see a program of work of a team focused on this. Developer experience means technical capabilities should be built with the same approaches used for end-user product development, applying the same research, prioritization, outcome-based thinking, and consumer feedback mechanisms.

To motivate developers, highly effective organizations franchise; that means developers should have the ability to improve their day to day lives. They have a policy for teams to make incremental technical improvements and manage technical debt. There should be a healthy data-backed discussion between developers and product management. Highly effective organizations also provide the ability for developers to innovate; when their teams have clear goals and a clear idea of bottlenecks, developers can be creative in solving problems. These organizations also create ways for the best ideas to “bubble to the top”, and then double down, using data to evaluate what is best.

After commitment, measurement and empowerment comes scaling.

At a certain organizational size, there is a need to create efficiency through economies of scale. Organizations do this by applying platform thinking - creating an internal platform specifically focused on improving effectiveness. They invest in engineering teams that build technical capabilities to improve developer effectiveness. They regard other development teams as their consumers, and the services they provide are treated like products. The teams have technical product managers and success metrics related to how they are impacting the consuming teams. For example, a platform capability team focused on observability creates monitoring, logging, alerting, and tracing capabilities so that teams can easily monitor their service health and debug problems in their product.

The need for governance is still a priority. However, this does not need to be seen as a dirty word since its application is very different in highly effective organizations. They move away from centralized processes to a lightweight approach. This is about setting and communicating the guardrails, and then nudging teams in the right direction rather than lengthy approval process approaches. Governance can have a critical role in effectiveness when it is implemented via:

• Clear engineering goals
• Specifying ways that teams and services communicate with each other
• Encouraging useful peer review
• Baking best practices into platform capabilities
• Automating control via architecture fitness functions

Essentially, effective organizations shorten the governance feedback loop. I will be expanding on this in a future article.

Case Study: Etsy

Etsy was one of the pioneers of the DevOps movement. Its leaders have worked to embed developer effectiveness into their culture, with the belief that moving quickly is both a technical and a business strategy. They actively measure their ability to put valuable products into production quickly and safely, and will adjust their technical investments to fix any blockers or slowness.

One of Etsy’s key metrics is lead time, which is measured, monitored, and displayed in real-time throughout their offices. When lead time reaches above a certain key threshold, the release engineering team will work to lower it to a reasonable level. Their CTO, Mike Fisher, talks about Etsy engineers being “fearless” to move forward quickly, having a safety net to try new things.

Deploying software fast is only half of the story. To be truly effective that software has to be valuable to consumers. Etsy does this by taking a data-driven approach, with each feature having measurable KPIs.

Code changes go through a set of checks, so that developers have confidence the change meets Etsy’s SLAs for performance, availability, failure rate, etc. Once the change is in production, Etsy’s experimentation platform is able to capture user behavior metrics. Teams use the metrics to iterate on products, optimizing for the associated KPIs and user satisfaction. If eventually the change is proven not to be valuable, it would be cleaned up, thereby avoiding technical debt.

Etsy has a current initiative that prioritizes the developer experience. It has four key pillars:

This policy represents a true commitment from Etsy’s leadership to their developers. They continually verify their effectiveness by tracking metrics including the 4 key metrics, and conducting monthly surveys with developers to capture net promoter scores (NPS).

Conclusion

The beginning of this article speaks to the importance of developer effectiveness and its impact on developer happiness and productivity. I focused on the outcomes developers aim to achieve and not just the tools and techniques. In pushing this examination further we see a series of feedback loops that developers often use while developing a product.

I also spoke to how inefficiencies in micro-feedback loops compound to affect larger indicators such the four key metrics and product development speed. As well as highlighted the importance of developer experience as a principal and how platform thinking will help maximize efficiency and effectiveness at scale.

In the coming series, a deeper dive into developer effectiveness and the individual feedback loops will be further explained through case studies. These will provide concrete details in how organizations have been able to achieve these numbers and the resulting outcomes. In addition to describing the organizational structures and processes that enable these optimizations at a local and a global level.

The next article will start with the smallest micro-feedback loops.