A Guide to Threat Modelling for Developers

Start from the technology

The first recommendation is to focus primarily on technical rather than broad threats, at least at first.

• Broad threats and threat sources include hacker groups, bad actors, disillusioned employees, human error or epidemics of new worm-like malware. These kinds of causes emerge from the world at large and are extremely various, uncertain and unpredictable. They are relative to the value of your system's data and services to your organisation and to others. These are the kinds of dramatic risks it is easy to talk about with non-technical folks.
• Technical threats and vulnerabilities are much more granular, such as particular weaknesses in software or missing security controls such as encryption or authorisation. These kinds of threats emerge from the structure and data-flow inherent in the system your team is building. Usually a bunch of technical threats combine together to allow a broad threat to impact your system.

By following this guide you will mainly focus on finding technical threats. This helps simplify the elaboration process, because the structure and data-flow of your system is something about which you can be certain. But it also means you can start from your existing strengths as a software developer, understanding technical stuff. This is a much stronger ground to start on than high-level risk analysis of threat sources, about which you may know little.

Don't forget about the bigger picture entirely though. A pragmatic and risk-based understanding of what broad threats are possible helps prioritise one technical threat over another. For example simple human error is usually much more likely than a nation state attack (see sidebar). That thinking can go into selecting what security scope to start examining first. When you focus on identifying technical threats first, it is then much easier to relate them back to broader threats that justify fixes and additional controls.

Take a collaborative approach

The second recommendation is to adopt a collaborative, team based approach. Identifying security requirements is not easy, and a diversity of perspectives will lead to better decision making. There will always be another vulnerability or technical threat to find, so bringing a wide variety of perspectives to the exercise makes brainstorming more robust. It also increases the likelihood you will identify the most important threats. Threat modelling in a group helps address risk holistically and helps the whole team to learn how to think and talk effectively about security.

Getting product owners involved is a great oppotunity from a risk management perspective. Product owners have insights into user behaviour and business context that software developers simply lack. They should know about the value of particular services to the business and the impact if that data was exposed or lost. When cyber security losses occur they are business losses. If the worst does happen then the causes will likely be particular to your organisation and the technology you are using. The cyber security problem is not just about ticking technical boxes, its about making good investment decisions to protect the business.

Threat modelling 'little and often'

The third recommendation is to start threat modelling 'little and often'. Each threat modelling session with the team should be short and focussed enough to be quickly digested into something that can be delivered. Start by analysing the thinnest slice of your system possible; just what you are working on right now. Rather than trying to analyse your entire system upfront, build your team's muscle memory with threat modelling a little bit at a time.

Practises which require a completely specified software design do not match how agile teams work. There is no reason why threat modelling needs to be an exhaustive upfront analysis. Too often teams are overwhelmed by comprehensive and highly structured approaches to threat modelling1. I have seen teams try such approaches and run out of time and patience before any real threats were identified- let alone fixes delivered!

Rather than creating and maintaining an exhaustive 'threat model' document, do threat modelling 'little and often'. When you work this way each threat modelling session is tiny, having little impact. Yet the cumulative effect of doing them has a huge impact. When you know you'll be doing this again every iteration, there's less incentive to try to do everything at once and more incentive to prioritize the most important work right now.

The three key questions

Understanding the simple structure of a threat modelling session, and doing a little bit of planning goes a long way to getting a great result.

The first thing to introduce is a simple and flexible structure for threat modelling 2. This is based on three key questions. It helps to commit this structure to memory. You can use the three question structure as a guide whenever you need to assess threats. Like riding a bike, once you have mastered the basics you will be able to apply and grow those skills.

This guide follows the three question structure. In each threat modelling session, you should spend about a third of your time answering each question. Then you will come out with a useful result. The rest of the guide will break this basic structure into more detailed steps, pointers and explanations to help you run successful threat modelling sessions.

Practical considerations

There are some things you need to get straight before you run a threat modelling session. The following pointers should help you plan.

Running sessions face-to-face vs. running remotely

A face-to-face threat modelling session could happen in a meeting room, or more informally in the team's normal work area- if you have space. A typical session involves drawing a diagram to explain and explore the scope, brainstorming threats, then prioritising fixes for the backlog. However, a face-to-face session is not always possible.

When you run a session remotely you just need to plan a little differently so everyone can participate virtually. You will need video conferencing and collaboration tools. Agree and setup these tools ahead of time. Teams at Thoughtworks have had success with a variety of tools, including Mural, Miro, Google Jamboard and Google Docs.

Get accustomed to your tools ahead of the session and get participants to test they have access. Whichever tool you choose, ensure you have security approval from your organisation to use the tool. Threat modelling outputs represent sensitive information for a number of reasons and must be protected.

Here are some more pointers to bear in mind when working remotely:

• It can help to create diagrams asyncronously before the exercise. This is because it can consumes a good amount of time to draw diagrams on virtual boards.
• Pay even more attention to creating a common understanding of the concepts and symbols you are using to illustrate the system. Explain diagram symbols, data flow arrows and colours of digital stickies.
• Be more intentional to ensure everyone is engaged in the exercise. Perhaps use some security related brain teasers as an ice-breaker. Refer to broader guides on remote facilitation.
• If you have a large group of people, it may make sense to create smaller groups and then consolidate the output. A couple of small sessions is better and more sustainable than one big session.
• You will need more breaks than required in a face-to-face session. Remote work is tiring.

Regardless if your session is remote or face-to-face, you should aim to finish on time. And with some concrete outcomes! This needs discipline - facilitating timings could be a role taken by a delivery manager or someone experienced making sure workshop sessions succeed.

Mona Fenzl and Sarah Schmid from Thoughtworks Germany have had some success using a collaboration tool called Mural. They used it to create a threat modelling template to help other teams get started with that tool.

Scoping the session

Deciding the right focus and level of detail for your session is called 'identifying the scope'. Make sure you have decided this ahead of getting people together to perform the activity. Be guided by what has most value right now. Perhaps its simply the user stories you are working on this iteration?

Do not try and bite off too much scope at once! If you try threat modelling the entire system at once, either you will make no findings in the time available or you will overrun dramatically and there will be no appetite or budget to do it again. It is much better to timebox threat modelling into manageable chunks, performing the activity 'little and often'.

Here are some examples of scopes which have worked well:

• Scope in current iteration.
• An upcoming security sensitive feature, such as a new user registration flow.
• The continuous delivery pipeline and delivery infrastructure.
• A particular microservice and its collaborating services
• A high level overview of a system to identify security tech debt.

Whatever scope your team chooses, make sure it is not too big for you to cover in the time available.

A worked example: scoping

The rest of the guide uses a real feature to show the concrete steps involved in threat modelling. There is a development team at a retail organisation which is building a platform to sell groceries for home delivery. Here is the epic they have in the upcoming sprint:

As a customer, I need a page where I can see my customer details, So that I can confirm they are correct

If you have ever used an online store, you will be able to imagine a page which is used to update address details and perhaps view a loyalty card balance.

From experience, a feature of this size is a pretty reasonable scope for threat modelling session.

Draw a 'lo-fi' technical diagram

Drawing a picture gets everyone on the same page. Before you can start thinking of threats, risk and mitigations you need a shared technical understanding of the software or infrastructure you are dealing with.

A worked example: Lo-fi diagram

Returning to the real feature introduced above, lets see how the team chose to illustrate the new 'customer details page' functionality. They drew the following diagram.

It illustrates an understanding that the system:

• is based on a microservice architecture
• already has an identity provider in place which allows the customer to authenticate.
• has a backend service for customer details (which is written in Java)
• has a Backend for Frontend (BFF) service and frontend UI (which are written in Javascript and React)
• has users who are customers, and want to edit their profile

No detailed knowledge of these technologies is required to follow this guide, but these facts illustrate the level of detail you should discuss while drawing the diagram.

Show how data flows

It is important to add details to show how data flows around your system.

A worked example: Show Data-flow

In our case arrows are added to illustrate data-flows from the customer who wants to see their details, to the UI, then on to the BFF service and onto the customer details resource server to obtain or update the data. There is also a data-flow to the Identity server which issues a token that authorises the session.

There is also an authorisation boundary because the UI is on the Internet, whereas the other components are within the organisation's cloud hosting.

A worked example: Show your assets

The team identified the personally identifable information (PII) stored by the Customer Service and the credential store in the Identity provider as the assets with most business value.

Use STRIDE to help

If your team is beginning with threat modelling, STRIDE is perfect. STRIDE is a very light framework that gives you a head-start brainstorming security threats. It is a mnemonic, where each letter refers to a security concept. The point is not about categorising what you find, but helping you brainstorm effectively.

Invest some time understanding and discussing each of the six security concepts with the team before you start. To help learn, Thoughtworks has created a set of STRIDE cue cards. These six cards are linked immediately below. They also include lists of examples on the reverse side.

On the Internet, nobody knows you are a dog

Can the data overflow and become instructions?

If there's no evidence, its easy to deny it happened

Who else could be looking?

Could the service be taken down?

How easy is it to circumvent protections?

Evil brainstorming

Coming up with ways to attack, break or frustrate a particular bit of software is threat modelling at its essence. It can also be great fun!

A worked example: Threats identified

Using the diagram created earlier, the team brainstormed around each data-flow in the system, using STRIDE to help.

As we discover potential threats in the mechanisms of the software, write them on stickies and annotate the lo-fi diagram.

Here's what they had come up with, at the end of the brainstorming part of the session:

Data-flow

Threat

Customer→Identity Service

• authentication is password based, no two-factor authentication

Customer→UI

• a DOM based cross-site scripting (XSS) attack

UI→Bff

• absence/weakness of identity token validation
• injection attack, such as SQL injection, stored XSS
• lack of logging of identity of caller
• badly configured TLS transport encryption
• misconfigured graphQL introspection enabled in production
• network layer flood of traffic from a botnet
• failure to prevent authenticated user from accessing someone else's details
• lack of regular patching could lead to remote code execution

Bff→Customer Service

• absence or weakness of 'server to server' authentication
• lack of logging of identity of caller
• overly permissive security groups allows customer service to be accessed from Internet
• failure to prevent authenticated user from accessing someone else's details

Notice that many of the threats occurred where the data-flow crossed the authorisation boundary from the Internet into the system. However threats were identified in the browser-based UI and within the backend network also.

Prioritise threats by risk

i. Share knowledge useful for prioritisation

It can help to get a shared understanding of what we know about the system's risk profile. Spend no more than 5 minutes on this. Roughly, there are two types of knowledge that is helpful when prioritising threats. Product owners or security teams often have excellent insights to share.

• Business value - what kind of losses put the organisation's objectives into jeapody? Is it having the customer database stolen? Reputational impact due to lost business?
• Broader threat - what are the likely root causes of losses due to cyber security issues for this organisation? Are we worried about fraud? Malicious insiders? Particularly capable hackers?

If nobody in the group has any insight into broader risk, then that is OK too. Access to this knowledge is not a pre-requisite. You can prioritise based on technical risk alone and still get a big benefit from threat modelling.

ii. Everyone vote for top riskiest threats!

This should be familiar to anyone who has done 'dot voting' in a retrospective or workshop before. Everyone gets some votes and casts them for the riskiest threats. Perhaps start with three votes each for your first session. But it can depend on how many people are in the group and how many threats you found. The goal is to whittle down to the most valuable threats to start with. Remember that risk is not just about how probable the threat is, but also the scale of potential loss.

Everyone casts their votes using dots, according to their own perception of the risk.

Everyone needs to cast their votes by marking dots on the sticky notes. Everyone gets a fixed number of votes. It is fine to vote on the same threat more than once if you think that is right.

Voting in this way will yield good risk decisions for low investment, reflecting the diverse perspectives in the group. People have an intuitive sense of risk and will be able to cast their votes with minimal prompting.

iii. Identify the top riskiest threats

You will need to count up the number of votes for each threat, and mark those with the highest votes somehow. Perhaps circle the riskiest threats with a pen.

Often I am asked how many threats we should seek to identify in a session. The first time, three can be a good number. That provides a good balance for the amount of time invested. But use your judgement and experiment. There may be one very risky item- right now it makes sense just to address that. Equally four or five threats can emerge from a single session.

iv. Take a photo of the annotated diagram, and record threats

Take a photograph using a mobile phone camera at this stage to capture the output of the brainstorming and prioritisation. If you are working with remote tools you can take a screenshot or an export. It makes a lot of sense to upload the image to your Wiki or store them in a repository somewhere.

Add fixes to your backlog

It is essential the time investment by the team leads to followup work. Software teams already have a powerful way to represent and sequence activity to deliver software: the backlog. It is now time to work out what fixes you actually need to reduce risk. Use the team's existing processes and ways of working to support this (see sidebar).

Reflect and keep improving

Feedback and continuous improvement is central to managing risk. Neither the systems we build nor the threats they face are simple, as I stressed at the start of this guide. And every team is different- with different skills, tools, constraints and personalities. There is no single way to threat model, this guide simply provides some basics to get you started. Much like test-driven development or continuous delivery, threat modelling rewards investment.

One way to improve is to perform a retrospective on your threat modelling efforts, once you have run a few sessions. Ask what went well and what could be improved. Is the timing right? Was the scope too granular? Not granular enough? What about the location or remote tools you have used? What issues cropped up after the session? How long did the scope take to deliver? By asking such questions, the team will adapt and build mastery over time, doubling down on what works and discarding what adds little value.

Over time you can grow a practise best suited to your team or organisation. Here are just a few ideas for next steps:

• Experiment with different types of diagram. For something like an OAUTH2 authentication flow, something like a UML sequence diagram might be better than a simple component diagram.
• Use domain specific threat libraries. There are many resources that can help you brainstorm threats. OWASP has some great resources around Mobile or APIs. There has been lots of recent interest in using the ATT&CK framework.
• No practise is a silver-bullet. Threat modelling is not an efficient way to find basic coding or dependency issues. Complement threat modelling using automated tools in the software delivery pipeline.
• Like everything else in software, if it isn't tested then you can't consider it done. Use threat modelling to discover the failure conditions to test for.
• Run a session on your system's risk profile. This kind of analysis is often focussed on the type of data you are processing and the value of your services to others- in security jargon, take a deep-dive into the business value of your 'assets'.
• Run a session on broader threats to your system. It is common to analyse the capabilities and motivations of potential attackers based on the best available information, for example threat intelligence, documentation of real attack techniques and real incidents from similar systems.
• Join threat modelling communities such as the Threat Modelling channel on OWASP Slack, or follow the Threat Modelling SubReddit. Follow other folks doing threat modelling on Twitter.

In conclusion

Many 'solutions' in security seem designed to keep security out of the hands of developers. That does not make them bad solutions. Automated checks in the pipeline are effective at finding vulnerabilties- you should use them. Penetration tests can find issues you would not yourself. Platforms with secure defaults can eliminate many common threats. However each 'solution' only addresses a limited class of threat. Whereas cyber-risk is not a simple thing, it is multi-facated and in constant flux. Understanding this risk is at the core of effective approaches to managing it.

The killer application of Threat Modelling is promoting security understanding across the whole team. This is the first step to making security everyone's responsibility. Just like business outcomes, quality, integration and infrastructure - security can be central to how teams think about software delivery. Rather than something bolted on at the end. In my experience, a team with the muscle memory of threat modelling is a team that proactively manages cyber-risk.

I remember being told that Threat Modelling was just too hard for most development teams. I just wasn't willing to accept that was the case. I understood the chance to improve how developers address security. Since then I have helped loads of teams get started with Threat Modelling and their journey to understand security properly. I have never come across a team who found threat modelling or security too hard. Particularly when explained in an accessible way. That is exactly what I have tried to do in this guide.

I believe threat modelling is a transformative practise for software development teams. A collaborative and risk-based practise which can be applied continuously. If you are part of a software team, please do send this article to your team and suggest a threat modelling session. Or forward it to whoever is responsible for software security at your organisation. By setting aside a little time to run a session you can get started. By starting to understand the threats, risk and security fixes needed for your system, you are taking a step closer to effective cyber security.

I hope this guide helps your team to start threat modelling. At the very least a session should provide value straight away- with good security stories and acceptance criteria added to your backlog. This method has helped many development teams at Thoughtworks' clients adopt an holistic approach to cyber security. I hope it is useful to you also.