Traffic peaked around 2020

Here's the plot of monthly page views since I started to track traffic in 2011.

We see the overall traffic flow shows a steady increase from 2011 to 2020, followed by a similarly steady decrease. Traffic levels in 2023 were about the same as 2015.

If you look carefully at the plot, you might notice there are points missing in 2019. This is due to a strange phenomenon in the data I got from Google. During a few months in 2019 the traffic of some of the navigation pages mentioned in my navigation bar (eg /agile.html) shot up from around 5000 monthly views to 221,000. It only affected a handful of pages, but a jump that big throws off monthly (and yearly) totals.

Here's the totals for the year.

† due to the strange google phenomenon above, the annual total for 2019 is meaningless

It's interesting to see the overall traffic, but that's only the start, and probably not the most interesting information about what's happening with the site. But to delve further, we need to understand a bit more about the general shape of the website's traffic.

Traffic drops rapidly after publication

The first, important, general pattern about this site (and I suspect most websites) is the traffic pattern for any page after it's released.

The following graph plot shows how many views an average article on the site gets on the days following its publication.

The thing to note here is the extreme drop off from the traffic after the first days. Not just is it an exponential curve, it's an exponential curve when plotted on logarithmic scale.

To plot an “average article”, I'm actually looking at the every article published since 2020, finding the page views for each day after publication, and find the median for each day.

The second thing to note is that things tend to stabilize after about six months.

Traffic is lower at weekends (and Christmas)

You'll notice that I've said weekdays above. The next graph shows why.

Here I've plotted the daily traffic, and color-coded between weekdays and weekends. The graph clearly shows how weekend traffic is much lower than weekday traffic. An example consequence of this is that if I want to compare articles for their post-launch traffic, I need to stick to weekdays, otherwise there will be strange differences between articles published on Tuesdays and articles published on Thursdays.

There's another traffic variation, which isn't obvious in the graphic above, but shows up if I just plot weekday daily traffic

Here we can see a notable drop-off at the end of year, due to the Christmas and new year holidays. This is why monthly figures for December are always lower than the rest of the year.

A small amount of articles get most of the traffic

How much traffic do pages on the site get? To start answering this question, I actually have to ask what counts as a page on the site. If I look at a list traffic counts, I'll see a lot of paths that are requested that don't lead to real pages, such as /articles/colelction-pipeline or /articles/cmljaGFyZH. Those requests are actually a remarkably large proportion of requested pages on the site. In January 2024, 936 or the 2506 paths requested for the site were unknown

Fortunately when it comes to overall traffic analysis, I don't have to worry about it that much, because the amount of requests to these paths is rather smaller

It turns out to be about 0.5%

From that I think it's reasonable to concentrate on the known paths, which in January 2024, numbered some 1570 URLs.

My usual first step when looking at something like is to plot a histogram, so I can get a sense for the frequency distribution of the data, in this case the traffic for each of these pages in the month of January.

What this tells me is that we have a really skewed distribution, skewed even though I'm plotting it here with a square root y axis. I can tell that most pages get very few views, but a few pages get a lot.

That's useful information, but I want a bit more granularity than that. I can do this by plotting the cumulative traffic - which I do by sorting the pages by how many views they get, calculating the cumulative traffic for each page, and making this plot

Each dot on this plot is a single page, towards the right of the plot all the dots squish together into a line. I can use this by following the 10% line from the y axis and seeing that it hits the squished points at roughly 520K page views. What this tells me is that top 10% of pages on the site generated roughly 520K page views. By looking at the initial dots I can see that 150K page views were generated by the top 5 pages.

Seeing this in terms of page views helps us understand the graph, but once we've got the hang of it, I find it more useful to think in terms of proportion of page views.

This is exactly the same curve, but now I read that the top 10% of pages generated just over 80% of the traffic on the site.

I don't know enough about statistics to know what how a real data scientist would react to a plot like this. It's very similar to a Cumulative Distribution Function.

I can also show the main points in a table.

The top pages on the site

Here's a table of the top 10% of pages on the site for 2023

Total is the total number of page views for that path in 2023. Monthly median is the median views for a month. Launch date is the is the date the article was first published. Not all pages have a launch date, often because navigation pages don't really get this kind of launch, but also because some older articles don't have publication dates assigned to them (on my todo list to fix, as I examine the place of old articles). For more on the type see the section below.

Mostly the total and monthly median track each other, but the cases where they don't are interesting, and I'll want to look at those.

Different types of pages

There are a few different types of pages on the site. I can classify them into three broad groups:

• posts: these are the core content of site, full length articles, posts in the bliki section, infodecks, and some older pattern-oriented material.
• navigation page (nav): pages used primarily to help the reader find posts. These are the guide pages (eg /agile.html), including the home page, and the tags pages.
• catalog pages are pages that are summaries of material that isn't on the site, such as the refactoring catalog, books pages, etc.

There are a few pages (eg faq.html that don't quite fit into any of these, for this analysis I'm clumping them in with catalog pages.

The proportion of the overall traffic to the page looks like this.

Most of the traffic to site is to post pages, which are the ones with the real content of the site. But we can get more information by plotting the cumulative frequency curves for each of the three types.

How much does traffic vary with age

I've been working on this site for a long time, starting it in the late 1990's, not long before I joined Thoughtworks. Many of the articles, thus go back a long way. When thinking about the age of pages on the site, it doesn't make much sense to think about navigation pages, so I concentrate on the posts: articles and bliki entries. Here's a plot of posts made each year.

The date information on the very oldest material is rather fuzzy. Some posts, particularly bliki posts, don't have a date in the source. I can figure these out by looking at version control, but before 2004 I was using CVS, and didn't feel motivated to go back to CVS files to figure out the real commit dates, since they didn't survive the transition to Subversion, Mercurial, and git.

Recall the earlier graph that shows how an article's traffic drops exponentially over time. Given this, we would assume that most older articles don't generate as much traffic an newer ones, indeed I've heard anecdotally that this is commonly the case, which is they so many sites bombard us with lots of new material. I've always preferred to concentrate on material with longer lasting value, how does this work in terms of traffic?

One way to assess this is to look at articles that get a lot of traffic. In the past I've formulated “evergreen” articles as articles that have generated at least 1000 monthly pageviews for at least six months in a year. I prefer 6 months of high monthly traffic to a high yearly total, because if an article gets suddenly noticed it can generate lots of page views for just a month or two, but not sustain it.

Here's the above plot, but this time marking out these evergreen articles (in green, of course).

This shows that plenty of older articles get this kind of traffic. But what proportion of traffic goes to older articles? To get a sense of the traffic of individual articles, we can plot each post's median monthly traffic for 2023 against its publication date.

To understand what proportion of traffic goes to older articles, we can make a plot of cumulative traffic.

This traffic analysis is based on traffic for November 2023. I didn't use January 2024, because that month included publishing a rewrite of the continuous integration article, a very popular article originally written in 2006. Before the rewrite it generated a couple of thousand views per month, but when the rewrite was published it generated over thirty thousand views. I felt this would distort the picture. I then didn't use December 2023, since Decembers suffer from lower traffic due to the Christmas and New Year break.

This reads in a similar way to the earlier cumulative plots. By following the 30% proportion line, we can see that 30% of the traffic comes to posts written in the last three years.

The rows here are roughly every 5%, but don't hit exactly the 5s because a single article can generate more than one percentage point of traffic, and thus cross the lines (something that the graph reveals). But it still tells us the 51% of post traffic comes to articles 5 years 9 months or younger.

This makes it clear how much the old articles are still read by readers today. Posts that are over five years old get about half the traffic that comes to posts on the site. Although that's the main thing to note here I should not forget that younger posts do grab a big hunk, 13% of traffic came from brand new articles.