The CD Foundation Survey, 2024

Today’s survey: “State of CI/CD Report 2024: The Evolution of Software Delivery Performance,” CD Foundation and SlashData, April, 2024.

Are people getting better at frequently releasing software and fixing problems in production? The most recent CD Foundation survey says…no:

• On average, 29% of respondents say they release software once a week or even more frequently; 40% take a more than month. The numbers here have been pretty stable over the past 4 years. This suggests that (a) improvement has stalled, and/or, (b) this is the normal baseline.

• 59% of respondents can fix production in a week or less, 40% in a day or less. This has gotten worse over the past 4 years. In 2020, 65% of respondents could fix production in a week or less.

Looking at the past 4 years of these surveys I also get the sense that we haven’t progressed very far with two of the goals of DevOps: more frequent app releases and fixing problems in production faster. This matches older surveys that show a very slow increase in release frequency, so slow that it might as well have stopped changing.

Does every organization need to release software frequently (weekly to daily)? Probably not, but in general what this survey tells us is not great. Does this means DevOps is a failure? Not at all: if you think these release cycles are long, you should have seen them in the 2000s and the 2010s!

My quick, hunch-made (read: “unscientific”) conclusion is that these numbers are “the new normal.” That means if asked how often enterprises release their apps, I’d say something like “about 50% of release their apps once a month or even daily (maybe 13% to 15% do daily or less?), and the other 50% take longer than a month, I’d guess most of those are quarterly.”

How frequently do most organizations really need to deploy, especially if you don’t count at-will bug and security fixes and patches? Does my pharmacy need to add new features every day, every week? My bank? The maintenance scheduling app my power company uses to send out trucks? Probably not. Maybe deploying once a month or more is actually perfect for most enterprises. What you’d like to see are survey questions that ask “and are you happy with this?” or “what are you plans, if any, to speed these up?” You’d want to divide these up by individual contributor and management/executives. I’d suspect the first would be like “meh,” and the second would like “fuck yeah!”1

To summarize: “You’re improving only getting slightly worse. Let’s put it like that.”

Deployment to production frequency

• The main question for app release frequency is: “On average, how often do you or your team deploy code to production?”

• In 2020Q3, 35% were at a month or more. In 2024Q1, it went up to 40% (this is bad, the wrong direction).

• The other time periods are mostly stable, but multiple deploys a day went from 12% to 9% over that time range.

• You glass half-full this and read it as: 60% of people release their software once a month or even more frequently. That reads pretty good, actually!

• If you want to get real detailed, you could pay close attention to the “you or your team” part of the question. This is different than “your entire organizations’ average across all apps.” For my purposes, the difference doesn’t matter.

How long does it take to fix problems in production?

• Fixing problems in production is measured her by time to restore service (MTTR)

• “The proportion of developers who can restore service in less than an hour has remained at around 11% since Q3 2022, though down from 17% in Q3 2020.

• However, the proportion of the worst-performing developers – more than one week to restore service – has been steadily increasing and is now the condition for 41% of developers.”

• Same point about “you and your team” versus “your entire org” as above.

Build automation and testing are not the norm

I use deployment frequency as the quickest/best metrics for measuring how we’re doing with getting better at software. Like I said above, you may not need to deploy at will. But if you’re doing all The Best Practices and DevOps Things, you should be able to deploy at least once a day if not more.

Can you deploy daily with manual deployments? Sure, we’ve all scp’ed files to production. Is that good? No. It is very not good. You want to automate your deployments, you know: Continuous Deployment. In an enterprise, like a big bank or pharmaceutical company, this is especially important because it’s the only way you’re going to reliably speed up the audit, compliance, and security bottlenecks.

Before CD, though, you need to automate your build and testing, you know: Continuous Integration.2

This has been known and recommended for a long time, 35+ years if I did the math right:

Grady Booch first proposed the term CI in his 1991 method, although he did not advocate integrating several times a day. Extreme programming (XP) adopted the concept of CI [circa 1989] and did advocate integrating more than once per day – perhaps as many as tens of times per day. Wikipedia.

I’d add that Thought Works championed this, as did many other people, and it became a one of the foundational principles of DevOps.

So. How are CI and CD usage doing? NOT GOOD.

Let’s look at people who agreed with the statement “I test my applications for security vulnerabilities” across the last four years of the survey:3

Above you can see people who answered yes/agree to the statement In 2024Q1, 71% of people said they do not “use continuous integration to automatically build and test my code changes.”

One of the people who I asked to look over this pointed out that the wording of the question is for individuals (“I use continuous integration” instead of “My team uses continuous integration”). It could be that individuals on a team don’t use CI, but that their team does. This also points out another survey nuance: you have to be careful about questions that ask “does your organization…” versus “does your app/team…” For example, for the first you could ask “does your organization use Generative AI?” And if there’s just one person in a 300,000 person company using it, the answer could be yes. That “yes,” of course, is much different than if 40,000 people are using AI, or if the top 3 products in that company rely on AI. And, indeed, dear readers, you’ll know that I am always interested in how many applications/workloads are, for example, running on Kubernetes, not how many organizations are using Kubernetes.

For more on CI and CD usage, let’s look at a different survey that goes back further, the State of Agile survey.4 They stopped asking about tools usage in 2022, but we have surveys that go back to 2007:

These numbers are (much) better, but there’s a problem with the results from both surveys: improvement is flat. Things haven’t gotten better or worse in a ~10% band.

So much ROI in your future

So, something like 47% to 71% of people don’t do CI. I’m not sure what to do with that. I hesitate to even think it’s real.

¯\_(ツ)_/¯

But, let’s say this analysis is right.

What it means is that:

1. Need something to boost your annual bonus? If you don’t have CI in place, it should be easy for enterprises to get better at software by simply automating builds and testing. If you don’t have CI in place (never mind CD!), set aside everything else you’re doing and put CI in place. Anything new you’re doing in apps (cloud native, DevOps, Kubernetes, etc.) is going to fail (or at least fail to achieve ROI) if you don’t have CI.

2. If you can get people to pay for CI/CD, you’ve got a market goin’! Selling these tools is difficult because you’re selling to a group that thinks they can do it on their own and have small budgets - you’ve got to go in with a freemium/cheap option and at some point, you add in RBAC, and you can sell big, company wide deals.

Other Details

• This report is done by Slashdata. They have huge data sets and do many of the reports for the CNCF. If you dig around on their site you can find PDFs of their general survey, Developer Nation.

• Demographics aren’t covered very well. I can’t tell what percentage is tech companies versus “normal,” company size, or the split between management and staff in respondents. But:

• “We survey 30,000+ developers annually.” And: “The report is based on a

  large-scale, online developer survey designed, produced, and carried out by

  SlashData over a period of ten weeks between November 2023 and February 2024… from 136.”

• Everyone is doing the DevOps: “As of Q1 2024, 83% of developers are involved in DevOps-related activities.” // There’s some variation by industry and org. size, but it doesn’t really matter. People think of themselves as doing DevOps, and many of the practices are likely practices. DevOps Thought Leading wins!

• BUT! If you look at how they determine this, they ask what DevOps practices people follow. There are 9 practices, some of which are just general programming practices, e.g., using CI, automating testing. This is different than asking “do you do the DevOps”? I don’t really like this method, so I wouldn't draw any conclusion about how many people are “doing the DevOps.” Maybe how many people are using DevOps practices if narrowed down from those nine.

• It’s probably better to use fewer (if not just one) CI/CD tools than more: “using multiple managed CI/CD platforms has minimal impact on improving deployment frequency, but is much more likely to lead to an increase in low performance.” More: “One possible reason for this may be that lead time for code changes is impacted not just by platform usage, but also by their CI and development processes. Multiple managed CI/CD platforms may introduce fragmentation of the CI process, leading to greater negative impacts. Similarly, development practices like code review, collaboration, and testing may be impacted by having to adapt to multiple platforms throughout the workflow, and this challenge has a larger impact on lead-time performance.” I’m all for centralizing and standardizing tools/platforms/stack, so, sure, I like that one.

• I think the PDF is saying that if you use hosted (public cloud) CI/CD tools, things are better.

• There’s an attempt (which I’m really interested in) to see if running your tool stack on public cloud versus on your own is better. The answer is that doing both (“multi-cloud”!) is the best. I don’t think this tells us enough.

//

If you liked this survey review, check out the two recent ones on platform engineering (the Perforce Puppet one and the one from Port, also, a bunch of other ones from over the years).

🪵 Logoff

I’ll put some links and waste book stuff into the next episode. In the meantime:

• Yesterday, I had the wrong link for the “Things I Like” post I mentioned: here is the correct link.

• On this week’s Software Defined Talk, I interview Jana Werner about some digital transformation cards she and her AWS friends have been using when they work with executives. It’ll post this Friday (April 26th at 7am Amsterdam time. It’s good stuff! You can find it in our YouTube channel or by subscribing to the podcast.

• Here’s my case for buying you cloud native platform instead of building it. I’ll probably regurgitate this in a future newsletter as well.