Container, DevOps, and Generative AI for Testing Usage

My work sponsored an IDC paper going over ways to use generative AI (and ML) at various stages of software development (the “Application Development and Product Life-Cycle Management”). There’s some interesting ideas in there, you should check them out.

As always, I like to collect the numbers from surveys and estimates. There’s some good ones in there! Here they are:

• “26% of organizations are using GenAI to support application development, testing, and management and 25% are utilizing machine learning with the greatest focus on leveraging AI to support DevOps analytics and process, governance, and security testing”

• “32% of application portfolios are built on containers and microservices today and, in five years [2028?], enterprises estimate that 39% of their portfolios will be built on containers and microservices” // I’m always after how many apps/workloads are running in Kubernetes. This is a new one! IDC’s numbers here don’t tell you which are running in production versus just in dev/test, but, whatever. Also, it’s “containers and microservices,” not just containers. // These are different estimates than a recent Gartner one: “By 2027, 25% of all enterprise applications will run in containers, an increase from fewer than 10% in 2021.”1 But, given that IDC’s is containers and microservices, maybe if you throw out microservices that are not running in containers, the IDC and Gartner numbers would be closer. I think the beers after work estimate, then, is something like: I don’t know, I’d guess companies are running something like 10% to 20% of their apps in Kubernetes? 20% seems, high - I mean, I just read the other day that 71% of developers don’t automate their builds and testing, right? So, who knows what’s going in enterprises. Anyway…want another beer?”

• “[E]nterprises are using DevOps methodologies to manage roughly 35% of their application portfolio today, with the anticipation of using DevOps to manage more than 47% of their portfolios in five years” // This seems like a better estimate than the headline one in the CD Foundation survey I looked at yesterday, which was (I don’t know the surveying name for this) synthetic conclusion based on the tools and practices people use, not the saying “we do the DevOps.” // Also, they’ll be spreading to 10% more of there portfolio over five years, to just under 50% of their portfolio under management. Change takes a long time! If we put the invention of DevOps as “[a]round 2007 and 2008,” we’re 16 to 17 years into DevOps and creeping towards 40% adoption. Probably fine, actually. It also shows you why we’re still talking about DevOps all these years later, and why there’s big windows for DevOps remixes like platform engineering.

• “[O]rganizations estimate that nearly 40% of their testing activities are automated, with ambitions to make almost 50% of testing activities automated in three years.” Now, this one feels bonkers. Who doesn’t automate their tests? Who doesn’t see the value in automating their tests? Well, could be that the someone in their team (or their org.) is automating tests and not the person answering “no” to this question. In my CD Foundation survey overview yesterday, I didn’t highlight the practices people said they used. “Test automation/management” use in the last 12 months was at 22%! So…I guess we’re to conclude that universal test automation is not universal. Bit of a WTF? there.

Above from “Applying Artificial Intelligence to Strengthen Application Development and Product Life-Cycle Management,” Pete Marston, IDC, November 2023.

If you enjoyed the above, check out my newsletter for more analysis like that, and plenty of other goofy things.

Relative to your interests

• Spring Now Offers Free Access for the Spring Academy Pro Content - Free Spring training for all: “The Spring team has announced that the Pro Content from their Spring Academy will no longer require a paid subscription, effective April 5th 2024, to improve the learning experience for the Spring community. The Spring Academy will continue to provide new content in the future.”

• A Shift in LLM Marketing : The Rise of the B2B Model - Enterprise AI requirements are different than consumer AI requirements: “For a data-focused customer base, SQL generation, code completion for Python, & following instructions matter more than encyclopedic knowledge of Napoleon’s doomed march to Moscow.”

• IBM to acquire Hashi for $6.4 billion, seeks software boost - Enterprise AI businesses case are difficult: “But he also said buyers’ initial enthusiasm for generative AI has eased as they ponder whether it can generate return on investment. Users are finding that applying generative AI to a single business process could cost as much as $300 million, a figure Krishna said is unlikely to produce positive ROI.”

• Inside TSMC’s Phoenix, Arizona expansion struggles - Sounds like a shitty place to work! Long hours, sometimes filled with low value work. The TSMC people must think Europeans are insane. In contrast, for the American work-culture, this feels like a “thank the unions” story.

• The Chilling of TikTok - “the bigger concern will simply be the distraction of it all. Product decisions will take longer. Timelines will slip. Executives will be absent. Employees will leave.” // Yeah, never good for a tech company to feel under attack. They’re just not used to it.

Wastebook

• “The characters shouldn’t be able to end the entire endless night by killing a big bird.” Sly.

• “The Coddling of the American Parent.” Here.

• “You don’t need to raise a cow to have the milk. You just have to make sure that the milk can be delivered to your doorstep.” Prof. Hsu.

• “In LLMs as in humans, context is that which is scarce.” Here.

• Kubernetes Kopacetic.

Conferences, Events, etc.

Talks I’m giving, places I’ll be, and other plans.

Atlanta Executive Dinner, May 22nd. DevOpsDays Amsterdam, June 20th, speaking. NDC Oslo, speaking, June 12th. SpringOne, August 26–29, 2024.

This year, I won’t be at Cloud Foundry Day (May 15th in NYC), but if you’re going you can get 20% off with the code CFNA24VMW20.

🪵 Logoff

I’m starting to think I could put together a whole presentation based on these surveys. I wouldn’t want it to be negative, maybe something more like “hey, slow down, in the real world, everything is fine.” I need to come up with a talk to pitch for DevOpsDays Ghent - maybe that’d be good! A sort of “DevOps isn’t dead, it’s just super-chill”…?

Spring Now Offers Free Access for the Spring Academy Pro Content - Free Spring training for all: “The Spring team has announced that the Pro Content from their Spring Academy will no longer require a paid subscription, effective April 5th 2024, to improve the learning experience for the Spring community. The Spring Academy will continue to provide new content in the future."

The Chilling of TikTok - Yeah, never good for a tech company to be get its feelings hurt: “the bigger concern will simply be the distraction of it all. Product decisions will take longer. Timelines will slip. Executives will be absent. Employees will leave."

A Shift in LLM Marketing : The Rise of the B2B Model - Enterprise AI requirements are different than consumer AI requirements: “For a data-focused customer base, SQL generation, code completion for Python, & following instructions matter more than encyclopedic knowledge of Napoleon’s doomed march to Moscow."

Inside TSMC’s Phoenix, Arizona expansion struggles - Rest of World - Sounds like a shitty place to work! Long hours, sometimes filled with low value work. The TSMC people must think Europeans are insane. In contrast, for the American work-culture, this feels like a “thank the unions” story.

IBM to acquire Hashi for $6.4 billion, seeks software boost - Enterprise AI businesses case are difficult: “But he also said buyers’ initial enthusiasm for generative AI has eased as they ponder whether it can generate return on investment. Users are finding that applying generative AI to a single business process could cost as much as $300 million, a figure Krishna said is unlikely to produce positive ROI."

The CD Foundation Survey, 2024

Today’s survey: “State of CI/CD Report 2024: The Evolution of Software Delivery Performance,” CD Foundation and SlashData, April, 2024.

Are people getting better at frequently releasing software and fixing problems in production? The most recent CD Foundation survey says…no:

• On average, 29% of respondents say they release software once a week or even more frequently; 40% take a more than month. The numbers here have been pretty stable over the past 4 years. This suggests that (a) improvement has stalled, and/or, (b) this is the normal baseline.

• 59% of respondents can fix production in a week or less, 40% in a day or less. This has gotten worse over the past 4 years. In 2020, 65% of respondents could fix production in a week or less.

Looking at the past 4 years of these surveys I also get the sense that we haven’t progressed very far with two of the goals of DevOps: more frequent app releases and fixing problems in production faster. This matches older surveys that show a very slow increase in release frequency, so slow that it might as well have stopped changing.

Does every organization need to release software frequently (weekly to daily)? Probably not, but in general what this survey tells us is not great. Does this means DevOps is a failure? Not at all: if you think these release cycles are long, you should have seen them in the 2000s and the 2010s!

My quick, hunch-made (read: “unscientific”) conclusion is that these numbers are “the new normal.” That means if asked how often enterprises release their apps, I’d say something like “about 50% of release their apps once a month or even daily (maybe 13% to 15% do daily or less?), and the other 50% take longer than a month, I’d guess most of those are quarterly.”

How frequently do most organizations really need to deploy, especially if you don’t count at-will bug and security fixes and patches? Does my pharmacy need to add new features every day, every week? My bank? The maintenance scheduling app my power company uses to send out trucks? Probably not. Maybe deploying once a month or more is actually perfect for most enterprises. What you’d like to see are survey questions that ask “and are you happy with this?” or “what are you plans, if any, to speed these up?” You’d want to divide these up by individual contributor and management/executives. I’d suspect the first would be like “meh,” and the second would like “fuck yeah!”1

To summarize: “You’re improving only getting slightly worse. Let’s put it like that.”

Deployment to production frequency

• The main question for app release frequency is: “On average, how often do you or your team deploy code to production?”

• In 2020Q3, 35% were at a month or more. In 2024Q1, it went up to 40% (this is bad, the wrong direction).

• The other time periods are mostly stable, but multiple deploys a day went from 12% to 9% over that time range.

• You glass half-full this and read it as: 60% of people release their software once a month or even more frequently. That reads pretty good, actually!

• If you want to get real detailed, you could pay close attention to the “you or your team” part of the question. This is different than “your entire organizations’ average across all apps.” For my purposes, the difference doesn’t matter.

How long does it take to fix problems in production?

• Fixing problems in production is measured her by time to restore service (MTTR)

• “The proportion of developers who can restore service in less than an hour has remained at around 11% since Q3 2022, though down from 17% in Q3 2020.

• However, the proportion of the worst-performing developers – more than one week to restore service – has been steadily increasing and is now the condition for 41% of developers.”

• Same point about “you and your team” versus “your entire org” as above.

Build automation and testing are not the norm

I use deployment frequency as the quickest/best metrics for measuring how we’re doing with getting better at software. Like I said above, you may not need to deploy at will. But if you’re doing all The Best Practices and DevOps Things, you should be able to deploy at least once a day if not more.

Can you deploy daily with manual deployments? Sure, we’ve all scp’ed files to production. Is that good? No. It is very not good. You want to automate your deployments, you know: Continuous Deployment. In an enterprise, like a big bank or pharmaceutical company, this is especially important because it’s the only way you’re going to reliably speed up the audit, compliance, and security bottlenecks.

Before CD, though, you need to automate your build and testing, you know: Continuous Integration.2

This has been known and recommended for a long time, 35+ years if I did the math right:

Grady Booch first proposed the term CI in his 1991 method, although he did not advocate integrating several times a day. Extreme programming (XP) adopted the concept of CI [circa 1989] and did advocate integrating more than once per day – perhaps as many as tens of times per day. Wikipedia.

I’d add that Thought Works championed this, as did many other people, and it became a one of the foundational principles of DevOps.

So. How are CI and CD usage doing? NOT GOOD.

Let’s look at people who agreed with the statement “I test my applications for security vulnerabilities” across the last four years of the survey:3

Above you can see people who answered yes/agree to the statement In 2024Q1, 71% of people said they do not “use continuous integration to automatically build and test my code changes.”

One of the people who I asked to look over this pointed out that the wording of the question is for individuals (“I use continuous integration” instead of “My team uses continuous integration”). It could be that individuals on a team don’t use CI, but that their team does. This also points out another survey nuance: you have to be careful about questions that ask “does your organization…” versus “does your app/team…” For example, for the first you could ask “does your organization use Generative AI?” And if there’s just one person in a 300,000 person company using it, the answer could be yes. That “yes,” of course, is much different than if 40,000 people are using AI, or if the top 3 products in that company rely on AI. And, indeed, dear readers, you’ll know that I am always interested in how many applications/workloads are, for example, running on Kubernetes, not how many organizations are using Kubernetes.

For more on CI and CD usage, let’s look at a different survey that goes back further, the State of Agile survey.4 They stopped asking about tools usage in 2022, but we have surveys that go back to 2007:

These numbers are (much) better, but there’s a problem with the results from both surveys: improvement is flat. Things haven’t gotten better or worse in a ~10% band.

So much ROI in your future

So, something like 47% to 71% of people don’t do CI. I’m not sure what to do with that. I hesitate to even think it’s real.

¯\_(ツ)_/¯

But, let’s say this analysis is right.

What it means is that:

1. Need something to boost your annual bonus? If you don’t have CI in place, it should be easy for enterprises to get better at software by simply automating builds and testing. If you don’t have CI in place (never mind CD!), set aside everything else you’re doing and put CI in place. Anything new you’re doing in apps (cloud native, DevOps, Kubernetes, etc.) is going to fail (or at least fail to achieve ROI) if you don’t have CI.

2. If you can get people to pay for CI/CD, you’ve got a market goin’! Selling these tools is difficult because you’re selling to a group that thinks they can do it on their own and have small budgets - you’ve got to go in with a freemium/cheap option and at some point, you add in RBAC, and you can sell big, company wide deals.

Other Details

• This report is done by Slashdata. They have huge data sets and do many of the reports for the CNCF. If you dig around on their site you can find PDFs of their general survey, Developer Nation.

• Demographics aren’t covered very well. I can’t tell what percentage is tech companies versus “normal,” company size, or the split between management and staff in respondents. But:

• “We survey 30,000+ developers annually.” And: “The report is based on a

  large-scale, online developer survey designed, produced, and carried out by

  SlashData over a period of ten weeks between November 2023 and February 2024… from 136.”

• Everyone is doing the DevOps: “As of Q1 2024, 83% of developers are involved in DevOps-related activities.” // There’s some variation by industry and org. size, but it doesn’t really matter. People think of themselves as doing DevOps, and many of the practices are likely practices. DevOps Thought Leading wins!

• BUT! If you look at how they determine this, they ask what DevOps practices people follow. There are 9 practices, some of which are just general programming practices, e.g., using CI, automating testing. This is different than asking “do you do the DevOps”? I don’t really like this method, so I wouldn't draw any conclusion about how many people are “doing the DevOps.” Maybe how many people are using DevOps practices if narrowed down from those nine.

• It’s probably better to use fewer (if not just one) CI/CD tools than more: “using multiple managed CI/CD platforms has minimal impact on improving deployment frequency, but is much more likely to lead to an increase in low performance.” More: “One possible reason for this may be that lead time for code changes is impacted not just by platform usage, but also by their CI and development processes. Multiple managed CI/CD platforms may introduce fragmentation of the CI process, leading to greater negative impacts. Similarly, development practices like code review, collaboration, and testing may be impacted by having to adapt to multiple platforms throughout the workflow, and this challenge has a larger impact on lead-time performance.” I’m all for centralizing and standardizing tools/platforms/stack, so, sure, I like that one.

• I think the PDF is saying that if you use hosted (public cloud) CI/CD tools, things are better.

• There’s an attempt (which I’m really interested in) to see if running your tool stack on public cloud versus on your own is better. The answer is that doing both (“multi-cloud”!) is the best. I don’t think this tells us enough.

//

If you liked this survey review, check out the two recent ones on platform engineering (the Perforce Puppet one and the one from Port, also, a bunch of other ones from over the years).

🪵 Logoff

I’ll put some links and waste book stuff into the next episode. In the meantime:

• Yesterday, I had the wrong link for the “Things I Like” post I mentioned: here is the correct link.

• On this week’s Software Defined Talk, I interview Jana Werner about some digital transformation cards she and her AWS friends have been using when they work with executives. It’ll post this Friday (April 26th at 7am Amsterdam time. It’s good stuff! You can find it in our YouTube channel or by subscribing to the podcast.

• Here’s my case for buying you cloud native platform instead of building it. I’ll probably regurgitate this in a future newsletter as well.

This feels like an accurate depiction of the halfling druid character I’ve been playing in D&D. Here, probably, enjoying Elevensies (11am), maybe Afternoon Tea (3 pm).