202306 - apenwarr - ’If you take a single pull request (PR) that adds a new feature, and launch it without tests or documentation, you will definitely get the benefits of that PR sooner. Every PR you try to write after that, before adding the tests and docs (ie. repaying the debt) will be slower because you risk creating undetected bugs or running into undocumented edge cases. If you take a long time to pay off the debt, the slowdown in future launches will outweigh the speedup from the first launch. This is exactly how CFOs manage corporate financial debt. Debt is a drain on your revenues; the thing you did to incur the debt is a boost to your revenues; if you take too long to pay back the debt, it’s an overall loss.’ Also: “Tech debt, in its simplest form, is the time you didn’t spend making tasks more efficient. When you think of it that way, it’s obvious that zero tech debt is a silly choice."

IT spending soars, generative AI investments barely leave a mark - These forecasts have been confusing in recent years. They’re always increasing, and yet the trends are always cost savings: ‘“Digital business transformations are beginning to morph,” said Lovelock. “IT projects are shifting from a focus on external-facing deliverables such as revenue and customer experience, to more inward facing efforts focused on optimization." The trend is reflected in where spending growth is highest. Gartner expects software, the fastest-growing segment, to achieve a double-digit growth rate of 14% on the year, as organizations reallocate spending to squeezing more value out of ERP and CRM applications, as well as other core platforms that deliver efficiency gains.’ // Also, yeah: with AI it’s way too early in the corporate planning and strategy cycle to be allocating lots of cash to it. It’ll just be small PoCs for at least a year.

No one has an “appetite for risk” - “I think there is a better way to express what we aim to express when we say ‘risk appetite.’ What we are talking about is the organization’s failure tolerance. How often is it okay for the organization to experience security failures? How big can the failures be (impact) and still be tolerable?"

We’re back from a three day weekend in London for our anniversary. I tried very hard to eliminate all “productivity,” so I have very little to say today. I did want to ADVERTISE for a thing I have tomorrow. So I’ve gathered up some waste book for you today.

Wastebook

• We should revise the Peter Principle not to be insulting (reached the level of your incompetence), but burnout driven: reached the level of being able to put up with more bullshit.

• “I know my price. Because I developed my identity outside of work, there’s a cost that comes when work infringes upon it—if it ever costs me a larger part of my identity and my life, I know it’s not worth it.” Here.

• I like systems that can be successfully applied, with only medium effort. It must be executable by many organizations, not just the exceptional ones. This principle weeds out the systems that sound good, but can never be done widely. It can be hard for the genius to come up with the perfect systems, but should be easy for us regular folk to implement it. This applies to individuals as well. Related:

• If you’re so smart, why are we all still so dumb?

• There’s tasks that need to be done by a date. And then there are tasks that have the potential to help me meet goals I don’t even know I have yet.

• In praise of source cream - It is good. Despite growing up eating Tex-Mex, it is something I never really got into until recently.

• Travel hack: you only have to keep the tray table up during landing and take off. So if you’re waiting for the plane to load and take off, you can put it down. I like to hunch over while I’m waiting, you know reading books or news or whatever. So I put the tray table down to lean on it.

• This is a good book, and I enjoyed reading it.

Tomorrow Darran Rice and I are going to be talking about how cloud native apps can make your financial services strategy better. Plus, a little security, legacy software fixes, etc. Watch it live, July 19th at 3pm Amsterdam time, and you can ask questions. Or, just save it to watch later.

Upcoming

Talks I’ll be giving, places I’ll be, things I’ll be doing, etc.

July 19th Improving FinTech with cloud native think, speaking. July 19th Stop Tech Debt and Start Using Faster, More Secure Paths to Production. Sep 6th to 7th DevOpsDays Des Moines, speaking. Sep 13th, stackconf, Berlin. Sep 14th to 15th SREday, London, speaking Sep 18th to 19th SHIFT in Zadar, speaking. Oct 3rd Enterprise DevOps Techron, Utrecht, speaking.

Logoff

See y’all next time!

personal organization - “Here’s my one piece of advice about personal organization: (calendars, tasks, planning, tracking): Think hard about your needs, pick a system, and then do not under any circumstances change it until at least one full year has passed."

Checking In on the State of TDD - Great round-up or the current thinking and research! But: Kids these days! “Look at it this way: 19 out of 20 startups fail. That means that odds are that you will never see this code again. You’d be a fool to spend any more time on it than absolutely necessary.” This is the kind of advice that’s fine for failed startups. And maybe even strategic if you’re just looking for a valuations run-up cash-out. But for #20, you’re just building your own legacy trap for you or the company that awaited you. Also, for enterprises, it’s worse: a bank is going to have that code for a decade or more and will be fucked in three to five years if it’s not well tested.

Also, even for the startups you’re just passing that tech debt onto whoever acquired you. “Merry Christmas,” to your new coworkers as you’re sitting in the pool all those TDD-free options bought you, and your new pals are working through the holidays to add some simple code ther should have only taken 15 minutes…if only there was a way to verify that it didn’t break anything.

What the Government Email Account Hack Says About the Future of Cybersecurity - Always be securing all the things.

Beware of High Expectations in Enterprise Tech

Here are three charts.

First, while everyone is freaking out about AI, it’s had little impact thus far:

Second, since 2020, usage of cloud seems to have leveled off at a 50/50 split between on-premises and public cloud:

But don’t worry, everyone is planning to move more to public cloud in the next three years. At least that’s been the case for the past seven years:

There are some rules of thumb for the impact of new enterprise technology here:

1. It will probably take much longer than you think.

2. The business value you achieve will probably be less than you think.

3. There’s not as much urgency to “transform” as us vendors would like you to believe.

The AI craze is an especially interesting case for me. It is clearly a big deal. I never used Docker, but the first usage of ChatGPT is what it must have been like to use Docker. You don’t believe what’s happening, it’s so effortless, and you can immediately start getting value from it.

For enterprises, however, things are more complicated. Just as you wouldn’t want your developers doing whatever they wanted with Docker - especially just downloading images of the Internet and, like, using them to run your paycheck processing, missile guidance, or just where to ship this box of celery systems - you don’t want ChatGPT just running around willynilly inside your enterprise. Even more: it can’t. You can’t just upload everything to ChatGPT and have it do wonderful things. Believe me: I’ve tried!

You can imagine all sorts of stupendous improvements to how a company operates if they unleash AI on everything. But, it will be less stupendous that we think, and take much longer. First, you’ll need to make sure all the governance and (as we’d say in the consumer world) privacy concerns are taken care of. This isn’t just about annoying lawyers and security people being annoying, it’s often to comply with regulations and laws. And it’s often a good idea!

Then, you have to figure out a way to actually load all that data into THE AI MACHINE. Access to all that corporate data across old ERP systems, CRM systems, Salesforce, email, Sharepoint, Google Docs, etc., etc. is already fraught and difficult. We’ve tried “data lakes” and…well? One of the main issues here is paying for all of that.

If you’ve ever tried to do what seems like simply pulling information from Salesforce or other ERP systems, you know what I mean. You rarely can just do what should be simple. First, you need access. And you’re going to have a get budget for that. Second, you need to actually understand the format the data is in. Third, you need the skills to actually find, clean, transform, and build reports out of that data. What this means is that you can’t do it alone: you always need to know “a guy” who can do it on the side for you. (Sidenote: if you’re working in a big company, always be looking out for this “guy.” If you see some really good charts or data in a presentation, ask who put that together and take that person to lunch so that you can ask them for help in the future.)

ChatGPT like systems could make this all much, much better. But it will take time, and it might not just, like happen as fast or as broadly as you think. Compare that to the public cloud usage in the above charts. We all know that using public cloud makes totally sense for any given apps. We all know that there are very few technical reasons to do it. We all know that the costs are, at worst, net-neutral. And yet…it’s going to take us three years to get there, apparently.

Why so slow?

This emerging technology adoption ass-dragging is driven by numerous things. The price to use new technologies and switch over to use them is often too high to easily deal with. If you’re serious about building an app platform on-top of Kubernetes, you’re going to pay a lot more money than you think. For a the smallest enterprises (let’s say, 5,000 to 10,000 people or more), you’re talking about millions of dollars a year. That can be either in staff if you want to DIY, buying vendor/public cloud services, etc. Just paying for the operating systems - a “free,” open source operating systems! - is going to cost millions. I have no idea how RHEL pricing works at this scale, but if you look at Salesforce’s numbers (moving 200,000 servers to RHEL), they feel like what big banks, large retailers, large government agencies, and other enterprises would face. That’s got to be, again, millions of dollars a year.

These costs are often worth it. You need computers and I’ve heard software is eating the world. However, when you introduce a brand new technology with such a steep price, you’re talking 12 to 18 months to get going. Enterprises work on annual budgeting cycles. If they’re using a regular calendar year for their fiscal year, you start “socializing” your budget needs around October, figuring out what you need, making deals with peers, sniffing out what your budget approvers actually value and will approve, getting your PowerPoints all prettied up with arts and crafts. Then you pitch it towards the end of the year. The budget is approved, then everyone disappears for two to three weeks for the December holidays. You come back in January, sort of putter around for a couple of weeks, and then you start finish talking with your vendors and public cloud people (who, hopefully, you started talking with back in October), finally selecting one to go with. Then you need to install the thing. But first you’ll need to talk with he networking and storage admins. The compliance and security people are going to want to see things. Also, you’ll need to update some training in the cloud center of excellence. And before you know it, a quarter has passed. You’ll have to spend a week or so preparing your quarterly review - more arts and crafts - to report on the status of it.

And so forth. (Also, I forget dealing with procurement. It’s hard enough to get an enterprise to pay for a $100/year expense, let alone $1.4 million.)

Then October rolls around again. Us vendors are like, “hey, how’s your consumption of this stuff: I gotta maintain the ARR, do some LTV farming…could you introduce me to that other group? I they’d like some steak dinners….”

Then, all of the sudden, those goofballs in the bond department screwed up their inflation estimates, and next thing you know another bank is buying you. “We’ve got some other things we need to handle now,” you tell your steak dinner friend.

You’ll get the stuff going eventually. You’ll get benefits from the new technology for sure - just ask us vendors to carpet bomb you with case studies! This shit is for real. It’s just that it’s for real once you get it all setup and learn how to do it.

It’ll take a lot longer than you think.

The revelation of reality

The Gartner hype cycle describes another phenomena here: inflated expectations. You’ve heard that AI (or public cloud, or cloud native, or CRM systems) is stupendous, revolutionary. There’s case studies from early adopters. But you haven’t actually used it yet. The hundreds (thousands, even) of people in your organization haven’t used it yet. You customers have not experienced its effects yet, and so forth.

Until you actually get your hands on it, at scale (not just some early PoC’s) you won’t know how stupendous it is in your organization working within your constraints and taking advantage of your capabilities. Getting to that point takes at least a year, often more to get more organization wide usage.

You can certainly get a better and better sense as you go along. You have to build in bureaucratic humbleness that is willing to learn and observe itself in action to get that sense. You have to be careful with your first impressions. What the hype cycle is telling you is that they’ll be inflated.

ChatGPT is a good example here. Once you use it day in and day out for months, you adjust your astonishment at it. It is amazing! But it’s now where near the level of earth shattering that the anti-AI people make it out to be.

Listen, it sounds like I’m being a wet-sandwich here. But it’s the opposite. The hype cycle has another mind-trick in it. After the peak of inflated expectations, everyone gets disappointed and you go into the trough of disillusionment. You get upset! You get letdown! (And when it comes to public cloud, all the on-premses vendors smell the delicious disappointment at migrating to the new and swarm - steak dinners galore!)

If you know that expectations are false, that it will take a long time to get the stupendous returns, you won’t get disillusioned. You’ll be smarter, reality will be revealed.

The rotting core

The other thing you have to keep on eye on is spending too much attention on the stupendous new thing. Meanwhile, you have the existing business to run, the existing systems to operate and optimize. I suspect that most of the benefits you can achieve from IT come just from doing what you’re currently doing better. For example, if you look at how you do software development, you probably call it “agile,” but most people are probably not following the actual practices.

A cool trick is to ask these agile teams how frequently they ship software. If it’s not at least monthly, you can probably do some improvements. And the improvement is probably just putting in some automation, talking with auditors to show them that that automation means they can trust you systems more and, thus, have less meetings about each release. Are you actually doing pair programming? What are your test coverage numbers? Are you actually standing up in your stand-up meetings?

I’ve studied the problem of rotting, neglected systems in large enterprises for the past two years and it is a big, big problem. One of my co-workers said at his previous job, at a big bank, they were spending near 50% of their time just on fire-fighting, upkeep, and dealing with compliance. As he said, time spent on that is time stolen from making the business better. And, also as he said, there’s well known, proven practices and technologies that could improve that 50%…you’d just have to actually decide to do it.

The issue with this rotting core is that (a) it’s boring compared to, like, AI, and, (b) it’s really hard to make a business case for addressing rot in your IT systems and software. It’s not going generate direct revenue, so unless your organization thinks in terms of running more efficiently (speeding the workflows your software does - for example, speeding up approving insurance applications or doing case management) and values agility, the business case is going to be difficult. Corporate strategy is a zero-sum game: those who win take from those who lost. There’s only so much budget and time that a company has. So if you’re pitching “just do what we do now, but better” versus “implement AI to revolutionize our business!”…it’s going to be tough.

The software factory

Most corporate strategy think doesn’t value well maintained software. The people making strategy decisions in enterprises seem to of software as one-off, install it and forget it projects. You have to think about your overall IT estate as a factory. Just as a you would spend a lot of time maintaining and optimizing your factory, you have to do the same with your IT “factory.”

I suspect that many organizations could get huge improvements just by getting better at what they’re currently doing.

That’s seemed to be the case business wise in the past decade. For example, banking startups were supposed to destroy traditional banking. Instead, existing banks just got better…without really changing how their business functions and what they do. And, as the public cloud charts show above. Well. That hasn’t rocked the world (but it will over the next three years!)

Two stupendous beguilement traps you should navigate right now

There’s two things going on right now that you can immediately apply this to.

Kubernetes

First, make sure you get realistic expectations about Kubernetes. The overall ecosystem has hyped the shit out of it, myself included many times. But if you look at actual adoption, it’s incredibly small. There’s probably about 10% of enterprise applications running in containers now, less in Kubernetes. There simply has not been enough use of kubernetes in enterprises to know if it’s a good or bad idea, let alone how much of a good or bad idea it is.

Don’t let Kubernetes migrations become a distraction from just improving how you currently work.

Instead, you need to setup a system - a culture, if you’re a DevOps fan - of learning and experimenting to figure out if Kubernetes will work for you. There’s a huge knowledge and experience vacuum when it comes to Kubernetes, and the strategic engineering move now to is fill that vacuum and then start making big decisions.

Platform engineering

Second, make sure you understand what platform engineering is before you launch big plans to transform how you do everything. Platform engineering is just DevOps under a new name. (At least that’s what I think this week.) At most, platform engineering just adds in internal developer collaboration tools (in the form of IDPs). Platform engineering might include building and running your platform-as-a-service…eventually.

But if you think through all the things in the umbrella of platform engineering, it’s too much. In a large enterprise (or even a small one), one group doesn’t have enough time, enough cognitive load to manage the tools developers use, the runtime environment they use, the middleware they use, and also do all that magic DevSecOps (pardon me, I mean PlatformEngineeringSecOps), and…all the other stuff. If you have a mature, proven platform like the Cloud Foundry ones, sure. There’s all sorts of people building platforms like that on-top of Kubernetes right now - so you can start with one of those instead of building your own. (You could also just short-circuit all of that and hand over the platform building and running to the public cloud - the dream!)

But people in enterprise IT are obsessed with building custom, bespoke platforms. I mean, builders like to build.

Most every enterprise I’ve ever talked with has numerous reasons why they’re different, why they can’t just use an off-the-shelf platform, and, thus, why they need to customize it. For the most part, their reasons are false. And, in most cases, they’re very common reasons that all of their peers and other enterprises have. And, as such, those reasons have been taken care of in the off-the-shelf platforms. Rather than change the unique constraints that drive the need to build their own platforms, these organizations work keep those constraints and are forced to build around it. It’d be like if you had to build a custom bed around an old radiator instead of just removing the radiator.

With the DIY platform mentality, you’re building a complex system, and you’re going to need a complex meatware system to manage it. “Complex systems build complex systems.” You’re probably already in a flywheel of doom made up of two complex systems - your software and your meatware - that’s making things more and more complex as the quarters tick by.

In that kind of system, you’re always going to need separate developer tools groups and separate platform operations groups. You can only break down silos if you put, as we used to say, an opinionated platform in place that standardized and centralizes what you do.

Have you tried actually standing?

So if you’re launching a Kubernetes and/or platform engineering, uh, “strategy” or “program,” figure all that out. Don’t have inflated expectations. It’s probably better to just, like, call in the DevOps Engineering team and the application development team, let’s see, the infrastructure management team, the security team…hell, get all the silos in one room, and ask them “so, tell me some daily tasks that we could be doing better. Are y’all actually standing up in your stand-up meetings?”

IncrementalOps

You’ve got to keep an eye on the new stupendous stuff. You should always be doing several projects to try it out. Not just “pilot” it or do a “PoC,” but to actually do real work and make sure you’re following a process to learn from it. If you think AI is going to be a big deal, spin up three projects that are directly connected to how the business runs and check in every month to learn what works and doesn’t work. You can’t do a business case for that. You just have to do it.

But, at the same time, spend a lot of your time just improving how you’re currently working. Remember, five years ago you did all of that transformation work, right? Maybe it’s time to go back and make sure you’re still following the practices and doing the things. That you haven’t just reverted to the old habits.

You know, instead of sitting, try actually standing up in your next stand-up meeting.

If you’re into the above kind of stuff, you should subscribe to my newsletter. It’s full of topics like these a few times a week, plus links I’ve found and other fun oddities to look at.

Subscribe!

Upcoming

Talks I’ll be giving, places I’ll be, things I’ll be doing, etc.

July 19th Improving FinTech with cloud native think, speaking. July 19th Stop Tech Debt and Start Using Faster, More Secure Paths to Production. August 21st to 24th SpringOne & VMware Explore US, in Las Vegas. Sep 6th to 7th DevOpsDays Des Moines, speaking. Sep 13th, stackconf, Berlin. Sep 14th to 15th SREday, London, speaking Sep 18th to 19th SHIFT in Zadar, speaking. Oct 3rd Enterprise DevOps Techron, Utrecht, speaking.

I’ve got what I hope will be a podcast-y type version of a webinar series I just wrapped up. I don’t know: something like a after show special where we talk about the series?

It’ll be live next week, on July 19th at 3pm Amsterdam time (check you local listings).

We did a good job in the series talking about how cloud native apps and thinking gives banks and other financial organizations new capabilities, how it changes, for the better, how you do security and compliance, and how to start thinking about modernizing all those legacy applications.

Get a reminder to watch it

You can watch it in LinkedIn, or YouTube if you prefer that. Come with some questions! We didn’t get any during the webinar series.

Logoff

I think I accidentally worked out the outline for my keynote at DevOpsDays Des Moines!

Finding an estimate of how many developers there are in the world is difficult. Oh, there’s plenty of people making estimates, but those estimates vary so much that the estimates are suspicious.

For example:

1. Microsoft/GitHub recently said they now have 100 million developers using GitHub.

2. “We estimate that, as of Q1 2023, there are 35.6 million active software developers in the world,” says Slashdata (who the CNCF uses for surveys).

3. Evans Data (by way of Statista) estimated that 25.6 million in 2021 and forecast 27.7 million in 2023.

4. In 2022, IDC said: “worldwide professional developer census, which IDC places at just under 16 million in 2022.” This is up from 11 million in their estimate for 2014.

Those spreads are pretty bonkers, huh!

Out of those, I’d trust IDC the most. This isn’t exactly apple-to-apples: IDC’s numbers are for paid developers, not just “hobbyists.” That’s fine, though, because when I’m interested in this number, I’m usually wearing my “pay the bills” hat and am only interested in professional developers.

If I could find the breakout between pro and hobbyists in the Evans data, I’d probably go with the Evans data. I usually go with IDC on any IT estimates, but Evans specialize in developers and have for years.

If you look back at IDC’s 2014 estimate, as summarized in The Register then you can get a sense for the hobbyiests: “11 million working as software developers and a further 7.5 million ‘hobbyist’ [as] developers.” So if we take those percentages, it’s 40%/60% between hobbyists and pros. Applied to the 2022 estimate, that’d get you, like, 26.6 million developers total. And, then, that lines up well enough with Evans' too.

(You could make an argument that since 2014 interest in programming has risen, access to computers and education has gone up, etc., etc., and say that the hobbyist share has grown, but, whatever: I’m a round numbers, “smells about right” kind of person once things get into the tens of millions.)

So, if I was going to use something, I’d say “around 16 million professional developers, probably something like 26 million total developers if you include people doing unpaid farting around.”

I’ve got what I hope will be a podcast-y type version of a webinar series I just wrapped up. I don’t know: something like a after show special where we talk about the series?

It’ll be live next week, on July 19th at 3pm Amsterdam time (check you local listings).

We did a good job in the series talking about how cloud native apps and thinking gives banks and other financial organizations new capabilities, how it changes, for the better, how you do security and compliance, and how to start thinking about modernizing all those legacy applications.

Get a reminder to watch it

You can watch it in LinkedIn, or YouTube if you prefer that. Come with some questions! We didn’t get any during the webinar series.

Wastebook

Relative to your interests

• Security Team Culture Matters - Being in security should be a happy job. // “Security and risk teams are more motivated and purpose-driven than others. As a 25-year cybersecurity veteran, this totally checks out for me. Almost everyone I know in [IT security] is mission- and purpose-driven. They took on this job to protect others!”

• Deploying the Swift Method to Modernize a Singapore Government Legacy System - Good description of what it feels like to be stuck in the legacy trap: “The [Singapore] government agency in this case study faced a similar issue with a legacy system that supported critical business processes, integrated with other business-critical applications, and was developed and maintained by third-party vendors. Over time, the codebase had become highly coupled within different business domains and contexts, making it difficult for developers to work on. This situation led to product development squads being slowed down by dependencies on the support team for this legacy system. The development squads also lacked confidence to make changes to this system themselves—given the low automated test coverage—and faced uncertainty about what they would be able to deliver for their own work streams independently.” You can see the original talk this is based on here.

More: The importance of on-premises applications and infrastructure; Managing negative emotions and cultivating happiness; VMware Tanzu and VMware Aria adapting to multi-cloud industry trends; Potential discussion on DevOps trends; WE CAN VIRAL IT FOR YOU WHOLESALE.

Upcoming

Talks I’ll be giving, places I’ll be, things I’ll be doing, etc.

July 19th Improving FinTech with cloud native think, speaking. July 19th Stop Tech Debt and Start Using Faster, More Secure Paths to Production. August 21st to 24th SpringOne & VMware Explore US, in Las Vegas. Sep 6th to 7th DevOpsDays Des Moines, speaking. Sep 13th, stackconf, Berlin. Sep 14th to 15th SREday, London, speaking Sep 18th to 19th SHIFT in Zadar, speaking. Oct 3rd Enterprise DevOps Techron, Utrecht, speaking

Logoff

Now that summarizing is working again in ChatGPT, I thought I’d try putting in a list of things I asked it to summerize, but did not bookmark to list an official link. That’s what the “More” paragraph is in the links section above. It’s like that Matt Levine/Harper’s style of mashing it all together. An “official link” means I’ve extracted some quote and/or added a comment of my own, bookmarked it, posted it to my blog and Mastodon, and probably promoted it on some other social holes. These ones are from the cutting room floor.

Security Team Culture Matters - Being in security should be a happy job. // “Security and risk teams are more motivated and purpose-driven than others. As a 25-year cybersecurity veteran, this totally checks out for me. “Almost everyone I know in [IT security] is mission- and purpose-driven. They took on this job to protect others!"