Much like financial debt, technical debt is helpful when managed responsibly, but like real debt, tech debt can also stop growth and innovation in its tracks.

My colleague Bryan Ross has a piece out on tech debt with five ways to address is.

Here’s a summary of it.

Think of technical debt as the accumulation of compromises on development quality to save time. Some examples of these choices are postponing unit tests, running outdated software, or focusing on end-user features at the expense of internal processes. When you don’t address these your process will eventually slow down each release cycle.

For example: testing new features will be more difficult because of dependencies on untested code, so when you’ll have to release untested code…or go back and fix those tests, which will take longer because you’ve forgotten how the code works, and will keep uncovering more and more untested code.

To address technical debt, organizations should adopt a low-risk, gradual approach. Over ten or so years, VMware Tanzu Labs has done just that with the Swift method. It involves selecting a non-critical application, but customer facing application to modernize. By working on this small app, you learn the Swift method, you discover changes that need to be made in your organization, and also build up one success case. This systematic approach helps businesses discover, experiment, and learn how to tackle their technical debt.

Too much tech debt comes from unhealthy habits. So you need to change those too. Here’s five steps to health:

1. Map your end-to-end process - one of the hidden types of tech debt it lack of knowledge and coordination about how your software goes from idea, to code, to production. Figure that out.
2. Be serious about automate testing: Implement automated testing to catch problems early and introduce a metric for software quality.
3. Agree on better coding practices: encourage standardized approaches, pair programming, and peer reviews to reduce defects and improve collaboration.
4. Plan for the future: Adopt microservices architecture to make applications more modular and adaptable. We don’t talk about this part of microservices a much as we used to. Perhaps the best mearware part of microservices is how they can decouple different groups. Here, this means making it easier to change parts of your code without having to change other parts.
5. Standardize the environment: rtegularly rebuild the development environment using automated tools to prevent temporary fixes from becoming permanent.

You’ve got to address technical debt or you’ll end up stifling innovation and quickly slow down the business.

Check out his overview for more details.

I’m a Texan living in Amsterdam, so when I come back to the States (Dallas and Austin this time), I noticed things about American that I never did:

• People are eager to be helpful, especially when you have a bunch of kids. Everyone offers to help with luggage. A cashier will race back to the shelves to find a replacement for a broken item, or verify the sales price.
• People are apologizing all the time, and people tell them it’s not a big deal, and apologize back. At a haircut place, someone forgot at a jacket and came back. That person said “sorry,” the haircutter said “sorry,” and then there was another sorry in there.
• Everything is huge.
• SUVs and sedans are the norm, not hatchbacks. In Europe, the hatchback and small station wagon are the norm for cars. There are many Teslas too. In the States, there are Teslas, but almost no hatchbacks and very few station wagons.
• The eggs, even the organic ones, are a pale yellow.
• Also, I almost forgot that you refrigerate eggs in the US.
• Litigation lawyers love billboards. Hit by a truck? Workplace injury? Just drive down the highway and you’ll find five or ten lawyers with stern looks ready to help YOU GET MONEY. Sometimes they’re even holding sledge hammers or baseball bats. So weird.
• There’s a definite Austin look. Clothes are cotton, but outdoors-y. Tattoos for sure, but low-key ones. Shorts, t-shirts and even sleeveless t-shirts. Lots of dogs on long leashes with poop-bags artfully tied to the leashes. Which I guess is to say: very little designer clothes.
• Men do not, at all really, wear murses, man purses. This is so normal in Europe, that I don’t even notice it anymore

Originally in my newsletter.

I’ve started writing this bit two times (see the community one and the one on ICs vs. managers for what happened instead).

If Twitter fails - or I stop using it - I’m looking forward to recalibrating my sense of urgency.

One thought going around is that no one would want to rebuild Twitter (I hear it most in the Ben Thompson Podcast Universe). I haven’t verified the business-side, but that seems true from the business angle: Twitter isn’t, like, that great of a business and has failed to figure it out like the Facebook Conglomerate.

Then there’s all the negative things too. Here, Twitter is like guns: too easy to use for bad outcomes.

After using Twitter for so many years, my sense of urgency is too short. I think in quick cycles and don’t have that “slow work” or “slow living” approach to most things.

I notice this with my kid’s education the most. School for kids is a long, long game. They have to learn to “do” school, which can take years and isn’t really taught in school. You know: keep track of assignments, due dates, make sure you understand what you need to do for each assignment (e.g., a five paragraph essay format; showing your work in math; adding those extra layers of work that show you’ve understood the higher-level ideas and synthesized them into some output), etc. These are all things office workers do without thinking, but kids don’t even know they exist - they’re unknown unknowns.

When my kids don’t learn these things the first time I explain it to them, or the 20th time, or the 30th time where I’ve tried to orchestrate some tricky way of teaching them, I get frustrated. But my timescale is just an hour, a week at most. It should really be, like, five years…ten years.

The same goes for that publishing vs. managing bit also here.

Another example is the rise of platform engineering and how quickly it’s gone through the full thought-leader cycle.

Backstage has been out for awhile, but it wasn’t until about February of this year (when Gartner wrote a paper on internal developer portals), and then this summer with all the devrel/PR-campaigns that Humanitec has done, culminating in BackstageCon that platform engineering came to the top of the heap.

About three weeks ago, there were the usual series of marketing-opportunistic articles saying “DevOps is Dead!” You can tell when this crest happens when there’s 1+n articles on The New Stack stating it as such - or if both The New Stack and InfoQ publish the same article, just written differently.

The response to the dead-thing was swift, and actually pretty good. It was like, “hey man, don’t harsh our vibe and all the work people have put into this thing for the past ten plus years. Be kind!”

This was a very fast loop!

I don’t remember how long that loop took with SRE, but I think it was longer and more…thoughtful?

But, the urgency model we use now made us all think too quickly about platform engineering and try to create a whole new category. Indeed, my theory is that, really, platform engineering is purely about internal developer portal and the tools team (the “DX team” if you like) - basically, the SDLC tool suite developers use. With a name like “platform engineering,” thought, it’s easy to also pull in the runtime environment - the IaaS and PaaS layer. (And, boy, kubernetes is really looking for it’s PaaS layer - so that world will grab onto any thing that floats by.)

But, this conflation of the two probably isn’t accurate or helpful. And I’ve done it myself!

With the Twitter-speed urgency though, you have to process and come up with takes quickly like this. You can’t sort of just let it play out and see what happens.

Anyhow: it’d be great to have less urgency. The thing with Twitter-speed urgency is that, really, it doesn’t matter. I mean, you could summarize the last two weeks of Twitter like this:

The long awaited PE buyout closed and was followed quickly by layoffs and uncertainty. For example, after laying off half of the staff, some had to be hired back. The new management team was unusually brash and rude, damaging the brand reputation of Twitter. Some big-name advertisers who were already unsure about the value of putting ads in Twitter began to worry about the association with Twitter, paused spend, and went into a wait and see posture. The new team followed the philosophy of “move fast and break things” when exploring new features, but because their theories were wrong, eventually slowed down to take a more considered approach. Meanwhile, the service stayed up, defying expectations. Clearly, the new management team has a lot to learn and even more interest to pay. More than likely, this was all a bad idea.

Following the minutia is fun, but I have to say, there’s not enough of it to make it rewarding. There’s only about one interesting thing every two days - so I’ve realized, there’s no payoff to checking all the time.

…the point being: my urgency calibration is all screwed up from Twitter-brain. And, you know, it’d be great to slide back to blog-brain urgency, at the very least.

This urgency is more about the expectations I set on myself rather than my craving for “news.” One day, I hope to be satisfied with one, solid piece of work published weekly rather than worrying about doing a quick-trickle of a bunch of small things.

From the report “Top Strategic Technology Trends for 2023: Platform Engineering,” Paul Delory and Oleksandr Matvitskyy, Gartner, Oct 2022.

• The authors don’t take a strong position here (?), but I think their vision of platform engineering sits above the infrastructure layer. See the diagram above, for example. The platform engineering group doesn’t mess with that stuff. This seems right to me.
• Everyone loves a Gartner prediction: “By 2026, 80% of software engineering organizations will establish platform teams as internal providers of reusable services, components and tools for application delivery.”
• “Cost savings are unlikely. The platform should improve productivity, cycle time and speed to market, among other important metrics. Expect a good return on investment, but not less investment overall. Direct, cash-out-of-pocket savings are unlikely to materialize.”

This report is free to download, my work licensed it. If you’re reading this, you’ll find it useful, so you should go read it!

Securing Your Environment with Tools Before Rules

My colleague Bryan Ross talk about security in the whole cloud native world. There’s plenty go shift left, and something called “shield right.” Also, he concisely explains the value of having a container-native build service (here, the Tanzu Build Service) and how you can get developers securing their code (better) from the start with Accelerators (templates), and buildpacks.

I think buildpacks are one of the more impressive, under appreciated things from the Cloud Foundry community. The idea of, well, preventing developers from building their own containers is huge if you want control over security, governance, and even basic things like instrumentation and other production management concerns.

You may recall this Bryan from this excellent talk he gave about his experience running a platforms in large organizations. If you’re in this whole cloud thing, you should really watch it if you haven’t.

I’ve tracked at least three different definitions of DevOps since the days of “agile infrastructure”:

1. Using Puppet and Chef (and then Ansible and Chef) to replace Opsware and BladeLogic.
2. Full stack engineers to setup EC2, load-balancers, and other Morlock shit.
3. Full stack engineers are bad, but sort of the same thing. Also, you can’t have a DevOps “group” or title. But, you know, someone should do all that automation.
4. Putting all the people on one team, having them focus on a product, and establishing a culture of caring and learning.
5. SRE is not DevOps.

So…actually five. Maybe some of them just being footnotes on the evolving concept. (And, if you, dear reader, feel these are wrong, then let’s compromise and make the list six.)

All of them evolved around bringing down The Wall of Confusion, allowing “developers” to deploy their software to production more frequently, weekly, if not daily. And, of course, making sure production stays up. (You’re supposed to call that “resiliency” and instead of SLAs use SLOs and some other newly named metrics that answer the question “IS MY SHIT WORKING?” Whatever you do, just don’t say “uptime," or you’re in for it and will be relegated to running the AS/400’s.)

I used to snide that the developers seemed to have been yanked out of DevOps, sometime around 2014 and 2015. All the talks I saw were, basically, operations talks. I haven’t really checked in on DevOps conference talks recently, but at the time, I don’t think there was much application development stuff. (I’m not sure if there ever was?)

None of this means that DevOps is not a thing. Not at all. It just means that the enterprise finds its own use for things. It also means there’s still weekly write-ups of what DevOps is - you know, those ones that are always lists of ideas, things you’re getting wrong, and how to start.

Autonomous product teams

Nowadays, I try to stick to that fourth one: you want to set up autonomous teams that have all the skills and responsibility/authority/tools needed to “own” the software being specified, designed, developed, and run. This means you have to, basically, remove-by-automating all the operations stuff it takes to stand-up environments, deploy things, and do all that “day 2” stuff.

(HEY! HEY! WANT TO BUY SOME ENTERPRISE SOFTWARE?!)

Now, I think this product-centric notion of DevOps is, well, kind of an over-extension of the term “DevOps.” But since SRE has sucked out the “ops” part (but, remember, dear reader, don’t commit the embarrassing act of saying SRE is DevOps - no, no, you’d never do that, right? SO SHAMEFUL! (SRE is totally different - no overlap or similar goals shared between them at all. I mean, they have separate groups, silos! COME ON!)), slicing “DevOps” back to just “Dev,” but with a product-not-project focus isn’t too shabby.

Anyhow. I came across a good overview of this product notion of DevOps, all the way back from 2016, while re-reading Schwartz’s evergreen excellent The Art of Business Value:

Agile approaches attempt to bring together developers and the business in an atmosphere of mutual respect and joint contribution. Until now, however, the focus has been on users of the software, product visionaries, and developers. Recent developments in the Agile world—notably DevOps—have broadened this idea of respect and inclusion to encompass Operations and Security. The DevOps model, in other words, looks to break down the silos that have resulted from technical specialization over the last few decades. But the DevOps spirit goes further, looking to eliminate the conflicting incentives of organizational silos and the inhumane behaviors that can result from those conflicting incentives.

Perhaps we can take this idea even further still. There is no reason why the DevOps team’s responsibility needs to stop at the border of what used to be considered IT. The team is part of a broader enterprise, whose collective knowledge, skills, and judgment need to be part of the value creation process.

Look a' that guy! Business Value just effortlessly jets out of his pores like a peripatetic thought-monarch!

This is from an executives' perspective, but it drives home the point we’re always trying to get to with software: doing whatever it takes to figure out, create, and give users features that are actually useful to them. Somewhere beyond that, if you’re lucky, it’ll help out “the business.” Also, it should implement The Unspoken User Story: user would like software to actually work.

As a DevOpsDays sponsor you’re often given the chance to give a one minute pitch to the entire audience. Back stage at DevOps Rex, this week, I was talking with a first timer. One minute seems like such a small amount of time: how could you say anything consequential in 60 seconds? You’re presenting in front of the full audience, anywhere between 150 to 500 people. They probably also loath vendors, or, at least are bored by them. The stage in Paris is intimidating. It’s a huge room in an old cinema, imagine the most stereotypical movie theater from whatever “the golden age” is: double decked seats, a huge screen. Plus, the organizers are meticulous: there’s a rehearsal for these 1 minute pitches in the morning. Like a full one where you’re given a minute to talk. Normally, these pitches are very informal. Overall, it can be a public speaking challenge…plus you have to get up an hour earlier than normal.

People get rattled by these 1 minute talks and they can also give boring pitches. Here’s what how I think about them and what I do:

1. The goal of this pitch is to tell people the name of your company, what you do, and to get them to come to your table to talk more.
2. So, tell them the name of your company, what you do, if you have time, the story of a customer who did something remarkable, and give people a reason to come to your booth.
3. People will come to your booth if you give them something: books, socks, flash lights, whatever. More senior, “decision makers” also want free stuff (they’re not monsters!), but they’ll also come to see how you can help them accomplish their work goals.
4. If you can tell a joke, even a lame one (of course, not an offensive one), do it, usually towards the start of your pitch. Getting a room full of people to laugh gets them engaged and listening closer. Your pitch more memorable, and also will give you a confidence boost to coast off for the next 45 to 30 seconds.
5. Finally, if you screw up, it does’t matter. It’s just 60 seconds so it’ll be over quickly and people will still come by your booth if the organizers have done their job for vendors: arranged the sponsor booth placement to drive foot traffic.

Figuring out what to say

The people like socks.

Despite how disorganized and spontaneous I may appear (that’s part of my well planned out and cultivated schtick, a safety valve for when I haven’t prepared, plus it’s a fantastically caustic feedback loop for my self-loathing — yay!) I usually prepare content before each talk.

I write a bunch of points down and reduce it to three points that I want to make. As I wait to get on the stage, I go over these three points my head; I usually write them down and look at them. Ask the local sales people what the make up of the audience is (are the developers, ops people, management, or just a general audience?), and any local events they want to drive people to.

Now, I often forget most, if not all, of that content, but that’s fine, really. Some of it will show-up. And definitely don’t let your three points constrict you, just use them as a fallback and a suggestion.

Being at a DevOps event, you should probably talk about how your company relates to DevOps. I tell people that Pivotal Cloud Foundry removes all the toil of lower-level automation that DevOps is looking to eliminate, the A in CAMS. It makes DevOps real, solves you DevOps problems, et. al., so you can get to the whole reason (the “outcome,” in business speak) for doing DevOps: creating better software and running it reliability in production.

De-wooding

The main thing you want to avoid is being stuffy. If you’re wearing a sports jacket (without being ironic), I’ve found that you’re more likely to give a stuffy talk — someone like Damon Edwards can sports-jacket all day, but he’s the exception that proves the rule.

If you’re just naturally wooden in public speaking situations, try to say something about your involvement in the pitch: how does it make you feel and how do you relate to it? Talking about yourself is easy as you’re the expert on the topic and have hopefully been there the whole time.

A little bit of humor goes a long way in these tiny talks. For example, Pivotal’s main product is well known for being more expensive than free, but it works and changes the fortunes of organizations that use it. That’s a good thing to joke about (“good thing it actually works ’cause it ain’t cheap”), or weird branding names (“for some reason, we call these ‘platform engineers’ rather than ‘SREs’”). I sometimes make a joke about PaaS, the cloud category Pivotal Cloud Foundry is in: “remember PaaS from five or so years back? It was terrible! Well, we’re a PaaS, but we doesn’t suck so much this time, it actually works!”

Don’t worry about it

The stakes of this pitch are extremely low. Look at it as more of a learning experience for yourself, practice for next time. If you biff, nothing bad will happen unless you work for shitty management that punishes you for 60 seconds of time (start looking for a new job — Pivotal is hiring!).

Some people like to memorize pitches, which is fine if that helps you. Most of all, the way to succeed at these pitches it to have fun, be playful.

You’ll be fine. Good luck!

There are two types of digital transformation. First, literally. You had an analog process (booking appointments at the barber shop with phone and paper, planning tanker refueling schedules with a white board), and now they’re replaced by pure software. These transformations are often about optimizing an existing business process, gaining huge cost and time efficiencies (bottom line revenue, profit) and resulting in higher customer productivity and satisfaction (top line revenue, “growth”). You might call this sustaining in the innovator’s dilemma canon.

Then there’s creating new business and innovating existing ones. Here, you use custom written software to create a new line of business (in-Home hair cuts) or make an existing one better (ordering pizza from an Apple Watch, using drones to do auto accident claims, entering a new market, wealth Management for lower income people, on-demand [uber like] Long-hail trucker scheduling).

The line between the two can be blurry (is using drones for claims inspections really just changing an analog process to digital? A better one might be doing spot/temples insurance [i want to buy a three hour policy for broken limbs and such minutes before I climb into a zip line, grocery curb-side or home delivery]). But don’t get too worked up about it: the work you end up doing ends up being the same.

The point of the distinction is more to make sure you don’t forget about one or the other. Sometimes you need a dramatically new line of business enabled by software, and sometimes you just need to transform a whiteboard into an app.

An airport is a time pause. It’s an excuse to not stress or try. You’re trapped in the system and will eventually get there. You can’t leave or you’ll have to re-humiliate yourself through security. Airports are even powerful enough to make you cancel meetings if your flight is late, canceled…or you pretend it is. Your wedding could be delayed because of the airport and no one would really fault you.

Everyone is transiting, coming and going, and while the entry fee might exclude the very poor (and the super rich fly their own), you see everyone.

At a major hub, you’ll see people from all over: the guy with the “Ragin' Cajun” hat, domestic and international grandmas, the harried big-city lawyer, the dad-jeans set, and the local staff. People dress in all manners of business-business or super casual for comfort.

The mix of experienced and novice travels creates a crackly dynamic, paired with either overly friendly or direct gate agents. While some can escape to airline lounges, even those environments are little different from the actual terminal: you just get much friendly staff and free drinks and peanuts.

Airports can be calming if you look at them as escapes and the sort of delightful, enforced boredom that I understand meditation to be.

They can be toxic if you stress out about delays, lines, other people, overhead bin space, and how flight delays affect]] your plans outside the airport. And they can be distracting like an opium den if you let their peaceful hum shut-out your real life.

Don’t ruin your time at the airport. If you let it, it’ll make sure you get back out right where you wanted to go.

I get all ants-in-pants about this whole bi-modal discussion because I feel like it’s a lot of energy spent talking about the wrong things.

This came up recently when I was asked about “MVP”, in a way that basically was saying “our stuff is dangerous [oil drilling], so ‘minimal’ sounds like it’d be less safe.” I tried to focus them on the “V” and figure out what “viable” was for their situation. The goal was to re-enforce that the point of all this mode 2/small batch/DevOps/PDCA/cloud native/OODA nonsense is to keep iterating to get to the right code.

Part of the continual consternation around bi-modal IT - sad/awesome mode - is misalignment around that “viability” and scoping on “enterprise” projects. This is just one seam-line around the splits of the discussion being unhelpful

Bi-strawperson

The awesome mode people are like:

You should divide the work into small chunks that you release to production as soon as possible - DevOps, Agile, MVP, CI/CD - POW! You have no idea what or how you should implement these features so you need to iteratively do it cf. projectcartoon.com

And the sad mode folks are like:

Yes, but we have to implement all this stuff all at once and can’t do it in small slices. Plus, mainframes and ITIL.

Despite often coming off as a sad mode apologist, I don’t even know what the sad mode people are thinking. There’s this process hugger syndrome that, well on both sides really, creates strawpeople. The goal of both methods is putting out software that makes users more productive, including having it actually work, and not overpaying for the whole thing.

The Enemy is finding any activity that doesn’t support those goals and eliminated it as much as possible. In this, there was some good scrabbling from the happy mode people laughing at ITSM think early on, but at this point, the sad people have gotten the message, have been reminded of their original goal, and are now trying to adapt. In fact, I think there’s a lot the “sad mode” people could bring to the table.

To play some lexical hopscotch, I don’t think there is a “mode 1.” I think there’s just people doing a less than awesome job and hiding behind a process-curtain. Sure, it may to be their choice and, thus, not their fault. “Shitty jobs are being done,” if you prefer the veil of passive voice.

Fix your shit

When I hear objections to fixing this situation, I try to b nice and helpful. After all, I’m usually there as part of an elaborate process to get money from these folks in exchange for helping them. When they go all Eeyore on me, I have to reframe the room’s thinking a little bit without getting too tough love-y.

“When I put these lithium batteries in this gas car, it doesn’t seem to work. So electric cars are stupid, right?

You want to walk people to asking “how do we plan out the transition from The Old Way That Worked At Some Point to The New Way That Sucks Less?” They might object with a sort of “we don’t need to change” or the even more snaggly “change is too hard” counter-point.

I’m not sure there are systems that can just be frozen in place and resist the need to change. One day, in the future, any system (even the IRS’!) will likely need to change and if you don’t already have it setup to change easily (awesome mode), you’re going to be in a world of hurt.

The fact that we discuss how hard it is to apply awesome mode to legacy IT is evidence that that moment will come sooner than you think.

(Insert, you know, “where’s my mobile app, Nowakowski?” anecdote of flat-footedness here.)

ITIL end in tears(tm)

The royal books of process, ITIL, are another frequent strawperson that frothy mouthed agents of change like to light up. Few things are more frustrating than a library of books that cost £100 each. There’s a whole lot in there, and the argument that the vendors screw it all up is certainly appetizing. Something like ITIL, though, even poorly implemented falls under the “at least it’s an ethos” category.

I’m no IT Skeptic or Charles T. Betz, but I did work at BMC once: as with “bi-modal,” and I really don’t want to go re-read my ITIL books (I only have the v2 version, can someone spare a few £100’s to read v3/4?), but I’m pretty sure you could “do DevOps” in a ITIL context. You’d just have to take out the time consuming implementation of it (service desks, silo’d orgs, etc.).

Most of ITIL could probably be done with the metaphoric (or literal!) post-it notes, retrospectives, and automated audit-log stuff that you’d see in DevOps. For certain, it might be a bunch of process gold-plating, but I’m pretty sure there’s no unmovable peas under all those layers of books that would upset any slumbering DevOps princes and princesses too bad.

Indeed, my recollection of ITIL is that it merely specifies that people should talk with each other and avoid doing dumb shit, while trying to improve and make sure they know the purpose/goals of and “service” that’s deployed. They just made a lot of flow charts and check lists to go with it. (And, yeah: vendors! #AmIrightohwaitglasshouse.)

Instead of increasing the volume, help spray away the shit

That gets us back to the people. The meatware is what’s rotting. Most people know they’re sad, and in their objections to happiness, you can find the handholds to start helping:

Yes, awesome mode people, that sounds wonderful, just wonderful. But, I have 5,000 applications here at REALLYSADMODECOGLOBAL, Inc. - I have resources to fix 50 of them this year. YOUR MOVE, CREEP!

Which is to say, awesome mode is awesome: now how do we get started in applying it at large orginizations that are several fathoms under the seas of sad?

The answer can’t be “all the applications,” because then we’ll just end up with 5,000 different awesome modes (OK, maybe more like 503?) - like, do we all use Jenkins, or CircleCI, or Travis? PCF, Docker, BlueMix, OpenShift, AWS, Heroku, that thing Bob in IT wrote in his spare time, etc.

Thus far, I haven’t seen a lot of commentary on planning out and staging the application of mode 2. Gartner, of course, has advice here. But it’d be great to see more from the awesome mode folks. There’s got to be something more helpful than just “AWESOME ALL THE THINGS!”

Thanks to Bridget for helping draw all this blood out while I was talking with her about the bi-modal pice she contributed to.

I have a larger piece on common objections to “cloud native” that I’ve encountered over the last year. Put more positive, “how to get your digital transformation started with a agile, DevOps, and cloud native” or some such platitudinal title like that. Here’s a draft of the dread-ROI section.

The most annoying buzzkill for changing how IT operates (doing agile, DevOps, taking “the cloud native journey,” or whatever you think is the opposite of “waterfall”) is the ROI counter-measure. ROI is a tricky hurdle to jump because it’s:

1. Highly situational and near impossible to properly prove at the right level — do you want to prove ROI just within the scope of the IT department, or at the entire business-level? What’s the ROI of missing out on transitioning Blockbuster to Netflix? What’s the ROI of a mobile app for a taxi company when Uber comes along? What’s the ROI for investing in a new product that may or may not work within three years, but will save the company’s bacon in five years?
2. Compounded by the fact that the “value” of good software practices is impossible to predict. Drawing the causal lines between “pair programming” and “we increased market-share by 3% in Canada” can be a hard line to draw. You can back-think a bunch of things like “we reduced defects and sped up code review time by pairing,” but does that mean you made more money, or did you make more money because the price of oil got halved?

In my experience, when people are asking you about ROI, what they’re asking is “how will I know the time and money I’m going to spend on this will pay off and, thus, I won’t lose time and money? (I don’t want to look like a fool, you see, at annual review time)”

What they’re asking is “how do I know this will work better than what I’m currently doing or alternatives.” It also usually means, “hey vendor, prove to me that I should pay you.”

As I rambled through last year, I am no ROI expert. However, I’ve found two approaches that seem to be more something than nothing: (1.) creating a business case and trusting that agile methods will let you succeed, and, (2.) pure cost savings from the efficiencies of agile and “cloud native.”

A Business Case

A business case can tell you if your approach is too expensive, but not if it will pay for itself because that depends on the business being successful.

Here, you come up with a new business idea, a product, service, or tweak to an existing one of those. “We should set up little kiosks around town where people can rent DVDs for a $1 a day. People like renting DVDs. We should have a mobile app where you can reserve them because, you know, people like using mobile. We should use agile to do this mobile app and we’re going to need to run it somewhere, like ‘the cloud.’ So, hey, IT-nerds, what’s the ROI on doing agile and paying for a cloud platform on this?”

In this case, you (said “IT-nerds”) have some externally imposed revenue and profit objectives that you need to fit into. You also have some time constraints (that you’ll use to push back on bloated requirements and scope creep when they occur, hopefully). Once you have these numbers, you can start seeing if “agile” fits into it and if the cost of technology will fit your budget.

One common mis-step here is to think of “cost” as only the licensing or service fees for “going agile.” The time it takes to get the technology up and running and the chance that it will work in time are other “costs” to account for (and this is where ROI for tech stuff gets nasty: how do you put those concerns into Excel?).

To cut to the chase, you have to trust that “agile” works and that it will result in the DVD rental mobile app you need under the time constraints. There’s no spreadsheet friendly thing here that isn’t artfully dressed up qualitative thinking in quantitate costumes. At best you can point to things like the DevOps reports to show that it’s worked for other people. And for the vendor expenses, in addition to trusting that they work, you have to make sure the expenses fit within your budgets. If you’re building a $10m business, and the software and licensing fees amount to $11m, well, that dog won’t hunt. There are some simple, yet helpful numbers to run here like the TCO for on-premises vs. public cloud fees.

Of course, a major problem with ROI thinking is that it’s equally impossible to get a handle on competing ways to solve the problem, esp. the “change nothing” alternative. What’s the ROI of how IT currently operates? It’d be good to know that so you can compare it to the proposed new way.

If you’re lucky enough to know a realistic, helpful budget like this, your ROI task will be pretty easy. Then it’s just down to horse-trading with your various enterprise sales reps. Y’all have fun with that.

Efficiency

Focus on removing costs, not making money.

If you’re not up for the quagmire of business case-driven ROI, you can also discuss ROI in terms of “savings” the new approach affords. For things like virtualizing, this style of ROI is simple: we can run 10 servers on one server now, cutting our costs down by 70–80% after the VMware licensing fees.

Doing “agile,” however, isn’t like dropping in a new, faster and cheaper component into your engine. Many people I encounter in conference rooms think about software development like those scenes from 80s submarine movies. Inevitably, in a submarine movie, something breaks and the officer team has to swipe all the tea cups off the officer’s mess table and unfurl a giant schematic. Looking over the dark blue curls of a thick Eastern European cigarette, the head engineer gestures with his hand, then slams a grimy finger onto the schematics and says “vee must replace the manifold reducer in the reactor.”

Solving your digital transformation problems is not like swapping “agile” into the reactor. It’s not a component-based improvement like virtualization was. Instead, you’re looking at process change (or “culture,” as the DevOps people like to say), a “thought technology.” I think at best what you can do is try to calculate the before and after savings that the new process will bring. Usually, this is trackable in things like time spent, tickets opened, number of staff needed, etc. You’re focusing on removing costs, not making money. As my friend Ed put it when we discussed how to talk about DevOps with the finance department:

In other words, if I’m going to build a continuous integration platform, I would imagine you could build out a good scaffolding for that and call it three or four months. In the process of doing that, I should be requiring less help desk tickets get created so my overtime for my support staff should be going down. If I’m virtualizing the servers, I’ll be using less server space and hard drive space, and therefore that should compress down. I should be able to point to cost being stripped out on the back end and say this is maybe not 100% directly related to this process, but it’s at least correlated with it.

In this instance, it’s difficult to prove that you’ll achieve good ROI ahead of time, but you can at least try to predict changes informed by the savings other people have had. And, once again, you’re left to making a leap of faith that qualitative anecdotes from other people will apply to you.

For example, part of Pivotal’s marketing focuses on showing people the worth of buying a cloud platform to support an agile approach to software deliver (we call that “cloud native”). In that conversation, I cite figures like this:

• Developers at Allstateused to spend only 20% of their time coding and now it’s closer to 90%
• A federal government agency wanted to save money on call-centers by converting the workflow to a web app. They’d scheduled to complete the project in 9 months, but after converting to agile, delivered it in 6 weeks.
• When doing agile because testing is pushed down to the team level and automated, you can expect to reduce your traditional QA spend. In fact, many shops on the cloud native journey have massively eliminated their QA department as a stand-alone entity.
• ING’s savings from transforming to a more cloud-y IT setup: “Investment of €200m to further simplify, standardize and automate IT; Decommissioning 40% of application landscape; Moving 80% of applications to zero-touch private cloud.” Resulting in savings of €270m starting in 2018.
• From Orange: “Who isn’t happy to continue working when projects are delivered on average six times faster than with a waterfall approach?”
• “[R]espondents from a recent government study who have already used PaaS say they save 47% of their time, or 1 year and 8 months off a 3.5 year development cycle. For those who have not deployed PaaS, respondents believe it can shave 31% off development time frames and save 25% of their annual IT budget, a federal savings of $20.5 billion.”
• 14 months down to 6 months, 16 staff down to 8 staff: “[w]hen planning the first product developed on Pivotal Cloud Foundry, CoreLogic allocated a team of 12 engineers with four quality assurance software engineers and a management team. The goal was to deliver the product in 14 months. Instead, the project ultimately required only a product manager, one user experience designer and six engineers who delivered the desired product in just six months.”
• “We did an analysis of hundreds of projects over a multiyear period. The ones that delivered in less than a quarter succeeded about 80% of the time, while the ones that lasted more than a year failed at about the same rate. We’re simply not very good at large efforts.”
• From a 1999 study: “software projects that use iterative development deliver working software 38% sooner, complete their projects twice as fast, and satisfy over twice as many software requirements.”
• After switching over to “the new way,” one large retailer has already seen 80% Reduction in cycle time and scope and reduced cycle time from 123 days to 23 days.
• One large insurance company can now manage 1,500 apps with just two operators. There were many, many more before that. Another large bank could manage 145 apps with just 2 operators, and so on.

In most of these cases, once you switch over to the new way, you end up with extra capacity because you can now “do IT” more efficiently. Savings, then, come from what you decide to do with that excess capacity: (a.) doing more with the new capacity like adding more functionality to your existing businesses, creating new businesses, or entering new markets, or, (b.) if you don’t want to “grow,” you get rid of the expense of that excess capacity (i.e., lay-off the excess staff or otherwise get them off out of the Excel sheet for your business case).

But, to be clear, you’re back into the realm of imagining and predicting what the pay-off will be (the “business case” driven ROI from above) or simply stripping out costs. It’s a top-line vs. bottom-line discussion. And, in each case, you have to take on faith the claims about efficiencies, plus trust that you can achieve those same savings at your organizations.

With these kinds of numbers and ratios, the hope is, you can whip out a spreadsheet and make some sort of chart that justifies doing things the new way. Bonus points if you use Monte Carlo inspired ranges to illustrate the breadth of possibilities instead of stone-code line-graph certainty.

Everything is up when there’s no bottom

As an added note of snark: all of these situations assume you know the current finances for the status quo way of operating. Surely, with all that ITIL/ITSM driven, mode 1 thinking you have a strong handle on your existing ROI, right? (Pause for laughs.)

More seriously, the question of ROI for thought technologies is extremely tricky. In that conversation on this topic that I had with Ed last year, the most important piece of advice was simple: talk with the finance people more and explain to them what’s going on.

That’s the most effective (and least satisfying!) advice you get about any of this “doing things the new way” change management prattle: whether it’s auditors, DBAs, finance, PMO people, or whoever is throwing chaff in your direction: just go and talk with them. Understand what it is they need, why they’re doing their job, and bring them onto the team instead of relegating them to the role of The Annoying Others.

Check out another take on this over in my September 2016 column at The Register.

Most of what we do as white-collar workers is help our organization come to a decision. What new geographies to sell our enterprise software and toothpaste in, what pricing to make our electric razors and cranes, which people to fire and which to promote, or how much budget is needed according to the new corporate strategy. Even in the most cynical corporate environment, asking questions — and getting answers! — is the best, primary way to set the stage for making a decision.

You have to be careful, however, of how many questions you ask, and on what topic. If you ask too many questions, you may find that you’ll just create more work for yourself. Before asking just any old question, ask yourself if you’re willing to do the work needed to answer it…because as the asker, you’ll often be tasked with doing that work. We see this in life all the time, you ask someone “want to go to lunch?” and next thing you know, you’re researching in all the restaurants within five miles that have gluten-free, vegan, and steak options.

I’ve seen countless “staffers” fall prey to this in meetings with managers who’re putting together plans and strategies. The meeting has spent about 30 minutes coming up with a pretty good plan, and then an eager staffer pipes up and suggests 1–3 other options. The manager is intrigued! Quickly, that staffer is asked to research these other options, but, you know, the Big Meeting is on Monday, so can you send me a memo by Sunday morning?

In some situations, this is fine and expected. But in others, conniving management will just use up as much as your energy as possible: it’s always better to have done more research, so why not let that eager staffer blow-up their Saturday to have more back-up slides? Co-workers can also let you self-assign homework into burnout if they find you annoying: you’ll notice that when you, the eager staffer, pipe up, they go suddenly quiet and add no input.

As always, you have to figure out your corporate culture. But, just make sure that before you offer up alternatives and otherwise start asking The Big Questions, you’re read to back up those questions by doing the extra homework to answering them yourself.

A kingmaker in the making.

I’ve talked with an old colleague about pitching a developer-based strategy recently. They’re trying to convince their management chain to pay attention to developers to move their infrastructure sales. There’s a huge amount of “proof” an arguments you can make to do this, but my experience in these kinds of projects has taught me that, eventually, the executive in charge just has to take a leap of faith. There’s no perfect slide that proves developers matter. As with all great strategies, there’s a stack of work, but the final call has to be pure judgement, a leap of faith.

“Why are they using Amazon instead of our multi-billion dollar suite?”

You know the story. Many of the folks in the IT vendor world have had a great, multi-decade run in selling infrastructure (hardware and software). All the sudden (well, starting about ten years ago), this cloud stuff comes along, and then things look weird. Why aren’t they just using our products? To cap it off, you have Apple in mobile just screwing the crap out of the analogous incumbents there.

But, in cloud, if you’re not the leaders, you’re obsessed with appealing to developers and operators. You know you can have a “go up the elevator” sale (sell to executives who mandate the use of technology), but you also see “down the elevator” people helping or hindering here. People complain about that SOAP interface, for some reason they like Docker before it’s even GA’ed, and they keep using these free tools instead of buying yours.

It’s not always the case that appealing to the “coal-facers” (developers and operators) is helpful, but chances are high that if you’re in the infrastructure part of the IT vendor world, you should think about it.

So, you have The Big Meeting. You lay out some charts, probably reference RedMonk here and there. And then the executive(s) still isn’t convinced. “Meh,” as one systems management vendor exec said to me most recently, “everyone knows developers don’t pay for anything.” And then, that’s the end.

There is no smoking gun

If you can’t use Microsoft, IBM, Apple, and open source itself (developers like it not just because it’s free, but because they actually like the tools!) as historic proof, you’re sort of lost. Perhaps someone has worked out a good, management consultant strategy-toned “lessons learned” from those companies, but I’ve never seen it. And believe me, I’ve spent months looking when I was at Dell working on strategy. Stephen O’Grady’s The New Kingmakers is great and has all the material, but it’s not in that much needed management consulting tone/style. (I’m ashamed to admit I haven’t read his most recent book yet, maybe there’s some in there.)

Of course, if Microsoft and Apple don’t work out as examples of “leaders,” don’t even think of deploying all the whacky consumer-space folks out like Twitter and Facebook, or something as detailed as Hudson/Jenkins or Oracle DB/MySQL/MariaDB.

I think SolarWinds might be an interesting example, and if Dell can figure out applying that model to their Software Group, it’d make a good case study. Both of these are not “developer” stories, but “operator” ones; same structural strategy.

Eventually, they just have to “get it”

All of this has lead me to believe that, eventually, the executives have to just take a leap of faith and “get it.” There’s only so much work you can do — slides and meetings — before you’re wasting your time if that epiphany doesn’t happen.

The transformation is complete.

If this is your bag, come check out a panel on the developer relations at the OpenStack Summit on April 28th, in Austin — I’ll be moderating it!

Hey, I’ve not only seen this movie before, I did some script treatments:

Chief Executive Officer John Chambers is aggressively pursuing software takeovers as he seeks to turn a company once known for Internet plumbing products such as routers into the world’s No. 1 information-technology company. … Cisco is primarily targeting developers of security, data-analysis and collaboration tools, as well as cloud-related technology, Chambers said in an interview last month.

Good for them. Cisco has consistently done a good job to fill out its portfolio and is far from the one-trick pony people think it is (last I checked, they do well with converged infrastructure, or integrated systems, or whatever we’re supposed to call it now). They actually have a (clearly from lack of mention in this piece) little known-about software portfolio already.

In case anyone’s interested, here’s some tips:

1.) Don’t buy already successful companies, they’ll soon be old tired companies

Software follows a strange loop. Unlike hardware where (more or less) we keep making the same products better, in software we like to re-write the same old things every five years or so, throwing out any “winners” from the previous regime. Examples here are APM, middleware, analytics, CRM, web browsers…well…every category except maybe Microsoft Office (even that is going bonkers in the email and calendaring space, and you can see Microsoft “re-writing” there as well [at last, thankfully]). You want to buy, likely, mid-stage startups that have proven that their product works and is needed in the market. They’ve found the new job to be done (or the old one and are re-writing the code for it!) and have a solid code-base, go-to-market, and essentially just need access to your massive resources (money, people, access to customers, and time) to grow revenue. Buy new things (which implies you can spot old vs. new things).

2.) Get ready to pay a huge multiple

When you identify a “new thing” you’re going to pay a huge multiple of 5x, 10x, 20x, even more. You’re going to think that’s absurd and that you can find a better deal (TIBCO, Magic, Actuate, etc.). Trust me, in software there are no “good deals” (except once in a lifetime buys like the firesale fro Remedy). You don’t walk into Tiffany’s and think you’re going to get a good deal, you think you’re going to make your spouse happy.

3.) “Drag” and “Synergies” are Christmas ponies

That is, they’re not gonna happen on any scale that helps make the business case, move on. The effort it takes to “integrate” products and, more importantly, strategy and go-to-market, together to enable these dreams of a “portfolio” is massive and often doesn’t pan out. Are the products written in exactly the same programming language, using exactly the same frameworks and runtimes? Unless you’re Microsoft buying a .Net-based company, the answer is usually “hell no!” Any business “synergies” are equally troublesome: unless they already exist (IBM is good at buying small and mid-companies who have proven out synergies by being long-time partners), it’s a long-shot that you’re going to create any synergies. Evaluate software assets on their own, stand-alone, not as fitting into a portfolio. You’ve been warned.

4.) Educate your sales force. No, really. REALLY!

You’re thinking your sales force is going to help you sell these new products. They “go up the elevator” instead of down so will easily move these new SKUs. Yeah, good luck, buddy. Sales people aren’t that quick to learn (not because they’re dumb, at all, but because that’s not what you pay and train them for). You’ll need to spend a lot of time educating them and also your field engineers. Your sales force will be one of your biggest assets (something the acquired company didn’t have) so baby them and treat them well. Train them.

5.) Start working, now, on creating a software culture, not acquiring one

The business and processes (“culture”) of software is very different and particular. Do you have free coffee? Better get it. (And if that seems absurd to you, my point is proven.) Do you get excited about ideas like “fail fast”? Study and understand how software businesses run and what they do to attract and retain talent. We still don’t really understand how it all works after all these years and that’s the point: it’s weird. There are great people (like my friend Israel Gat) who can help you, there’s good philosophy too: go read all of Joel’s early writing of Joel’s as a start, don’t let yourself get too distracted by Paul Graham (his is more about software culture for startups, who you are not — Graham-think is about creating large valuations, _not_ extracting large profits), and just keep learning. I still don’t know how it works or I’d be pointing you to the right URL. Just like with the software itself, we completely forget and re-write the culture of software canon about every five years. Good on us. Andrew has a good check-point from a few years ago that’s worth watching a few times.

6.) Read and understand Escape Velocity

This is the only book I’ve ever read that describes what it’s like to be an “old” technology company and actually has practical advice on how to survive. Understand how the cash-cow cycle works and, more importantly for software, how to get senior leadership to support a cycle/culture of business renewal, not just customer renewal.

7.) There’s more, of course, but that’s a good start

Finally, I spotted a reference to Stall Points in one of Chambers’ talks the other day which is encouraging. Here’s one of the better charts you can print out and put on your wall to look at while you’re taking a pee-break between meetings:

That charts all types of companies. It’s hard to renew yourself, it’s not going to be easy. Good luck!

Figuring out the market for PaaS has always been difficult. At the moment, I tend to estimate it at $20–25bn sometime in the future (5–10 years from now?) based on the model of converting the existing middleware and application development market. Sizing this market has been something of an annual bug-bear for me across my time at Dell doing cloud strategy, at 451 Research covering cloud, and now at Pivotal.

A bias against private PaaS

This number is in contrast to numbers you usually see in the single digit billions from analysts. Most analysts think of PaaS only as public PaaS, tracking just Force.com, Heroku, parts of AWS, Azure, and Google, and bunch of “Other.” This is mostly due, I think, to historical reasons: several years ago “private cloud” was seen as goofy and made-up, and I’ve found that many analysts still view it as such. Thus, their models started off being just public PaaS and have largely remained as so.

I was once a “public cloud bigot” myself, but having worked more closely with large organizations over the past five years, I now see that much of the spending on PaaS is on private PaaS. Indeed, if you look at the history of Pivotal Cloud Foundry, we didn’t start making major money until we gave customers what they wanted to buy: a private PaaS platform. The current product/market fit, then, for PaaS for large organizations seems to be private PaaS

(Of course, I’d suggest a wording change: when you end-up running your own PaaS you actually end-up running your own cloud and, thus, end up with a cloud platform. Also, things are getting even more ambiguous at the infrastructure layer all the time — perhaps “private PaaS” means more “owning” the PaaS layer, regardless of who “owns” the IaaS layer.)

How much do you have budgeted?

With this premise — that people want private PaaS — I then look at existing middleware and application development market-sizes. Recently, I’ve collected some figures for that:

• IDC’s Application Development forecast puts the application development market (which includes ALM tools and platforms) at $24bn in 2015, growing to $30bn in 2019. The commentary notes that the influence of PaaS will drive much growth here.
• Recently from Ovum: “Ovum forecasts the global spend on middleware software is expected to grow at a compound annual growth rate (CAGR) of 8.8 percent between 2014 and 2019, amounting to $US22.8 billion by end of 2019.”
• And there’s my old pull from a Goldman Sachs report that pulled from Gartner, where middleware is $24bn in 2015 (that’s from a Dec 2014 forecast).

When dealing with large numbers like this and so much speculation, I prefer ranges. Thus, the PaaS TAM I tent to use now-a-days is something like “it’s going after a $20–25bn market, you know, over the next 5 to 10 years.” That is, the pot of current money PaaS is looking to convert is somewhere in that range. That’s the amount of money organizations are currently willing to spend on this type of thing (middleware and application development) so it’s a good estimate of how much they’ll spend on a new type of this thing (PaaS) to help solve the same problems.

Things get slightly dicey depending on including databases, ALM tools, and the underlying virtualization and infrastructure software: some PaaSes include some, none, or all of these in their products. Databases are a huge market (~$40bn), as is virtualization (~$4.5bn). The other ancillary buckets are pretty small, relatively. I don’t think “PaaS” eats too much database, but probably some “virtualization.”

So, if you accept that PaaS is both public and private PaaS and that it’s going after the middleware and appdev market, it’s a lot more than a few billion dollars.

Satisfying the mythical auditors is often one of the first barriers to spreading DevOps initiatives more widely inside an organization. While these process-driven barriers can be annoying and onerous, once you follow the DevOps tradition of empathetic inclusion — being all “one team” — they can not only stop slowing you down but actually help the overall quality of the product. Indeed, the very reason these audit checks were introduced in the first place was to ensure overall quality of the software and business. There’s some excellent, exhaustive overviews out there of dealing with audits and the like in DevOps. In this column, I wanted to go through a little mental re-orientation for how to start thinking about and approaching the “compliance problem.”

Three-Ring Binder Ninjas

In this context, I think of “auditors” as falling into the category of governance, risk and compliance (GRC) — any function that acts as a check on code as and how the code is produced and run as it goes through its lifecycle. I would put security in here as well, though that tends to be such a broad, important topic that it often warrants its own category (and the security people seem to like maintaining their occultic silo-tude, anyhow).

The GRC function(s) may impose self-created policies (like code and architectural review), third party and government imposed regulations (like industry standard compliance and laws such as HIPAA), and verification that risky behavior is being avoided (if you write the code, you can’t be the same person who then uses that code for cash payouts, perhaps, to yourself, for example). In all cases, “compliance” is there to ensure overall quality of the product and the process that created it. That “quality” may be the prevention of malicious and undesired behavior; that is, in a compliance-driven software development mindset, the ends rarely justify the means.

In many cases, the GRC function is more interested in proof that there is a process in place than actually auditing each execution of that process. This is a curious thing at first. Any developer knows that the proof is in the code, not the documentation. And, indeed, for some types of GRC the amount of automation that a DevOps mindset puts into place could likely improve the quality of GRC, ironically.

Establishing trust and automating compliance

Indeed, automation is one of the first areas to look at when reducing DevOps/GRC friction. First, treat complying with policies as you would any other feature. Describe it, prioritize it and track it. Once you have gotten your hands around it, you can start figure out how to best implement that “feature.” Ideally, you can code and automate your way out of having to do too much manual work.

There’s work being done in the US Federal government along these lines that’s helpful because it’s visible and at scale. First, as covered in a recent talk by Diego Lapiduz, part of what auditors are looking for is to trust the software and infrastructure stack that apps are running on. This is especially true from a security standpoint. The current way that software is spec’d out and developed in most organizations follows a certain “do whatever,” or even YOLO principal. App teams are allowed to specify which operating systems, orchestration layers and middleware components they want. This may be within an approved list of options, but more often than not it results in unique software stacks per application.

As outlined by Diego, this variation in the stack meant that government auditors had to review just about everything, taking up to months to approve even the simplest line of code. To solve this problem, 18F standardized on one stack — Cloud Foundry — to run applications on, not allowing for variance at the infrastructure layer. They then worked with the auditors to build trust in the platform. Then, when there was just the metaphoric or literal “one line of code” to deploy, auditors could focus on much less, certainly not the entire stack. This brought approval time down to just days. A huge speed up.

When it comes to all the paperwork, also look to ways to automate the generation of the needed listings of certifications and compliance artifacts. This shouldn’t be a process that’s done in opaque documents, nor manually, if at all possible. Just as we’d now recoil in horror at manually deploying software into production, we should try to achieve “compliance as code” that’s as autogenerated (but accurate!) as possible. To that end, the work being done in the OpenControl project is showing an interesting and likely helpful approach.

The lessons for DevOps teams here is clear: Standardize your stack as much as possible and work with auditors to build their trust in that platform. Also, look into how you can automate the generation of compliance documents beyond the usual .docx and .pptx suspects. This will help your GRC process move at DevOps speed. And it will also allow your auditors to still act as a third party governing your code. They’ll probably even do a better job if they have these new, smaller batches of changes to review.

Refactoring the compliance process

To address the compliance issue fully, you’ll need to start working with the actual compliance stakeholders directly to change the process. There’s a subtle point right there: Work with the people responsible for setting compliance, not those responsible for enforcing it, like IT. All too often, people in IT will take the strictest view of compliance rules, which results in saying “no” to virtually anything new — coupled with Larman’s Law, you’ll soon find that, mysteriously, nothing new ever happens and you’re back to the pre-DevOps speed of deployment, software quality levels and timelines. You can’t blame IT staff for being unimaginative here — they’re not experts in compliance and it’d be risky for them to imagine “workarounds.” So, when you’re looking to change your compliance process, make sure you’re including the actual auditors and policy setters in your conversations. If they’re not “in the room,” you’re likely wasting your time.

As an example, one of the common compliance problems is around “developers deploying to production.” In many cases and industries, a separation of duties is required between coding and deploying. When deploying code to production was a more manual, complicated process, this could be extremely onerous. But once deployments are push-button automated with a good continuous delivery pipeline,you might consider having the product manager or someone who hasn’t written code be the deployer. This ensures that you can “deploy at will,” but keeps the actual coders’ fingers off the button.

As another intriguing compliance strategy, suggested by Home Depot’s Tony McCulley (who also suggested the above approach to the separation of duties) is to give GRC staff access to your continuous delivery process and deployment environment. This means instead of having to answer questions and check for controls for them, you can allow GRC staff to just do it on their own. Effectively, you’re letting GRC staff peer into and even help out with controls in your software. I’d argue that this only works if you have a well-structured platform supporting your CD pipeline with good UIs that non-technical staff can access.

It might be a bit of a stretch, but inviting your GRC people into your DevOps world, especially early on, may be your best bet at preventing compliance slowdowns. And, if there’s any core lesson of DevOps, it’s that the real problems are not in the software or hardware, but the meatware. Figuring out how to work better with the people involved will go a long way towards addressing the compliance problem.

(I originally wrote this December 2015 for FierceDevOps, a site which has made it either impossible or impossibly tedious to find these articles. Hence, it’s now here.)

Working at home, with a family, is a challenge, as this nice overview piece at The Register goes over. You think you’re trading all those interruptions from co-workers talking about the sportsball or just complaining about the daily grind, but you’re actually trading in for a different set of co-workers, your family. And their requests for your attention are harder to stonewall than chatty cube-mates.

And then there’s the whole “out of site, out of mind” effect with management at work. I’ve worked at home on and off (mostly at home) over the past decade and it has it’s challenges. I lead a public enough work-life, along with remote working aware folks, that Management forgetting about me rarely comes up. However, as my kids have grown up and there’s, consequently, more going on at home, figuring out how to shut-out my family is a constant challenge. You see, that’s the taboo part! “Shut-out” - you could say “manage” or all sorts of things, but if you follow the maker/manager mentality that most individual contributor (non-managers) knowledge workers must, you have to shut people (“distractions”) out.

Achieving flow considered a luxury

On the other hand, this “flow” is a luxury us privileged folks have been experiencing for a long time:

What I didn’t know at the time was that this is what time is like for most women: fragmented, interrupted by child care and housework. Whatever leisure time they have is often devoted to what others want to do – particularly the kids – and making sure everyone else is happy doing it. Often women are so preoccupied by all the other stuff that needs doing – worrying about the carpool, whether there’s anything in the fridge to cook for dinner – that the time itself is what sociologists call “contaminated.”

I came to learn that women have never had a history or culture of leisure. (Unless you were a nun, one researcher later told me.) That from the dawn of humanity, high status men, removed from the drudge work of life, have enjoyed long, uninterrupted hours of leisure. And in that time, they created art, philosophy, literature, they made scientific discoveries and sank into what psychologists call the peak human experience of flow.

Women aren’t expected to flow.

It’s like there’s a maker/manager/_mother_ time management paradigm. (Speaking of that privilege: here I am, with time to type this very post.)

Context switch like Nietzsche

What I’ve been doing is tying to reprogram my mind to think in slices of time fragments and to gorge on 60 minute time spans when they come up. I recall learning that one of the reasons Nietzsche wrote so many aphorisms was because he didn’t have time to write longer pieces; his chronic sickness conditions (whatever they were) gave him little “flow” time.

When I shifted to work at Dell and was on the road the at 451 Research, I was similarly afflicted with fragmented time (at Dell, you’d be in meetings all day because that’s how things ran). I remember one time when I was 451 Research I’d been trying to finish a piece on SUSE and was walking down a ponderously long casino hallway: I just stopped, pulled out my laptop, and started typing for about ten minutes. Finding those little slices that adds up to a full 90 to 120 minutes is hard…but, at least with non-programming knowledge work, you can get over the tax of context switcthing enough to make it worth it.

However, this is all within a large context: the computer. All of that partial attention swapping on the Internet over these years has helpfed warp my brain to work in fragments, but now I need to train my mind to swap between computer and “real life.” So far, it’s slow going.

Resisting the shut-out

All of this on the other hand, I really value working from home. I enjoy seeing my kids and wife all day long (so much more so than all those random run-ins with people in the office). I like being in my own environment, being able to eat at home, and on those rare occasions when I’m in a boring, useless, but obligatory meeting, doing something more useful with my time as I listen in. I have one of the better situations I’ve ever had at work right now: everyone on my team, including my boss, is remote. This means we all know the drill, use the tools, and coordinate.

As my wife is fond of telling me, I should just lock my office door more, which is true. The other part that you, as a remote worker, have to program your brain for is: you’re going to be interrupted while you’re in “flow” a lot. Just accept it. In the office there’s plenty of fire-alarms, going to lunch, people stopping by your desk, and so on. We can’t all be on the flat food diet. My other bit of advice is to take advantage of being at home and a flexible work schedule to do more with your family. If you’re like me, you travel a fair amount as well. So just as I have to gobble up every long span of time greedily, when I’m home and have the chance to do things with family, I try to.

There’s just as much pull for DevOps in government as there is in the private sector. While most of our focus around adoption is on how businesses can and are using DevOps and continuous delivery, supported by cloud, to create better software, many government agencies are in the same position and would benefit greatly from figuring out how to apply DevOps in their organizations.

Just 13% of respondents in a recent MeriTalk/Accenture survey of 152 US Federal IT managers believed they could “develop and deploy new systems as fast as the mission requires.” The impact of improving on that could be huge. For example, the US Federal government, by conservative estimates, spends $84 billion a year on IT. And yet, the Standish Group believes that 94% of government IT projects fail. These are huge numbers that, with even small improvements, can have massive impact. And that’s before even considering the benefits of simply improving the quality of software used to provide government services.

As with any organization, the first filter for applicability is whether or not the government organization is using custom written software to accomplish it’s goals. If all the organization is doing is managing desktops, mobile, and packaged software, it’s likely that just SaaS and BYOD are the important areas to focus on. DevOps doesn’t really apply, unless there’s software being written and deployed in your organization or, as is more common in government agencies, for your organization as we’ll get to when we discuss “contractors.”

When it comes to adopting and being successful with DevOps, the game isn’t too different than in the business world: much of the change will have to do with changing your organization’s process and “culture,” as well as adopting new tools that automate much of what was previously manual. You’ll still need to actually take advantage of the feedback loop that helps you improve the quality of your software, in respect to defect, performance in production, and design quality. There are a few things that tend to be more common in government organizations that bear some discussion: having to cut through red-tape, dealing with contractors, and a focus on budget.

Living with red-tape

While “enterprise” IT management tasks can be onerous and full of change review boards and process, government organizations seem to have mastered the art of paperwork, three ring binders, and red tape in IT. As an example, in the US Federal government, any change needs to achieve “Authority To Operate” which includes updating the runbook covering numerous failure conditions, certifying security, and otherwise documenting every aspect of the change in, to the DevOps minded, infinitesimal detail. And why not? When was the last time your government “failed fast” and you said “gosh, I guess they’re learning and innovating! I hope they fail again!” No, indeed. Governments are given little leash for failure and when things go terribly wrong, you don’t just get a tongue lashing from your boss, but you might get to go talk to Congress and not in the fun, field-trip how a bill is made kind of way. Being less cynical, in the military, intelligence, and law enforcement parts of government, if things go wrong more terrible things than denying you the ability to upload a picture of your pot roast to Instagram can happen. It’s understandable — perhaps, “explainable” — that government IT would be wrapped up in red-tape.

However, when trying to get the benefits of continuous delivery, DevOps, and cloud (or “cloud native” as that tryptic of buzzwords is coming to be known), government organizations have been demonstrating that the comforting mantle of red-tape can be stripped. For example, in the GSA, the 18F group has reduced the time it takes to get a change through from 9–14 months to just two to three days.

They achieved this because now when they deploy applications on their cloud native platform (a Cloud Foundry instance that they run on Amazon Web Services) they are only changing the application, not the whole stack of software and hardware below the application layer. This means they don’t need to re-certify the he middleware, runtimes and development frameworks, let alone the entire cloud platform, operating systems used, networking, hardware, and security configurations. Of course, the new lines of application code need to be checked, but because they’re following the small batch principles of continuous delivery, those net-new lines are few.

The lesson here is that you’ll need to get your change review process — the red-tape spinners — to trust the standard cloud platform you’re deploying your applications on. There could be numerous ways to do this from using a widely used cloud platform like Cloud Foundry, building up trusted automation build processes, or creating your own platform and software release pipelines that are trusted by your red-tape mavens.

Contractors & Lost Competency

If you want to get staff in a government IT department ranting at you all night long, ask them about contractors. They loathe them and despise them and will tell you that they’re “killing” government IT. Their complaints is that contractors cannot structurally deal with an Agile mentality that refuses to lock-down a full list of features that will be delivered on a specific date. As you shift to not even a “DevOps mindset,” but an Agile mindset where the product team is more discovering with each iteration what the product will be and how to best implement it, you need the ability to change scope throughout the project as you learn and adapt. There is no “fail fast” (read: learning) when the deliverables 12 months out are defined in a 300 page document that took 3–6 months to scope and define.

Once again, getting into this state is likely explainable: it’s not so much that any actor is responsible, it’s more that the management in government IT departments is now responsible to fix the problem. The problem is more than a square peg (waterfall mentalities from contractors) in a round-hole (government IT departments that want to be more Agile) issue. After several decades of outsourcing to contractors, there’s also a skills and cultural gap in the IT departments. Just as custom written software is becoming strategically important to more organizations, many large IT departments find themselves with little experience and even less skill when it comes to software and product development. I hear these same complaints frequently from the private sector who’ve outsourced IT for many years, if not decades.

The Agile community has long discussed this problem and there are always interesting, novel efforts to get back to insourcing. A huge part is simply getting the terms of outsourcing agreements to be more compatible. The flip-side of this is simplifying the process to become a government contractor: it’s sure not easy at the moment. Many of the newer, more Agile and DevOps minded contractors are smaller shops that will find the prospect of working with the government daunting and, well, less profitable than working with other organizations. Making it easier for more shops to sign up will introduce more competitions rather than the more limited strangle-hold by paperwork, smaller market that exists now. The current pool of government contractors seems mostly dominated by larger shops that can navigate the government procurement process and seem to, for whatever reason, be the ones who are the most inflexible and waterfall-y.

Another part is refusing to ceed project management and scoping management to external partiers; and, making sure you have the appropriate skills in-house to do so. Finally, the management layers in both public and private sector need to recognize this as a gap that needs to be filled and start recruiting more in-house talent. Otherwise, the highly integrated state of DevOps — let alone a product focus vs. a project focus — will be very hard to achieve.

Addressing budgetary concerns with waste removal

Every organization faces budget problems. We call them “unicorns” because they have this mythical quality of seemingly unlimited budget. The spiral horn-festooned are the exception that proves the rule that all organizations are expected to spend money wisely. Government, however, seems to operate in a permanent state of shrinking IT budgets. And even when government organizations experience the rare influx of cash, there’s hyper-scrutiny on how it’s spent. To me, the difference is that private sector companies can justify spending “a lot” of money if “a lot” of profit results, where-as government organizations don’t find such calculations as easily. Effectively, government IT departments have to prove that they’re spending only as much money as necessary and strategically plan to have their budget stripped down in each budgetary cycle.

Here, the Lean-think part of DevOps can actually be very helpful and, indeed, may become a core motivation for government to look to DevOps. My simplification of the goals of DevOps are to:

1. Ensure that the software has good availability (which it does by focusing on resilience vs. perfection, the ability to recover from failure quickly rather than avoiding all failure by rarely changing anything). This is something that recent failures in US Federal government IT can appreciate.
2. Enable the weekly, if not daily, deployment of new code into production with continuous delivery. The goal here is to improve the quality of the software, both bugs and “design” quality, ensuring that the software is what users actually want by iterating over features frequently.

Those two goals end up working harmoniously together (with smaller batches of code deployed more frequently, you reduce the risk of each causing major downtime, for example). For government organizations focused on “budget,” the focus on removing as much “waste” from the system to speed up the delivery cycle starts to look very attractive for the cost-cutting minded. A well functioning DevOps shop will spend much time analyzing the entire, end-to-end cycle with value-stream mapping, stripping out all the “stupid” from the process. The intention of removing waste in DevOps think is more about speeding up the software release process and helping ensure better resilience in production, but a “side effect” can be removing costs from the system.

Often, in the private sector we say that resources (time, money, and organization attention) saved in this process can be reallocated to helping grow the business. This is certainly the case in government, where “the business” is, of course, understood not as seeking profits but delivering government services and fulfilling “mission” requirements. However, simply reducing costs by finding and removing unneeded “waste” may be an highly attractive outcome of adopting DevOps for governments.

“Bureaucracy” doesn’t have to be a bad word

As with any large organization, governments can be horrendous bureaucracies. Pulling out the DevOps empathy card, it’s easy to understand why people in such government bureaucracies can start to stagnate and calcify, themselves becoming grit in the gears of change if not outright monkey-wrenches.

In particular, there are two mind-sets that need to change as government staff adopt DevOps:

1. Analysis paralysis — The almost default impulse to over analyze and specify with ponderous, multi-100 page documents the shifting to a more Agile and DevOps mindset. A large part of the magic of DevOps and Agile think is avoiding analysis paralysis and learning by doing rather than thinking in .docx. Government teams not familiar with smaller batch, experiment-based approaches to software development would do well to read up on Lean Startup think, perhaps checking out Lean Enterprise for a compendium of current best practices and, well, mindsets.
2. Stagnant minds — large organizations, particularly government ones, can breed a certain learned helplessness and even laziness in individuals. If things are slow moving, impossible to change, and managed in a tall blade of grass gets cut style, individuals will tune out rapidly. If DevOps is understood as a practice to help jump-start all too slow IT organizations, it’ll often be the case that individuals in that organization are in this stagnated mindset. One of the key challenges becomes inspiring and then motivating staff to care enough to try something new and stick with it.

Again, these problems frequently happen in the private sector. But, they seem to be larger problems in government that bear closer attention. Thankfully, it seems like leaders in government know this: in a recent Gartner, global survey, 40% of government CIOs said they needed to focus more on developing and communicating their vision and do more coaching. In contrast, 60% said they needed to reduce the time spent in command-and-control mode. Leading, rather than just managing, the IT department, as ever, is key to the transformative use of IT.

More than rats dragging pizza

In any given time, it’s easy to be dismissive of government as wasteful and even incompetent. That’s the case in the U.S. at least, if you can judge based on the many politicians who seem to center their political campaigns around the idea of government waste. In contrast, we praise the private sector for their ability to wield IT to…better target ads to get us to buy sugar coated corn flakes. Don’t get me wrong, I’m part of the private sector and I like my role chasing profit. But we in the “enterprise” who are busy roaming the halls of capitalism don’t often get the chance to positively effect, let alone simply help and improve the lives of, everyone on a daily basis. Government has that chance and when you speak with most people who are passionate about using IT better in government, they want to do it because they are morally motivated to help society.

The benefits of adopting DevOps have been clearly demonstrated in recent years, and for businesses we’re seeing truth in the statement that you’re either becoming a software organization or losing to someone who is. As government organizations start to think about improving how they do IT, they have the chance to help all of us, “winning” isn’t zero-sum like it can be in the business world. To that end, as we in the industry find new, better ways to create and deliver software, it behoves us to figure out how government can benefit as well. That’ll get us a even closer towards making software suck less something we’ll all benefit from.

(I originally wrote this September 2015 for FierceDevOps, a site which has made it either impossible or impossibly tedious to find these articles. Hence, it’s now here.)

I’m always wanting to do a talk or write a series of items on the white-collar toolchain, or surviving in big companies. Here’s one principal about presentations in corporate settings.

Slides must stand on their own

Much presentation wisdom of late has revolved around the actual event of a speaker talking, giving the presentation. In a corporate setting, the actual delivery of the presentation is not the primary purpose of a presentation. Instead, a presentation is used to facility coming to a decision; usually you’re laying out a case for a decision you want the company to support. Once that decision is made, the presentation is often used as the document of record, perhaps being updated to reflect the decision in question better.

As a side-note, if your presentation doesn’t argue for a specific, “actionable” decision, you’re probably doing it wrong. For example, don’t just “put it all on the table” without suggesting what to do about it.

Think of presentations as documents which have been accidentally printed in landscape and create them as such. You will likely not be given the chance to go through your presentation from front to end like you would at a conference, You’ll be interrupted, go back and forth, and most importantly, end up emailing the presentation around to people who will look at it without you presenting.

You should therefore make all slides consumable without you being there. This leads to the use of McKinsey titles (titles that are one-liners explaining the point you’re making) and slides that are much denser than conference slides. The presentation should have a story-line, an opening summary of the points you want to make, and a concluding summary of what the decision should be (next steps, launching a new project, the amount needed for your budget, new markets to enter, “and therefore we should buy company X,” etc.).

This also gives rise to “back-up” slides which are not part of the core story-line buy provide additional, appendix-like information for reference both during the presentation meeting and when others look at the presentation on their own. You should also put extensive citations in footnotes with links so that people consuming the presentation can fact check you; bald claims and figures will be defeated easily, nullifying your whole argument to come to your desired decision.

Also remember that people will take your slides and use them in other presentations, this is fine. And, of course, if successful, your presentation will likely be used as the document of record for what was decided and what the new “plan” was. It will be emailed to people who ask what the “plan” is and it must be able to communicate that accordingly.

Remember: in most corporate settings, a presentation is just a document that has been printed in landscape mode.

Donkey teamwork

What’s the point of it all? Why are we doing this? These questions pop up frequently in IT teams where the reason for doing your daily activities — like churning through tickets, whizzing up builds, or “doing the DevOps” — seems only that someone, somewhere told you to do it.

If you’re in this situation — you have no idea how your activities are helping your organization make money — you should stop and find out quickly what your company’s goals and strategies are to make sure you’re not wasting time. The good news is the confusion is probably not your fault; the bad news is that you’ll have to convince management that the fault is theirs.

Gratuitous optimization by technology

The adoption of things like DevOps or the cloud sometimes happens for wrong or unknown reasons — gratuitous plans without a tight connection to business goals. We used to call this “management by magazine,” and it happens now more than ever. A process — even “cultural” — change like DevOps is not like the easy improvement fodder of virtualization. But you can’t blame IT management for trying gratuitous optimization by technology. The magic of VMware was that you just installed it, and things got better because it improved resource utilization. You didn’t need to figure out why or match it to goals. If you inject DevOps into an organization expecting it to just improve things without tightly coupling to strategy, you’ll get weird results. You’ll probably just create more work!

If you don’t know where you are going, any road will get you there

Agile, DevOps, and now “cloud native” (I hope you’re updating your buzzword lexicons!) need strong connections to the business goals — some would say “strategy” — to be successful. In order to operate in a lean fashion, you want to only do things that are valuable and useful to the customer (or obligatory to stay in business, like compliance and auditability). Indeed, being able to sort out what’s valuable and useful to the business is a key tool for doing DevOps successfully. You want to cut out all the stuff that doesn’t matter, or at least minimize it. Otherwise, you just sort of do everything and anything because there’s no way to determine if any given activity is helpful.

So how do you align your work with the overall business strategy?

There are tried and true (though seemingly new to the IT department) techniques like value-stream mapping: take any given business process and map out all the activities that happen from end-to-end, questioning if each is needed. Most people are shocked at how much “stupid” is going on in such maps and it’s a great technique for finding and removing bottlenecks.

If you’re in the consumer business — like so many “unicorns” are — it’s easy to understand the mission and the goals: get more people buying books, downloading your app, streaming more videos, and so forth. But in other, more traditional settings, it’s common to find a willful disentanglement between how IT is used and how it contributes to customer value. More than not, the stasis-inducing ludlum of time and success just numbs people’s collective minds and sets them into auto-pilot here.

You see this happen most often around decision making processes in business: things that need approval, planning processes and market assessments. People in large companies love cogitating and wrapping process around activities that cause change in the company; it feels like they almost like to slow down change and activity. You might even codify in a whole process with change review board meetings and careful inspection of all the changed components by a panel of architectural and security audit wizards.

Cultivating complainers

You can also identify where your processes aren’t matching with business goals and strategies by cultivating squeaky wheels.

When change happens, individuals often pipe up asking, “Why are we doing this? Why is this valuable to the customer?” More than likely, they’re seen as troublemakers or sand in the gears, and are shut down by the group, Five Monkeys style. At best, these individuals cope with learned helplessness; at worst, they leave, kicking off a sort of Idiocracy effect in the remaining organization.

These “complainers” are actually a valuable source of data for testing out how well understood a company’s goals and strategies are. You want to court these types of people to continually test out how effective the organization is at setting goals and strategy. One fun practice, as mentioned by Ticketmaster’s Jody Mulkey, is to interview new employees a month after starting to ask them what seems “screwy around here” before they get used to it.

Blame management

So what do you do when they or any other process you’ve tried identify real disconnects between what you’re doing and why? The fun begins — because it’s management’s job to fix this bug. The role of mid- and upper-level management in the cloud native era is poorly understood and documented (its always been so, of course, in creative-driven endeavors like software). To be successful at these types of initiatives, management has a lot of work to do and the managers who are overseeing DevOps teams can’t assume things will just proceed as normal. This is why, as with software, you need to continually test the assumption that people know the business goals and strategy.

This point has been stuck in my brain after reading Leading the Transformation (an excellent book for managers figuring out how to apply DevOps “in the large”), which states the point more plainly than I can:

Management needs to establish strategic objectives that make sense and that can be used to drive plans and track progress at the enterprise level. These should include key deliverables for the business and process changes for improving the effectiveness of the organization.

What I like about this advice (and the rest in the book) is that it’s geared to defining management’s job in all this DevOps hoopla. In said hoopla, we spend a lot of time on what the team does, but we don’t spend too much time on what management should do. It’s lovely thinking about flattening the organization and having everyone act as responsible peers, but in large organizations, this isn’t done easily or quickly. Just as with all those zippy containers, you need an orchestration layer to make sense of all the DevOps teams working together. That’s the job of management in the cloud native era: setting goals and orchestrating all the teams to make sure they’re all pulling in the right direction.

(I originally wrote this August 2015 for FierceDevOps, a site which has made it either impossible or impossibly tedious to find these articles. Hence, it’s now here.)

Think you can show DevOps ROI? Think again

“What is the ROI for DevOps?” is a question that has been tossed my way frequently of late. There are numerous reasons why this is at the same time an absurd but also important question.

Modeling DevOps ROI is absurd because predicting the gains and costs of a process, let alone one as new as DevOps, is difficult and dependent on all sorts of unique variables per organization.

However, thinking through DevOps ROI is an important step for adoption because the promises of DevOps are so grandiose and the changes needed sound large and almost impossible to achieve for “normal” people.

That is, DevOps is an unmeasurable process with respect to ROI (it has value, to be sure, but is nearly impossible to measure independently and precisely) and, yet, because “doing DevOps” seems to be such a big change, organizations need assurances that transformation will be “worth it.”

So, if you’re asked to help show the ROI for DevOps, what can you do? Let’s cover three ways to approach the problem. I don’t think any of them are a real answer, but they get closer to satisfying some possible motivations for asking for ROI in the first place.

What is ROI?

First, what is ROI? I misuse economic and accounting terms all the time, but I think of “ROI” as showing the profit you achieve after a given period of time, for our purposes, by doing something new and different with IT: you might buy some new software (running on a cloud platform like Pivotal Cloud Foundry instead of just IaaS), do your software development and delivery differently (like, “doing DevOps”), and so forth.

With ROI, you’re not only interested in the question “does it work,” you’re interested in the question “did this make me money?” Oftentimes, you’re also interested in comparing the costs of competing approaches, or just inflicting vendors with the thrill of “bake offs” and ROI spreadsheet fights.

To figure that basic ROI, you use a brutally simple formula:

(Gain — Cost)/Cost = ROI

You can convert the end result to a percentage if you’re not into the whole decimal thing.

As a simple example, let’s say you sell an app that allows people to track how many apples they eat each day, so they can keep those ravenous doctors out of the way. After it’s shipped for a month, you’ve made $20,000 in sales for the app. To get to that point, it costs them $5,000 in paying for developer time and $5,000 in infrastructure charges (the back-end that analyzes the data, mashes it up with Facebook and Twitter profiles, and then sells that data to the Apple Sellers Association of Tomorrow takes some horse-power and storage!).

So, the ROI for the apple muncher app is:

($20,000 — $10,000)/$10,000 = 100%

A pretty good return on your investment! It’s certainly better than the rate I get on any of my personal investments.

So, what would be the ROI of introducing DevOps to that process? More importantly, how could you predict it? There are many ways to answer the ROI question, including the favorite “that’s a bad question, you shouldn’t want that” which can take on all sorts of subtle and helpful forms. Let’s look at three possible approaches.

1. Bottoms-up ROI: We know everything and have put it in this spreadsheet

If you have clear inputs and outputs — your gains and costs — then things can be realistically simple. This is the favorite approach of ROI spreadsheets: they’ll cost out software license costs, hardware/IaaS costs, and people costs (employees and consultants).

Once you’ve figured out costs, you need to estimate what your gains will be: either based on historic run rates, or, more likely, on a mixture of a prediction and hope for how much you’ll make in the future. Tracking the demand for software can be hard and this estimate is one of the most dangerous parts of this simple method. If all you want to do is track the ROI for saving money, perhaps things are a little easier. And while this implies that you’re not looking to DevOps to support a revenue growth strategy, perhaps that’s good input: if you’re not looking to grow your business, maybe it’s not right for you and will have negative ROI.

You then have to pick a period of time to snap-shot and you just run the math.

Of course, few, if any, of the things you’re costing out here are “DevOps.” You might spend money on a commercial continuous integration tool, on a cloud platform or a DevOps consultant. You’ll certainly spend money on people…but you didn’t really spend money on “doing DevOps.”

You might be tempted to simply ascribe gains to DevOps. “For this release, we were doing DevOps, and we made $30,000 with apple muncher! DevOps brought us $10,000 in new revenue.” But that doesn’t feel right.

Still, if you have a good handle on the costs during some period of time where you were doing DevOps and the gain that resulted from that period of time, you could come up with a bottoms-up ROI analysis. I think it’ll be somewhat dicey since it’s so hard to attribute costs and gains directly to DevOps but, hey, it’s better than either telling people they’re asking the wrong question or its mute cousin: nothing.

2. Were the efforts to change worth it? That is, DevOps is all cost

As you might be teasing out, one of the problems with ROI is that it doesn’t really take time into account. You need to draw clear lines around the time period in which you’re including the factors that create your gains and costs. (If you’re interested in an approach that does take time into account, check out Rex Morrow’s suggestion to use IRR instead of ROI.)

Using this lack of time problem as a generative constraint, you could instead study the ROI of changing to DevOps. What did switching over to DevOps cost us? What did it cost us compared to maintaining our current process state?

Here, you’re taking whatever your regular ROI calculation is and just adding the one-time cost of time and money it took to change to DevOps. Figuring out what your gain is will be problematic. Again, what you’ll be gaining are new capabilities (to deliver software faster and increase your uptime in production); how those contribute to gains is still left as a mysterious exercise to the reader.

Still, if you want to run the numbers on something like “they tell me it will take three months and $50,000 in training and consultants to ‘do the DevOps’” this might satisfy your ROI craving. Again, you’ll need to have a pre-existing ROI at hand to simply plug your DevOps costs into.

3. Pain avoidance and remediation

In the “DevOps is all cost” ROI scenario, we avoided ascribing gain to changing to DevOps. Again, while this is overly simplified, the deliverables of DevOps are to provide a continuous delivery process for your product and ensure that your product has excellent uptime (that is, “it works”). How could you account for the gain of those two desirables? You could create a way of assigning value to the knowledge you gain from weekly iterations about how to improve your product. You could also calculate the savings from avoided downtime.

It’s fun to model-out placing value on the first part, “knowing,” but most people asking for ROI will likely look at that as a “soft” metric and, therefore, not really useful to their “hard”-centric minds. Including money saved (or generated?) by avoiding downtime could be interesting in a point in time (if a trading system goes down, money is lost when no one can trade), but how do you account for it ongoing?

The issue with including DevOps in this “easy” type of ROI calculation is figuring out how much gain and cost to attribute to DevOps.

As with uptime, sometimes it can be easy: before we did DevOps, the system was down two hours a day, now it’s only down five –10 minutes a day, if at all. If there’s a pain you’re seeking to remove, then perhaps this model will work.

Your pain might also be “it takes us too long to deliver software,” which is a common problem for DevOps adopters. If you know how to measure the gain of time to market, for example, then you can do one of these bottoms-up ROI cases: “We were able to deliver a third release of apple muncher in two weeks instead of the six it had been taking. This means we could start charging for the new in-app purchases sooner, gaining us $5,000 more over that two week period.”

If you like this kind of figuring, check out Zend’s suggestion for how to do continuous delivery ROI for some inspiration. Like all “good” ROI calculations, it requires changing ROI around slightly to fit what’s measurable…and some good estimating.

So, can you send me a spreadsheet with all this in it?

I’ve deftly avoided actually giving you anything actionable here. Calculating ROI is a very numbers-, spreadsheet-friendly exercise and any answer should really include at least a starter spreadsheet to get you calculating things. However, as the above hopefully shows, it is indeed the case that asking for “DevOps ROI” is the wrong question. The “ROI” is getting the process and tools in place to create a better product. Obviously, as you rack up the costs associated with DevOps (both in money and time spent), you can start to model the overall ROI of the project versus the revenue and profit you generate, but there’s little DevOps specific about that.

Beyond such obvious answers, when I see people asking for “DevOps ROI,” what we can offer them is over-thinking like the above and examples of it working at organizations. Examples like Allstate and Humana are good mainstream cases, and you can listen in to more on the excellent Goat Farm podcast.

Additionally, I would suggest focusing on looking at DevOps as a continual improvement initiative rather than trying to predict ROI. Much of the problem with figuring out ROI is in having to predict costs and gains. Instead I would try to focus on tracking and trying to improve how you’re doing things in short intervals. In my experience, most organizations devalue the idea of continuously learning and trying to improve their process. Focusing on that might be a better use of time than summoning up a solid case for DevOps ROI.

I’d love to see examples of how you did an DevOps ROI case…or avoided it all together. If we can accumulate enough after-the-fact studies, then at least we could make a “rule of thumb” collection. Leave a comment below!

(I originally wrote this July 2015 for FierceDevOps, a site which has made it either impossible or impossibly tedious to find these articles. Hence, it’s now here.)

Can DevOps declare victory yet? Not quite, but soon.

Figuring out when a technology inflection point happens is always hard, if not impossible, in real-time. It’s easy to point backwards and say when ERP, agile software development, the Web, business intelligence, mobile or cloud suddenly became “normal.” I think DevOps is right at the door of that point, and as some recent Gartner predictions have proffered, we could see something like a quarter of all large enterprises using DevOps next year.

But it won’t be easy. The same house of industry sages also threw some cold water on that exuberance by predicting a 90% failure among organizations attempting to do DevOps if they fail to properly address process and culture.

As DevOps spreads to more and more IT shops, what can we in the DevOps community do to help? Clearly, we need to keep up the overall conversation about what DevOps is and the process/cultural changes needed to be successful. Another critical element is to start telling more and more stories of how non-technology companies are succeeding with cloud and DevOps. I think the recent Humana profile provides an interesting template here, as does Standard Bank’s forays into DevOps.

In addition to keeping up the good work, there are four key areas that will be helpful.

1. Define the goal properly

In one of my favorite straw-polls, groups who focus on the wrong outcomes and goals with private cloud have similar failure rates as those Gartner describes for organizations attempting to do DevOps.

What exactly those right goals are, for both DevOps and cloud adoption, is a new theory of mine I wanted to road-test with the DevOpsDays Austin crowd. As far as I can tell, the best goal of both a “cloud project” and “doing DevOps” is to do continuous delivery. So, cloud and DevOps let businesses set up the process and technologies needed to deliver custom written software on weekly, if not shorter turns, and actively study, learn and adapt software from the feedback of actual people using their software. This is the path of becoming a software defined business and DevOps is the definitive “how” of how that’s done.

To that end, I suggested to the audience in Austin that we should start, more or less, thinking of continuous delivery and DevOps as synonymous. Once you frame what DevOps is — what DevOps enables — as that, the conversation becomes crisper and, I believe, easier for everyone to understand and do something about. As I discussed last time, becoming a software defined business entails (a.) starting to think in a product-oriented manner (greatly facilitated by continuous delivery), and, (b.) ensuring that you have the overall cloud platform in place that provides the feathered, infrastructure bed for everything.

And, to add to the tracking of DevOps’ ascension to the mainstream, if you think of it as continuous delivery, some recent studies have shown that while overall CD use is low, growth has been ramping up year over year, just like DevOps.

2. Clearly explain the DevOps stack(s)

While even the best tools without the proper process (or “culture”) are ineffective, most people think in terms of stacks and tool-chains. So many of the DevOps conversations I’ve been involved in over recent years start by talking about tools and technologies. We’re in IT: it’s what we know.

I’d really like to see us start discussing common tool-chains and patterns of use (“cookbooks” to use an older, common programming documentation metaphor) for doing DevOps. Reference implementations even! Vendors do well telling you what they think the toolchain should be — please, oh please, feel free to ask me! ;). In fact, I’d say there’s almost an unhelpful amount of fragmentation in the infrastructure management layer at the moment: there are so many options that one can be left confused and overwhelmed.

Instead of letting us vendors define those stacks, I’d like to see the overall community get even more involved. Don’t be afraid to talk about tools in the face of all this culture talk! And don’t let us vendors steal the show.

3. Work with legacy code

Almost by definition, the IT shop at a non-technology company will be chock full of existing IT and “legacy code.” That’s the very IT that was once the growth-engine darling of the company and laid the foundation for where they are now.

As we all know but try to shy away from admitting too loudly, the new cycle of code and tech rarely works with the previous cycle’s code. I talk with companies almost weekly that are very interested in the question of how to integrate new cloud-native and mobile applications with five, 10, even 15+ year old centralized IT services. They want all the power of cloud and continuous delivery, but need help rationalizing and working with what they already have.

In my view, this is a conversation that doesn’t happen often enough in the DevOps community. It first starts with a — wait for it! — seemingly dottering old term: “IT portfolio management.” That is, taking the time to assess what IT you have and understand the business priorities around it. Without that kind of big picture, systems-based understanding of what you have, any whiz-bang awesomeness you get with DevOps will pale in comparison to the rumbling rebar-festooned concrete ball of legacy IT you have to deal with. (Damon Edwards gave a great talk right before mine introducing one method of getting down and dirty with portfolio management.)

There are many thought-technologies of how to approach this, from Gartner’s bi-modal IT approach, to some interesting work going on over at the Cutter Consortium. The point is to have the discipline and maturity to actually do portfolio management so you can start to improve everything and better prioritize your time and projects.

And, to point out the obvious: we need to start documenting how the application being written and supported by DevOps teams is integrating and co-existing with non-DevOps (or “legacy”) applications and services.

4. Keep up the Land-grab

As its name implies, DevOps has been on a land-grab mission that started back in the Agile days. If agile had gone the portmanteau route in naming itself, we might have seen DevQA, or even ProductDevQA. Agile development very consciously crossed silos and unified product management and QA with development, so much so that by the time we came up with DevOps, “Dev” represented all of those traditional roles.

Now, as companies are looking to IT and custom written software to help them become software defined businesses, DevOps-minded folks need to start thinking about how they can get more involved with “the business.” Do you know who these mythical business people are, what they’re worried about, how they think? Can you speak their language and help them learn yours?

To pick one very specific item that’s always a punji pit of IT despair: what KPIs and metrics should you use to communicate “up the chain”? (Ernest Mueller and Karthik Gaekwad have a great presentation on just this topic from last year.)

Think of it this way: what is the “API” for your business, and how can you start programming it…if not designing the API? Once DevOps is tightly integrated with the business side, and most companies are activity thinking about how custom written software can help run, grow, and innovate their business…then we’ll be able to declare DevOps success in the mainstream.

(I originally wrote this May 2015 for FierceDevOps, a site which has made it either impossible or impossibly tedious to find these articles. Hence, it’s now here.)

(I originally wrote this April 2015 for FierceDevOps, a site which has made it either impossible or impossibly tedious to find these articles. Hence, it’s now here.)

Quick tip: if you’re in a room full managers and executives from non-technology companies and one of them asks, “what kind of company do you think we are?”…no matter what type of company they are, the answer is always “a technology company.” That’s the trope us in the technology industry have successfully deployed into the market in recent years. And, indeed, rather than this tip being backhanded mocking, it’s praise. These companies are taking advantage of the opportunity to use software and connected devices in novel ways to establish competitive advantage in their businesses. They’re angling to win customer cash by having better software and technology than their competitors.

What does it look like “on the ground,” though when it comes to “being a technology company”? I’d argue that the traditional ways we think about structuring the IT department is different than how technology companies structure themselves. To massively simplify it, traditional IT departments are oriented around working on projects, where-as technology companies are oriented around working on products.

The project-oriented IT department

Project oriented thinking takes in requests from an outside entity and works on solving an immediate, well understood problem. There’s often a definitive end to the project: the delivery of the new “service.” Project oriented thinking is good for creating an initial version of an application, installing and upgrading existing packaged software, setting up new offices, on-boarding employees, and other things that have a definitive completion date and well known tasks.

When organizing around this type of work, you setup a functional organization that can be assembled to implement the specific problem. Here, by “functional organization,” I mean groups of people who are defined by their expertise in something: networking, server administrator, software development, project management, audit and compliance, security, and so on. These people are typically shared across various projects as needed and usually are responsible for just what they know about. (For a very different take on how to use functional organizations, see Horace Dediu’s discussion of how Apple organizes itself.)

On-top of this, you take a request-driven approach to change management, which defines when to launch a new project or make “small” changes to an existing one (like adding a user). In the 2000s, we fancied up this concept by calling it “service management.”

Once the project is up and running, there may be something called “maintenance mode” which sees IT making sure, for example, that the ERP application has enough disk space available, that new users are added to the application, and that extra capacity is added when needed.

This mind-set is very handy when you’re dealing with keeping a bunch of products from tech vendors up and running. It’s even good if you have custom written applications that are not changed frequently. What’s also great, is that because each project is well defined at the start and has a definitive end, you can measure success and financial metrics easily: how many requests did we handle (tickets closed) this month? did we deliver on time? did we deliver on-budget? is the project profitable (thus, did we pay too much for that software or get a good deal?)

However, two things have been changing this state of affairs, pushing IT to be more exploratory in nature. As a consequence, the structure of the IT department will need to change as well to maximize ITs value to the overall business.

Cloud changes the organization into a software eating the world type situation

I often joke that it’s been impossible to see a keynote in recent years without seeing the horsemen of the digital apocalypse. These are the cliche topics that seem to come up in every keynote. Two of these lay the groundwork for why the structure of the IT department needs to change:

1. Software is eating the world — Cloud technologies and practices have made huge improvements in productivity and costs when it comes to creating and running custom written applications. It’s easier to write and run software now, and the rise in “always on” devices (all those super-computers in our pockets that are on the Internet 24/7) creates a massive foot-print for computation: an endless buffet for software.
2. Change or die! — with this huge buffet of opportunity, there’s a rallying call for companies to invent new business models that rely heavily on software. This means that most every business has the opportunity to use custom written software to change the nature of their business. Think of the opportunity for taxi companies to use software to change how they operate, or for the hotel industry to come up with a brand new business model to sell empty capacity…and you’re thinking of Uber and AirBnB. The “or die” part is a rhetorical trick to position this imperative as dire. And, indeed, studies have shown that remaining on-top has become harder in recent decades. Change is needed to survive.

These two alone create a pull for more custom written software in businesses. It’s fast and cheaper to create software, and competition is relying on that to create new business models that challenge incumbents or, rather, those businesses that are not evolving how they run their business with software. Again: think of all those taxi services versus Uber.

IT — SaaS = what?

There’s a third “horseman” in the broader industry that’s driving the need to change how IT departments are structured: the rise of SaaS. Before the advent of SaaS across application categories, software had to be run and managed in-house (or handed off to outsources to run): each company needed its own team of people to manage each instance of the application.

Source: Source: Two studies, first with 1,137 respondents, second with 1,097, involved in their company’s IT buying decisions participated in the Jan 2014 and July 2014 survey, including 470 and 445 whose company currently use public cloud. “Corporate Cloud Computing Trends,” 451 ChangeWave, Feb, 2014 & “Corporate Cloud Computing Trends,” 451 ChangeWave, Aug 2014.

As SaaS use grows more and more, that staffing need changes. How many IT staff members are needed to keep Google Apps or Microsoft’s Office 365 up and running? How many IT staff do you need to manage the storage for Salesforce or Successfactors? Indeed, I would argue that companies use more and more SaaS instead of on-premises packaged software, the staffing needs change dramatically: they lessen. You can look at this in a cost-cutting way, as in “let’s reduce the budget!” Hopefully you can look at it in a growth way instead: we’ve freed up the budget to focus on something more valuable to the business. In most cases, that thing will writing custom software. That is: developers.

The product oriented IT department

This is where the shift to thinking like a product organization is vital. First of all, if you feel the need to develop more custom software — as you should! — you’ll need to hire and train more software developers, product managers, QA staff, and related folks. You’ll also want to cultivate an environment where new ideas can be explored, user-tested in production, and then quickly refined in a loop that spans mere weeks if not one week. You’ll need to become a continuous delivery and learning organization. Jonathan Murray has called this type of organization a “software factory” and has explained how he implemented the change while at Warner Music. More recently, books like Lean Enterprise have explained how this type of thinking can be applied outside of “startup culture,” whose concerns tend to be more around achieving a high valuation to get the company acquired or IPO rather than building and maintaining sustainable business models.

Setting up an organization like this requires not only developers, but creating the actual “factory” that they operate in. I think of this factory as a “platform” and the folks responsible for standing up and caring for that platform are a new type of operations staff. They’re in charge of, really, providing the “cloud” that developers effortlessly deploy and run their applications in.

This new type of IT staff has to think about how they add in as many self-service and highly elastic services in their “cloud” as possible. They too are creating a “product,” one that’s targeted at the internal developer teams and which must continually have new features added to it.

Meanwhile, your developers will be arranged into product-centric teams, hopefully working more closely with line of business managers and staff who are helping craft and grow new applications. No doubt they’ll need operations skill on the team: staff who know how to properly architect and operationalize cloud-native applications.

This is where the now classic DevOps mentality comes in: in order to properly focus on a product, the team must be responsible for all parts of that product’s life, from development through production and back. With a proper cloud platform in place and the operations team to support it, these goals are more achievable than if the product team has to start from bare metal, or work with IT through a ticket system.

To be pragmatic, you probably can’t dedicate all people fully to a product and will need to share them. This carries large risks, however, namely, making sure you properly prioritize an individual’s time and realizing that they’ll have a harder time keeping up with fewer products rather than more. Quality and ability to deliver on time will likely decrease. It may seem like an impossible goal, but often in order to stay competitive — to survive — large, seemingly impossible changes are needed

Fun with market sizing

I’ve spent a lot of time over the years working with cloud market-sizings, and occasioanlly on them. They’re always a bit whackadoodle and can be difficult to pull apart. But, so long as they’re consistent year of year, they do give a good intedication of momentum and a comparision to other markets. This is what you should be using emerging technology marketsizing for: just indications of which way the wind is blowing and how strong that wind is relative to other breezes.

All too often, strategy and M&A people (and other “MBA” types who’re doing valuations and finances, plus all the hangers on in the chattering classes) get obsssed with market-sizing as if they’re “real” and start to do things like include them as fundamental parts of their business plan, e.g., “we’re going to capture 1% of the cloud market this year!” The implication there is that if they don’t, they’ve failed…but when you realize how corny the market-sizing are, you realize that basing your yearly plan on an Excel macro is a poor use of time.

With that disclaiming context established, I love finding market sizing numbers. They’re fun once you get enough of them and can start figuring out the relative size of markets. Which is to say: how much money is being spent each year across the world on various types of technology.

PaaS

Platform-as-a-Service is one of the more tricky markets to size and has spun all sorts of directions over the years. It’s always the smallest of the three aaS’s, but the highest growth. Even worse, most PaaS market-sizing you see is for only public PaaS. I’ve heard that Gartner has consternated much about this over the years: if you insert a simple “r” into it, they have an on-premises category called CrEAP that sizes what, I think, is “private PaaS,” and I believe they’re fixing it up further.

However, the problem with sizing the PaaS market is asking what you’re sizing. There’s “PaaS from SaaS” offerings like Force.com and then so called “first generation” PaaSes like Heroku (also at Salesforce) and EngineYeard. Then there’s hosted development tools (all the CI services otu there) that, for some reason, show up in PaaS market-sizing. But now there’s all the private PaaS offerings that happen to also run as public PaaS (it turns out the enterprise market really likes private cloud as well as public). And then there’s trouble-makers like us at Pivotal who bristle at the notion of being called (just) a PaaS. Layer Docker, Mesos, kubernetes and all those folks in there…and you’re head should be spinning.

The composition of this market has changed dramatically over the years. My theory is that it’s not well “shaken out” (defined) yet and it’ll change more. So, when I get asked for PaaS market-sizing, I sigh a bit inside.

Getting to a helpful PaaS market-sizing

I think the best process is to thinking through a what PaaS is used for and then try to figure out how much money is spent on solving that problem, not the exact technologies used to solve it. To me, that gets to some indication (whether it’s a ceiling, floor, or mid-point, I don’t know) of how much money there is up to grabs if you’re selling PaaS.

To that end, I always like this chart from a recent Goldman PDF:

As you can infer from my over contextualizing above, I think most PaaS market-sizing is bunk, but this chart a good way of thinking through getting to answer. It compares traditional packaged software and on-premises hardware spend to IaaS and PaaS to show how it starts to erode into “non-cloud” IT. I’m not sure if IaaS and PaaS and public cloud only (it probaly is, which is problematic).

When using this chart, the voice over is something like “tracking this market is difficult at this moment, as any analyst will tell you. I like to use something like this chart as a guide for thinking about it. With PaaS, what you’re interested in is how much traditional IT teams writing cloud-native applications are taking over, and this is one swag at it. Notice that the growth rates are drastically different too: there’s little to no growth in traditional.

The other thing (since most analysts don’t track private PaaS) this suggests is that all the traditional middleware money switches over to all PaaS (public and private) at some point…at least all the “new” money, the growth. Or, at least, that it’s a rough heuristic. It could be less (prices go down with cloud, right? ;>) or it could be more (software eating the world X IT - SaaS = what? == more customer software development at companies leading to more spend on the application development category).

So, if you add up the traditional markets of Appdev and middleare, you get something like a $35-40bn market in 2018 or so, depending how exuberant or dower you want to be, for pubic and private PaaS. Again, that wet-finger-in-the-winding is “bunk,” but it tells you type of money you should think about. Assume that over the next 10 years most “appdev and middleware” spend converts to “PaaS” and then you’ve got something that looks less shitty than the PaaS market-sizing analysts do now-a-days…and more real, I think.

I had lunch with Israel Gat yesterday. Lobster bisque in a sourdough bread bowl, to answer your first question. We were talking about the concept of a “software defined business” (and I was complaining about how HEB needs more of that, if only to get digital Buddy Bucks).

The question came up, so will companies really do this “software defined business” stuff (that’s the phrase I like for “third platform," “digital enterprise,” horseman style jabber-jargon)?

Well, over the next 3 years, I think much of the marketing efforts in tech will converge on exactly that. This is what tech companies will try to sell and the “thought lordship” they’ll try to deploy into the market. I think it’ll actually be to the tremendous benefit of customers, not just a hustle. Soon the egg will become a chicken, and the chicken will start making demands on the egg. Which one is egg and chicken? Indeed! One can never tell the causation directionality in these things.)

Why will tech companies focus on software defined businesses as a growth driver? Well, it’s kind of the only area for growth, at least interesting growth. Keep in mind that if you’re a big, publicly traded company, you have to grow, you need to find new money sources. Last year’s revenue can’t just sit there, staying stable. Otherwise you’re toast because investors will want to allocate their money in companies that are growing, not shrinking (they’ll dump your stock, and but another). This is true for any business, but very true for technology companies.

Here’s my rough sense of revenue streams tech companies will have:

1. Consumer churn.

You know, you get a new mobile phone ever 2-3 years. Instead of subscribing to a cable package, you subscribe to HBO Go. (You sort of end up paying the same amount, but who’s paying that close of attention when there’s so new Game of Thrones episodes to watch?) You buy subscription services. Games.

Basically, the “not enterprise” market. This is what most tech press covers and talks about; it’s (sadly?) what we think of as “tech” now.

There’s growth in here, but it’s a totally different space than traditional, let alone “enterprise” tech. Microsoft finally seems to have figured this out, but meanwhile Apple, Google, and Facebook are gobbling up all the revenue and growth…not to mention all the new companies that have come along.

2. How much more ERP can there be?

Businesses still need a lot of software, but I’d argue that “systems of record” are probably well saturated at this point and low growth. This used to be the fuel of the tech sector, but if everyone has a system of record in place…how much more more spend can there be there each year? (I’m sure we could look up some IDC or Gartner numbers about single digit growth in these markets.) Not much. One area of interest is shifting over to SaaS, or:

3. Rip-n-replace cloud.

“Well, I guess I need to replace all this ‘legacy’ stuff with cloud.” You know in storage, compute, and maybe networking. There’s lots of hardware and infrastructure software churn here. In the software category, I’d put migrating your on-premises ERP/systems of record stuff to SaaS here as well: moving to Salesforce, Successfactors, etc. In a squishy analogy, things like Adobe transforming from licensed sales to subscription sales.

This is a long term play with lots of cash; for businesses though: do they end up with anything net-new? Have you actually tried to use Salesforce? It removes the hassle of having the manage your own CRM instance, but you still have to manage how your company uses the application…otherwise it’s baffling what’s going on in there. My point is: the company ends up kind of back where it was before the great rip-n-replace, just more optimized IT, with hopefully HTML5 and native mobile apps instead of Flex.

4. Software Defined Businesses.

Here, companies are looking to create new custom written software that helps run their businesses in new ways or creates entirely new business models. It’s the thing we at Pivotal target, what you see coming out of the IBM/Apple partnership (mostly - some of it just the next step in the great rewrite the UI every decade journey of green-screen->HTML 1.0->Flex->HTML5->Swift), and it’s what will benefit us people most: companies get new businesses and, thus, growth themselves, us individuals get companies using software more to hopefully make working with them suck less. You know: Uber and all that.

There’s lots of “drag” (secondary spending) that gets to #3 above: you know, you’re gonna need a platform for all that stuff, and the hardware and services around it…but instead of just ending up with the same IT-driven capabilities, you’ll have new capabilities in your business.

Portfolio Management

So, if you’re a tech company, and you’re looking at the 4 sources of cash and growth above, the fourth option looks pretty good. #1 means competing with Apple, Google, and Facebook and then a dog’s breakfast of lower margin goods below the UI layer. #2 and #3 are good, known quantities, but probably with single-digit growth, if not tricky waters to traverse in the lower cloud infrastructure layers.

Then you look at the other option: a wide open field of possibilities where you “go up the elevator” and avoid the Morlocks. Large tech companies have to do all of these, of course, but I suspect you’ll see most of the razzle-dazzle spread on the fourth.

Those cigar makers have nothing to do with this, but cool picture, huh?

Never mind journalism, it’s industry analysts who are being disrupted.

I keep coming across a new crop of IT industry analysts who end up getting compared incorrectly to journalists. It’s little wonder as most people have little idea what an industry analyst does; it’s not like analysts, hidden behind their austere paywalls, help much there.

People like Horace Dediu, Ben Thompson, and others are experimenting with ways to disrupt industry analysts. They’re using new business models and tools that often seem bonkers to the more traditional analysts wrapped up all warm and tight in their blue blazers.

Their models focus on narrow topics with broad appeal (Apple, vendor sports among high profile tech companies [you can call this “strategy”], and “social”) and they tend to make much, if not all, of their content free. What they lack is the breadth of the overall industry analyst world (they have no opinion on what type of identity and access management or CRM system you might want to use), but that can could be fixed as more “independent” analysts like themselves pop up. There’s also not a lot of “short-listing” (ranking of vendors and products intended to be used by IT decision makers and buyers) that these folks do; this an area where incumbents can easily defend themselves.

One way of looking at it is the “consumerization of industry analysis:” focusing on selling and serving individuals rather than enterprises. Indeed,current industry analyst shops sell mostly to companies and are near impossible for individuals to work with.

Imagine Ben Thompson times 50

While Horace is patient zero here, the best example of this trend in action is Ben Thompson, or “stratechery.” For whatever reason - and his self-proclaimed Midwestern modestly would make him blush at this notion - he talks about his business more and, thus, provides a better view into the business side of this trend.

In the first episode of his podcast, Ben lays out the model he’s trying to execute (and how, you know, the Internet and blogging makes all this possible); he later elaborated on it with his rain forrest layer cake metaphor; and in an even more recent episode goes over how his business has evolved.

How well do these models work? Well, we have some data points from Ben since he’s discussed his momentum by subscriber numbers a few times. Let’s compare it to what I’ve made as analyst over the years to get a sense of what’s “normal”:

[Missing Image]

(Sources: my often shoddy memory [adding up salary and bonus approximations], and Ben Thompson talking about reaching 1,000 subscribers on November 13th, 2014, and then 2,000 on February 2nd, 2015).

This excludes a lot of thing: health insurance is the biggest and other non-cash compensation.

The point is to show that at the individual level, Ben is doing well. His business is performing well compared to what’s “normal” for analysts. The recent growth rate looks even more promising. I actually ended up at the high end of the analyst wage chart (I think). The average is a lot closer to $100,000 the more junior you get.

If this model can be replicated by other individuals , we’ll see the biggest disruption to the industry analyst business since the Web. What the established firms have is marketing reach, brand awareness, and lots of money and time. The first two are hard for individuals to achieve, but not impossible. The last two are harder.

I think there’s a lot of room for Gartner (the mega firm) and the RedMonks (boutique firms) of the world, but in the middle things will get harder. Forrester is always rearing to be a #2, but revenue-wise, they have a long way to go; IDC will probably keep winching the cost-cranks and double down on being Master of the PivotTable. My former friends at 451 Research have a lot of potential, but like all the other folks in the middle, they need to keep honing their strategies and go-to-market. I keep hearing that HfS is awesome, which could provide an interesting case.

With that bucket of points made for the tl;dr crowd, the rest is an extended treatment.

The Enterprise Gruber

I ran into Nick Muldoon a few years ago at a DevOpsDays (in 2012, right in the middle of my time at Dell) and he paid me a high compliment, loosely quoted from memory: “I always thought you could be the Gruber of enterprise IT.” Indeed, that’s what all my type dream about when we drive past those lottery billboards on the way to the airport at 4:30am: sitting at home, reading news, blogging, and being so awesome that it pays well.

To some extent, I did that at RedMonk; not at Dell, for sure: there’s no talking in public, really, when you work on strategy and M&A. And I did that on the pay side of the paywall at 451 Research. I loved it: writing up what I think about the IT industry targeted at helping my “audience” (we called them “clients”) make better decisions about IT, be it product management, competing, investing, or using IT. (There’s a minority that look towards analysts for entertainment, which is valid, but likely pays poorly.)

In part, that’s what I’ve been asked to do in my new job at Pivotal, except with a Pivotal bent, of course.

The RedMonk Disruption

Just a few years into it, RedMonk decided to do away with their paywall and ended up showing one path to disrupting the industry analyst market: the idea of providing free analyst reports through blogs seemed crazy, but it worked. James captured the, uh, esprit de corps in the analyst world well in a 2005 post:

I was at a recent event when a well known industry analyst, who used to run a firm well known for writing white papers in support of vendor positions, sat down. I was discussing how blogs, RSS splicing and aggregation were going to change industry analyst and other information-based businesses. They sniffed and said that bloggers had no credibility. This from someone that sold their credibility down the river long ago.

Yup, analysts are a friendly lot…

As with VCs, one of the problems an analyst has is generating enough flow to get the raw materials you need for your day-to-day work: getting people to talk to you enough, frequently enough, and deeply enough to gather all the information you need to usefully pontificate. You need raw fodder for your content creation. Ben alludes to this is another way: you have to create a pipe (or an overflowing Evernote notebook) of content ideas, things to write and talk about…to analyze.

For RedMonk, having no paywall meant that their marketing was done for “free.” The consequence was (and still is) that RedMonk can’t charge for content, it’s all free. Most firms in the industry analyst business charge a lot for content. My last analyst shop, 451 Research, charges a bundle, and people seem to like it: 451 writes great stuff and their large customer base shows that people value it. But, it does mean that 451 needs to do marketing separately; they don’t get those “zero marketing budget” dynamics RedMonk does. Neither model is better or worse, just different depending on what and how you’re running the business. Both models still get paid for consulting, webinars, and a multitude of other things.

Let’s look at three firms to peek into the bushes of the business a little bit.

RedMonk’s Differentiation

RedMonk thus differentiated itself from other analyst firms first by making all of its research free (at the time, very novel): it allowed RedMonk to build pull in the market, that is, it made marketing free. It wasn’t easy, and it took awhile, but it worked.

Their research topics matched this structural approach as well, namely:

1. focusing on bleeding edge technologies and practices, which, leads to the paid work on…
2. helping those bleeding edge companies talk to the “mainstream” IT world
3. helping “mainstream” IT vendors (IBM, Microsoft, Sun, Adobe, etc.) understand how to profit from and strategize/product plan with/around bleeding edge stuff.

They still do that and do it well, along with some of the usual analyst business models (like consulting, webinars, events, etc.)…but all of RedMonk’s activities revolve around knowing about the new shit sooner than the next analyst and being able to explain how to fit it into client thinking. Their events business (launched after I left) looks like an an adjacent business to the “knowing what the fuck we’re talking about” strategy: the tried and true come “hang out with the smart folks and drink your face of” business model.

RedMonk is cheap compared to other firms. The entry level is $5,000 for startups, and goes up from there. Companies like IBM, SAP, and Microsoft pay a lot more (and get a lot more!) but still get a really good deal compared to what other firms charge. You can check out their client logo page to estimate their revenue if you do a little estimating for the larger account sizes and Excel swagging: not too shabby, eh?

The 451 Differentiation

451, structurally, is similar to other analyst outfits: there’s a paywall for most everything. As with any analyst outfit, 451 does paid consulting, webinars, events, and other usual marketing driven stuff. 451 also does data center planning (they acquired The Uptime Institute some time ago) and has some interesting data-driven businesses that are being marshaled into proper quantitative analyst products. 451’s key differentiation is mixing its scale with the “the new shit” focus (perhaps a bit less bleeding edge than RedMonk, but not much), all stuck in the speed blender of publishing velocity.

451 seeks out new technologies, not old ones, and writes a lot: each analyst has to write somewhere between 40–60 reports a year, basically one ~1,500 word report a week…not including other deliverables. For the most part, if you brief a 451 analyst they’ll write a report on you, vendors love that and it helps with content flow (and gives analysts inbox heart-burn). I was terrible at that cadence coming from the RedMonk school (which emphasizes consulting, which I did a lot of at 451 instead of writing), but the best performers at 451 rarely take a briefing that resulting in no report being written.

451 is slightly cheaper than larger folks like Gartner and much more expensive than RedMonk. I was delightfully shocked at how much 451 charged coming from RedMonk; which is more a reflection of how cheap RedMonk is (I’m not sure they’ve raised prices, at least at the entry level, since 2006 when I started there - great for clients!). You get a lot more content, of all types, however, from 451 than from RedMonk due to 451’s sheer analyst bulk and core process of weekly report writing.

You’ll recall that my personal revenue was much higher at 451. I think that’s a reflection of the “leverage” a larger group of analysts can have: selling the same thing (reports and knowledge) over and over more.

Gartner Differentiation

Gartner is giant. It has breadth and has captured much marketshare. In analyst sales calls you often hear a variation on this:

Well, we’re signing up with Gartner because we have to, and IDC next because we need their PivotTables…the rest of you get to fight over what’s left (want to write a white paper for me?).

Gartner is really good at being the Microsoft of the analyst space…and I mean that as a compliment.

One of the key activities they do is ranking vendors. That may seem trivial, but it’s huge. Gartner tells you what the safe bet in IT acquisition is. It may not be the growth bet, or even the anti-disruption bet for your industry, but it’s the safe bet. And really, with the way most people use IT, that’s all they want. People don’t want to be Uber, they’re forced to compete with Uber, and they’d rather Uber didn’t exist at all.

If this beguiles you, think about your own buying habits outside the realm of computers. Do you prefer to buy your building materials at Home Depot, or some experimental shop on the side of the road? Like lumber and Shop-Vacs, most people look to computers for a function, not a complex belief system (I could have typed “paradigm”), let alone putting a business strategy in action.

(We at Pivotal like to think we help companies who are wise enough to take the first mover advantage when it comes to using IT to gain competitive advantage. This is shockingly not everyone in the world, which is fine: so far there’s been plenty of wise customers out there.)

Throw in their relative scale, and Gartner is in the hollowed “don’t fuck it up” position.

Gartner is expensive, from what I’ve heard and encountered when I’ve been on the vendor side. However, depending on what you need their content is good and the ability to influence (that is, educate, not make them parrot your messaging) analysts through working with them is nice. The IIAR seems to like Gartner, that group of analyst relations folks having ranked Gartner as #1 most every year since 2008 (it’s interesting to note that individual analyst winners are much different). Enterprises seem to like Gartner a lot as well from anecdotes I hear.

(See more analyst shop rankings from Kea Company’s 2014 survey if you like that kind of thing.)

The New “Bloggers-cum-analysts”

The new (or “new new,” if you’re RedMonk and crew) crop of analysts follows the “make all the good stuff free” rule of having no paywalls (though Ben Thompson is following a sort of open core model). And in making all that stuff free, they’re doing much of the type of work that industry analysts do now, mostly of the qualitative sort. Horace gets into forecasts and market-sizing a bit, actually, which makes him even more of a threat; the other folks don’t seem to spend time on that. (Of late Horace has been mixing in market numbers from traditional analyst shops as well, but he also doesn’t do surveys.)

Again, the reason this new crop of “bloggers” are threatening to industry analysts is because they’re serving some of the same purposes, with much of the same tools and outputs of traditional analysts. And for those analyst activities they don’t currently do: it’s not too far fetched to think that someone soon will. The core differences are similar to previous disruptors (RedMonk and 451 - see, that’s why I outlined them above, all you tl;dr ding-a-lings!), but with some tweaks, namely:

Narrow focus, but with broad appeal

Most of these analysts have a very narrow focus that appeals to a mainstream market. One individual can only cover so much, but if there’s a large audience for that topic, it will suffice.

Horace ostensibly is the world’s premier Apple analyst. That’s a bold claim as I don’t read any Apple analysts you have to pay to read, so maybe there’s some better than him locked behind paywalls; at the very least, he’s a big deal for his size. As a side effect of being an Apple analyst, he covers the mobile space in general. There’s two more tricks for him that build off this seemingly narrow focus:

1. In that smartphones and tablets are taking over how compute is done, he sort of covers “PCs of the future” and, thus, computers broadly.
2. “Innovation” is actually his main focus area, namely, figuring out what and how great companies actually do great things (as the intro to his podcast says, more or less). Apple is a wise choice for this question, but you can see his engineering-like “break it apart to find out how it works” curiosity at play when he looks at how Hollywood works (with Pixar as his Apple there), Amazon, and the mysteries of Google (cf. “The absence of a purpose rooted in profit makes Google resistant to analysis.”).

The combination of Apple, “PC of the future,” and “innovation” all amounts to a very large “audience.”

(Recently, Horace went to go work for a think-tank; it’s hard to tell if that invalidates some of the “this independent blogger-cum-analyst thing is a thing” thinking here or not.)

While Ben Thompson started out seemingly as another Apple/mobile analyst, I’d argue he’s become more of a “third platform” analyst, discussing how the “consumerization of IT,” is effecting the tech industry.

The efficiency of vendor sports

This narrow focus means that both (and most of these new types of analysts) cover “vendor sports”: they don’t give buyers advice about what products and services to buy, they instead tell you how various tech vendors are doing and explore the strategic possibilities of new types of technology.

There are very few of these new analysts that are prescriptive when it comes to buying. I’m not sure why, but I’d theorize that it has a lot to do with the cost structures of doing such work (in the cycle time of analysts learning, collecting epiphanies, publishing, and then collecting money form clients for sharing the results). This is a large part of why I think Gartner’s scale and established position is and will continue to be hard to beat, head on at least.

There are some challengers here:

• I look towards folks like WiseGate, TrustRadius, Peerlyst, and even Spiceworks to sort out short-listing out.
• Folks over at places like The Virtualization Practice can help in their niche; if you wade through sites like BrainMadden.com you can get a sense of wise virtual desktop choices; and Horses for Sources seems like they’d be spot on for outsourcing short-listing.

Once these new bloggers move beyond vendor sports - if they can - and start recommending what to buy, the dynamics will change a lot. Until then, they’re nibbling at the industry analyst business…but that’s how it all starts.

Back to the “Enterprise Gruber”

On that quest to find an enterprise Gruber, there’s been a rash of sites of late that go for that. There’s still a giant gap in the market for good enterprise tech coverage.

• TPM started up EnterpriseTech which seems to have taken the ethos of his wonderful IT Jungle (which seems to purposefully hide behind its ancient-feeling aesthetic - it’s some of the best coverage and analysis of IBM that you’ll find out there), and just announced a new site, The Platform.
• The Diginomica gang is having a run at enterprise applications.
• My friends at thenewstack.io are getting close and will soon give a view of how native advertising works the industry analyst business model blender.

I watch these sites closely to see how they pan out and if they fall into the usual journalistic traps that start to preclude good analysis.

How the small and mid-sized firms can react

What you’d really like to see is some dramatic business model hacking in the mid and small section of the industry analyst market. What would it mean to have a Ben Thompson or a RedMonk approach at a place like 451, or Forrester even? Those shops would have huge cultural issues to deal with (analysts are, ironically, a lot who’re the least interested in doing new things in their own processes: they hate changing), but the established brand/reach and capital (in money and time) those larger firms could bring to the strategies of the micro firms would be interesting.

How the big firms can react

The problem with the big shops taking in the blogger-cum-analysts is that big shops don’t like to create rock star analysts. The rock stars leave to become independent because they can make more money, or, at least have more freedom. They, like me, also get snatched up by vendors who can pay much more including something rarely seen in the analyst world: those mythical stock options which could worth anything between the title for a large house, the cost of a college diploma, or pack of novelty cup-cake papers.

Larger firms are better positioned to cement their position by upping their game by deeply evaluating and short-listing technologies. There are two examples right in front of us: OpenStack and Docker. Both of those vacillate between IaaS utopia (sometimes people come down from their buzz and realize that Docker often aspires to be a PaaS too) and shit-shows cracking in tire-fires all the way down.

Someone like a Gartner has the time, money, and (potential) authority to run labs to test technologies like these out and give solid recommendations on what to use and not use…per business use case, even. To quote the meme, one does not simply build an enterprise cloud…so how could you expect anyone who just creates PDFs about cloud to actually be credible?

So, what’s your problem, Coté?

With all this glee, you may be wondering why I’m now at a vendor. Good question, as they say when they’re buying time to think. The core of it is that my fixed expenses are too high. I have a family, a large house, and even a new dog (I resisted as long as I could - promise!). And, I’m the single earner for all that.

While I would love to bushwhack my way through this emerging analyst jungle, I don’t want to Mosquito Coast my family; and let’s be honest, myself either. The warm, bi-weekly embrace of a vendor is very comforting. So, like the analysts themselves who observe from the sideline, I’ll be eagerly watching how the industry analyst sports-ball brackets play out.

(Also, check out the two part podcast - part one and part two - with myself and some other analysts on this topic.)