We’re getting exactly the government IT we asked for

If there’s one complaint that I hear consistently in my studies of IT in large organizations, it’s that government IT, as traditionally practiced, is fucked. Compared to the private sector, the amount of paperwork, the role of contractors, and the seeming separation between doing a good job and working software drives all sorts of angst and failure.

Mark Schwartz’s book on figuring out “business value” in IT is turning out to be pretty amazing and refreshing, especially on the topic of government IT. He’s put together one of the better “these aren’t the Droids you’re looking for” responses to ROI for IT.

You know that answer: you just want to figure out the business case, ROI, or whatever numbers driven thing, and all the DevOps-heads are like “doo, doo, doo-doo – driving through a tunnel, can’t hear you!” and then they pelt you with Goldratt and Deming books, blended in with some O’Reilly books and The Phoenix Project. “Also, you’re argument is invalid, because reasons.”

A Zen-like calm comes over them, they close their eyes and breath in, and then start repeating a mantra like some cowl-bedecked character in a Lovecraft story: “survival is not mandatory. Survival is not mandatory. Survival is not mandatory!”

Real helpful, that lot. I kid, I jest. The point of their maniacally confusing non-answers is, accurately, that your base assumptions about most everything are wrong, so before we can even approach something as precise as ROI, we need to really re-think what you’re doing. (And also, you do a lot of dumb shit, so let’s work on that.)

But you know, no one wants to hear they’re broken in the first therapy session. So you have to throw out some beguiling, mind-altering, Lemarchand’s boxes to change the state of things and make sure they come to the next appointment.

Works as Designed

Anyhow, back to Schwartz’s book. I’ll hopefully write a longer book review over at The New Stack when I’m done with it, but this one passage is an excellent representation of what motivates the book pelters and also a good unmasking of why things are the way we they are…because we asked for them to be so:

The US government is based on a system of “checks and balances”—in other words, a system of distrust. The great freedom enjoyed by the press, especially in reporting on the actions of the government, is another indication of the public’s lack of trust in the government. As a result, you find that the government places a high value on transparency. While companies can keep secrets, government is accountable to the public and must disclose its actions and decisions. There is a business need for continued demonstrations of trustworthiness, or we might as well say a business value assigned to demonstrating trustworthiness. You find that the government is always in the public eye—the press is always reporting on government actions, and the public is quick to outrage. Government agencies, therefore, place a business value on “optics”—how something appears to the observant public. In an oversight environment that is quick to assign blame, government is highly risk averse (i.e., it places high business value on things that mitigate risk).

And then summarized as:

…the compliance requirements are not an obstacle, but rather an expression of a deeper business need that the team must still address.

Which is to say: you wanted this, and so I am giving it to you.

The Agile Bureaucracy

The word “bureaucracy” is something like the word “legacy.” You only describe something as legacy software when you don’t like the software. Otherwise, you just call it your software. Similarly, as Schwartz outlines, agile (and all software processes) are insanely bureaucratic, full of rules, norms, and other “governance.” We just happen to like all those rules, so we don’t think of them as bureaucracy. As he writes:

While disavowing rules, the Agile community is actually full of them. This is understandable, because rules are a way of bringing what is considered best practices into everyday processes. What would happen if we made exceptions to our rules—for instance, if we entertained the request: “John wants to head out for a beer now, instead of fixing the problem that he just introduced into the build?” If we applied the rules capriciously or based on our feelings, they would lose some of their effectiveness, right? That is precisely what we mean by sine ira et studio in bureaucracy. Mike Cohn, for example, tells us that “improving technical practices is not optional.”15 The phrase not optional sounds like another way of saying that the rule is to be applied “without anger or bias.” Mary Poppendieck, coauthor of the canonical works on Lean software development, uses curiously similar language in her introduction to Greg Smith and Ahmed Sidky’s book on adopting Agile practices: “The technical practices that Agile brings to the table—short iterations, test-first development, continuous integration—are not optional.” I’ve already mentioned Schwaber and Sutherland’s dictum that “the Development Team isn’t allowed to act on what anyone else [other than the product owner] says.”17 Please don’t hate me for this, Mike, Mary, Ken, and Jeff, but that is the voice of the command-and-control bureaucrat. “Not optional,” “not allowed,” – I don’t know about you, but these phrases make me think of No Parking and Curb Your Dog signs.

These are the kind of thought-trains that only ever evoke “well, of course my intention wasn’t the awful!” from the other side. It’s like with the ITIL and the NRA, gun-nut people: their goal wasn’t to put in place a thought-technology that harmed people, far from it.

Gentled nestled in his wry tone and style (which you can image I love), you can feel some hidden hair pulling about the unintended consequences of Agile confidence and decrees. I mean, the dude is the CIO of a massive government agency, so he be throwing process optimism against brick walls daily and too late into the night.

Learning bureaucracies

The cure, as ever, is to not only to be smart and introspective, but to make evolution and change part of your bureaucracy:

Rules become set in stone and can’t change with circumstances. Rigidity discourages innovation. Rules themselves come to seem arbitrary and capricious. Their original purpose gets lost and the rules become goals rather than instruments. Bureaucracies can become demoralizing for their employees.

So, you know, make sure you allow for change. It’s probably good to have some rules and governance around that too.

Making mainframe applications more agile, Gartner – Highlights

In a report giving advice to mainframe folks looking to be more Agile, Gartner’s Dale Vecchio and Bill Swanton give some pretty good advice for anyone looking to change how they do software.

Here’s some highlights from the report, entitled “Agile Development and Mainframe Legacy Systems – Something’s Got to Give”

Chunking up changes:

  1. Application changes must be smaller.
  2. Automation across the life cycle is critical to being successful.
  3. A regular and positive relationship must exist between the owner of the application and the developers of the changes.


This kind of effort may seem insurmountable for a large legacy portfolio. However, an organization doesn’t have to attack the entire portfolio. Determine where the primary value can be achieved and focus there. Which areas of the portfolio are most impacted by business requests? Target the areas with the most value.

An example of possible change:

About 10 years ago, a large European bank rebuilt its core banking system on the mainframe using COBOL. It now does agile development for both mainframe COBOL and “channel” Java layers of the system. The bank does not consider that it has achieved DevOps for the mainframe, as it is only able to maintain a cadence of monthly releases. Even that release rate required a signi cant investment in testing and other automation. Fortunately, most new work happens exclusively in the Java layers, without needing to make changes to the COBOL core system. Therefore, the bank maintains a faster cadence for most releases, and only major changes that require core updates need to fall in line with the slower monthly cadence for the mainframe. The key to making agile work for the mainframe at the bank is embracing the agile practices that have the greatest impact on effective delivery within the monthly cadence, including test-driven development and smaller modules with fewer dependencies.

It seems impossible, but you should try:

Improving the state of a decades-old system is often seen as a fool’s errand. It provides no real business value and introduces great risk. Many mainframe organizations Gartner speaks to are not comfortable doing this much invasive change and believing that it can ensure functional equivalence when complete! Restructuring the existing portfolio, eliminating dead code and consolidating redundant code are further incremental steps that can be done over time. Each application team needs to improve the portfolio that it is responsible for in order to ensure speed and success in the future. Moving to a services-based or API structure may also enable changes to be done effectively and quickly over time. Some level of investment to evolve the portfolio to a more streamlined structure will greatly increase the ability to make changes quickly and reliably. Trying to get faster with good quality on a monolithic hairball of an application is a recipe for failure. These changes can occur in an evolutionary way. This approach, referred to in the past as proactive maintenance, is a price that must be paid early to make life easier in the future.

You gotta have testing:

Test cases are necessary to support automation of this critical step. While the tooling is very different, and even the approaches may be unique to the mainframe architecture, they are an important component of speed and reliability. This can be a tremendous hurdle to overcome on the road to agile development on the mainframe. This level of commitment can become a real roadblock to success.

Another example of an organization gradually changing:

When a large European bank faced wholesale change mandated by loss of support for an old platform, it chose to rewrite its core system in mainframe COBOL (although today it would be more likely to acquire an off-the-shelf core banking system). The bank followed a component-based approach that helped position it for success with agile today by exposing its core capabilities as services via standard APIs. This architecture did not deliver the level of isolation the bank could achieve with microservices today, as it built the system with a shared DBMS back-end, as was common practice at the time. That coupling with the database and related data model dependencies is the main technical obstacle to moving to continuous delivery, although the IT operations group also presents cultural obstacles, as it is satis ed with the current model for managing change.

A reminder: all we want is a rapid feedback cycle:

The goal is to reduce the cycle time between an idea and usable software. In order to do so, the changes need to be smaller, the process needs to be automated, and the steps for deployment to production must be repeatable and reliable.

The ALM technology doesn’t support mainframes, and mainframe ALM stuff doesn’t support agile. A rare case where fixing the tech can likely fix the problem:

The dilemma mainframe organizations may face is that traditional mainframe application development life cycle tools were not designed for small, fast and automated deployment. Agile development tools that do support this approach aren’t designed to support the artifacts of mainframe applications. Modern tools for the building, deploying, testing and releasing of applications for the mainframe won’t often t. Existing mainframe software version control and conguration management tools for a new agile approach to development will take some effort — if they will work at all.

Use APIs to decouple the way, norms, and road-map of mainframes from the rest of your systems:

wrapping existing mainframe functions and exposing them as services does provide an intermediate step between agile on the mainframe and migration to environments where agile is more readily understood.

Contrary to what you might be thinking, the report doesn’t actually advocate moving off the mainframe willy-nilly. From my perspective, it’s just trying to suggest using better processes and, as needed, updating your ALM and release management tools.

Read the rest of the report over behind Gartner’s paywall.

The state of Java – My Feburary Register Column

This month, my column at The Register is on the state of Java and the evolving nature of J(2)EE:

Despite all the inside-bickering, lawsuits, a shotgun wedding to Oracle, drawn-out releases, and rivals from PHP, to Rails, to Swift, Java is still in wide use and shows no signs of finally dying. Jobs-wise, you’d be hard-pressed to find a better language than Java as your primary programming language if you wanted to switch from dropping off hot-pies to writing code.

Check out the rest!

Source: Java? Nah, I do JavaScript, man. Wise up, hipster, to the money • The Register

Hardware layoffs at Oracle

Oracle claims the company isn’t closing the Santa Clara facility with this reduction in force. Instead, “Oracle is refocusing its Hardware Systems business, and for that reason, has decided to lay off certain of its employees in the Hardware Systems Division.”

Those hardware employees appear to have been Oracle’s failing SPARC hardware department staffers. In mid 2016, Oracle claimed its new SPARC S7 processor would be offered on Oracle Cloud. The cloud is Oracle’s new revenue hope since its new software licensing revenue plummeted by 20 percent in its last quarter ended December 15. At the same time, Oracle’s hardware revenue had fallen 13 percent.


Choose your TAM wisely and remember to charge a high price, RethinkDB

[O]ur users clearly thought of us as an open-source developer tools company, because that’s what we really were. Which turned out to be very unfortunate, because the open-source developer tools market is one of the worst markets one could possibly end up in. Thousands of people used RethinkDB, often in business contexts, but most were willing to pay less for the lifetime of usage than the price of a single Starbucks coffee (which is to say, they weren’t willing to pay anything at all). Link

How big is the pie?

Any company selling developers tools needs to figure out the overall market size for what they’re selling. Developers, eager to work tools for themselves (typically, in their mid to late 20s developers work on at least one “framework” project) often fall prey to picking a market that has little to no money and, then, are dismayed when “there’s no money in it.”

What we’re looking for here is a market category and a way of finding how much money is being spent in it. As a business, you want grab as much as the money as possible. The first thing you want to do is make sure there’s enough money for you to care. If you’re operating in a market that has only $25m of total, global spend, it’s probably not worth your while, for example.

Defining your market category, too, is important to find out who your users and buyers are. But, let’s look at TAM-think: finding what the big pie of cash looks like, your Total Addressable Market.

The TAMs on the buffett

If you’re working on developer oriented tech, there are a few key TAMs:

Another interesting TAM for startups in the developer space is a combo one Gartner put out recently put together that shows public and private PaaS, along with “traditional” application platforms: $7.8bn in 2015. 451 has a similar TAM that combines public and private cloud at around $10bn in 2020.

I tried to come up with a public and private PaaS TAM – a very, very loose one – last year and sauntered up to something like $20 to $25bn over the next 5-10 years.

There are other TAMs, to be sure, but those are good ones to start with.

Bending a TAM to your will, and future price changes

In each case, you have to be very, very careful because of open source and public cloud. Open source means there’s less to sell upfront and, that, likely, you’ll have a hard time suddenly going from charging $0 to $1,000’s per unit (a unit is whatever a “seat” or “server” is: you need something to count by!). If you’re delivering your stuff over the public cloud, similar pricing problems arise: people expect it to be really cheap an are, in fact, shocked when it adds up to a high monthly bill.

But briefly: people expect infrastructure software to be free now-a-days. (Not so much applications, which have held onto the notion that they should be paid for: buy the low prices in the app store depress their unit prices too.)

In both cases (open source and public cloud delivery), you’re likely talking a drastically lower unit price. If you don’t increase the overall volume of sales, you’ll whack down your TAM right quick.

So, you have to be really, really careful when using backward looking TAMs to judge what your TAM is. Part of the innovation you’re expected to be doing is in pricing, likely making it cheaper.

The effect is that your marketshare, based on “yesterday’s TAMs,” will look shocking. For example, Gartner pegged the collective revenue of NoSQL vendors (Basho, Couchbase, Datastax, MarkLogic, and MongoDB) at $364M in 2015: 1% of the overall TAM of $35.9bn! Meanwhile, the top three Hadoop vendors clocked in at $323.2M and AWS’s DB estimate was $833.6M.

Pair legacy TAMs with your own bottoms-up TAM

In my experience, the most helpful way for figuring out (really, recomputing TAMs in “real time) is to look at the revenue that vendors in that space are having and then to understand what software they’re replacing. That is, in addition to taking analyst TAMs into perspective, you should come up with your own, bottoms-up model and explain how it works.

If you’re doing IT-lead innovation, using existing (if not “legacy”!) TAMs is a bad idea. You’ll likely end up over-estimating your growth and, worse, which category of software you are and who the buyers are. Study your users and your buyers and start modeling from there, not pivot tables from the north east.

The other angle here is that if you’re “revolutionizing” a market category, it means you’re redefining it. This means there will be no TAM for many years. For example, there was no “IaaS” TAM for a long time, at some point, there was no “Java app server TAM.” In such cases, creating your own TAMs are much more useful.

Finally, once you’ve figured out how big (or small!) your pie of money is, adjust your prices accordingly. More than likely you’ll find that you’ll need to charge a higher price than you think is polite…if you want to build a sustainable, revenue-driven business rather than just a good aggregation startup to be acquired by a larger company…who’ll be left to sort out how to make money.

“the obsolescence of Java EE” – Notebook

Bottom line: Java EE is not an appropriate framework for building cloud-native applications.

In preparation for this week’s Pivotal Conversations, I re-read the Gartner write-up on the decline of traditional JEE and the flurry of responses to it. Here’s a “notebook” entry for all that.

From Gartner’s “Market Guide for Application Platforms”

This is the original report from Anne Thomas and Aashish Gupta, Nov 2016. Pivotal has it for free in exchange for leag-gen’ing yourself.
What is an “application platform” vs. aPaaS, etc.?

Application platforms provide runtime environments for application logic. They manage the life cycle of an application or application component, and ensure the availability, reliability, scalability, security and monitoring of application logic. They typically support distributed application deployments across multiple nodes. Some also support cloud-style operations (elasticity, multitenancy and selfservice).

An “aPaaS,” is a public cloud hosted PaaS, of which they say: “By 2021, new aPaaS deployments will exceed new on-premises deployments. By 2023, aPaaS revenue will exceed that of application platform software.”

On the revenue situation:


Commercial Java Platform, Enterprise Edition (Java EE) platforms’ revenue declined in 2015, indicating a clear shift in the application platform market…. Application platform as a service (aPaaS) revenue is currently less than half of application platform software revenue, but aPaaS is growing at an annual rate of 18.5%, and aPaaS sales will supersede platform software sales by 2023.


Currently, the lion’s share of application platform software revenue comes from license sales of Java EE application servers. From a revenue perspective, the application platform software market is dominated by just two vendors: Oracle and IBM. Their combined revenues account for more than three-quarters of the market.

Decline in revenue for current market leaders IBM and Oracle over last three years (4.5% and 9.5% respectively), meanwhile uptick from Red Hat, AWS, and Pivotal (33.3%, 50.6% and 22.7% respectively).
Decline/shifting is driven by:

given the high cost of operation, the diminishing skill pool and the very slow pace of adoption of new technologies, a growing number of organizations — especially at the low end of the market — are migrating these workloads to application servers or cloud platforms, or replacing them with packaged or SaaS applications.


Java EE has not kept pace with modern architectural trends. Oracle is leading an effort to produce a new version of Java EE (version 8), which is slated to add a host of long-overdue features; however, Oracle announced at Oracle OpenWorld 2016 that Java EE 8 has been delayed until the end of 2017.3 By the time Java EE catches up with basic features required for today’s applications, it will be at least two or three years behind the times again.

Target for cloud native:

Design all new applications to be cloud-native, irrespective of whether or not you plan to deploy them in the cloud…. If business drivers warrant the investment, rearchitect existing applications to be cloud-native and move them to aPaaS.

Vendor selection:

Give preference to vendors that articulate a platform strategy that supports modern application requirements, such as public, private and hybrid cloud deployment, in-memory computing, multichannel clients, microservices, event processing, continuous delivery, Internet of Things (IoT) support and API management.


Oracle and Java: confusing

Oracle’s stewardship of Java has been weird of late:

It’s all about WebLogic and WebSphere

I think this best sums it all up, the comments from Ryan Cuprak: “What this report is trying to do is attack Oracle/IBM via Java EE.”

I wouldn’t say “attack,” but rather show that their app servers are in decline, as well as TP processing things. The report is trying to call the shift to both a new way of development (cloud native) and the resulting shifts in product marketshare, including new entrants like Pivotal.

I can’t speak to how JEE is changing itself, but given past performance, I’d assume it’ll be a sauntering-follower to adapting technologies; the variable this time is Oracle’s proven ambivalence about Java and JEE, and, thus, funding problems to fuel the change fast enough to keep apace with other things.

WTF is “digital transformation”? Beyond AI and VR for practical, software-driven innovation – My January Register Column

At the top of the year, companies are setting their IT agendas. Most high level executives seem to be lusting for “digital transformation,” but that phrase is super-squishy. In my Register column this month, I offered my advice on what it should be: simply “digitizing” existing, manual work-flows by perfecting how you do software.

This, of course, is the core of what I work on at Pivotal; see my wunderkammer of knowledge, the soon to be PDF’ed “Crafting your cloud native strategy,” for example.

What do these opportunities look like in businesses? Here’s a chunk that cut out of the piece that provides some examples:

A project to “digitize” the green card replacement program in the US provides a good example of the simple, pragmatic work IT departments should be curating for 2017. Before injecting software into process it’d “cost about $400 per application, it took end user fees, it took about six months, and by the end, your paper application had traveled the globe no less than six times. Literally traveled the globe as we mailed the physical papers from processing center to processing center.”

After discovering agile and cleaning up the absurd government contracting scoping (a seven year project costing $1.2bn, before accounting for the inevitable schedule and budget overruns), a team of five people successfully tackled this paper-driven, human process. It’s easy to poke fun at government institutions, but if you’ve applied for a mortgage, life insurance, or even tried to order take out food from the corner burger-hut, you’ll have encountered plenty of human-driven processes that could easily be automated with software.

After talking with numerous large organizations about their IT challenges, to me, this kind of example is what “digital transformation” should mostly about, not introducing brain-exploding, Minority Report style innovation. And why not? McKinsey recently estimated that, at best, only 29% of a worker’s day-to-day requires creativity. Much of that remaining 71% is likely just paid-for monotony that could be automated with some good software slotted into place.

That last figure is handy for thinking about the opportunity. You can call it “automation” and freak out about job stealing, but it looks like a huge percentage of work can be “digitized.”

Check out the full piece.

Not actually a DevOps talk

I get asked to talk on DevOps a lot. Here’s my current (late 2016 and 2017) presentation, going over the why’s, the how’s, the technologies, and the meatware that supports including some best and worst practices based on what Pivotal customers do. See the slides.

Also, here’s a more blatantly pro-Pivotal (and longer) version that you might have seen, esp. if the talk title was something like “Digital Transformation in the Streets.”

Much of it draws a lot on my cloud native journey booklets as well.

RFQ: Cloud Native Journey, 2nd edition WIP


Like reading about doing agile, DevOps, and “cloud native” in the real world? Help me finish up my current booklet on that topic by reviewing my almost finished draft.

I’ve been working on this since around August of this year. I’m almost done! There’s some of my content you might have seen around the web here and there, but most of it is new. I’ve tried to wrap up all the common topics I talk about with large organizations and put in as many cases and anecdotes – proof and data! – from “donkey” organizations as possible.

Help me get more eyes on this, and also read an “early edition” before you have to get your boy Johnny Leadgen on the case.