Pumping the digital transformation bunny at the US Air Force, an interview with Bryon Kroger

Few organizations have or rely on as much software the US Air Force. There’s plenty of it around and, thus, plenty to be improved. In recent years, one of the more spectacular digital transformation stories has come from the USAF’s work modernizing their Air Operations Control software. In this episode, USAF’s Bryon Kroger goes over how they’ve moved multi-year release cycles to just weeks in the Kessel Run projects. Much of the work is in the “fuzzy front” end of planning and procurement, but as Bryon says, an equally, hearty serving has to do with building up people’s skills, moral, and the overall culture.

One of my recent Pivotal Conversations episodes. There’s a play list collecting together other “customers” talking, rather than the usual of us Pivotal people just talking to ourselves.

Link: Mission capability delivered at startup speed

More coverage of the US Air Force going all in on digital XP, lean design, and cloud native to dramatically – almost unbelievably so – modernize their software.

The mission capabilities these war fighters received in 120 days or less span deliberate targeting, mission reporting, advanced target production, refueling operations and many more, saving over $6.4 million and 1,100 man-hours per month within the Air Force Central Command.

More:

Embracing agile software development processes, the AOC leverages a highly disciplined approach to software engineering called Extreme Programming (XP), to gain the benefits of test-driven development, pair programming, continuous integration and continuous delivery (CI/CD).

Original source: Mission capability delivered at startup speed

Link: Cyber Airmen fuel innovation

Any problems discovered in the software can be more easily corrected, a small failure being preferable to correcting the shortfalls of an entire software suite. Changes requested by the customer can sometimes be delivered in just a couple days.

Prior to Jigsaw, the tanker mission planners would use whiteboards to plot out their fueling rendezvous.

“Rather than taking hours to run the calculations by hand for the hundreds of sorties scheduled each day to find a feasible plan,” Tatro said. “The program logs events in order to detect and report errors in scheduling.”

It didn’t take long for the new software to start paying off.

“Before Jigsaw was delivered to the tanker planners, they would be spending 8 to 12 hours a day with a team of five planning a day of tanker missions,” Maung said. “Now they only need three people and it takes them 4 to 5 hours, usually done before lunch.”

Since the program was put into use in April 2017, Jigsaw has saved approximately $200,000 per day, just in fuel. There were other benefits as well.
Original source: Cyber Airmen fuel innovation

Link: Air Force wants to make ‘Kessel Run’ standard in tech acquisition

“You’re probably going to see maybe a directive from us that basically says every acquisition is going to have to have something that looks like Kessel Run from the primes. So you want to have [authority to operate] within three or four weeks, not six years.”
Original source: Air Force wants to make ‘Kessel Run’ standard in tech acquisition

Link: The Air Force Will Treat Computer Coding Like a Foreign Language

For example, when Defense Innovation Unit went to air operations centers in Middle East, the defense tech expert envisioned software changes that would optimize the way that airmen tracked refueling tankers. Teaming up with a commercial firm in Boston called Pivotal Labs, the new software is saving about $200,000 in fuel every month.

It’s usually reported at $200,000 a day.

Mostly, though, lots of AI talk:

During Operation Inherent Resolve, he led 1,800 Air Component Commander analysts to create the first-ever visualization of millions of data points from sources including unmanned aerial vehicles and open-source streams. That reduced 1.5 hours of daily target development to five minutes. He also provided intelligence support to the fight against ISIS that increased deliberate (i.e. pre-planned) target development by 68 percent.
Original source: The Air Force Will Treat Computer Coding Like a Foreign Language

A series of small projects, building momentum to scale

This post is an early draft of a chapter in my book,  Monolithic Transformation.

Not actually a picture of what’s described here, but it looks cool.

Every journey begins with a single step, they say. What they don’t tell you is that you need to pick your first step wisely. And there’s also step two, and three, and then all of the n + 1 steps. Picking your initial project is important because you’ll be learning the ropes of a new way of developing and running software, and hopefully, of running your business.

When it comes to scaling change, choosing your first project wisely is also important for internal marketing and momentum purposes. The smell of success is the best deodorant, so you want your initial project to be successful. And…if it’s not, you quietly sweep it under the rug so no one notices. Few things will ruin the introduction of a new way of operating into a large organization than initial failure. Following Larman’s Law, the organization will do anything it can — consciously and unconsciously — to stop change. One sign of weakness early, and your cloud journey will be threatened by status quo zombies.

In contrast, let’s look at how the series of small projects strategy played out in the US Air Force.

The USAF had been working for at least 5 years to modernize the 43 applications used in Central Air Operations Command, going through several hundreds of millions of dollars. These applications managed the US’s and allie’s daily air missions throughout Iraq, Syria, Afghanistan, and nearby countries. No small task of import. The applications were in sort need of modernizing, and some weren’t even really applications: the tanker refueling scheduling team used a combination of Excel spreadsheets and a whiteboard to plan the daily jet refueling missions.

Realizing that they’re standard 5 to 12 years cycle to create new applications wasn’t going to cut it, the US Air Force decided to try something new: a truly agile, small batch approach. Within 120 days, a suitable version of the tanker refueling application was in production. The tanker team continued to release new features on a weekly, even daily basis. The project was considered a wild success: the time to make the tanker schedule was reduced from 8 hours to 2, from 8 airmen to 1, and the USAF ended up saving over $200,000 a day in fuel that no longer needed to be flown around as backup for error in the schedule.

Number of USAF CAOC transformed applications over time, starting with 0 and ending with an estimated 18. Sources from several USAF presentations and write-ups.

The success of this initial project, delivered in April of 2017, called JIGSAW, proved that a new approach would work, and work well. This allowed the group driving change at the USAF to start another project, and then another one, eventually getting to 13 projects in May of 2018 (5 in production and 8 in development. The team estimates that by January of 2018 they’ll have 15 to 18 applications in production.

The team’s initial success, though just a small part of the overall 43 applications, gave them the momentum to starting scale change to the rest of the organization and more applications.

Project picking peccadilloes

Picking the right projects to start with is key. They should be material to the business, but low risk. They should be small enough that you can quickly show success in the order of months and also technically feasible for using cloud technologies. These shouldn’t be science projects or automation of low value office activities — no augmented reality experiments or conference room schedulers (unless those are core to your business). On the other hand, you don’t want to do something too big, like migrate the .com site. Christopher Tretina recounts Comcast’s initial cloud-native ambitions in this way:

We started out with a very grandiose vision… And it didn’t take us too long to realize we had bitten off a little more than we could chew. So around mid-year, last year, we pivoted and really tried to hone in and focus on what were just the main services we wanted to deploy that’ll get us the most benefit.

Your initial projects should also enable you to test out the entire software life cycle — all the way from conception to coding to deployment to running in production. Learning is a key goal of these initial projects and you’ll only do that by going through the full cycle.

The Home Depot’s Anthony McCulley describes the applications his company chose in the first six or so months of its cloud-native roll-out. “They were real apps. I would just say that they were just, sort of, scoped in such a way that if there was something wrong, it wouldn’t impact an entire business line.” In The Home Depot’s case, the applications were projects like managing (and charging for!) late tool rental returns and running the in store, custom paint desk.

A special case for initial projects is picking a microservice to deploy. Usually, such a service is a critical backend service for another application. A service that’s taken forever to actually deliver, or has been unchanged and ancient for years is an impactful choice. This is not as perfect a use case as a full-on, human-facing project, but it will allow you to test out cloud-native principals and rack up a success to build momentum. The microservice could be something like a fraud detection or address canonicalization service. This is one approach to migrating legacy applications in reverse order, a strangler from within!

Picking projects by portfolio pondering

There are several ways to select your initial projects. Many Pivotal customers use a method perfected over the past 25 years by Pivotal Labs called discovery. In the abstract, it follows the usual BCG matrix approach, flavored with some Eisenhower matrix. This method builds in intentional scrappiness to do a portfolio analysis with the limited time you can secure from all of the stakeholders. The goal is to get a ranked list of projects based on your organization’s priorities and the easiness of the projects.

First, gather all of the relevant stakeholders. This should include a mix of people from the business and IT sides, as well as the actual team that will be doing the initial projects. A discovery session is typically led by a facilitator, preferably someone familiar with coaxing a room through this process.

The facilitator typically hands out stacks of sticky notes and markers, asking everyone to write down projects that they think are valuable. What “valuable” means will depend on each stakeholder. We’d hope that the more business minded of them would have a list of corporate initiatives and goals in their heads (or a more formal one they brought to the meeting). One approach used in Lean methodology is to ask management this question: “If we could do one thing better, what would it be?” Start from there, maybe with some five whys spelunking.

Once the stakeholders have written down projects on their sticky notes, the discovery process facilitator draws or tapes up a 2×2 matrix that looks like the following:

Participants then put up their sticky notes in the quadrant, forcing themselves not to weasel out and put the notes on the lines. Once everyone finishes, you get a good sense of projects that all stakeholders think are important, sorted by the criteria I mentioned, primarily that they’re material to the business (important) and low risk (easy). If all of the notes are clustered in one quadrant (usually, in the upper right, of course), the facilitator will redo the 2×2 lines to just that quadrant, forcing the decision of narrowing down to just projects to do now. The process might repeat itself over several rounds. To enforce project ranking, you might also use techniques like dot voting which will force the participants to really think about how they would prioritize the projects given limited resources.

At the end, you should have a list of projects, ranked by the consensus of the stakeholders in the room.

Planning out the initial project

You may want to refine your list even more, but to get moving, pick the top project and start breaking down what to do next. How you proceed to do this is highly dependent on how your product teams breaks down tasks into stories, iterations, and releases. More than likely, following the general idea of a small batch process you’ll

  1. Create an understanding of the user(s) and the challenges they’re trying to solve with your software through personas and approaches like scenarios or Jobs to be Done.
  2. Come up with several theories for how those problems could be solved.
  3. Distill the work to code and test your theories into stories.
  4. Add in more stories for non-functional requirements (like setting up build processes, CI/CD pipelines, testing automation, etc.).
  5. Arrange stories into iteration-sized chunks without planning too far ahead (least you’re not able to adapt your work to the user experience and productivity findings from each iteration)

Crafting your hockey stick

Starting small ensures steady learning and helps contain the risk of a fail-fast approach. But as you learn the cloud-native approach better and build up a series of successful projects, you should expect to ramp up quickly. This chart shows The Home Depot’s ramp up in the first year:

Chart shows the number of application instances, which is not 1:1 to applications. The end-point represents about 130 applications, composed of about 900 services. Source: “Cloud-Native at Home Depot, With Tony McCulley,” Pivotal Conversations #45.

The chart measures application instances in Pivotal Cloud Foundry, which does not map exactly to a single application. As of December 2016, The Home Depot had roughly 130 applications deployed in Pivotal Cloud Foundry. What’s important is the general shape and acceleration of the curve. By starting small, with real applications, The Home Depot became learned the new process and at the same time delivered meaningful results that helped them scale their transformation.

See also another post of mine on this topic.

This post is an early draft of a chapter in my book,  Monolithic Transformation.

Link: Transforming the Air Force into a software company with more airpower

“We have a small team that was tired of working this way and tired of not being able to provide this capability to our warfighters,” said Adam Furtado (pictured), chief product officer at the U.S. Air Force. “Basically, Congress told us to figure something new out.”

And:

“We’ve been designing a brand new system to modernize for about 10 years, and we just haven’t been able to get it to the field for a ton of Department of Defense, bureaucratic and acquisitions reasons,” Furtado explained.
Original source: Transforming the Air Force into a software company with more airpower

Link: Airmen given direct access to AOC development process

“Air Force acquisition leaders recognized the current acquisition strategy, in progress since 2009, will not deliver capability to the warfighter fast enough. Today, we terminated the current AOC 10.2 contract with Northrop Grumman in order to take a different approach.”

And:

“Once we field this platform and establish the software pipeline, we will begin the iterative improvement process,” said Sanders.  “That’s where this acquisition process really makes a difference.  The customers, in this case Airmen at the AOCs, are able to communicate their needs directly to developers and see the changes they request within weeks.
Original source: Airmen given direct access to AOC development process

Link: Air Force looks to rapidly develop software with Project Kessel Run

More coverage of the USAF modernizing their approach to software. Here, what some of the apps are: “Kessel Run has been able to push five applications to the classified network, Kroger said…. The project is currently working on a number of things, including how the Air Force plans air tasking orders, a document which tasks units to fly their aircraft, Kroger said. It’s also working on building a tool that automates mission reports, which have to be written for every mission that flies, Kroger said.”

Original source: Air Force looks to rapidly develop software with Project Kessel Run

Link: Innovation at the edge: the top air defence trends by domain

“Software company Pivotal, backed by Dell EMC, VMWare, GE, Microsoft and Ford, has developed a tanker refuelling solution for the USAF with the US Defense Innovation Unit Experimental (DIUx); Running on the firm’s Pivotal Cloud Foundry platform, the software solution was built for under $2m in 90 days and is now being used in operational areas including Qatar. It currently saves the US Air Force $1 million per day in fuel costs, with the software being managed by just one person. It also aligns with USAF’s Air Operations Centre (AOC) capabilities via a continuous delivery software development pipeline to a hybrid cloud-based platform alongside the legacy AOC 10.1 system.”
Original source: Innovation at the edge: the top air defence trends by domain

Link: Innovation at the edge: the top air defence trends by domain

“Software company Pivotal, backed by Dell EMC, VMWare, GE, Microsoft and Ford, has developed a tanker refuelling solution for the USAF with the US Defense Innovation Unit Experimental (DIUx); Running on the firm’s Pivotal Cloud Foundry platform, the software solution was built for under $2m in 90 days and is now being used in operational areas including Qatar. It currently saves the US Air Force $1 million per day in fuel costs, with the software being managed by just one person. It also aligns with USAF’s Air Operations Centre (AOC) capabilities via a continuous delivery software development pipeline to a hybrid cloud-based platform alongside the legacy AOC 10.1 system.”
Original source: Innovation at the edge: the top air defence trends by domain