Figuring out fixing federal government IT – Notebook

In the US, we love arm-chair strategizing government IT, in particular federal IT. Getting your arms around “the problem” is near impossible.

What do we think is wrong, exactly?

As citizens, our perceptions seem to be that government IT has poor user experience, none at all (there’s no app to do things, you have to go to an office to fill something out, etc.), and that it costs too much. More wonky takes are that there’s not enough data provided, nor insights generated by that data to drive better decision making.

When I’ve spoken with government IT people, their internal needs revolve around increasing (secure) communication, using more modern “white-collar” tools (from simply upgrading their copies of Office, to moving to G Suite/Office 365 suites, or just file sharing), and addressing the citizen perceptions (bringing down costs, making sure the software, whether custom made or “off the shelf,” have better customer experiences.

Is it so hard, really?

It’s also easy to think that government is a special snow-flake, but, really, they have mostly the same problems as any large organization. As highlighted below, the government contracting, procurement, and governance processes are more onerous in government IT, and the profile of “legacy” systems is perhaps higher, but, worse, more of a pull down into the muck.

From my conversations, one of the main barriers to change is systemic inertia, seemingly driven by avoidance of risk and overall lack of motivation to do anything. This lack of motivation is likely driven by the lack of competition: unlike in the private sector, there’s no other government to go to, so there’s no fear of loosing “business,” so what care to change or make things better?

Anyhow, here’s a notebook of federal government IT.

“Legacy”

  • “92 percent of Federal IT managers say it’s urgent for their agency to modernize legacy applications, citing the largest driving factors as security issues (42 percent), time required to manage and/or maintain systems (36 percent), and inflexibility and integration issues (31 percent)” from an Accenture sponsored 2015 survey of “150 Federal IT managers familiar with their agency’s applications portfolio”
  • Theres a large pool of legacy IT, though not as large as you might think: ~60% of portfolio are from before 2010(https://www.gartner.com/document/3604417).
  • That said, the same report says that ~25% of portfolios are pre-1999, with 5% from the 1980’s.
  • On spending: “The government has been reporting that 75 to 80 percent of the federal IT budget is spent on running legacy (or existing) systems.”
  • But, actually, that’s pretty normal: “That may sound alarming to those who aren’t familiar with the inner-workings of a large IT organization. However, the percentage is in-line with the industry average. Gartner says the average distribution of IT spending between run, grow and transform activities — across all industries — is 70 percent, 19 percent and 11 percent respectively. Those numbers have been consistent over the past decade.”
  • However, the spending items above are from Compuware’s CEO, who’s clearly interested in continuing legacy spending, mostly on mainframes.

Priorities

Source: “2017 CIO Agenda: A Government Perspective,” Rick Howard, Gartner, Feb. 2017.

Other notes:

  • In the same survey, data & analytics skills are the leading talent gap, with security coming in second. Everything else is in the single digits.
  • Why care about data? On simply providing it (and you, know, the harder job of producing it), the UN e-Government survey says “Making data available online for free also allows the public – and various civil society organizations –to reuse and remix them for any purpose. This can potentially lead to innovation and new or improved services, new understanding and ideas. It can also raise awareness of governments’ actions to realize all the SDGs, thus allowing people to keep track and contribute to those efforts.”
  • And, on analytics: “Combining transparency of information with Big Data analytics has a growing potential. It can help track service delivery and lead to gains in efficiency. It can also provide governments with the necessary tools to focus on prevention rather than reaction, notably in the area of disaster risk management.”
  • Reducing compliance and overall “bureaucracy” is always a problem. My benchmark case is an 18F project that reduced the paperwork time (ATO) down from 9-14 months to 3 days.

The workloads – what’s the IT do?

  • And, while it’s for the Australian government, check out a good profile of the kinds of basic services, and, therefore, applications that agencies need, e.g.: booking citizenship process appointments, getting permits to open businesses, and facilitating the procurement process.
  • If you think about many of the business services governments do, it’s workflow process: someone submits a request, multiple people have to check and co-relate the data submitted, and then someone has to approve the request. This is a core, ubiquitous thing handled by enterprise software and, in theory, shouldn’t be that big of a deal. But, you know, it usually is. SaaS offerings are a great fit for this, you’d hope.

The problems: the usual old process, expensive COTs, contractors, compliance

  • If you accept that much of government IT is simple workflow management, much of the improving the quality and costs of government IT would likely come from shifting off custom, older IT to highly commoditized, cheap (and usually faster evolving, and more secure), SaaS-based services.
  • Jennifer Pahlka: “When you consider that much of what ails government today is the use of custom development at high cost when a commodity product is readily and cheaply available, we must acknowledge that agile is one useful doctrine, not the doctrine. “
  • So, if you do the old “IT – SaaS = what?” you suck out a lot of resources (money, attention, etc.) by moving from janky, expensive COTs systems (and all the infrastructure and operations support needed to run them). You can both cut these costs (fire people, shut down systems), and then reallocate resources (people, time, and money) to better customizing software. Then, this gets you back to “agile,” which I always read as “software development.”
  • In my experience, government IT has the same opportunities as most companies, taking on a more “agile” approach to IT. This means doing smaller, faster to release batches, with smaller, more focused, “all in teams.” Again, the same thing as most large organizations.
  • An older survey (sponsored by Red Hat): “Just 13% of respondents in a recent MeriTalk/Accenture survey of 152 US Federal IT managers believed they could ‘develop and deploy new systems as fast as the mission requires.’”
  • Mikey Dickerson, 2014: “We’ll break that up by discouraging government contracts that are multibillion-dollar and take years to deliver. HealthCare.gov would have been difficult to roll out piecemeal, but if you, a contractor, have to deliver some smaller thing in four to six weeks while the system is being constructed, you’ll act differently.”
  • Government contractors and procurement are a larger problem in government IT, though. The structure of how business is done with third parties, and the related procurement and compliance red-tape causes problems, and, as put by Andrew McMahon, it creates “a procurement process that has become more important than the outcome.”
  • While there’s “too much” red-tape, in general we want a huge amount of transparency and oversight into government work. In the US, we don’t really trust the government to work efficiently. This become frustrating ironic and circular, then, if your position is that all of that oversight and compliance is a huge part of the inefficiency.
  • As put by one government CIO: “Government agencies, therefore, place a business value on ‘optics’—how something appears to the observant public. In an oversight environment that is quick to assign blame, government is highly risk averse (i.e., it places high business value on things that mitigate risk)…. the compliance requirements are not an obstacle, but rather an expression of a deeper business need that the team must still address.”

Success story

  • Tom Cochran: “While running technology for Obama’s WhiteHouse.gov, open-source solutions enabled our team to deliver projects on budget and up to 75% faster than alternative proprietary-software options. More than anything, open-source technology allows governments to utilize a large ecosystem of developers, which enhances innovation and collaboration while driving down the cost to taxpayers.”
  • As with “agile,” it’s important to not put all your eggs-of-hope in one basket on the topic of open source. My theory is that for many large organizations, simply doing something new and different, upgrading – open or not – will improve your IT situation:
  • While open source has different cost dynamic, I’d suggest that simply switching to new software to get the latest features and mindset that the software imbues gives you a boost. Open source, when picked well, will come with that community and an ongoing focus on updates: older software that has long been abandoned by the community and vendors will stall out and become stale, open or not.
  • One example of success, from Pivotal-land, is the IRS’s modernization of reporting on diligent taxes. It moved from a costly, low customer service quality telephone based system to an online approach. As I overuse in most of my talks, they applied a leaner, more “agile” approach to designing the software and now “taxpayers have initiated over 400,000 sessions and made over $100M in payments after viewing their balance.”

If you’re really into this kind of thing, you should come to our free Pivotal workshop day in D.C., on June 7th. Mark Heckler and I will be going over how to apply “cloud-native” thinking, practices, and technologies to the custom written software portion of all this. Also, I’ll be speaking at a MeetUp later that day on the overall hopes and dreams of cloud-native, DevOps, and all that “agile” stuff.

The fight for money between states and cities

The miniature culture wars fought between cities and states—such as North Carolina’s tussle with Charlotte over its anti-discrimination rules—are well known. The financial tensions between them are quieter but as important. “Money is usually the main problem,” says Larry Jones of the United States Conference of Mayors, and especially divisive in lean times.

Link

With no competition, government websites often have no incentive to be good

In contrast to agile, private-sector companies, the public sector does not face any pressure from competition. When it comes time to renew your license, there is only one place for you to do that: and, unfortunately for Americans, that’s the DMV. With no competitive forces, government agencies do not have to innovate or take bold risks when it comes to digital.

And, as ever, being smart about using updated tools and new methods yield huge productivity results:

While running technology for Obama’s WhiteHouse.gov, open-source solutions enabled our team to deliver projects on budget and up to 75% faster than alternative proprietary-software options. More than anything, open-source technology allows governments to utilize a large ecosystem of developers, which enhances innovation and collaboration while driving down the cost to taxpayers.

While open source has different cost dynamic, I’d suggest that simply switching to new software to get the latest features and mindset that the software imbues gives you a boost. Open source, when picked well, will come with that community and an ongoing focus on updates: older software that has long been abandoned by the community and vendors will stall out and become stale, open or not.

With most large organizations, and especially government, simply doing something will give you a huge boost in all your KPIs in the short term. Picking a thriving, vibrant stack is critical for long term success. Otherwise, five or ten years from now, whether using open or closed source, you’ll end up in the same spot, dead in the water and sucking.

Link

Australian government’s Cloud Foundry apps in production

Delivery teams are now able to build services faster and easier. In July 2016, DTA had 14 apps in production and 50 apps in development. In October 2016, the numbers increased to 47 apps in production and 225 apps in development.

Australian Government Cuts Release Time with Cloud Foundry, Iterates Faster – Cloud Foundry Live

We’re getting exactly the government IT we asked for

If there’s one complaint that I hear consistently in my studies of IT in large organizations, it’s that government IT, as traditionally practiced, is fucked. Compared to the private sector, the amount of paperwork, the role of contractors, and the seeming separation between doing a good job and working software drives all sorts of angst and failure.

Mark Schwartz’s book on figuring out “business value” in IT is turning out to be pretty amazing and refreshing, especially on the topic of government IT. He’s put together one of the better “these aren’t the Droids you’re looking for” responses to ROI for IT.

You know that answer: you just want to figure out the business case, ROI, or whatever numbers driven thing, and all the DevOps-heads are like “doo, doo, doo-doo – driving through a tunnel, can’t hear you!” and then they pelt you with Goldratt and Deming books, blended in with some O’Reilly books and The Phoenix Project. “Also, you’re argument is invalid, because reasons.”

A Zen-like calm comes over them, they close their eyes and breath in, and then start repeating a mantra like some cowl-bedecked character in a Lovecraft story: “survival is not mandatory. Survival is not mandatory. Survival is not mandatory!”

Real helpful, that lot. I kid, I jest. The point of their maniacally confusing non-answers is, accurately, that your base assumptions about most everything are wrong, so before we can even approach something as precise as ROI, we need to really re-think what you’re doing. (And also, you do a lot of dumb shit, so let’s work on that.)

But you know, no one wants to hear they’re broken in the first therapy session. So you have to throw out some beguiling, mind-altering, Lemarchand’s boxes to change the state of things and make sure they come to the next appointment.

Works as Designed

Anyhow, back to Schwartz’s book. I’ll hopefully write a longer book review over at The New Stack when I’m done with it, but this one passage is an excellent representation of what motivates the book pelters and also a good unmasking of why things are the way we they are…because we asked for them to be so:

The US government is based on a system of “checks and balances”—in other words, a system of distrust. The great freedom enjoyed by the press, especially in reporting on the actions of the government, is another indication of the public’s lack of trust in the government. As a result, you find that the government places a high value on transparency. While companies can keep secrets, government is accountable to the public and must disclose its actions and decisions. There is a business need for continued demonstrations of trustworthiness, or we might as well say a business value assigned to demonstrating trustworthiness. You find that the government is always in the public eye—the press is always reporting on government actions, and the public is quick to outrage. Government agencies, therefore, place a business value on “optics”—how something appears to the observant public. In an oversight environment that is quick to assign blame, government is highly risk averse (i.e., it places high business value on things that mitigate risk).

And then summarized as:

…the compliance requirements are not an obstacle, but rather an expression of a deeper business need that the team must still address.

Which is to say: you wanted this, and so I am giving it to you.

The Agile Bureaucracy

The word “bureaucracy” is something like the word “legacy.” You only describe something as legacy software when you don’t like the software. Otherwise, you just call it your software. Similarly, as Schwartz outlines, agile (and all software processes) are insanely bureaucratic, full of rules, norms, and other “governance.” We just happen to like all those rules, so we don’t think of them as bureaucracy. As he writes:

While disavowing rules, the Agile community is actually full of them. This is understandable, because rules are a way of bringing what is considered best practices into everyday processes. What would happen if we made exceptions to our rules—for instance, if we entertained the request: “John wants to head out for a beer now, instead of fixing the problem that he just introduced into the build?” If we applied the rules capriciously or based on our feelings, they would lose some of their effectiveness, right? That is precisely what we mean by sine ira et studio in bureaucracy. Mike Cohn, for example, tells us that “improving technical practices is not optional.”15 The phrase not optional sounds like another way of saying that the rule is to be applied “without anger or bias.” Mary Poppendieck, coauthor of the canonical works on Lean software development, uses curiously similar language in her introduction to Greg Smith and Ahmed Sidky’s book on adopting Agile practices: “The technical practices that Agile brings to the table—short iterations, test-first development, continuous integration—are not optional.” I’ve already mentioned Schwaber and Sutherland’s dictum that “the Development Team isn’t allowed to act on what anyone else [other than the product owner] says.”17 Please don’t hate me for this, Mike, Mary, Ken, and Jeff, but that is the voice of the command-and-control bureaucrat. “Not optional,” “not allowed,” – I don’t know about you, but these phrases make me think of No Parking and Curb Your Dog signs.

These are the kind of thought-trains that only ever evoke “well, of course my intention wasn’t the awful!” from the other side. It’s like with the ITIL and the NRA, gun-nut people: their goal wasn’t to put in place a thought-technology that harmed people, far from it.

Gentled nestled in his wry tone and style (which you can image I love), you can feel some hidden hair pulling about the unintended consequences of Agile confidence and decrees. I mean, the dude is the CIO of a massive government agency, so he be throwing process optimism against brick walls daily and too late into the night.

Learning bureaucracies

The cure, as ever, is to not only to be smart and introspective, but to make evolution and change part of your bureaucracy:

Rules become set in stone and can’t change with circumstances. Rigidity discourages innovation. Rules themselves come to seem arbitrary and capricious. Their original purpose gets lost and the rules become goals rather than instruments. Bureaucracies can become demoralizing for their employees.

So, you know, make sure you allow for change. It’s probably good to have some rules and governance around that too.

Cull your rules and regulations, or be frozen by them

screenshot-2017-02-25-09-21-23

Instead, the post emphasized new ways NASA is thinking about acceptable risk for these launches. During the space shuttle era, launch sites were governed by some 2,200 safety requirements. NASA worked to “strip down the expensive rules and came up with just 500 core safety requirements for launches.”

But, the post notes, ”at commercial launch sites, there are only 55 safety requirements.” This “underscores perhaps the single biggest difference between NASA and its commercial partners: risk tolerance. NASA likes to have next to none, and pays for it, while the companies tend to embrace calculated risk as a way to iterate and learn while still developing their products.”

This a major point in companies fixing up their software process: you probably have all sorts of rules and regulations that no longer apply. Just like with managing a legacy application portfolio, you have to aggressively manage your compliance portfolio to eliminate rules that no longer apply.

Matt Curry covers this point with a funny boat-metaphor at the start of his brand talk from last year. You also hear some cases of federal agencies forcing this culling by simply not following the rules and seeing if anyone complained: the compliance equivilent of just turning off old, dusty servers to see who’s using it.

Link

TrumpTech: $450bn in annual fed spend in limbo

There is a lot of uncertainty in the air,” said one consultant close to the Office of Management and Budget’s IT efficiency initiatives who asked not to be identified. “The whole IT industry and federal IT operations are in a wait-and-see holding pattern,” he said, anticipating official word on key federal IT initiatives and leadership positions.

In my amateur analysis of Trump’s effect on IT spend, it seems like there’s three options:

  1. More of the same with big contractors and vendors, just wrapped up in myths of change.
  2. Complete shut down of everything with respect to growth; they just stop spending and let government IT age.
  3. Start working with new government contractors and doing things differently; the “Space X” option.

Who knows what’ll happen?

Link

TrumpTech: If the rocket scientists can do it cheaper, surely IT can too

Boeing and Lockheed were already worried about their costs long before the election. If I were the United Space Alliance, I would be even more terrified of the danger of losing government business now. And those of us in federal IT need to realize that our time may be around the corner.

As we discussed in the 2017 predictions show last month, the Trump adminstration is clearly not reliable in it’s agenda or principals for any reliable, let alone logical, predictions. Cutting spending in favor of “non-traditional” options, though, seems like something they’d goof into.

Link

The internet of food nutrition labels

“So anyone who is producing food that ends up in our grocery stores, we’re working with them to get the data from their labels and the packaging information to come right into the database for us,” Pamela Stark Reed, deputy administrator for Nutrition, Food Safety and Quality, said on Information Management month.

The database has actually existed for over a century, Reed said. But before starting the initiative, it only had about 8,000 entries. Since opening it up to manufacturer submission, ARS has received 80,000 new items, a 1000 percent increase.
And on future plans:

The goal for the database is to eventually expand to 1,000,000 items. Reed said ARS anticipates getting store brand and international food items into the database soon. Some items from chain restaurants may follow.

Because of this, the agency is looking into cloud services to increase its storage capacity.

Just imagine, globally, how many data sets like that there are. Throw in the workflow to injest and cleanup the data, and change it, plus APIs to access it, and you have an almost endless amount of projects for software eating.

http://federalnewsradio.com/information-management-month/2016/11/usda-turns-nutritional-data-open-data/

Saving $20m and going agile in the process

From an interesting sounding panel on government IT:

“We do discovery on a small chunk and then development, and then while that’s going on, we’re starting discovery on the next small chunk, and so on and so forth,” Smith said. “And then when the development is done, we loop back and we do user testing on that piece that’s done. But we don’t release it. That’s … one of the differences between agile and the way we did it. At the end of the phase we release everything.”

Also, some fun notes on consolidating legacy systems and resistance to going agile.