I have an extra piece in The Register this month. I was asked to frame the history of software product theory between the Cluetrain, Andrew Clay Shafer’s agile infrastructure talk, DevOps, and the “we’re a software company now” trope.
Try to go beyond hand waving and opinions and find out what really is happening. A good way to start is to ask people to picture what their scenario would look like if everything was perfect. This puts them into a positive frame and helps focus on great outcomes. Once you’re sure you’re working on an improvement opportunity that’s worth your time, try small time-bound experiments that you actually follow through on. Use what you learn to come up with the next step. I’ve found that the combination of being bold with the vision but taking small steps to get there is a good combination.
In the US, we love arm-chair strategizing government IT, in particular federal IT. Getting your arms around “the problem” is near impossible.
What do we think is wrong, exactly?
As citizens, our perceptions seem to be that government IT has poor user experience, none at all (there’s no app to do things, you have to go to an office to fill something out, etc.), and that it costs too much. More wonky takes are that there’s not enough data provided, nor insights generated by that data to drive better decision making.
When I’ve spoken with government IT people, their internal needs revolve around increasing (secure) communication, using more modern “white-collar” tools (from simply upgrading their copies of Office, to moving to G Suite/Office 365 suites, or just file sharing), and addressing the citizen perceptions (bringing down costs, making sure the software, whether custom made or “off the shelf,” have better customer experiences.
Is it so hard, really?
It’s also easy to think that government is a special snow-flake, but, really, they have mostly the same problems as any large organization. As highlighted below, the government contracting, procurement, and governance processes are more onerous in government IT, and the profile of “legacy” systems is perhaps higher, but, worse, more of a pull down into the muck.
From my conversations, one of the main barriers to change is systemic inertia, seemingly driven by avoidance of risk and overall lack of motivation to do anything. This lack of motivation is likely driven by the lack of competition: unlike in the private sector, there’s no other government to go to, so there’s no fear of loosing “business,” so what care to change or make things better?
Anyhow, here’s a notebook of federal government IT.
- “92 percent of Federal IT managers say it’s urgent for their agency to modernize legacy applications, citing the largest driving factors as security issues (42 percent), time required to manage and/or maintain systems (36 percent), and inflexibility and integration issues (31 percent)” from an Accenture sponsored 2015 survey of “150 Federal IT managers familiar with their agency’s applications portfolio”
- Theres a large pool of legacy IT, though not as large as you might think: ~60% of portfolio are from before 2010(https://www.gartner.com/document/3604417).
- That said, the same report says that ~25% of portfolios are pre-1999, with 5% from the 1980’s.
- On spending: “The government has been reporting that 75 to 80 percent of the federal IT budget is spent on running legacy (or existing) systems.”
- But, actually, that’s pretty normal: “That may sound alarming to those who aren’t familiar with the inner-workings of a large IT organization. However, the percentage is in-line with the industry average. Gartner says the average distribution of IT spending between run, grow and transform activities — across all industries — is 70 percent, 19 percent and 11 percent respectively. Those numbers have been consistent over the past decade.”
- However, the spending items above are from Compuware’s CEO, who’s clearly interested in continuing legacy spending, mostly on mainframes.
Source: “2017 CIO Agenda: A Government Perspective,” Rick Howard, Gartner, Feb. 2017.
- In the same survey, data & analytics skills are the leading talent gap, with security coming in second. Everything else is in the single digits.
- Why care about data? On simply providing it (and you, know, the harder job of producing it), the UN e-Government survey says “Making data available online for free also allows the public – and various civil society organizations –to reuse and remix them for any purpose. This can potentially lead to innovation and new or improved services, new understanding and ideas. It can also raise awareness of governments’ actions to realize all the SDGs, thus allowing people to keep track and contribute to those efforts.”
- And, on analytics: “Combining transparency of information with Big Data analytics has a growing potential. It can help track service delivery and lead to gains in efficiency. It can also provide governments with the necessary tools to focus on prevention rather than reaction, notably in the area of disaster risk management.”
- Reducing compliance and overall “bureaucracy” is always a problem. My benchmark case is an 18F project that reduced the paperwork time (ATO) down from 9-14 months to 3 days.
The workloads – what’s the IT do?
- And, while it’s for the Australian government, check out a good profile of the kinds of basic services, and, therefore, applications that agencies need, e.g.: booking citizenship process appointments, getting permits to open businesses, and facilitating the procurement process.
- If you think about many of the business services governments do, it’s workflow process: someone submits a request, multiple people have to check and co-relate the data submitted, and then someone has to approve the request. This is a core, ubiquitous thing handled by enterprise software and, in theory, shouldn’t be that big of a deal. But, you know, it usually is. SaaS offerings are a great fit for this, you’d hope.
The problems: the usual old process, expensive COTs, contractors, compliance
- If you accept that much of government IT is simple workflow management, much of the improving the quality and costs of government IT would likely come from shifting off custom, older IT to highly commoditized, cheap (and usually faster evolving, and more secure), SaaS-based services.
- Jennifer Pahlka: “When you consider that much of what ails government today is the use of custom development at high cost when a commodity product is readily and cheaply available, we must acknowledge that agile is one useful doctrine, not the doctrine. “
- So, if you do the old “IT – SaaS = what?” you suck out a lot of resources (money, attention, etc.) by moving from janky, expensive COTs systems (and all the infrastructure and operations support needed to run them). You can both cut these costs (fire people, shut down systems), and then reallocate resources (people, time, and money) to better customizing software. Then, this gets you back to “agile,” which I always read as “software development.”
- In my experience, government IT has the same opportunities as most companies, taking on a more “agile” approach to IT. This means doing smaller, faster to release batches, with smaller, more focused, “all in teams.” Again, the same thing as most large organizations.
- An older survey (sponsored by Red Hat): “Just 13% of respondents in a recent MeriTalk/Accenture survey of 152 US Federal IT managers believed they could ‘develop and deploy new systems as fast as the mission requires.’”
- Mikey Dickerson, 2014: “We’ll break that up by discouraging government contracts that are multibillion-dollar and take years to deliver. HealthCare.gov would have been difficult to roll out piecemeal, but if you, a contractor, have to deliver some smaller thing in four to six weeks while the system is being constructed, you’ll act differently.”
- Government contractors and procurement are a larger problem in government IT, though. The structure of how business is done with third parties, and the related procurement and compliance red-tape causes problems, and, as put by Andrew McMahon, it creates “a procurement process that has become more important than the outcome.”
- While there’s “too much” red-tape, in general we want a huge amount of transparency and oversight into government work. In the US, we don’t really trust the government to work efficiently. This become frustrating ironic and circular, then, if your position is that all of that oversight and compliance is a huge part of the inefficiency.
- As put by one government CIO: “Government agencies, therefore, place a business value on ‘optics’—how something appears to the observant public. In an oversight environment that is quick to assign blame, government is highly risk averse (i.e., it places high business value on things that mitigate risk)…. the compliance requirements are not an obstacle, but rather an expression of a deeper business need that the team must still address.”
- Tom Cochran: “While running technology for Obama’s WhiteHouse.gov, open-source solutions enabled our team to deliver projects on budget and up to 75% faster than alternative proprietary-software options. More than anything, open-source technology allows governments to utilize a large ecosystem of developers, which enhances innovation and collaboration while driving down the cost to taxpayers.”
- As with “agile,” it’s important to not put all your eggs-of-hope in one basket on the topic of open source. My theory is that for many large organizations, simply doing something new and different, upgrading – open or not – will improve your IT situation:
- While open source has different cost dynamic, I’d suggest that simply switching to new software to get the latest features and mindset that the software imbues gives you a boost. Open source, when picked well, will come with that community and an ongoing focus on updates: older software that has long been abandoned by the community and vendors will stall out and become stale, open or not.
- One example of success, from Pivotal-land, is the IRS’s modernization of reporting on diligent taxes. It moved from a costly, low customer service quality telephone based system to an online approach. As I overuse in most of my talks, they applied a leaner, more “agile” approach to designing the software and now “taxpayers have initiated over 400,000 sessions and made over $100M in payments after viewing their balance.”
If you’re really into this kind of thing, you should come to our free Pivotal workshop day in D.C., on June 7th. Mark Heckler and I will be going over how to apply “cloud-native” thinking, practices, and technologies to the custom written software portion of all this. Also, I’ll be speaking at a MeetUp later that day on the overall hopes and dreams of cloud-native, DevOps, and all that “agile” stuff.
In a report giving advice to mainframe folks looking to be more Agile, Gartner’s Dale Vecchio and Bill Swanton give some pretty good advice for anyone looking to change how they do software.
Here’s some highlights from the report, entitled “Agile Development and Mainframe Legacy Systems – Something’s Got to Give”
Chunking up changes:
- Application changes must be smaller.
- Automation across the life cycle is critical to being successful.
- A regular and positive relationship must exist between the owner of the application and the developers of the changes.
This kind of effort may seem insurmountable for a large legacy portfolio. However, an organization doesn’t have to attack the entire portfolio. Determine where the primary value can be achieved and focus there. Which areas of the portfolio are most impacted by business requests? Target the areas with the most value.
An example of possible change:
About 10 years ago, a large European bank rebuilt its core banking system on the mainframe using COBOL. It now does agile development for both mainframe COBOL and “channel” Java layers of the system. The bank does not consider that it has achieved DevOps for the mainframe, as it is only able to maintain a cadence of monthly releases. Even that release rate required a signi cant investment in testing and other automation. Fortunately, most new work happens exclusively in the Java layers, without needing to make changes to the COBOL core system. Therefore, the bank maintains a faster cadence for most releases, and only major changes that require core updates need to fall in line with the slower monthly cadence for the mainframe. The key to making agile work for the mainframe at the bank is embracing the agile practices that have the greatest impact on effective delivery within the monthly cadence, including test-driven development and smaller modules with fewer dependencies.
It seems impossible, but you should try:
Improving the state of a decades-old system is often seen as a fool’s errand. It provides no real business value and introduces great risk. Many mainframe organizations Gartner speaks to are not comfortable doing this much invasive change and believing that it can ensure functional equivalence when complete! Restructuring the existing portfolio, eliminating dead code and consolidating redundant code are further incremental steps that can be done over time. Each application team needs to improve the portfolio that it is responsible for in order to ensure speed and success in the future. Moving to a services-based or API structure may also enable changes to be done effectively and quickly over time. Some level of investment to evolve the portfolio to a more streamlined structure will greatly increase the ability to make changes quickly and reliably. Trying to get faster with good quality on a monolithic hairball of an application is a recipe for failure. These changes can occur in an evolutionary way. This approach, referred to in the past as proactive maintenance, is a price that must be paid early to make life easier in the future.
You gotta have testing:
Test cases are necessary to support automation of this critical step. While the tooling is very different, and even the approaches may be unique to the mainframe architecture, they are an important component of speed and reliability. This can be a tremendous hurdle to overcome on the road to agile development on the mainframe. This level of commitment can become a real roadblock to success.
Another example of an organization gradually changing:
When a large European bank faced wholesale change mandated by loss of support for an old platform, it chose to rewrite its core system in mainframe COBOL (although today it would be more likely to acquire an off-the-shelf core banking system). The bank followed a component-based approach that helped position it for success with agile today by exposing its core capabilities as services via standard APIs. This architecture did not deliver the level of isolation the bank could achieve with microservices today, as it built the system with a shared DBMS back-end, as was common practice at the time. That coupling with the database and related data model dependencies is the main technical obstacle to moving to continuous delivery, although the IT operations group also presents cultural obstacles, as it is satis ed with the current model for managing change.
A reminder: all we want is a rapid feedback cycle:
The goal is to reduce the cycle time between an idea and usable software. In order to do so, the changes need to be smaller, the process needs to be automated, and the steps for deployment to production must be repeatable and reliable.
The ALM technology doesn’t support mainframes, and mainframe ALM stuff doesn’t support agile. A rare case where fixing the tech can likely fix the problem:
The dilemma mainframe organizations may face is that traditional mainframe application development life cycle tools were not designed for small, fast and automated deployment. Agile development tools that do support this approach aren’t designed to support the artifacts of mainframe applications. Modern tools for the building, deploying, testing and releasing of applications for the mainframe won’t often t. Existing mainframe software version control and conguration management tools for a new agile approach to development will take some effort — if they will work at all.
Use APIs to decouple the way, norms, and road-map of mainframes from the rest of your systems:
wrapping existing mainframe functions and exposing them as services does provide an intermediate step between agile on the mainframe and migration to environments where agile is more readily understood.
Contrary to what you might be thinking, the report doesn’t actually advocate moving off the mainframe willy-nilly. From my perspective, it’s just trying to suggest using better processes and, as needed, updating your ALM and release management tools.
Read the rest of the report over behind Gartner’s paywall.
I always like his focus in speeding up the release cycle as a forcing function for putting continuous integration in place, both leading to improving how an organization’s software:
I try not to get too caught up in the names. As long as the changes are helping you improve your software development and delivery processes then who cares what they are called. To me it is more important to understand the inefficiencies you are trying to address and then identify the practice that will help the most. In a lot of respects DevOps is just the agile principle of releasing code on a more frequent basis that got left behind when agile scaled to the Enterprise. Releasing code in large organizations with tightly coupled architectures is hard. It requires coordinating different code, environment definitions, and deployment processes across lots of different teams. These are improvements that small agile teams in large organizations were not well equipped to address. Therefore, this basic agile principle of releasing code to the customer on a frequent basis got dropped in most Enterprise agile implementations. These agile teams tended to focus on problems they could solve like getting signoff by the product owner in a dedicated environment that was isolated from the complexity of the larger organization.
You can hide a lot of inefficiencies with dedicated environments and branches but once you move to everyone working on a common trunk and more frequent releases those problems will have to be address. When you are building and releasing the Enterprise systems at a low frequency your teams can brute force their way through similar problems every release. Increasing the frequency will require people to address inefficiencies that have existed in your organization for years.
On how organization size changes your managerial tactics:
If it is a small team environment, then DevOps is more about giving them the resources they need, removing barriers, and empowering the team because they can own the system end to end. If it is a large complex environment, it is more about designing and optimizing a large complex deployment pipeline. This is not they type of challenges that small empowered team can or will address. It takes a more structured approach with people looking across the entire deployment pipeline and optimizing the system.
The rest of the interview is good stuff. Also, I reviewed his book back in November; the book is excellent.
“We do discovery on a small chunk and then development, and then while that’s going on, we’re starting discovery on the next small chunk, and so on and so forth,” Smith said. “And then when the development is done, we loop back and we do user testing on that piece that’s done. But we don’t release it. That’s … one of the differences between agile and the way we did it. At the end of the phase we release everything.”
Also, some fun notes on consolidating legacy systems and resistance to going agile.
[A]fter pairing, I realized how much of my time I spend staring into space when I program alone.
Companies commonly make one of two mistakes when selecting a product owner. Often they tap a junior employee with limited experience and therefore a limited understanding of how the project fits into the larger mission. Product owners need enough seniority to inspire and motivate peers across multiple business units. By earning the respect of teams in customer experience, enterprise architecture, and risk and compliance, for example, the product owner can help ensure that projects move smoothly without costly bottlenecks. Other companies err in the opposite direction, selecting a senior executive who is too harried to devote adequate time and may not adapt well to the highly responsive, iterative nature of agile development.
So what should companies look for when appointing product owners? In our view, the key is to find people who think and behave like entrepreneurs.
Much of the advice here falls under the category of “if you do good things, good things happen”:
success comes from simply managing a sound process: conducting market research, understanding the customer’s needs, identifying where the product will create the most value, prioritizing the most important features, testing ideas, capturing customer feedback, and continuously refining their vision over time.
The tasks is setting up and environment, processes, even “culture” that encloses and rewards good behavior like this. And the protecting that structure from corporate barbarians. That’s a job – and the responsibility – of management. So, perhaps it’s good to get some management consulting advice on what good looks like.
Diego Lo Giudice, vice president and principal analyst with Forrester Research, is one of those who thinks that companies haven’t really adopted Agile as they should have done. “I would say that we’re going to see more Agile because we haven’t done it well enough yet,” he told The Reg. “Organisations say they’re doing it, but they’re struggling to scale it.”
One of the more shocking “turns out” to most of my talks is just this: just under 20 years later, the industry still doesn’t do that much agile.