Notes on the 2019 DevOps Report

Some quick notes and callouts from this year’s 2019 DevOps Report:

  • Four key metrics: lead time, deployment frequency, mean time to restore (MTTR) and change fail percentage.
    • Med, High, and Elite all have a change fail rate of 0-15%. So, expect 15% change fail as benchmark worst case to shoot for…?
  • Demographics: 30% are devs, 26% “DevOps or SRE” – [so, lots of ICs self-evaluating]. 16% “managers,” and then it goes down from there…
  • Top industries are Technology at 38% and FinServe at 12%. Retail is 9%.
  • Mostly North American (50%)and Europe (29%)
  • Org. size: 100-499 (21%), 500-1,999 (15%), and 10,000+ (26%)
  • “A key goal in digital transformation is optimizing software delivery performance: leveraging technology to deliver value to customers and stakeholders.”
  • [I’m not sure if age of company, and, thus, an indication of governance and tech debt, is tracked. With 38% being tech companies, it’d be good know how young they are. But, most FinServ companies are large and old (unless it was mostly FinServ startups!).
  • Very prescriptive this year, a maturity model to put a strategy in place, etc.
  • A lot on paying down tech debt:
    • Bounded contexts, APIs, SOA and microservices. Using and testing out of team services without having to work with that team (sort of like mocking for runtime).
    • Also: “Teams that manage code maintainability well have systems and tools that make it easy for developers to change code maintained by other teams, find examples in the codebase, reuse other people’s code, as well as add, upgrade, and migrate to new versions of dependencies without breaking their code”
  • Very little prod chaos monkey stuff: less than 10% across the board.
  • CABs still bad: those that have them are 2.6x more likely to be low performers.
    • Instead, do peer reviews and automate governance: “peer review-based approval during the development process. In addition to peer review, automation can be leveraged to detect, prevent, and correct bad changes much earlier in the delivery lifecycle. Techniques such as continuous testing, continuous integration, and comprehensive monitoring and observability provide early and automated detection, visibility, and fast feedback. In this way, errors can be corrected sooner than would be possible if waiting for a formal review.”
    • CABs should instead focus on process and practices change: ” the CAB should focus instead on helping teams with process- improvement work to increase the performance of software delivery. This can take the form of helping teams implement the capabilities that drive performance by providing guidance and resources. CABs can also weigh in on important business decisions that require a trade-off and sign-off at higher levels of the business, such as the decision between time-to- market and business risk.”
    • [I’m pretty sure that was the original point, esp. when you look at RUP and ITIL stuff: setting the process to be used. Tooling to automate governance wasn’t really available. Policing it those prescriptive processes took over as it always does. And I’m not sure there are industry standard frameworks to use there yet either. There must be lots of hand-crafting.]
    • “Survey respondents with a clear change process were 1.8 times more likely to be in elite performers.” – [as ever, garbage in, garbage out.]
    • The people who work on governance are not the ones who can actually do the coding to automate it: “only our technical practitioners have the power to build and automate the change management solutions we design, making them fast, reliable, repeatable, and auditable…. Leaders at every level should move away from a formal approval process where external boards act as gatekeepers approving changes, and instead move to a governance and capability development role. After all, only managers have the power to influence and change certain levels of organizational policy. We have seen exponential improvements in performance— throughput, stability, and availability—in just months as a result of technical practitioners and organizational leaders working together.”
  • This is a different measure of “productivity”: “Productivity is the ability to get complex, time-consuming tasks completed with minimal distractions and interruptions.”
    • It doesn’t track amount of work done, but the environment people are working in…?
  • Tools use is all across the board: DIY stuff, COTs, open source, etc. [This sort of excludes the IaaS and other runtime layers, focusing on just CI/CD and test automation]
  • “Multi-tasking” across roles and projects might be OK: “we cannot conclude that how well teams develop and deliver software affects the number of roles and projects that respondents juggle.”
  • Being able to find things and ask questions [and, presumably, getting answers!], having search, is important.
  • From my read (slide 74), the methods of transforming orgs are all across the board with Big Bang and Training Center as the only low ranked ones. Communities of practice are high, part of the Spotify model.
  • Pg. 75 tries to derive some advice nonetheless: mostly that separate education and training groups don’t work well/widely, that grassroots is used a lot, and that communities of practice are good, as well as PoCs that get cloned.
  • [This is an instance where the high level of individual contributors in the answers might have an effect. They see the positive change in their own team, but don’t have the big picture view to see if the practices scale up to 1,000’s of people. On the other hand, they might follow the “my congressperson is perfect, all the other ones are corrupt and terrible” pattern. Also, those 5,000+ people orgs struggle.]
  • [We still don’t know how to change an engine in flight.]

5 Definitions of DevOps, or, ¯\_(ツ)_/¯

DevOpsDays Amsterdam - Thursday June 25th

I’ve tracked at least three different definitions of DevOps since the days of “agile infrastructure”:

  1. Using Puppet and Chef (and then Ansible and Chef) to replace Opsware and BladeLogic.
  2. Full stack engineers to setup EC2, load-balancers, and other Morlock shit.
  3. Full stack engineers are bad, but sort of the same thing. Also, you can’t have a DevOps “group” or title. But, you know, someone should do all that automation.
  4. Putting all the people on one team, having them focus on a product, and establishing a culture of caring and learning.
  5. SRE is not DevOps.

So…actually five. Maybe some of them just being footnotes on the evolving concept. (And, if you, dear reader, feel these are wrong, then let’s compromise and make the list six.)

All of them evolved around bringing down The Wall of Confusion, allowing “developers” to deploy their software to production more frequently, weekly, if not daily. And, of course, making sure production stays up. (You’re supposed to call that “resiliency” and instead of SLAs use SLOs and some other newly named metrics that answer the question “IS MY SHIT WORKING?” Whatever you do, just don’t say “uptime,” or you’re in for it and will be relegated to running the AS/400’s.)

I used to snide that the developers seemed to have been yanked out of DevOps, sometime around 2014 and 2015. All the talks I saw were, basically, operations talks. I haven’t really checked in on DevOps conference talks recently, but at the time, I don’t think there was much application development stuff. (I’m not sure if there ever was?)

None of this means that DevOps is not a thing. Not at all. It just means that the enterprise finds its own use for things. It also means there’s still weekly write-ups of what DevOps is – you know, those ones that are always lists of ideas, things you’re getting wrong, and how to start.

Autonomous product teams

This kind of thing is happening all the time

Nowadays, I try to stick to that forth one: you want to setup autonomous teams that have all the skills and responsibility/authority/tools needed to “own” the software being specified, designed, developed, and run. This means you have to, basically, remove-by-automating all the operations stuff it takes to stand-up environments, deploy things, and do all that “day 2” stuff.

(HEY! HEY! WANT TO BUY SOME ENTERPRISE SOFTWARE?!)

Now, I think this product-centric notion of DevOps is, well, kind of an over-extension of the term “DevOps.” But since SRE has sucked out the “ops” part (but, remember, dear reader, don’t commit the embarrassing act of saying SRE is DevOps – no, no, you’d never do that, right? SO SHAMEFUL! (SRE is totally different – no overlap or similar goals shared between them at all. I mean, they have separate groups, silos! COME ON!)), slicing “DevOps” back to just “Dev,” but with a product-not-project focus isn’t too shabby.

Anyhow. I came across a good overview of this product notion of DevOps, all the way back from 2016, while re-reading Schwartz’s evergreen excellent The Art of Business Value:

Agile approaches attempt to bring together developers and the business in an atmosphere of mutual respect and joint contribution. Until now, however, the focus has been on users of the software, product visionaries, and developers. Recent developments in the Agile world—notably DevOps—have broadened this idea of respect and inclusion to encompass Operations and Security. The DevOps model, in other words, looks to break down the silos that have resulted from technical specialization over the last few decades. But the DevOps spirit goes further, looking to eliminate the conflicting incentives of organizational silos and the inhumane behaviors that can result from those conflicting incentives.

 

Perhaps we can take this idea even further still. There is no reason why the DevOps team’s responsibility needs to stop at the border of what used to be considered IT. The team is part of a broader enterprise, whose collective knowledge, skills, and judgment need to be part of the value creation process.

Look a’ that guy! Business Value just effortlessly jets out of his pores like a peripatetic thought-monarch!

This is from an executives perspective, but it drives home the point we’re always trying to get to with software: doing whatever it takes to figure out, create, and give users features that are actually useful to them. Somewhere beyond that, if you’re lucky, it’ll help out “the business.” Also, it should implement The Unspoken User Story: user would like software to actually work.

We’re getting exactly the government IT we asked for

If there’s one complaint that I hear consistently in my studies of IT in large organizations, it’s that government IT, as traditionally practiced, is fucked. Compared to the private sector, the amount of paperwork, the role of contractors, and the seeming separation between doing a good job and working software drives all sorts of angst and failure.

Mark Schwartz’s book on figuring out “business value” in IT is turning out to be pretty amazing and refreshing, especially on the topic of government IT. He’s put together one of the better “these aren’t the Droids you’re looking for” responses to ROI for IT.

You know that answer: you just want to figure out the business case, ROI, or whatever numbers driven thing, and all the DevOps-heads are like “doo, doo, doo-doo – driving through a tunnel, can’t hear you!” and then they pelt you with Goldratt and Deming books, blended in with some O’Reilly books and The Phoenix Project. “Also, you’re argument is invalid, because reasons.”

A Zen-like calm comes over them, they close their eyes and breath in, and then start repeating a mantra like some cowl-bedecked character in a Lovecraft story: “survival is not mandatory. Survival is not mandatory. Survival is not mandatory!”

Real helpful, that lot. I kid, I jest. The point of their maniacally confusing non-answers is, accurately, that your base assumptions about most everything are wrong, so before we can even approach something as precise as ROI, we need to really re-think what you’re doing. (And also, you do a lot of dumb shit, so let’s work on that.)

But you know, no one wants to hear they’re broken in the first therapy session. So you have to throw out some beguiling, mind-altering, Lemarchand’s boxes to change the state of things and make sure they come to the next appointment.

Works as Designed

Anyhow, back to Schwartz’s book. I’ll hopefully write a longer book review over at The New Stack when I’m done with it, but this one passage is an excellent representation of what motivates the book pelters and also a good unmasking of why things are the way we they are…because we asked for them to be so:

The US government is based on a system of “checks and balances”—in other words, a system of distrust. The great freedom enjoyed by the press, especially in reporting on the actions of the government, is another indication of the public’s lack of trust in the government. As a result, you find that the government places a high value on transparency. While companies can keep secrets, government is accountable to the public and must disclose its actions and decisions. There is a business need for continued demonstrations of trustworthiness, or we might as well say a business value assigned to demonstrating trustworthiness. You find that the government is always in the public eye—the press is always reporting on government actions, and the public is quick to outrage. Government agencies, therefore, place a business value on “optics”—how something appears to the observant public. In an oversight environment that is quick to assign blame, government is highly risk averse (i.e., it places high business value on things that mitigate risk).

And then summarized as:

…the compliance requirements are not an obstacle, but rather an expression of a deeper business need that the team must still address.

Which is to say: you wanted this, and so I am giving it to you.

The Agile Bureaucracy

The word “bureaucracy” is something like the word “legacy.” You only describe something as legacy software when you don’t like the software. Otherwise, you just call it your software. Similarly, as Schwartz outlines, agile (and all software processes) are insanely bureaucratic, full of rules, norms, and other “governance.” We just happen to like all those rules, so we don’t think of them as bureaucracy. As he writes:

While disavowing rules, the Agile community is actually full of them. This is understandable, because rules are a way of bringing what is considered best practices into everyday processes. What would happen if we made exceptions to our rules—for instance, if we entertained the request: “John wants to head out for a beer now, instead of fixing the problem that he just introduced into the build?” If we applied the rules capriciously or based on our feelings, they would lose some of their effectiveness, right? That is precisely what we mean by sine ira et studio in bureaucracy. Mike Cohn, for example, tells us that “improving technical practices is not optional.”15 The phrase not optional sounds like another way of saying that the rule is to be applied “without anger or bias.” Mary Poppendieck, coauthor of the canonical works on Lean software development, uses curiously similar language in her introduction to Greg Smith and Ahmed Sidky’s book on adopting Agile practices: “The technical practices that Agile brings to the table—short iterations, test-first development, continuous integration—are not optional.” I’ve already mentioned Schwaber and Sutherland’s dictum that “the Development Team isn’t allowed to act on what anyone else [other than the product owner] says.”17 Please don’t hate me for this, Mike, Mary, Ken, and Jeff, but that is the voice of the command-and-control bureaucrat. “Not optional,” “not allowed,” – I don’t know about you, but these phrases make me think of No Parking and Curb Your Dog signs.

These are the kind of thought-trains that only ever evoke “well, of course my intention wasn’t the awful!” from the other side. It’s like with the ITIL and the NRA, gun-nut people: their goal wasn’t to put in place a thought-technology that harmed people, far from it.

Gentled nestled in his wry tone and style (which you can image I love), you can feel some hidden hair pulling about the unintended consequences of Agile confidence and decrees. I mean, the dude is the CIO of a massive government agency, so he be throwing process optimism against brick walls daily and too late into the night.

Learning bureaucracies

The cure, as ever, is to not only to be smart and introspective, but to make evolution and change part of your bureaucracy:

Rules become set in stone and can’t change with circumstances. Rigidity discourages innovation. Rules themselves come to seem arbitrary and capricious. Their original purpose gets lost and the rules become goals rather than instruments. Bureaucracies can become demoralizing for their employees.

So, you know, make sure you allow for change. It’s probably good to have some rules and governance around that too.

Think your organization is too caustic for “agile”? That actually makes it perfect

Many organizations object to the idea of doing anything remotely similar to “agile” based on the lack of communications between different groups and an overall caustic corporate culture. While more cynical than most agile adherents would like, the functional practices in agile may actually be a perfect fit for such corrupted cultures. As Grady Beaubouef ingeniously explains it:

The customer should perform a realistic assessment of the relationship between their Business and the internal IT organization. If the relationship does not foster effective collaboration or there is a known alignment challenge, then this presents a significant project risk. An iterative approach should be considered in this situation – especially when it comes to software developments for packaged software enhancements and customizations.

I finished up Beaubouef’s book on the plan today. It’s somewhat tedious to get through – the topic of planning and project managing big COTS installs isn’t exactly thrilling – it’s chock full of useful thinking.

Moving beyond the endless debate on bi-modal IT

I get all ants-in-pants about this whole bi-modal discussion because I feel like it’s a lot of energy spent talking about the wrong things.

This came up recently when I was asked about “MVP”, in a way that basically was saying “our stuff is dangerous [oil drilling], so ‘minimal’ sounds like it’d be less safe.” I tried to focus them on the “V” and figure out what “viable” was for their situation. The goal was to re-enforce that the point of all this mode 2/small batch/DevOps/PDCA/cloud native/OODA nonsense is to keep iterating to get to the right code.

Part of the continual consternation around bi-modal IT – sad/awesome mode – is misalignment around that “viability” and scoping on “enterprise” projects. This is just one seam-line around the splits of the discussion being unhelpful

Bi-strawperson

The awesome mode people are like:

You should divide the work into small chunks that you release to production as soon as possible – DevOps, Agile, MVP, CI/CD – POW! You have no idea what or how you should implement these features so you need to iteratively do it cf. projectcartoon.com

And the sad mode folks are like:

Yes, but we have to implement all this stuff all at once and can’t do it in small slices. Plus, mainframes and ITIL.

Despite often coming off as a sad mode apologist, I don’t even know what the sad mode people are thinking. There’s this process hugger syndrome that, well on both sides really, creates strawpeople. The goal of both methods is putting out software that makes users more productive, including having it actually work, and not overpaying for the whole thing.

The Enemy is finding any activity that doesn’t support those goals and eliminated it as much as possible. In this, there was some good scrabbling from the happy mode people laughing at ITSM think early on, but at this point, the sad people have gotten the message, have been reminded of their original goal, and are now trying to adapt. In fact, I think there’s a lot the “sad mode” people could bring to the table.

To play some lexical hopscotch, I don’t think there is a “mode 1.” I think there’s just people doing a less than awesome job and hiding behind a process-curtain. Sure, it may to be their choice and, thus, not their fault. “Shitty jobs are being done,” if you prefer the blamelesss-veil of passive voice.

Fix your shit

When I hear objections to fixing this situation, I try to b nice and helpful. After all, I’m usually there as part of an elaborate process to get money from these folks in exchange for helping them. When they go all Eeyore on me, I have to reframe the room’s thinking a little bit without getting too tough love-y.

“When I put these lithium batteries in this gas car, it doesn’t seem to work. So electric cars are stupid, right?

You want to walk people to asking “how do we plan out the transition from The Old Way That Worked At Some Point to The New Way That Sucks Less?” They might object with a sort of “we don’t need to change” or the even more snaggly “change is too hard” counter-point.

I’m not sure there are systems that can just be frozen in place and resist the need to change. One day, in the future, any system (even the IRS’!) will likely need to change and if you don’t already have it setup to change easily (awesome mode), you’re going to be in a world of hurt.

The fact that we discuss how hard it is to apply awesome mode to legacy IT is evidence that that moment will come sooner than you think.

(Insert, you know, “where’s my mobile app, Nowakowski?” anecdote of flat-footedness here.)

ITIL end in tears(tm)

The royal books of process, ITIL, are another frequent strawperson that frothy mouthed agents of change like to light up. Few things are more frustrating than a library of books that cost £100 each. There’s a whole lot in there, and the argument that the vendors screw it all up is certainly appetizing. Something like ITIL, though, even poorly implemented falls under the “at least it’s an ethos” category.

Climbing the Value Chain

I’m no IT Skeptic or Charles T. Betz, but I did work at BMC once: as with “bi-modal,” and I really don’t want to go re-read my ITIL books (I only have the v2 version, can someone spare a few £100’s to read v3/4?), but I’m pretty sure you could “do DevOps” in a ITIL context. You’d just have to take out the time consuming implementation of it (service desks, silo’d orgs, etc.).

Most of ITIL could probably be done with the metaphoric (or literal!) post-it notes, retrospectives, and automated audit-log stuff that you’d see in DevOps. For certain, it might be a bunch of process gold-plating, but I’m pretty sure there’s no unmovable peas under all those layers of books that would upset any slumbering DevOps princes and princesses too bad.

Indeed, my recollection of ITIL is that it merely specifies that people should talk with each other and avoid doing dumb shit, while trying to improve and make sure they know the purpose/goals of and “service” that’s deployed. They just made a lot of flow charts and check lists to go with it. (And, yeah: vendors! #AmIrightohwaitglasshouse.)

Instead of increasing the volume, help spray away the shit

That gets us back to the people. The meatware is what’s rotting. Most people know they’re sad, and in their objections to happiness, you can find the handholds to start helping:

Yes, awesome mode people, that sounds wonderful, just wonderful. But, I have 5,000 applications here at REALLYSADMODECOGLOBAL, Inc. – I have resources to fix 50 of them this year. YOUR MOVE, CREEP!

Which is to say, awesome mode is awesome: now how do we get started in applying it at large orginizations that are several fathoms under the seas of sad?

The answer can’t be “all the applications,” because then we’ll just end up with 5,000 different awesome modes (OK, maybe more like 503?) – like, do we all use Jenkins, or CircleCI, or Travis? PCF, Docker, BlueMix, OpenShift, AWS, Heroku, that thing Bob in IT wrote in his spare time, etc.

Thus far, I haven’t seen a lot of commentary on planning out and staging the application of mode 2. Gartner, of course, has advice here. But it’d be great to see more from the awesome mode folks. There’s got to be something more helpful than just “AWESOME ALL THE THINGS!”

Thanks to Bridget for helping draw all this blood out while I was talking with her about the bi-modal pice she contributed to.

Doing ROI for agile, DevOps, et. al. is difficult

I wrote up a rant-y style summary of thinking about ROI when it comes to planning out Agile, DevOps, cloud native, and otherwise “new ways of doing things in IT” schemes.

A little excerpt:

Doing “agile,” however, isn’t like dropping in a new, faster and cheaper component into your engine. Many people I encounter in conference rooms think about software development like those scenes from 80s submarine movies. Inevitably, in a submarine movie, something breaks and the officer team has to swipe all the tea cups off the officer’s mess table and unfurl a giant schematic. Looking over the dark blue curls of a thick Eastern European cigarette, the head engineer gestures with his hand, then slams a grimy finger onto the schematics and says “vee must replace the manifold reducer in the reactor.”

Solving your digital transformation problems is not like swapping “agile” into the reactor. It’s not a component based improvement like virtualization was. Instead, you’re looking at process change (or “culture,” as the DevOps people like to say), a “thought technology.” I think at best what you can do is try to calculate the before and after savings that the new process will bring. Usually this is trackable in things like time spent, tickets opened, number of staff needed, etc. You’re focusing on removing costs, not making money. As my friend Ed put it when we discussed how to talk about DevOps with the finance department:

In other words, if I’m going to build a continuous integration platform, I would imagine you could build out a good scaffolding for that and call it three or four months. In the process of doing that, I should be requiring less help desk tickets get created so my overtime for my support staff should be going down. If I’m virtualizing the servers, I’ll be using less server space and hard drive space, and therefore that should compress down. I should be able to point to cost being stripped out on the back end and say this is maybe not 100% directly related to this process, but it’s at least correlated with it.
In this instance, it’s difficult to prove that you’ll achieve good ROI ahead of time, but you can at least try to predict changes informed by the savings other people have had. And, once again, you’re left to making a leap of faith that qualitative anecdotes from other people will apply to you.

Read the rest over in my Medium feed.

Maybe “bi-modal” is about temperament, not just enabling “sad IT”

More for the “I don’t think bi-modal is that far from what all us methodology hipster talk about” file here:

Q. Have you seen organizations that have officially reorganized around a Bimodal approach? What kind of job titles/job descriptions are you seeing?

It’s much less an organizational structure change, and much more about culture, process and method. But yes, it is not unusual in the short to mid-term to differentiate teams. Mode 2 usually starts life as being quite small, and often needs “protection”, to prevent the organizational anti-bodies killing it off. It is better to go narrow and deep and focus the capability on a particular part of the enterprise, or on particular domains like mobile for example. Small IT organizations (<50) don’t have the option to create any kind of split of course).

I'd still wager that most people who complain about bi-modal haven't read up on it too much (it's hard to do, and there's bat-shit expensive paywalls around it). Also, I think Gartner has been filling out the nuance of the original, clear idea: softening it and making it more practical, which makes it more ambiguous.

My theory here – based on conversations with people transforming large organizations – is that the focus here should be on the temperament of the people, not the process. For any given project, the proven process of small batches applies, but you’ll often want “cow-pokes” (risk takers who hate long term commitment) for new projects and “city-folk (enjoy stability) for post-1.0 projects.

There’s a chance for judgement about who the cool kids are vs. the old folks there, but I’d suggest finding that distinction is all about the perceiver. In my career, I’ve found people who are perfectly happy being in one of those two boxes. Problems arise (I think Apple’s negligence for apps their no longer interested in is a good bad result to ponder here), but I feel like it fits the staffing and prioritization needs of most large organizations.

But, I don’t call it “theory” for nuthin’.

What is “waterfall”?

“Waterfall” comes up a lot in my line of work: talking with orginizations about how to do software better. People like myself take the definition of “waterfall” for granted and usually don’t spend too much time talking about it, let alone when it might be a good idea of how to do it well.

Someone asked me the simple question “what is waterfall?” as a follow-up to one of my talks recently. Here’s what I typed back:

Continue reading “What is “waterfall”?”

What I’m looking forward to at SpringOne Platform

The biggest cloud native conference is coming up at the first week of August, SpringOne Platform. To plan out my time I took at look at the sessions. Here’s what I’m looking forward to and what I think you, dear readers, will find interesting as well. Doing a list like this, of course, ends up excluding some awesome sessions, so be sure to check out the talk list yourself as well.

Also, if you’re interested and haven’t registered yet, be sure to use the code pivotal-cote-300 to get $300 off.

Dealing with legacy

Almost every conversation I have with large organizations involves a discussion about dealing with legacy software. While you may not be running JFK era IT, you probably have to deal with legacy. Here’s some sessions on that topic:

Cloud Native Coding

Moving to The New, New Thing requires different ways of architecting and coding your software. Here’s some sessions that go over those new ways:

Case Studies

While cooked up demos of Pet Stores and Breweries are education, I’m most interested in hearing tales of what’s actually happened out in the world. Here are some of the case studies that look interesting:

The Usual Chuckle-heads

And, to highlight talks from my team:

(And, remember: if you want to come, you can get $300 if use the code pivotal-cote-300 when you register.)

Agile Methodology In-depth Review, Government Edition

I’m giving a 90 minute overview of agile for an agency later this month. Here’s my slides, so far. As ever, “government” is just an extra layer of sprinkles on-top of advice for all large organizations.

There’s some extensive (for me) talking points in the slide notes if you’re into that kind of thing.

Getting collaboration right in Agile & DevOps – Press Pass

I don’t do press passes as much as I did when I was an analyst, but here’s one from a recent email interview for a ProjectsAtWork story:

Q: What’s your favorite tip to improve collaboration when an organization moves to agile and DevOps?

A: I think the core DevOps thing with collaboration is getting people to trust each other. Most corporate cultures are not built on people trusting each other and feeling comfortable: they’re based in competitive, zero sum structures or command and control management at best.

Organizations that are looking to DevOps for help are likely trying to innovate new software and services and so they have to shift to a mode of operating that encourages collaboration and creativity. Realizing that is a critical step: we want to create and run new software, so we need to understand and become a software producing organization.

In contrast, if you operate differently if you’re just driving down costs each quarter and not creating much with IT. We’d counter-argue that if you’re a large organization and you’re not worrying about software then you’ll be creamed by your competition who is becoming a software organization.

If forced to pick one tip to increase collaboration I would say: do it by starting to work. How you do this is to pick a series of small projects and slowly expand the size of the projects. These projects should be low profile, but have direct customer/revenue impact so that they’re real. It’s important for these projects to be actual applications that people use, not just infrastructure and back-end stuff. It will help the team understand the new way of operating and at the same time help build up momentum and success for company wide transformation later down the road.

As a basic tactic, Andrew Shafer has a fun, effective tactic about having each people on the team wrote fantasy press releases about each other to start to build trust.

(See the full piece by Will Kelly over on the site.)

Use agile for speed, not cutting costs – Agile survey from Gartner AADI

I’m at Gartner AADI this year, the first time I’ve been to a Gartner conference. One of the sessions was a read-out of a recent survey about Agile. While a small sample set – “167 IT leaders in 33 countries” – it was screened for people who were familiar with or doing agile of some kind. As with all types of surveys like this, it’s interesting to look at the results both with respect to “what people are doing” and how they’re thinking about what they’re doing. Here’s some slides I took pictures of during the talk:

Organization's Profile - Gartner AADI agile survey readout

Why organizations do agile - Gartner AADI agile survey readout

Agile methodologies in use  - Gartner AADI agile survey readout

When agile practices were adopted - Gartner AADI agile survey readout

Agile stumbling blocks - Gartner AADI agile survey readout

My first take-aways are:

  • Well, Scrum is popular.
  • Most of the “stumbling blocks” are, of course, meatware problems: people and “culture.”
  • Pair programming, as always, gets no respect.
  • Organizations want to use agile for speed, not for cutting costs.

It's 10:35PM. Do You Know Where Your DNS Records Are? Or, HA, The Wealth of Networks, The $100 Laptop, and Generational Change

Thanks to the valiant efforts of Mr. Steve O’Grady, the RedMonk blogs will be up-ish tomorrow. I say “up-ish” because, as most you know, dear readers, switching domain names around on the internet is not a speedy science. Indeed, I’m often taken aback at how controlled and yet how chaotic the ‘net seems.

Then again, I’d be willing to be that there are teams of jack-booted thugs with hex screw-drivers and Cisco certifications ready to keep the network up. I mean, how terrible would that be if it went down?

While we wait for The Switchover, I still have this scrappy old thing. In the world of SaaS, High Availability means having two blogs.

Other Meanings for “High Availability”

I used to work at a company. Let’s call it WXYZ, Inc. One of the class clowns there made this joke one day:

High Availability? Baby, if you wanna get high…WXYZ is available!

Remarks like this were often followed by, “Waitress! Another Dewar’s!”

The Wealth of Networks

I started reading The Wealth of Networks last night. It’s nice, dense yet concise, academic talk about how content-producers controlling the means of production and distribution changes things. Information Marxism? Sure, sign me up as long as I can have a swanky Paris flat to go with it.

I’ve read a scant 20-30 pages, and there’s already a great conclusion: the physical distribution constraints of the “industrial information age” (pre-net) were the requirements driver of all that nasty, hegemony friendly IP law we created and now have.

Now, of course:

The removal of the physical constraints on effective information production has made human creativity and the economics of information itself the core structuring facts in the new networked information economy.

At least, that’s my understanding of those pages.

PC Means “Personal Computer”

As I read it, I keep thinking about the other 4-5 billion people who aren’t on the network. I had the pleasure of hanging out with Mr. Koranteng Ofosu-Amaah last week while he was in town. We had some Ruby’s BBQ and then some coffee (well, he had tea) at Spider House.

Somehow we got to talking about the $10040 laptop. He had two interesting things to say (side-note: a longer post on the hangin’ out is much over-due):

  • The notion of a 1:1 mapping between a computer and user may not be universal.
  • Hey, how ’bout them cellphones?

Which was interesting, because we talked with Nokia this morning. Now there’s a mega-platform for you: cellphones. The strange thing about The Wealth of Networks thinking (my 20-30 pages understanding of it) is that the people who run and own the networks seem a few successful startups away from being PanAm and TWA to the analogous Southwests. I mean: telcos! Come on!

In America, we always wave off madness in the telco world — Korea is light years ahead of us, I hear, and they have some sort of crazy cool network in Europe — as regulation and FUD. Really, it’s probably just 50-100 well paid people who’re waiting to retire until they screw with the golden goose.

Generations


And there we have one of my new pet-theories. (Inquire within for more pet-theories.) Technological change is generational. You have to wait for one generation to hand over the reins before real change can happen. Excited about Agile Software Development? Keep your eye on the retire date of all those managers and “decision makers.” Want better cellphone networks in America? Wait for “insiders” to retire.

If good software takes 10 years, seachanges in software take 30-40. As the man said, “get used to it.”

Of course, firing people works too, but it feels so nasty. And really, wouldn’t you just be sticking it to yourself in that case?

Luby’s Upgraded to STRONG BUY <eom>

The problem for us youngin’s is that retirement is soon to retire itself as a concept. Aside from all the fretting about not being able to live out The Golden Years in an RV or finally getting to writing That Novel, the generations in the tech world need to make a pact. A sort of realpolitik:

OK, we’re all going to keep our minds flexible and updated, right? I mean, if you’re going to stay in the work force forever, you’ll voraciously take on new ideas throughout your term, not just in the first 10 years. Maybe in exchange we’ll slow down a bit and focus on creating technology that allows you to work just 40 hours a week instead of 60. And we won’t say “no” next time you suggest Luby’s…. Okey-okey, and we’ll be less — just a little — snarky if you’re letting us play in your lawn.

Of course, we should probably be lining up to thank the generation ahead of us for working to pay the bills instead of bankrupting society. So, let me be the first: Thanks! …but now can we get on with some badly needed changes in core thinking?

Disclaimer: I actually like Luby’s. One word: okra. Bonus words: tarter sauce, deviled eggs.

Tags: , , , , , , , , , .

Notes from an Agile Presentation

Waterfall is plan driven — fixed requirements, estimated time and resources.

Agile is value driven — fixed time and resources, prioritized features.

“Would you rather have 100% of your features 80% complete or 80% of your features 100% complete?”

When the Dogs Make Their Own Food, or, Bottom-up Agile Innovation

An unbending tenant of eXtreme Programming is YAGNI, “you aren’t gonna need it.” As Ron Jeffries says:

YAGNI says, it never makes sense for a developer to implement things that aren’t asked for. It is always an explicit waste of the company’s money. It does not save money over doing the same thing later: the cost is the same, just spent later, which is a good thing. Making the marketing guys grin will not put money in the company’s pocket.

Much of Chapman’s In Search of Stupidity is an extended proof of why you’d want to use YAGNI to keep developers (and fame-crazed execs) from running wild with features. Any developer who’s suffered through hours long design meetings and protracted “debates” about how baroque to make a system become fast adherents to YAGNI as well. See KISS and Good Enough Software.

On the other hand, we have flickr. And del.icio.us.* And, oh yeah, Mr. $434.26, the GOOG.

“Some of the most intense users”

Cauvin linked to a set of interesting notes on Google’s product and project management taken in Jan. 2003 by Evelyn Rodriguez. On the topic at hand were these tidbits:

  • Employees (called Googlers) are best source for ideas at Google — also some of
    the most intense users of Google with well-formed opinions.
  • We get a lot of our ideas from our users (including customer support queue)
  • Next iteration [of Google News]: Didn’t make it to user studies because Googlers hated it.

While these notes are from 2003, that was obviously a critical time for Mr. $434.26, so the software development approach they were taking at the time is obviously of interest.

And what’s interesting is how involved the “Googlers” (man, I’d kill myself if I had to go by that moniker) themselves are in playing the role of customer. In the world of Agile, that’d be a big poop-on-that no-no: in most Agile implementations, the product owner is the mouth-piece of the God of Features, The Customer.

At the other end of the spectrum are developers who are writing and using the software: those “intense users” at Google.

Eating Your Own Dog Food

In software, we call this “eating your own dog food.” While I obviously have infinite respect for everything else good product managers do, the eating your own dog food principle is the most important one to follow, and yet the one least used.

If you’re working on some big, complex piece of enterprise software, for example, ask yourself, “would I want to install this and configure it?” Chances are, the answer is no: in an enterprise software shop, one of the hardest things to do is to find a running copy of your software (dev instances don’t count). No one wants to go through the hassle of installing that beast, which usually involves several hours, cryptic error codes, and dedicating a whole machine.

The CEO Test

Never mind the ever insulting “my mother/aunt/some clueless old lady” test; a product manager I know has a real answer for this problem: “The CEO should be required to install every piece of software their company makes.” In other words, the CEO would have such a terrible time installing most pieces of software that they’d use their bully-pulpit to make install easy.

Install is just the first part of your software (and as the first impression a customer gets, ask yourself, “how much time do we spend assuring that the install makes the best impression?”). The next step, of course, would be to require the CEO to use the application weekly, if not daily, with the same hopefully result.

The flip-side is that as a buyer of software, one of your first questions should be, “tell me how your company uses this software. Give me details.” If a company — esp. a large company that has all the same IT concerns as any other company — doesn’t use it’s own software, they’re sales people better be ready to dance up a storm to explain why.

Agile Dog Food?

Charles and I have talked about the lack of dog food eating in Agile quite a lot in the podcast (esp. in the first two episodes: 1 and 2). We’ve never come to a good conclusion, except that there’s something missing.

As the Google notes above indicate, employees are often one of the best sources for features, esp. if they’re users of the system. There’s certainly some shuckin’ and jivin’ that can be done to say that Agile thinking addresses this (we listen to customers, programmers could be customers, ergo we listen to programmers). But, as of yet, figuring out how to harness the innovative abilities of the developers is something that most Agile software development methodologies as practiced don’t do well. Of course, you could knock out the word “Agile” from that sentence, and it’d be even more true.

* My gut tells me that flickr and del.icio.us “allow” their developers to drive features. A couple of quick searches didn’t find anything to confirm it, so for now my proof is simply my instincts. For Google, on the other hand, I have the above “proof.”

The Waste Approach to Explaining Agile

I’m pretty much burned out on reading about Agile software development. I spent 2-3 years rabidly reading about it, and now that we’ve been applying it at The Day Job, I’m more interested in solutions and improvements to specific problems and processes I encounter. So the generalized stuff and platitudes you find out there gets to be repetitive and not immediately applicable.

That aside, I liked this post entitled “Application Integration and Agility (1): Understanding Agility” because it wove together Agile think very tightly with Lean think, e.g.:

[T]he core of waste in software comes from the traditional approaches to project management.

[As one example:] Tighter scope controls convert the “change control boards” into change prevention boards. And oftentimes teams deliver “to the scope” and not to the real business need, producing useless products that need substantial time to be augmented with functionality that is actually needed.

That is, adding extra process to slow down change can be wasteful because it produced software that (a.) the customer doesn’t want and, worse, (b.) that requires lots of work to make worthwhile.

The first items is the core of what it means to be Lean: you produce only what the customer wants. While the second is a consequence of what you might call The Grand Mistake of Agile Software Development Practitioners: the sense of immediacy and minimalism in delivering “only what the customer wants” often causes you to create software that is not flexible, nor easy to change.


“But there are also unknown unknowns, the ones we don’t know we don’t know.”

That is, despite what the phrase “deliver only what the customer wants” seems to imply, you do need to think of the future: customers always want more, and they want their software to be quickly modifiable to get that more. They just don’t know they want that until it’s too late. Part of being a good software developer — or any service job — is giving the customer what they don’t know they want. Put another way: the customer is rarely 100% right or complete, where “complete” means “can give you a list of every single thing they want.”

Small Groups in One Room

At the moment, my cure-all for all woes of applying Agile boils down to the well-worn Agile precept: small groups of people, all jammed into one room have the best chance, out of all project management strategies, of creating great software. Really, this practice is what you might call a “theory grandchild” of my favorite project management “law,” Conway’s Law, the conclusions of which point towards having as little hierarchy and lines of communication as possible…assuming you want to produce simple, elegant software. (As an aside, those two things are easy to loose track of when you focus on the naive understanding of “deliver only what the customer wants.”)

Sadly, the legacy world of plan-driven, people-heavy, product-based/shrink-wrap software we exist in is effectively anathema to small groups of people in one room. It’s an odd thing that.

Again With the Google!

As an industry, our prime hope is that The People Who Decide start looking to Google as a successful, new way of doing software development. If old GOOG can stay in the $100’s long enough, it just might stick. Otherwise, it’ll just be a flash in the pan, and we’ll stick with the boring, wasteful ways of developing software. Customers will continue to be “happy” because nothing better exists. The bears will dance!

(With Yahoo!’s last few acquisitions and partnerships with small software development shops, the old Y! might be a beacon of project management hope as well. We’ll have to see.)

Wasteful

So, yes, check the post, it has a refreshing interlacing of Lean and Agile.

And if you figure out how to disrupt the boring, legacy way of doing things, drop us a comment below.

Tags: , , .

[DrunkAndRetired.com Podcast] Episode 01: The Life Agile, Part 1

(Click Here to Download the Podcast)

I’ve finally put together the first Drunk and Retired podcast. This is part one of some back porch-chattin’ with my good friend, Charles Lowell about his time at ThoughtWorks, agile thinking, Star Control, and a few other things. It’s about 1/2 an hour, and you know you’ll love it!

Update: also, of course, the RSS feed contains an enclosure for this episode. So, you can just add the URL http://feeds.feedburner.com/cote to iPodder, or whatever podcast aggregator you use.