How a BigCo actually got some innovation done – The Longer Story of Crowbar

Here’s the slides for the talk I gave this morning on lessons learned from working on a DevOps product in a large company, that is, Crowbar in Dell.

The abstract:

Sometimes it seems like It’s near impossible to get anything innovative, interesting done in a large company – it’s as if BigCos are goaled to prevent just that. While you can’t type a URL without hearing how a Ramen-fueled startup got ground breaking product out the door, you rarely hear about how the other side of the exit lives in Large Company Land. This talk will use the story of Crowbar at Dell to grope out how to get good things done in big technology companies, esp. when it comes to something as BigCo esoteric as DevOps!

I’m amazed when I find a skunk-worked project that’s blossomed into a valuable, strategic asset for a company. In the case of Dell and Crowbar, it’s even more astonishing: Dell has traditionally been a stone-cold hardware company focused on shipping more boxes each quarter, Crowbar is an open source piece of software whose business model depends on the nuanced dynamics of open platforms strategy. You’d never think these two things would go together. And yet, Crowbar exists and has had amazing success (both externally and internally) in an extremely short time. With the access I have to the “real story,” being at Dell now after six years at RedMonk covering tech from the outside, I’ll go over lessons learned on getting DevOps and a DevOps product through the Brazil-like pneumatic tubes of a $62.1B company.

[ Podcast] Episode 110 – I got a retarded Leopard

Halloween Party

In this episode we talk about a whole passel of things, most interesting of which are:

Finally, go check out DFoF!

(This episode edited by Coté.)

[ Podcast] Episode 76 – Micro-Modularity, Dressing like a Mac, plus: The Scotch and Soda Use Case

In this episode, on the way to the airport, Coté starts by telling the worst re-telling of a movie moment event, we sing several “work songs,” talk about test driven development and refactoring, and then sort out how to dress while you travel.

(This episode edited by Charles.)

Tags: , , , , .

It's 10:35PM. Do You Know Where Your DNS Records Are? Or, HA, The Wealth of Networks, The $100 Laptop, and Generational Change

Thanks to the valiant efforts of Mr. Steve O’Grady, the RedMonk blogs will be up-ish tomorrow. I say “up-ish” because, as most you know, dear readers, switching domain names around on the internet is not a speedy science. Indeed, I’m often taken aback at how controlled and yet how chaotic the ‘net seems.

Then again, I’d be willing to be that there are teams of jack-booted thugs with hex screw-drivers and Cisco certifications ready to keep the network up. I mean, how terrible would that be if it went down?

While we wait for The Switchover, I still have this scrappy old thing. In the world of SaaS, High Availability means having two blogs.

Other Meanings for “High Availability”

I used to work at a company. Let’s call it WXYZ, Inc. One of the class clowns there made this joke one day:

High Availability? Baby, if you wanna get high…WXYZ is available!

Remarks like this were often followed by, “Waitress! Another Dewar’s!”

The Wealth of Networks

I started reading The Wealth of Networks last night. It’s nice, dense yet concise, academic talk about how content-producers controlling the means of production and distribution changes things. Information Marxism? Sure, sign me up as long as I can have a swanky Paris flat to go with it.

I’ve read a scant 20-30 pages, and there’s already a great conclusion: the physical distribution constraints of the “industrial information age” (pre-net) were the requirements driver of all that nasty, hegemony friendly IP law we created and now have.

Now, of course:

The removal of the physical constraints on effective information production has made human creativity and the economics of information itself the core structuring facts in the new networked information economy.

At least, that’s my understanding of those pages.

PC Means “Personal Computer”

As I read it, I keep thinking about the other 4-5 billion people who aren’t on the network. I had the pleasure of hanging out with Mr. Koranteng Ofosu-Amaah last week while he was in town. We had some Ruby’s BBQ and then some coffee (well, he had tea) at Spider House.

Somehow we got to talking about the $10040 laptop. He had two interesting things to say (side-note: a longer post on the hangin’ out is much over-due):

  • The notion of a 1:1 mapping between a computer and user may not be universal.
  • Hey, how ’bout them cellphones?

Which was interesting, because we talked with Nokia this morning. Now there’s a mega-platform for you: cellphones. The strange thing about The Wealth of Networks thinking (my 20-30 pages understanding of it) is that the people who run and own the networks seem a few successful startups away from being PanAm and TWA to the analogous Southwests. I mean: telcos! Come on!

In America, we always wave off madness in the telco world — Korea is light years ahead of us, I hear, and they have some sort of crazy cool network in Europe — as regulation and FUD. Really, it’s probably just 50-100 well paid people who’re waiting to retire until they screw with the golden goose.


And there we have one of my new pet-theories. (Inquire within for more pet-theories.) Technological change is generational. You have to wait for one generation to hand over the reins before real change can happen. Excited about Agile Software Development? Keep your eye on the retire date of all those managers and “decision makers.” Want better cellphone networks in America? Wait for “insiders” to retire.

If good software takes 10 years, seachanges in software take 30-40. As the man said, “get used to it.”

Of course, firing people works too, but it feels so nasty. And really, wouldn’t you just be sticking it to yourself in that case?

Luby’s Upgraded to STRONG BUY <eom>

The problem for us youngin’s is that retirement is soon to retire itself as a concept. Aside from all the fretting about not being able to live out The Golden Years in an RV or finally getting to writing That Novel, the generations in the tech world need to make a pact. A sort of realpolitik:

OK, we’re all going to keep our minds flexible and updated, right? I mean, if you’re going to stay in the work force forever, you’ll voraciously take on new ideas throughout your term, not just in the first 10 years. Maybe in exchange we’ll slow down a bit and focus on creating technology that allows you to work just 40 hours a week instead of 60. And we won’t say “no” next time you suggest Luby’s…. Okey-okey, and we’ll be less — just a little — snarky if you’re letting us play in your lawn.

Of course, we should probably be lining up to thank the generation ahead of us for working to pay the bills instead of bankrupting society. So, let me be the first: Thanks! …but now can we get on with some badly needed changes in core thinking?

Disclaimer: I actually like Luby’s. One word: okra. Bonus words: tarter sauce, deviled eggs.

Tags: , , , , , , , , , .

Notes on IT Labor

  • You can rarely pick the people on your team, or pick those that your team must work with and depend on. “If we had all the right people, of course all the right things would happen.”
  • Putting people in one big room is a defensive move against the above. Open source software is developed by geographically dispersed groups (the word “team” is even too strong to use). Sales is driven by geographically dispersed, ad hoc teams.
  • Theory: we haven’t figured out how to do software in business, on a large scale. It may be done, but it’s not normal, or thought of as “good” (by programmers at least).
  • Theory: worse, we haven’t come up with re-usable ways to solve the people problems associated with not being face to face. We expect our geographically developers to work as well as open source developers, but it hasn’t worked out yet on a large scale.
  • Given all that, your best bet is one big room.
  • Is it true that technical skill is disappearing in the US? MSFT and others are always saying “we have to hire globally because those in North America who’re smart enough aren’t plentiful enough.” Is business skill is suffering the same thing? What skills do we have in North America?

When the Dogs Make Their Own Food, or, Bottom-up Agile Innovation

An unbending tenant of eXtreme Programming is YAGNI, “you aren’t gonna need it.” As Ron Jeffries says:

YAGNI says, it never makes sense for a developer to implement things that aren’t asked for. It is always an explicit waste of the company’s money. It does not save money over doing the same thing later: the cost is the same, just spent later, which is a good thing. Making the marketing guys grin will not put money in the company’s pocket.

Much of Chapman’s In Search of Stupidity is an extended proof of why you’d want to use YAGNI to keep developers (and fame-crazed execs) from running wild with features. Any developer who’s suffered through hours long design meetings and protracted “debates” about how baroque to make a system become fast adherents to YAGNI as well. See KISS and Good Enough Software.

On the other hand, we have flickr. And* And, oh yeah, Mr. $434.26, the GOOG.

“Some of the most intense users”

Cauvin linked to a set of interesting notes on Google’s product and project management taken in Jan. 2003 by Evelyn Rodriguez. On the topic at hand were these tidbits:

  • Employees (called Googlers) are best source for ideas at Google — also some of
    the most intense users of Google with well-formed opinions.
  • We get a lot of our ideas from our users (including customer support queue)
  • Next iteration [of Google News]: Didn’t make it to user studies because Googlers hated it.

While these notes are from 2003, that was obviously a critical time for Mr. $434.26, so the software development approach they were taking at the time is obviously of interest.

And what’s interesting is how involved the “Googlers” (man, I’d kill myself if I had to go by that moniker) themselves are in playing the role of customer. In the world of Agile, that’d be a big poop-on-that no-no: in most Agile implementations, the product owner is the mouth-piece of the God of Features, The Customer.

At the other end of the spectrum are developers who are writing and using the software: those “intense users” at Google.

Eating Your Own Dog Food

In software, we call this “eating your own dog food.” While I obviously have infinite respect for everything else good product managers do, the eating your own dog food principle is the most important one to follow, and yet the one least used.

If you’re working on some big, complex piece of enterprise software, for example, ask yourself, “would I want to install this and configure it?” Chances are, the answer is no: in an enterprise software shop, one of the hardest things to do is to find a running copy of your software (dev instances don’t count). No one wants to go through the hassle of installing that beast, which usually involves several hours, cryptic error codes, and dedicating a whole machine.

The CEO Test

Never mind the ever insulting “my mother/aunt/some clueless old lady” test; a product manager I know has a real answer for this problem: “The CEO should be required to install every piece of software their company makes.” In other words, the CEO would have such a terrible time installing most pieces of software that they’d use their bully-pulpit to make install easy.

Install is just the first part of your software (and as the first impression a customer gets, ask yourself, “how much time do we spend assuring that the install makes the best impression?”). The next step, of course, would be to require the CEO to use the application weekly, if not daily, with the same hopefully result.

The flip-side is that as a buyer of software, one of your first questions should be, “tell me how your company uses this software. Give me details.” If a company — esp. a large company that has all the same IT concerns as any other company — doesn’t use it’s own software, they’re sales people better be ready to dance up a storm to explain why.

Agile Dog Food?

Charles and I have talked about the lack of dog food eating in Agile quite a lot in the podcast (esp. in the first two episodes: 1 and 2). We’ve never come to a good conclusion, except that there’s something missing.

As the Google notes above indicate, employees are often one of the best sources for features, esp. if they’re users of the system. There’s certainly some shuckin’ and jivin’ that can be done to say that Agile thinking addresses this (we listen to customers, programmers could be customers, ergo we listen to programmers). But, as of yet, figuring out how to harness the innovative abilities of the developers is something that most Agile software development methodologies as practiced don’t do well. Of course, you could knock out the word “Agile” from that sentence, and it’d be even more true.

* My gut tells me that flickr and “allow” their developers to drive features. A couple of quick searches didn’t find anything to confirm it, so for now my proof is simply my instincts. For Google, on the other hand, I have the above “proof.”

Technical vs. Non-technical, or, Re: The Process Variant of Conway’s Law

Scott Sehlhorst has a thorough reply to the Document Mobile example in one of last week’s posts.

I like his re-thinking of the cynical idea that layers of docs are a way for non-technical people to hide-out. Indeed, the docs do provide a sort of “common language” for the technical and non-technical people to talk in.

In fact, Sehlhorst’s post serves well as a “what you should be doing” reply to the problem of Document Mobiles.

Non-Technical People

As I’ve discussed these ideas with people over the last few days, I’ve realized that my definition of “non-technical” is broader than most folks think. For example, I don’t expect a “technical” person to know how to program, or even debug a hairy system problem. Those types of things are what “programmers,” or, at least, “system administrators” do.

What Do Technical People Know?

I do expect “technical” people, for example, to know things like:

  • Right clicking on a web page is an extreamly weird and uncommon thing: like a 3 wheeled car.
  • Changing the wording on a screen isn’t a big deal.
  • When to use a GUI vs. a web app.
  • Most aspects of web app usability and design, e.g.: methods of text overflow controlling and scrolling issues, or what “AJAX” means for making web apps more dynamic and realtime.

Of course, there are exceptions, variations, and thin lines to walk for those, and other, things depending on the system and requirements. But, being able to sort out those issues is the core of what makes someone “technical.”

Editors and Writers

To make an analogy: an editor isn’t a “writer” but they’re certainly in the business of helping put together articles, books, and stories. There’s a shit-load of shared knowledge between editors and writers. If you took some dude off the street and had them be an editor, chances are the best you’d get when they read the book would be “this books is long,” or, “I like this book.”

Non-technical people often give similar responses when confronted with demos of software: “could we move that header over a little to the left?”

An Example

I’ve had the pleasure to meet several “technical” people who aren’t programmers. For the issue at hand, Brandon stands out as a good example. He’s a product manager, though he did used to write code…in C++ I think of, all horrors. No wonder he stopped programming ;>.

Despite being a “mouse person” (as opposed to the “keyboard people” who type code), he knows his shit when it comes to software. For example, if you tell him you’re starting an application, he’ll throw a barrage of technical questions at you like:

  • Are you bundling a database? Oracle? MySQL? Hypersonic? Can I switch out the database? Can I move the database?
  • Is it a web application? Java? Ruby? Does it use JBoss? Just Tomcat? Rails?
  • OK, do you have group permissions? Fine grained permissions? Permissions at all? LDAP? Active Directory?
  • How about an XML over HTTP based API? On that note, do you implement the <insert your domain here> blah-blah data/process standard(s)?
  • You doing water-fall or Agile? How long’s the iteration?
  • Of course you’ll want to upgrade to the new Java. But do we really need to? Does it at least run faster?
  • Users don’t want 3,000 nodes, events, or 3,000 of anything. Let’s give them 5.
  • There is no right click on a web page. Shove that JavaScript up your ass.

(OK, maybe that last, ass part is what I’d say. Brandon has — what do they call it? — couth?)

The above is a good mix of “speeds and feeds and feature tick-lists” and to understanding how technologies are used and can be used, in both old and new ways.

Point being: he can actually add a shit-load of value to the process of writing kick-ass software. And that, of course, is the idea of a “technical” person. They don’t need to write code, but their work should directly inform and feed into the development and (if you’re into making money) monetization of code. Otherwise, there’s probably a degree of waste going on that could be cut out.

The Process Variant of Conway's Law

Recently I’ve found myself explaining what I call “The process Variant of Conway’s Law” to several people. Most of you, dear readers, have probably figured out that Conway’s Law is one of my top 5 favorite “laws” about software development (probably right behind “9 women can’t make a baby in one month”). I’ve mentioned it in several posts in the past, for example this one on sizing iterations to your desired goal, where I said:

You end up with a sort of process/methodology version of Conway’s Law: the software development process an organization uses will directly reflect the managerial and communication structure of that organization.

And there’s another mention in this post.

The Document Mobile as a Test

My neighbor’s have a mobile hanging in their tree made out of, well, what seems like junk: CD’s, card-board but-outs, old styrofoam cups, and Mentos tins. When I look at it I think of an org-chart with it’s single point at the top and branching out sub-trees with all sorts of stuff hanging from it. Branches and leaves to use nerd-talk.

Cauvin, as he often does in the space of project management, did a good job outlining an all too common “document mobile” in the software world: you start with an MRD, which branches out to a PRD, and then an FRS, then an SRS, and finally, when the coder is coding away, tasks and other “get the shit done” items. (Don’t worry if you don’t know all those TLA’s. The actual meaning — the answer to the question “what the fuck do I write in this?” — varies from group to group across companies).

Ostensibly, the two goals of this document mobile are:

  1. To start out very broad — “we need software that allows The Kids to live a mobile life-style!” — and slowly whittle down to the extremely specific — we need to use the Motorola Fizy5 TuneDaddy with the latest Upchuck IM’ing client, but disable the Bluetooth photo sharing” … “to accomplish that we’ll use our own OS with the following screens, functionality, wizards, etc.”
  2. Allow those at the top to keep track of how much those at the bottom are getting done. That is, workflow and dashboards!

In practice, this document mobile is the prime example of The Process Variant of Conway’s Law. That is, each of those document maps almost 1-1 to a level in your org-chart:


I obviously have a strong stance on the need for software companies to simplify their over-all approach and reduce waste, so you can imagine that document mobiles like the above drive me nuts.

The point is, when it comes to process and documents, whip out your org-chart, and I guarantee there’ll be endless “synergy” between that chart and all the artifacts and practices in your process/methodology.

What You Want

Put simply, when it comes to innovating software, we’d like something like this:


Tags: , , , .