Advice on introducing DevOps from Merrill Corp & SPS Commerce – Highlights

Nicely moderated by Bridget. Some of my notes and highlights:

  • Amy talks about pace of change, sustaining it in the beginning, etc.
    • The amount of time it took us to get going was a surprise – was longer.
    • If you can start to show results early, it helps build up momentum. “Having enough wins, like that, really helped us to keep the momentum going while we were having a culture change like DevOps.”
    • It takes the right people to keep that energy going, but also be able to go back to the business to show that why we are putting these changes in place.
    • You’re going to be able to see the changes to the business right away.
  • Peg – tools, don’t try to fix the old ones, like ITIL service desk tools. Instead we just had Jenkins open tickets and such, automating the toil of dealing with old tools
  • Global/offshore tactics, from Amy:
    • What with all the retrospective stuff, you need to be able to get teams together, physically. The collaboration angles are much better in person
    • Set-up each “shore” as an architecturally and management island, make them as independent as possible. They also need their own context, not held up by time zones so they don’t need to wait 24-48 hours for authorizations and collaboration. [To my mind, this means taking advantage of the organizational de-coupling you can get with microservices.]
  • Starting change, even when they company needs it. Amy: You have to start with the business need, what’s the big driver behind a change like DevOps. [Managers often don’t make sure they figure this out, let alone decimate it to staff.]

Vanguard’s thinking on microservices

Breaking up the monolith with good, old fashioned, OO-think:

Instead, Vanguard has begun a journey to break apart our monolithic legacy systems piece-by-piece by replacing them with microservices over time. With a microservices architecture, we remove the business logic and data logic from our applications and replace it with a set of re-usable modules of code that are built and deployed as independent entities. We then compliment this architecture by chunking out our user interfaces into modular purpose-built components.

De-coupling for stability and resiliency, among other things:

This service-based approach to application architecture provides a variety of advantages over the jumble of code that defines a non-modular monolithic application. First, services reduce redundancy by making sure there is only one copy of application logic for a given capability – regardless of how many applications leverage that logic. In the long run, this leads to lower development costs and increases speed to market. Second, since these services are deployed independently and built in a resilient manner, outages in one area of an application are less likely to bring down an entire system. In some instances, several of our services can be down without our clients being aware of a loss in functionality thanks to the ability of our applications to automatically react to a service that isn’t available. Finally, services enable our applications to scale easier. The marriage of cloud and services means we can quickly spin up infrastructure to handle surges in the number of transactions we need to handle without needing to scale up an entire application.

Vanguard CIO: Why we’re on a journey to evolve to a microservices architecture

Pacing cloud-native transformation, and actually doing the work to increase productivity

I like to tell large organizations that compared to the break-neck pace of “the silicon valley mindset,” they can operate at a leisurely pace. That pace is usually fast for these enterprises, but their problem set and risk profile is a lot different than hats on cats. Abby has a nice, short write-up that hits on this topic among others:

By the end of his first year, Safford and his teams had built prototypes and market tests and finished 16 new software projects.

At Home Depot, they were at about 140 to 150 projects after a year or so. However, it’s common in the first year to do a lot of replatforming of “simple,” mostly cloud-native compatible apps in there. You can do these at a pretty fast clip, with the rule of thumb being 10 apps in 10 weeks. This is in addition to new applications, but explains high numbers like those at Home Depot. I suspect the Allstate numbers are mostly net-new apps, though.


Safford’s eventual goal is to shift Allstate software development to 70 percent extreme agile programming and 30 percent traditional scrum and waterfall. Where developers used to spend only 20 percent of their time coding software, today up to 90 percent of their days are spent programming. Each of his CompoZed development labs around the world has the same startup look and feel, including scooters parked in the hallways. This is not your grandfather’s insurance company anymore.

What you hear over and over again from organizations going cloud-native is that developers were spending lots of time in meetings, checking email, and otherwise not coding (and, yes, by “coding” I don’t mean just recklessly LOC‘ing it up without design, and all that). Management had to spend much effort to get them back to coding.

As I fecklessly tell my seven year old when he’s struggling with homework: the only way to finish this quickly is to actually do the work.

(Also: nice write-up from Abby!)

Source: Don’t Forget People and Process in Your Digital Transformation

How JPMC is making IT more innovative with PaaS, public and private

wocintech (microsoft) - 154

A good, pretty long overview of JPMorgan Chase’s plans for doing cloud with a PaaS focus. Some highlights.

More than just private-IaaS and DIY-platforms:

Like most large U.S. banks, JPMorgan Chase has had some version of a private cloud for years, with virtualized servers, storage and networks that can be shared in a flexible way throughout the organization.

The bank is upgrading its private cloud to “platform as a service” — in other words, the cloud service will manage the infrastructure (servers, storage, and networks), so that developers don’t have to worry about that stuff.

On the multi-/hybrid-cloud thing:

By the second half of 2017, the bank plans to run proprietary applications on the public cloud. At the same time, it’s building a new, modern internal cloud, code-named Gaia.

While “hybrid-cloud” has been tedious vendor-marketing-drivel over the past ten years, pretty much all of the large organizations I work with at Pivotal have exactly this approach. Public, private, whatever: we want to do it all.

Shifting their emphasis innovation:

“We aren’t looking to decrease the amount of money the firm is spending on technology. We’re looking to change the mix between run-the-bank costs versus innovation investment,” he said. “We’ve got to continue to be really aggressive in reducing the run-the bank costs and do it in a very thoughtful way to maintain the existing technology base in the most efficient way possible.” …Dollars saved by using lower-cost cloud infrastructure and platforms will be reinvested in technology, he said.

On appreciating the scale of “large organizations” that drive their very real challenges with adopting new ways of running IT:

The bank has 43,000 employees in IT; almost 19,000 are developers.

Good luck having the “we have no process by design” process with that setup.

On security, there’s a nice, almost syllogistic re-framing of “cloud security here”:

For years, banks have worried about using the public cloud out of security concerns and fears of what their regulators will say. Ever since the 2013 Target data breach, in which hackers stole card information from 40 million customers by breaking into the computers of an air conditioning company Target used, regulators have strongly urged banks to carefully vet and monitor all third parties, with a specific focus on security.

“We’re spending a significant amount of time to ensure that any applications we choose to run on a public cloud will have the same level of security and controls as those run internally,” Deasy said.

Most notable corporate security breeches over the year have involved on-premises IT (like the HVAC example above). The point is not to make sure that “cloud is as secure as [all that on-prem IT that’s been the source of most security problems in the past], but to make sure that all IT has a rigorous approach to security. “Cloud” isn’t the security problem, doing a shitty job at security is the security problem.

Source: Unexpected Champion of Public Clouds: JPMorgan CIO Dana Deasy, Penny Crosman, American Banker

Automation at Goldman, The Computer takes out four people

Today, nearly 45 percent of trading is done electronically, according to Coalition, a U.K. firm that tracks the industry.


Average compensation for staff in sales, trading, and research at the 12 largest global investment banks, of which Goldman is one, is $500,000 in salary and bonus, according to Coalition. Seventy-five percent of Wall Street compensation goes to these highly paid “front end” employees, says Amrit Shahani, head of research at Coalition… Investment bankers working on corporate mergers and acquisitions at large banks like Goldman make on average $700,000 a year, according to Coalition, and in a good year they can earn far more.

Automating those $700,000+ meat-sacks:

Goldman Sachs has already begun to automate currency trading, and has found consistently that four traders can be replaced by one computer engineer, Chavez said at the Harvard conference. Some 9,000 people, about one-third of Goldman’s staff, are computer engineers.

Finding the things to automate:

Though those “rainmakers” won’t be replaced entirely, Goldman has already mapped 146 distinct steps taken in any initial public offering of stock, and many are “begging to be automated,” he said.

To be all double-turns-out about the grim automation stuff, in theory, this could mean hiring more programmers and people who support those robots, bringing down those big chunks of cash from “rainmakers” and spreading it down to “lower” grade staff. Of, you know, the bank can just keep that money and trickle it up to execs and share-holders.

Source: As Goldman Embraces Automation, Even the Masters of the Universe Are Threatened

WTF is “digital transformation”? Beyond AI and VR for practical, software-driven innovation – My January Register Column

At the top of the year, companies are setting their IT agendas. Most high level executives seem to be lusting for “digital transformation,” but that phrase is super-squishy. In my Register column this month, I offered my advice on what it should be: simply “digitizing” existing, manual work-flows by perfecting how you do software.

This, of course, is the core of what I work on at Pivotal; see my wunderkammer of knowledge, the soon to be PDF’ed “Crafting your cloud native strategy,” for example.

What do these opportunities look like in businesses? Here’s a chunk that cut out of the piece that provides some examples:

A project to “digitize” the green card replacement program in the US provides a good example of the simple, pragmatic work IT departments should be curating for 2017. Before injecting software into process it’d “cost about $400 per application, it took end user fees, it took about six months, and by the end, your paper application had traveled the globe no less than six times. Literally traveled the globe as we mailed the physical papers from processing center to processing center.”

After discovering agile and cleaning up the absurd government contracting scoping (a seven year project costing $1.2bn, before accounting for the inevitable schedule and budget overruns), a team of five people successfully tackled this paper-driven, human process. It’s easy to poke fun at government institutions, but if you’ve applied for a mortgage, life insurance, or even tried to order take out food from the corner burger-hut, you’ll have encountered plenty of human-driven processes that could easily be automated with software.

After talking with numerous large organizations about their IT challenges, to me, this kind of example is what “digital transformation” should mostly about, not introducing brain-exploding, Minority Report style innovation. And why not? McKinsey recently estimated that, at best, only 29% of a worker’s day-to-day requires creativity. Much of that remaining 71% is likely just paid-for monotony that could be automated with some good software slotted into place.

That last figure is handy for thinking about the opportunity. You can call it “automation” and freak out about job stealing, but it looks like a huge percentage of work can be “digitized.”

Check out the full piece.

The internet of food nutrition labels

“So anyone who is producing food that ends up in our grocery stores, we’re working with them to get the data from their labels and the packaging information to come right into the database for us,” Pamela Stark Reed, deputy administrator for Nutrition, Food Safety and Quality, said on Information Management month.

The database has actually existed for over a century, Reed said. But before starting the initiative, it only had about 8,000 entries. Since opening it up to manufacturer submission, ARS has received 80,000 new items, a 1000 percent increase.
And on future plans:

The goal for the database is to eventually expand to 1,000,000 items. Reed said ARS anticipates getting store brand and international food items into the database soon. Some items from chain restaurants may follow.

Because of this, the agency is looking into cloud services to increase its storage capacity.

Just imagine, globally, how many data sets like that there are. Throw in the workflow to injest and cleanup the data, and change it, plus APIs to access it, and you have an almost endless amount of projects for software eating.

AirBnB lowers hotel prices 8-10%, effects low end hotel more

There’s all sorts of fun findings and theories in this study of AirBnB’s effect in the hotel market in Austin and Dallas. The easiest one is that it lowers pricing by 8-10% for the non-business traveler segment:

As Airbnb has its roots in casual stays, including those involving shared accommodations, we expect it to be a more attractive option for travelers on a budget. Conversely, business travelers and vacationers who frequent high-end hotels are two examples of consumers we argue are less likely to substitute a hotel stay with an Airbnb stay.

There’s also some interesting commentary on the very fixed assets of traditional hotel companies verses the agility of AirBnB:
– It’s impossible to rapidly increase the supply of hotel rooms to meet demand: it takes an average of 4 years to build new hotels, so you can’t really meet rising demand even on an annual basis.
– In contrast, the AirBnB supply can expand and contract on a daily basis as people decided to list and delist their rooms and houses.
– Of course, AirBnB demand is cap’ed to the number of fixed houses and apartments in an areas…but companies to hotel rooms, that supply seems infinite. (There’s an interesting analogy to public cloud here.)

Cloud Platform Adoption: Lessons Learned — Philip Glebow, Gap

Gap’s Philip Glebow goes over their use of Pivotal Cloud Foundry, including things that worked well and need improvement. His list of the supporting tools they use – like APM and data virtualization – is handy as well.

Some items:

  • (~2:30) Fast deploys: “We can deploy changes faster than people can really consume them.”
  • (~18:00) Developer morale: “We could really push something in five minutes… and developers love it, you click the commit button and there you go.”
  • (~19:40) [Poor transcription by me] On the danger of changing too fast: …generally we want to have a little bit of control into what goes into that production environment… but we don’t want to change so rapidly so that users are confused… There’s also a little bit of cultural change that we need to go through… ((too rapid of change is jarring)) …and as we bring that capability forward, we want to be sensitive to those concerns.
  • (~23:58) Overview of their pipeline and testing.
    (~26:29) [Poor transcription by me]  Typically we’ve organized out teams around sort of domain concepts – so we have a pricing team – then there’s several squads, then that squad is responsible for optimization – price, packing the stuff, etc. That’s how we’ve organized the teams, two pizza teams, we’ve tried to that. Also, distributed teams… sometimes that’s a little bit complicated.

Allianz now deploying to production in minutes

By changing its development practices and investing in a private cloud platform as a service, there have been clear benefits to the business. “Historically it would take two or three days for a deployment to go to production, with lots of manual production. Now with the apps in the garages we can do it on the basis of Cloud Foundry within minutes.”

Source: Allianz app deployment goes from ‘days to minutes’ with PaaS and agile practices