I’m always interested to hear how management manages to change how software is done in large organizations – it can seem impossible! As ever, Allstate provides a fascinating stream of information here, and I was lucky to get the chance to interview Opal Perry there on how Allstate has been doing with all that cloud-native stuff.
Check out the listing on SoundCloud, and be sure to subscribe to the podcast if you like it.
Also, if you want to hear more, Matthew Curry and I had a similar conversation a few weeks ago at OSCON.
I’m doing a podcast with Comcast in a few weeks, so I’ve been going over all their public talks on their cloud-native efforts. They’ve been working with Pivotal since around 2014 and are one of the more impressive customer cases with over a 1,000 applications now on Pivotal Cloud Foundry.
Here are some highlights
from the talks I’ve been watching. As always
, things I put in square brackets are my own comments, the rest are quotes or summaries of what people said:
August, 2016 – Empowering Devops with Cloud Foundry – Sergey Matochkin, Neville George; Comcast
- Sergey Matochkin.
- (17:00) Every deployment to production took at least 6 weeks, but most commonly around 2 months end-to-end. Which also means you need to plan capacity much in advance.
- We started to use virtualization and containerization “well, well before Docker existed… it was some success, we had some improvements, but those improvements were marginal.”
- Traditionally, it’d take at least 4-6 months to setup your dev/test infrastructure. But, luckily, virtualization came along.
- (9:20) Business drivers… Comcast phone service, set-top boxes get DVRs, VoD, etc. All of these require apps on the backend, so the portfolio of apps starts to grow, and with they way they were before it meant they had to build a new datacenter every six months. Virtualization helped here, of course.
- Also, virtualization allowed us to put a service layer [think “platform”] on-top of the infrastructure.
- It’d take 4-6 weeks for testing environment, but now it takes 10-15 minutes in a self-service portal.
- Demo of using Pivotal Cloud Foundry for much of the automation needed to deploy and scale an application.
- (~32:00) We used to have things like “order servers” and “make load-balancer changes” and somewhere in the bottom of the backlog was “write some code and do some testing.” [That is, they were focusing on items with low business value, below “the value line,” rather than customer features.]
- “What Cloud Foundry essentially helped us with was to get all those unnecessary user stories out of our backlog so we can focus on the writing code, on testing, and deploying rather than managing infrastructure.”
- (33:45) momentum/proof-points:
- 9 PCF instances; 900+ developers; 2,000+ active apps “most of which are in “the critical path of our customer experience”; 4,100 application instances; 2,000 requests per second.
- Lots of Slack/ChatOps usage for monitoring and such.
August 3rd, 2016 – Transforming the monolith at 20M tph – Nick Beenham, Comcast
- Existing state:
- 250m transaction per day.
- Would take 3 months to get a server useful, from moment of purchasing to using.
- “Over a 100 services run by development teams.”
- In functional, silo roles.
- (3:45) “We knew we had that large, rigid infrastructure. [Pivotal] Cloud Foundry and it’s adoption really enables us to change that to gain the agility, to gain the elasticity at scale.
- Taking away roles to reduce finger-pointing and all the negative stuff, and unified team, of course.
- (7:35) Anecdote of Nick going from “ops guy” to writing code and liking coding.
- (12:18) ESP router that was a small router written in Go to translate SOAP requests as part of a strangler pattern. Decades old SOA layer that they wanted to modernize. But they couldn’t strip it out, would take so long. So, were going to duck-type as SOA, but do REST and micro services underneath. Strangler pattern, etc. This is what the ESP router does marshals and unmarshalls between microservices and SOAP stuff. But new things need to be done in new style.
- Also, “de-mingling data,” moving off Oracle RAC/GoldenGate for multi-site. Some simpler CRUD services to front the data.
- (~15:00) Used to take a week+ to deploy the entire stack, but with Pivotal Cloud Foundry it takes minutes. It gives us a great deal of velocity that we’ve never had before. “Sometimes we’ll deploy multiple times an hour.”
- (17:00) From 1,000’s of lines of bash to deploy out to various WebLogic clusters, which has for the most part moved to Cloud Foundry.
- Improving production updates: bringing new node up and shutting old node down slowly; canary updates, with a CI test suite, then switching over to a production install.
August 1st, 2016 – James Taylor – The Power of Partnership & Building a Cloud Native Tier-1 Platform
- “Sparrow, Service Activation Platform.”
- “Helping someone put a smile on their face is one of the greatest gifts we can give each other.”
- Their VP provides the feedback loop of things to focus on. Right now: reducing technical debt, reducing incidents, increasing velocity, experimentation.
- (~6:30) “You can’t move forward – innovate – if you don’t have time to try new things.”
- (~18:35) “If you’re spending time configuring a Docker container, that’s time you’re not spending coding or solving a problem.”
- (13:51): “At the end of the day, [business] value is what puts money in everyone’s pocket. If our company, Comcast, can’t create something of value, no one’s gonna pay for us…if we can’t create value. So it’s important for us to understand ‘how can you create value?’”
- (~22:02, starting epic rant!) “Who is our customer and what value do we bring to our customers…”
- If you’re spending money on support, that’s cutting into your margins. A call coming in costs $8 right off the bat, then more as it takes longer. So you want to figure out preventing customer support problems… which points to understanding your customers more.
- [A good overview of thinking about “value” in the context of a specific application, their customer activation center, Sparrow.] “If you have a [support] call rate of 30%, you’re probably cutting out all the value… So we try to figure out, how do we prevent calls?” [Very similar to IRS cloud-native story.]
- “We’ve been holding technical workshops”: Internal training things every month with Pivotal people, leveraging Pivotal knowledge. With our development teams every month: webinar, or on-site visit.
- Sparrow: 5 junior Java developers… we built it from scratch in parallel while existing teams maintained the platform… we then had to integrate the processes together… figure out decomposing the monolith platforms, etc….then we had to just cut off stuff when it was too much of a hassle.
August 17th, 2016 – Greg Otto SpringOne Platform keynote
- X1 boxes – a new release about once a month.
- Processing 10’s of millions of transactions on this new platform daily on Pivotal Cloud Foundry/new platform.
- “About a 75% lift in velocity as well as time to market, and the business is really feeling it.”
- Developer reactions:
- Momentum Stats:
- 40 apps to 900 apps, 2015 to 2016
- 300 AIs to 4,100 AIs, 2015 to 2016
- All with “zero outbound marketing from my team, this all word of mouth from all those happy developers.”
June 9th, 2016 – Greg Otto CF Summit keynote
- “Late last year in 2015” – live in production [on Pivotal Cloud Foundry] with business critical systems from our back-office systems on our Cloud Foundry environment.
- We put Pivotal Cloud Foundry directly in the customer critical path.
- Applications doing 30,000 event a second on Cloud Foundry.
- Started in 2014, met with Pivotal.
- Had sort of thrown all the people into the Pivotal Cloud Foundry pool, they had to do a lot of research and such.
- But, people were really interested in the ease of working with the platform [the productivity improvements].
- Successful prototype app 30 days after platform.
- Idea to feature, before after: “several weeks, at least”/“2-3 days”
- Time-line and summary:
June, 2016 – Open source at Comcast story
- “If Comcast has a problem to solve, there are three possible approaches: solve it themselves by making an investment in teams and resources; solve it through a commercial vendor that could build a product for them; or work with the open source community.”
- OpenStack: “In addition to Linux, Comcast is a heavy user of OpenStack. They use a KVM hypervisor, and then a lot of data center orchestration is done through OpenStack for the coordination of storage and networking resources with compute and memory resources. Muehl said that Comcast has roughly a petabyte of memory and around a million virtual CPU cores that they are running under the OpenStack umbrella. As an operator, Comcast does a lot of things around operations, and they use Ansible to deploy and manage OpenStack at scale.”
- Cloud Foundry: “They also use Cloud Foundry, but according to Muehl that work is in the very early stages at Comcast.”
May 2015 – Running Cloud Foundry at Comcast talk
- Neville George, Sam Guerrero, Tim Leong, Sergey Matochkin
- They wanted to make custom URLs.
- Used Puppet for stuff.
- (~8:30) Their requirements for a platform:
- A lot of emphasis on self-service and the micro services benefits of operating independently, product management wise.
- They use OpenStack, Docker, and [Pivotal] Cloud Foundry.
- Pre-provisioning resources for a pool of containers that are ready to go, etc.
- (~27) a couple applications in production today… we’ll be ramping up quickly.
- (Either this video or the 2016 one, a few minutes from the end) Q, training mode. A, Sergey: “I can’t say we have a really good training model…. We do brown-bags to have people aware. We focus on 12 factor application model… on overall microservices model, not just to shape application, but also data. Developers need to understand how they [do] applications for PaaS instead of traditional.
Container networking and storage is hella-complicated. Check out this conversation with Usha Ramachandran, Richard Seroter, and myself on the topic, including a discussion of why it’s so complicated and how Cloud Foundry addresses the problems.
It was a pretty good episode:
In preparation for his DevOpsDays Atlanta talk, Josh and Coté (well, mostly Coté) talk about the relationship between microservices and DevOps. They use the CAMS framing to go over how microservices could provide the architectural requirements to make DevOps possible.
Pivotal had a pretty major milestone this last quarter, over a quarter-billion dollars in 2016 bookings, up 130 percent year-over-year. The momentum with Pivotal in terms of digital transformation is white hot. Pivotal is now engaged in a third of the Fortune 100, and I would say the strategy so far has been to only go after the tallest buildings in the city. The opportunity to take Pivotal Cloud Foundry into Fortune 2000 and beyond is enormous. That’s going to represent a big opportunity for partners.
From a a recent interview.
Can we take that governance and work with the platform team to codify, to automate that which they were doing on a per application basis – that’s, quiet frankly slowing down the delivery of the software – can we take that governance and can we have them work with the platform team to codfiy, to actually automate on a per application basis, have them expose that as a service on the platform
–Cornelia Davis on governance and cloud-native, “Who Does What? Mapping Cloud Foundry Activities and Entitlements to IT Roles,” August 2016
In other words: you should not only automate the audit three-ring binders of compliance, but enforce as much as possible in the platform.
The rest of the talk is good stuff on how think through re-arranging your organization to be all DevOps-y, with the help of Pivotal Cloud Platform to automate all the infrastructure and middleware stuff:
My colleague Richard has a nice post suggesting the new functions enterprise architects can play in a cloud-native organization. I like this one in particular, help make the change:
Champion new team organization patterns. As an architect, you can bring developers and operations teams together. Recognize that functional silos slow down delivery. A DevOps-type approach really works. Architects are perfectly positioned to pioneer new team structures that increase velocity and customer attentiveness.
It’s brief, but there’s plenty of other good chunks of advice in there.
Source: How to Remaster Enterprise Architecture for a Cloud-Native World
Earlier this month I did a webinar with Nick at 451. He does a great job summarizing all the digital hoopla going around and I finish up with, predictably, why and how Pivotal can help out there, along with a few customer examples. Check it out!
Check out this talk on “cloud-native”:
We’ve got all your answers to “what exactly is ‘cloud-native’?” in this episode with special guests Pivotal’s Kenny Bastani and RedMonk’s James Governor. Kenny gives us a good overview of what cloud-native is, as Coté summarizes it: handling the configuration and automation for your applications along with all the supporting frameworks and platforms to do that. We then discuss the process (“culture”) angle, the origin of Spring Boot, the concept of “lock-in,” and if public cloud is needed or not. Bonus: serverless talk!
In this week’s episode, Richard and I talk with Dino about the work Pivotal does to help companies quickly start migrating applications to Pivotal Cloud Foundry. Check it out, and subscribe if you haven’t already.