I theorize about what the “dev” in DevOps has come to mean 15 years in.
I’ve proposed an open spaces for DevOpsDays Amsterdam, 2021. The idea is:
The DevOps community pushes for people to change how they think and operate. When it comes to working better, we have proven tools, techniques, and even big picture ways of thinking like CALMS. You’re more than likely eager to try these new things, get better, change. However, many more people seem less than eager to change – your co-workers, managers, and the countless “others” in your organization. In the discussions I have with change agents and executives in large organizations, this reluctance to change is one of the top three concerning topics. I invite you to this discussion to talk about why people are reluctant to change, how you’ve worked helped people change, or, perhaps given up, and, hopefully, to share stories about your own experience overcoming reluctance. Our goal will be to move beyond being frustrated with “frozen” minds and middles, and get a sense for what to do about it…if anything. To start the discussion, I’ll start with a few stories and methods for getting people to change that I’ve encountered in the past few years.Me!
In preparing for it, I typed in these notes:
Reluctance to change is one of the top concerns with executives and managers I talk with.
- Frozen middle, frozen minds – often means “I don’t like what they’re (not) doing” – is that kind?
- I want to talk about why people are reluctant to change; why you are; stories of success and failure in changing people’s minds, desires, and behavior/action.
- Examples to start with:
- Constant Planning, or, Analysis paralysis – they want to change, but think too much and don’t act. Fix: external problem/urgency, like (sadly) COVID, competition, dying/plateaued cash cow (not a very good motivator).
- Fear of change – demonstrate that it works.
- Seems like more work, or, won’t make their lives better, so why change? Show them that it make their lives better – automating things frees you up from tedium; automating for auditors saves them over-time; the new way can be more secure.
- Changing job/responsibilities/identity. The DBA likes being the DBA, the network admin likes that – prove to them that it’s better.
- Fear that they can’t change/learn the new thing – Coté doesn’t get around to learning Dutch, same fear. Related: embarrassment, e.g., Coté doesn’t buy from the butcher down the street cause he don’t speak Dutch (but, butcher doesn’t care – OF COURSE!) Fix: hard one, show success from peers.
- General fix for all of this: build up success stories of their peers doing it.
- Change motivations:
- like money;
- flow – removing friction (better quality of life and work-life);
- raising individual profile with OSS work/fame;
- closer to end-user to see value they provide.
- Change structure. “Culture follows structure” – Larman’s Law – points out a common pattern/behavior. Once you know the behavior, you can start thinking about how to change/improve it.
- “the organizational system (groups, teams, roles and responsibilities, hierarchies, career paths, policies, measurement and reward mechanisms, etc)”
You can also see me discuss these in one of my Tanzu Talk streams.
BT SpringOne Talk: developer first and a willingness to learn and change as needed
A keynote given by Rajesh Premchandran, BT
BT wants to get better at how their do business, through software. Their strategies are, of course, operational excellence, but also getting software that improves the customer’s experience. This means they need to simplify how business is done, which they did by transforming the way they do software: IT underpins this corporate transformation, Rajesh says.
His approach to telling this story is to talk about how BT questioned some initial assumptions. On some of their initial plans, they pivoted (adapted) when they discovered new needs and realitis. On others, they persisted when they really needed the change. For example, at first, they thought that having a PaaS would be good enough for all development and app needs.
A pivot example: they discovered that more fine grained control was needed for some apps (esp. legacy ones), so they added in kuberntes. Here. PCF/TAS and TKG.
A persist example: security had to move from security by static IP. Because they were moving to AWS and microservices, things needed to be more dynamic. There was no way to achieve their tech-stack transformation otherwise. So, they persisted.
The third area he describes is their approach to removing ops toil from application developer’s agenda. As ever, they want apps developers to focus on…apps! And have time to explore and innovate making the apps better to deliver on customer service goals. (You see several example from Comcast on this with their new TV apps and improving customer service with ChatBots, and even in-home wifi coverage calibration, etc.) He also comments on a discovery, or validation of a predictable state, in my rewording: developers aren’t good at production ops and don’t really realize all the new tasks they’ll need to do – responisbilities they have! – in a DevOps model. So, they had pivot to training them and putting in place tech stuff to make it work better for them.
He has some fantastic framing: "instead of letting [developers] grapple with operating model choices, we adopted a more human-centric approach to educating teams."
He touches briefly on some portfolio modernization strategy. The benefits of doing, I think, were that they spent their time wisely, modernizing apps that were feasible and valuable to modernize instead of all of them. I could use more detail here, but I think the point is made – I mean it’s a five or six minute talk, so it’s fine. For a lot more, check out the ever excellent Rohit explaining this kind of portfolio app modernization analysis.
There’s another, long BT talk that I haven’t fully dissected yet.
- BT business – Fixed line, mobile TV, and networking.
- Lead in converged activity by simplifying business.
- Build scalable platforms for growth.
- Creating leaner business models.
- IT underpins this corporate transformation.
- "The challenges arise from our ways of working, process, account policies, security postures, and even we inventory software assests."
- Three challenges and how they worked with them:
- A PaaS, PCF/TAS with Spring – time to deploy from 2 months to 2 minutes. Also, microservices.
- But, people wanted kubernetes for finer grain control, and needs for non/fuzzy cloud-native apps. Used TKG for this.
- Listening to devteam
- Moved from static IP to static IP, public IP addresses.
- AWS networking needs, i.e., elastic IPs and REST endpoints.
- "They also were not fully aware of the [new] shared responsibilities of managing infrastructure as code, or how their roles changed when adopting the cloud."
- E.g., with a more DevOps approach, "their responsibilities did not end with a cf push."
- Enterprise architects looked through portfolio to bucket apps into legacy, strategic, SaaS, PaaS, and IaaS to plan out the way of working, priorities, app modernization tasks.
- Drives concensous on "application treatment" and how to educate teams on their roles and new skills.
- Also, this eliminates scope overlap in our budgets and app modernization plans, allowing the PaaS team to focus on executation, [instead of work that wasn’t applicable to the various apps strategies and bucketing].
Also, an excellent InfoWorld interview with Rajesh where he talks about several of these points, even more stridently.
- Rajesh: "What really happens is you’ve got to tease out what is containerizable"
- "To overcome this challenge, BT has established a platform team, dedicated to helping application teams identify these containerizable elements and find the best environment in which to host them, be it in the public cloud or on a platform-as-a-service (PaaS)."
- Rajesh: "You have to handhold them, otherwise they will take the biggest unit they can handle, put that into a Docker container and then lo and behold you’ve got the same monster on better infrastructure — that doesn’t solve your business problem."
- "This is a constant tussle," he admits, "where people want Kubernetes by default, but I think you’ve got to be a bit more prescriptive in the beginning to developers and then as you see them maturing, have them make those independent choices."
- "the next task is to scale it out via documentation and word of mouth buzz, both for in-house engineers and with external partners."
- Rajesh: "If you look at how standards are democratized, you’ve got enterprise architecture that looks at the overall architecture standards and policies, then you have solution architects, who are a level down and are specific to a product, and finally we have distinguished engineers — we call them expert engineers — who are also key influencers for other developers, who look up to them."
A good, and fun talk from Paul. He tries to refocus DevOps-y energy from “culture change” to more practical things for individuals to do.
- You can’t change culture from the bottom. Leaders change the culture, they define it.
- Culture is behavior.
- If you want to change culture, you need to change your leadership.
- Culture: as a practitioner, you can’t change it. And if you’re in a leadership position, you’re incentivized not to change it.
- “If you choose to die on a hill, you’re gonna die on that fucking hill.”
- Corollary: “If you don’t decide to die on a hill, you won’t die on that hill.”
- Generative orgs can create microservices.
- Others will make monoliths.
- Most people spend more time justifying not working than working.
- People who do good work will be pulled down.
- Most people around you aren’t bad people, they’re just a product of their surroundings.
- As soon as you start fighting the organization, it’ll start fighting back harder than you.
- Often said: if your company is acquired by a larger company, leave.
- Then, some examples of automating governance.
Check out this recording of a recent talk of mine. Here’s the abstract:
Many Government organizations are getting better at software development, deployment and management by using techniques like DevOps, agile development, and product management. Cloud native technologies are making organizations’ software supply chains more efficient and reliable. Our substantial experience with open-source technology and continuous deployment approaches, offers a powerful accelerator for contact tracing and integrated citizen response solutions. Improvement is fragile, and scaling up in large organizations is difficult. This talk will discuss bottlenecks, challenges, and how Government agencies and organizations are succeeding.
There’s even a transcript!
36% of developers cited “resource allocation and management” as their chief impediment to productivity, while 34% cited the “lack of automated collaboration between software developers and IT operations.”
Some quick notes and callouts from this year’s 2019 DevOps Report:
- Four key metrics: lead time, deployment frequency, mean time to restore (MTTR) and change fail percentage.
- Med, High, and Elite all have a change fail rate of 0-15%. So, expect 15% change fail as benchmark worst case to shoot for…?
- Demographics: 30% are devs, 26% “DevOps or SRE” – [so, lots of ICs self-evaluating]. 16% “managers,” and then it goes down from there…
- Top industries are Technology at 38% and FinServe at 12%. Retail is 9%.
- Mostly North American (50%)and Europe (29%)
- Org. size: 100-499 (21%), 500-1,999 (15%), and 10,000+ (26%)
- “A key goal in digital transformation is optimizing software delivery performance: leveraging technology to deliver value to customers and stakeholders.”
- [I’m not sure if age of company, and, thus, an indication of governance and tech debt, is tracked. With 38% being tech companies, it’d be good know how young they are. But, most FinServ companies are large and old (unless it was mostly FinServ startups!).
- Very prescriptive this year, a maturity model to put a strategy in place, etc.
- A lot on paying down tech debt:
- Bounded contexts, APIs, SOA and microservices. Using and testing out of team services without having to work with that team (sort of like mocking for runtime).
- Also: “Teams that manage code maintainability well have systems and tools that make it easy for developers to change code maintained by other teams, find examples in the codebase, reuse other people’s code, as well as add, upgrade, and migrate to new versions of dependencies without breaking their code”
- Very little prod chaos monkey stuff: less than 10% across the board.
- CABs still bad: those that have them are 2.6x more likely to be low performers.
- Instead, do peer reviews and automate governance: “peer review-based approval during the development process. In addition to peer review, automation can be leveraged to detect, prevent, and correct bad changes much earlier in the delivery lifecycle. Techniques such as continuous testing, continuous integration, and comprehensive monitoring and observability provide early and automated detection, visibility, and fast feedback. In this way, errors can be corrected sooner than would be possible if waiting for a formal review.”
- CABs should instead focus on process and practices change: ” the CAB should focus instead on helping teams with process- improvement work to increase the performance of software delivery. This can take the form of helping teams implement the capabilities that drive performance by providing guidance and resources. CABs can also weigh in on important business decisions that require a trade-off and sign-off at higher levels of the business, such as the decision between time-to- market and business risk.”
- [I’m pretty sure that was the original point, esp. when you look at RUP and ITIL stuff: setting the process to be used. Tooling to automate governance wasn’t really available. Policing it those prescriptive processes took over as it always does. And I’m not sure there are industry standard frameworks to use there yet either. There must be lots of hand-crafting.]
- “Survey respondents with a clear change process were 1.8 times more likely to be in elite performers.” – [as ever, garbage in, garbage out.]
- The people who work on governance are not the ones who can actually do the coding to automate it: “only our technical practitioners have the power to build and automate the change management solutions we design, making them fast, reliable, repeatable, and auditable…. Leaders at every level should move away from a formal approval process where external boards act as gatekeepers approving changes, and instead move to a governance and capability development role. After all, only managers have the power to influence and change certain levels of organizational policy. We have seen exponential improvements in performance— throughput, stability, and availability—in just months as a result of technical practitioners and organizational leaders working together.”
- This is a different measure of “productivity”: “Productivity is the ability to get complex, time-consuming tasks completed with minimal distractions and interruptions.”
- It doesn’t track amount of work done, but the environment people are working in…?
- Tools use is all across the board: DIY stuff, COTs, open source, etc. [This sort of excludes the IaaS and other runtime layers, focusing on just CI/CD and test automation]
- “Multi-tasking” across roles and projects might be OK: “we cannot conclude that how well teams develop and deliver software affects the number of roles and projects that respondents juggle.”
- Being able to find things and ask questions [and, presumably, getting answers!], having search, is important.
- From my read (slide 74), the methods of transforming orgs are all across the board with Big Bang and Training Center as the only low ranked ones. Communities of practice are high, part of the Spotify model.
- Pg. 75 tries to derive some advice nonetheless: mostly that separate education and training groups don’t work well/widely, that grassroots is used a lot, and that communities of practice are good, as well as PoCs that get cloned.
- [This is an instance where the high level of individual contributors in the answers might have an effect. They see the positive change in their own team, but don’t have the big picture view to see if the practices scale up to 1,000’s of people. On the other hand, they might follow the “my congressperson is perfect, all the other ones are corrupt and terrible” pattern. Also, those 5,000+ people orgs struggle.]
- [We still don’t know how to change an engine in flight.]
This massive growth in scale has required an evolution in practices and organization to achieve success. Most of what technologists are aware of in this regard is labeled “DevOps,” but there is more nuance to it than that. The way infrastructure capacity is allocated becomes decoupled from specific hardware, so the infrastructure team has to adapt new tools. The way databases and message busses, among other things, are operated and made available to applications has become more “self-service”, and thus those teams have to see themselves as service providers rather than as infrastructure teams.
You’re basically breaking things down into smaller increments. I’m constantly getting in front of the customer. At the end of the day, you’re delivering really what the customer wants from you.
So you may be able to start giving value to your customer much quicker. Even though they’re not getting 100% of what they want, they get an MVP [minimum viable product] that they can start to utilize, they can start to realize value and give you feedback on that. That may influence the rest of what you’re building as well.
What all of those “unicorns” have in common are flat organizations with small teams that are responsible for a product or feature, including receiving feedback from their customers and guiding the future of the product. ING decided to transform its business to be more agile.
And this is where DevOps tool vendors now see opportunity. The mounting tax of maintaining tools has grown so extensive that offering an all-in-one solution may just be the trick to get development teams unstuck and the innovation train rolling even faster.
I’ve tracked at least three different definitions of DevOps since the days of “agile infrastructure”:
- Using Puppet and Chef (and then Ansible and Chef) to replace Opsware and BladeLogic.
- Full stack engineers to setup EC2, load-balancers, and other Morlock shit.
- Full stack engineers are bad, but sort of the same thing. Also, you can’t have a DevOps “group” or title. But, you know, someone should do all that automation.
- Putting all the people on one team, having them focus on a product, and establishing a culture of caring and learning.
- SRE is not DevOps.
So…actually five. Maybe some of them just being footnotes on the evolving concept. (And, if you, dear reader, feel these are wrong, then let’s compromise and make the list six.)
All of them evolved around bringing down The Wall of Confusion, allowing “developers” to deploy their software to production more frequently, weekly, if not daily. And, of course, making sure production stays up. (You’re supposed to call that “resiliency” and instead of SLAs use SLOs and some other newly named metrics that answer the question “IS MY SHIT WORKING?” Whatever you do, just don’t say “uptime,” or you’re in for it and will be relegated to running the AS/400’s.)
I used to snide that the developers seemed to have been yanked out of DevOps, sometime around 2014 and 2015. All the talks I saw were, basically, operations talks. I haven’t really checked in on DevOps conference talks recently, but at the time, I don’t think there was much application development stuff. (I’m not sure if there ever was?)
None of this means that DevOps is not a thing. Not at all. It just means that the enterprise finds its own use for things. It also means there’s still weekly write-ups of what DevOps is – you know, those ones that are always lists of ideas, things you’re getting wrong, and how to start.
Autonomous product teams
Nowadays, I try to stick to that forth one: you want to setup autonomous teams that have all the skills and responsibility/authority/tools needed to “own” the software being specified, designed, developed, and run. This means you have to, basically, remove-by-automating all the operations stuff it takes to stand-up environments, deploy things, and do all that “day 2” stuff.
(HEY! HEY! WANT TO BUY SOME ENTERPRISE SOFTWARE?!)
Now, I think this product-centric notion of DevOps is, well, kind of an over-extension of the term “DevOps.” But since SRE has sucked out the “ops” part (but, remember, dear reader, don’t commit the embarrassing act of saying SRE is DevOps – no, no, you’d never do that, right? SO SHAMEFUL! (SRE is totally different – no overlap or similar goals shared between them at all. I mean, they have separate groups, silos! COME ON!)), slicing “DevOps” back to just “Dev,” but with a product-not-project focus isn’t too shabby.
Anyhow. I came across a good overview of this product notion of DevOps, all the way back from 2016, while re-reading Schwartz’s evergreen excellent The Art of Business Value:
Agile approaches attempt to bring together developers and the business in an atmosphere of mutual respect and joint contribution. Until now, however, the focus has been on users of the software, product visionaries, and developers. Recent developments in the Agile world—notably DevOps—have broadened this idea of respect and inclusion to encompass Operations and Security. The DevOps model, in other words, looks to break down the silos that have resulted from technical specialization over the last few decades. But the DevOps spirit goes further, looking to eliminate the conflicting incentives of organizational silos and the inhumane behaviors that can result from those conflicting incentives.
Perhaps we can take this idea even further still. There is no reason why the DevOps team’s responsibility needs to stop at the border of what used to be considered IT. The team is part of a broader enterprise, whose collective knowledge, skills, and judgment need to be part of the value creation process.
Look a’ that guy! Business Value just effortlessly jets out of his pores like a peripatetic thought-monarch!
This is from an executives perspective, but it drives home the point we’re always trying to get to with software: doing whatever it takes to figure out, create, and give users features that are actually useful to them. Somewhere beyond that, if you’re lucky, it’ll help out “the business.” Also, it should implement The Unspoken User Story: user would like software to actually work.
I’m too wordy when I reply to reporters. This is mostly true everywhere I produce content. I don’t like trite, simple answers. Brevity and clarity makes me suspicious, especially on topics I know well. As a consequence, I don’t think this interview by email was ever published.
If you mean what I do, it means studying people and organizations who are trying trying to improve how they do software, summarizing all those, ongoing, into several different types of content, and then trying to help, advise, educate people on how they can improve how they do software. A loop of learning and then trying to teach, in a limited way. For example, I’m working on finish up a book that contains a lot of this stuff that I’ve found over the past couple of years.
Yes, those are the core tools. The traditional foundation is “CALMS” which means Culture, Automation, Lean, Measurement, and Sharing. Ultimately, these are things any innovation-driven process follows, but they’re called out explicitly because traditional IT has lost its way and doesn’t usually focus on these common sense thing. A lot of what DevOps is trying to do is just get people to follow better software development and delivery practices…ones they should have been doing all along but got distracted from with outsourcing, SLAs, cost cutting, and the idea of treating IT like a service, or utility rather than an innovation engine for “the business.”
- Culture – the norms, processes, and methodology IT follows. You want to shift from a project delivery culture to a product culture, from service management to innovation. Defining “culture,” let along how to change it and how to use it is slippery. I wrote up what I’ve figured out so far here.
- Automation – this is the easiest to understand of all the DevOps things. It means, to focus on automating as much as possible. If you find yourself manually doing some configuration or whatever, or relying on people opening a ticket to get something
- (Like a database, etc.), figure out how to automate that instead.
- Lean – software development has been borrowing a lot from Lean for the past 15 years. DevOps takes most all of it, but the key concepts it brings in are eliminated waste (effort spent that has “no value” to customers, in IT, often wait time for things like setting up servers and such) and working on incremental, more frequent (like weekly) releases rather than big, yearly releases.
- Measurement – DevOps, like agile, is actually very disciplined if done properly. In addition to monitoring your applications and such in production, in order to continuously improve, DevOps is interested in measuring metrics around process. How many bugs are in each release? How frequently do we deploy software? And so forth. The point is to use these measurements to indicate areas of improvement and figure out if you’re actually improving or not.
- Sharing – this was added after the initial four concepts. It’s straight forward and means that people across groups and even across organizations should share knowledge with each other. It also means, within organizations, having more unified teams of people rather than different groups that try to work with each other.
Shipping more frequently means you have more input on the usefulness of your software and it also adds much more stability and predictably into your software process. Because you’re shipping weekly, or daily, you can observe how people use your software and make very frequent changes to improve your software. There’s a loop of trying our a new feature, releasing it and observing how people use it, and then coming up with a new way to solve that problem better.
Stability and predictability are introduced because you establish a realistic rate of feature delivery each week. When you’re delivering each week, you quickly learn how much code (or features) you can do each week. This means that rather than having developers estimate how many features they can deliver in a year, for example, you learn how much they can actually deliver each week. Estimates are pretty much always wrong, and complete folly. But, once you calibrate and know how many features the team can deliver each week, they’re predictable and the overall process is more stable.
Yes, a monolithic architecture implies software that’s made of many different parts, but that all depend strongly on each other. To be frank, it also means software that’s complex, poorly tested, and, thus, not well understood. “Monolith” is often used for “software I’m scared to change,” that is, “legacy software.” In contrast, if you’re fine to change software and don’t fear doing so, you just call it “software.”
A microservice architecture is the current approach to break up “monoliths” into more independent components, different services that evolve on their own but are composed together for an application. Buying a product online is a classic example. If you look at the product page, it could be composed of many different services: pictures of the product, figuring out the pricing for your region, checking inventory for the product, listing reviews, etc. A monolithic architecture would find all of that information all at once, in “one” piece of code. An application following a microservices architecture would treat all of these things as third party, not under your control services and compose the page from calling all those services.
To over simplify it, we used to call this idea “mashups” in the Web 2.0 era: pulling data from a lot of different sources and “mashing” that data up into a web page. All the rotating ads and suggested content you see on news sites are a metaphoric example as well: each of those components are pulled in from some other service rather than managed and collected together by the news site CMS. This is why the ads and suggested content are often awful, of course: there’s no editorial control over them.
“Infrastructure as code” means using automation tools the building and configuring of servers (the software parts, not the hardware) and other “infrastructure” and then treating those automation workflows as if they were software code: you check them into version control and track them like a version of your application. This means that you can check out, for example, a version of the server you’re configuring and automatically create it. The point of doing this is get more visibility and control over that configuration by removing manual, human-driven configuring and such. Humans create errors, forget how things were done, have bad hair days, and otherwise foul things up. Computers don’t (unless those annoying humans tell them to).
An annoying, though accurate answer would be “it depends. I don’t really code anymore, so I couldn’t really say. Usually, you start with the minim needed and just add in more complex architectures as needed. That sounds like the opposite of architecture, but it’s worse to end up with something like all those giant, built out cities that end up having few people living in them.
Kanban is used a lot in DevOps, maybe not fully. But, the idea of having cards that represent a small feature, a backlog that contains those cards ranked by some priority, and then allowing people to pull those cards and put them in columns marked something like “working on” and “complete” is used all the time.
I’m not sure what “craftsmanship” is in this context, but it it means perfecting things like some master furniture maker, most DevOps people would encourage you to instead “release” the cabinets more frequently to find out how they should be designed than assuming you knew what was needed and working on it all at once: maybe they want brutalist square legs instead of elegant rounded legs topped with a swan.
And, of course, if “craftsmanship” means “doing a good job and being conscious of how you’re evolving your trade,” well, everyone would say they do that, right? :)
Private clouds owned and self-managed by enterprises can be cheaper than public cloud. The magic number to beat is about $25 per VM-month at 100% utilization. If the cost of the whole stack comes in under this number, then even with the addition of labor to manage that private cloud, it should be cheaper than public cloud. Obviously, with better labor efficiency, unit costs versus public cloud are lowered further, and the relative value of benefits increases. Enterprises unable to achieve a labor efficiency of 300 VMs per engineer are unlikely to beat public cloud on price.
Original source: THE EMERGING THREE-TIER PRICING MODEL FOR PRIVATE CLOUD
It doesn’t matter if DevOps has accelerated development. You need to accelerate the entire lead time from concept to cash.
Original source: 2018 Day One Recap: The “Project To Product” Movement Is In Full Swing
That’s why SRE is a Big Lie – because it enables people to say they’re doing a thing that could help their organization succeed, and their dev and ops engineers to have a better career and life while doing so – but not really do it. Yes, there have been Big Lies before, which is why I cite Kanban as another example – but even if the new criminal is pretty much like the old criminal, you still put their picture up on the post office wall.
If something you’re selling is profoundly misused it’s your responsibility to be more up front about the issues.
Original source: SRE: The Biggest Lie Since Kanban
Brief, but some names of apps they’ve modernized and high-level thinking in the broader portfolio modernization.
Original source: USAF’s plans for agile, DevOps, cloud
“Service management” considered legacy thing.
Original source: ITSM’s own “DevOps renaissance” won’t come from copying DevOps
Ops, still a thing.
Original source: No, Operations Isn’t Going Anywhere, But it’s Going to Look Different
“The metrics can be broken down into four broad categories — deployment rate metrics, response rate metrics, code quality metrics, and program management, assessment, and estimation metrics. The DIB also provides general timeframes for what a “good” score looks like for each metric.”
Original source: Defense Innovation Board proposes new metrics for assessing DOD software development