Source Code

I’ve almost figured out the drama around Red Hat redoing how they distribute the source code for RHEL.

Is this it?

Here is my understanding of it:

1. Before the changes, anyone could get the source code (and I assume configuration, build methods, etc.) so that they could create an exact copy of RHEL (“bug for bug,” as people like to say).

2. This means that “clones” of RHEL (or distros) could be relied on to act exactly like RHEL.

3. So, if you did not want to pay RHEL for access to RHEL and the support they’d give if things went wrong, you could get RHEL for free, or some discount from the clones (including Oracle).

4. The drama was: RHEL no longer provides that source code.

5. What this means, exactly, is incredibly confusing. Red Hat still develops future versions of RHEL all in the open in a project called CentOS.

6. HOWEVER - and this is key to everything, so if it’s not accurate, I am still super confused -, as Jonathan Bennett wrote here: “CentOS Stream just isn’t quite the same as RHEL, and there are bugfix patches that land in RHEL and not in CentOS Stream.”

7. That is, whatever is in CentOS is not exactly the same as what’s in RHEL.

8. Now, you can actually get access to the source code for RHEL, and thus the ability to build it on your own and get the exact copy as in #1 above.

9. BUT! There are new restrictions on that: you have to be a customer (I don’t think it matters if you pay or not) or be in their developer program (which is free).

10. There’s some open source licensing (GPL) “license interpretation innovation” going on here.

11. The thing about being a customer or in the developer program, is that Red Hat could decide to terminate you as a customer or your membership in the developer program at any point.

12. Thus, a cloner could sign up as either, get the current (and presumably past) RHEL source code. They could then build the exact copy in #1, and go about their business (literally and figuratively).

13. However, if Red Hat doesn’t like this, they could terminate that cloner, cutting off all future access to the source code.

Control

And that’s the drama: effectively, Red Hat now can control who gets ongoing access to RHEL source code.

It’s very hard for me to figure out if these details are correct. It all hinges on point #6 above. There’s another version of point #6 that is less “bad”:

It could be that CentOS is actually “bug for bug” compatible with RHEL. However, I think Red Hat said they will only sync up RHEL to CentOS for every major release of RHEL. That seems to happen every six months. This means that whatever happens in between every major release - in particular bug and security patches, the most important things you’d want! - is held back by Red Hat. So, if you went with a clone of RHEL, you’d have to wait that six months to get your bug and security patches. I don’t know though: that seems like some real tin-foil hat shit, and what does it even mean to “sync” code: is Red Hat maintain a private repo of all the patches and then they just do a pull request to the CentOS git repos every May and November. I mean…that’s not technically impossible.

To simplify it a lot, if I understand this correctly, first, even though RHEL development is done in the open, it’s only mostly done in the open. Some of the code is held back, particularly when it comes to bug fixes and security patches.

Second, Red Hat can control who gets access to the real RHEL source code.

This means that you can’t clone RHEL reliably, which means that if you’re running your organization on software that is only supported and verified to run on RHEL, you now have to run on RHEL and, thus, pay Red Hat.

I mean, that organization could run on not exactly RHEL clones and will probably be fine. The clones will have to figure out and distribute bug fixes and security patches, presumably. But no one wants say “yeah, the software that runs our factory, makes sure shipments get from Rotterdam to LA, makes sure the missiles don’t accidentally land in the suburbs will…probably work…there’s probably not a problem or incompatibility in the operating system.” Instead, those organizations will want the assurance (and, like, finger-pointing and support) that it’s all certified on RHEL and that they’re running actual RHEL.

Of course, you could come up with a scheme to get access to RHEL source code. The cloners could hire anyone off the street to sign up for an account and download the source code again and again, burning a developer account each time. Once that person had the source code, as I understand it due to the open source licensing, that person could give the code the cloners. Or just put in online for anyone to download it. Once you get a copy of the source code, you can then give it someone else if you don’t mind your developer account being canceled.

I assume you have to be 18 to get a legal developer account with Red Hat. There’s something like 2.6 billion people in the world over the age of 20. So, I don’t think the clones will run out of people to hire to fill out a form and click a button: the clones could automate the process as well.

You could even see people crowd funding this: you just pool together something like $5,000 a month to to hire someone for, like, $500 to fill out the developer program forum and then run a script to download the RHEL code. So whenever a patch comes out, you do that, and you can get an exact copy of RHEL, per #1 above. Someone like Oracle would probably be like, “here, how about $15 million?” Then you could even setup a foundation and get a fancy executive director!

This is a high degree of shenanigans, however. I can see the clone sales rep trying to explain it all to a CISO at, like, Citibank or the German Air Force and that CISO going like “hmmm…how about I just spend $20 million more every three years and totally eliminate a weird bucket of risks? I mean, I’m the CISO, all I have to do is sneak up behind the CEO and CFO and go ‘SECURITY! BOO!”, pick up the spare change that falls out of their pockets when they jump-scare, and then my budget is pretty much unlimited.

“I’ll get my Red Hat (or, even better, IBM!) rep to buy me some steak dinners a few times every year, fly me out to Vegas for the annual conference, be on a panel at RSA for them, and we’ll call it a day. (Also, we’re supposed to be running all our shit in THE CLOUD by 2028, where we won’t be using much, if any, RHEL, so…whatever?)”

So…?

Did I get that all correct?

Consequences

More than just control, all this means that more people would likely become paid RHEL customers. This is because the clones would no longer be reliable enough (see the “probably” scenarios above). There’s probably not too many enterprise customers that have reduced their bills by paying clones or just running without paid support. (I’ll let someone else look up the marketshare). I could see that a lot of telcos would be doing this, though. Some managed service providers, maybe even the big cloud providers. Still: how much extra revenue are we talking about: is it $15 million, or more like $2 billion?

The whole social contract, norms, feels and vibes thing is a completely “orthogonal” thing as the people involved might say. I just want to make sure I get the understand the new constraints in the system.

Developers

Also, in the developer program, you can only run 16 simultaneous instances of RHEL, which would put a damper on some companies, who, for example, would want to certify that storage arrays are compatible with RHEL. For example, if you’re a hardware manufacturer and you want to run 100 tests (I don’t know, for stats or some shit) for the new version of RHEL to make sure it works on your hard drives and computers, you’ll have to do that in chunks of 16.

Let’s assume that testing takes three hours. You need one hour of setup time, one hour to run the rest, and one hour to analyze the results. If you could run then all at once, then it’s just three or four hours to run those tests (plus the time to rig up the hardware and stuff, but let’s assume you’re really good at that by now, so it takes, like, two hours at most to do all of it).

If you have to do that in batches of 16 [is this the right math: ( (100 * 3)/16 ) * 3?] then it will take 56.25 hours instead. That’s 2.3 full days (24 hours), and seven working days (8 hour days).

So, you could say that your costs became 2x to 8x (mostly in labor), depending if you already were running on three shifts (day, evening, midnight) or just one (day). This would also mean that to fully certify a patch, you’d need seven days. This means that, if a security problem came out on any other day than a Monday, and in the morning, you’d need people to work on the weekends, or have them work overtime.

And then, if had to patch previous versions of RHEL, you need to multiple that by however many versions back you go that needs the patch. Obviously, all of these numbers are made up, but you can see where there’s a model here for figuring out how only being able to run 16 simultaneous RHEL instances (for free) would change things a lot.

Now, I would imagine that a lot of the people, rather, organizations that are doing this testing already have an extensive, long-term customer/partner relationship with Red Hat. These would be from the likes of Dell, EMC, Intel, AMD, SAP, Microsoft, likely the public cloud providers, etc., etc. So, they probably don’t care. It’d be the smaller, even down to just one individual who’d suffer from the above.

And, I guess, if you were working a system or software that was intended to run more than 16 instances of RHEL - like most parts of Kubernetes, OpenStack, scaled out databases, etc. - you’d be kind of screwed too if you wanted to do it all for free.

Wastebook

• Goofball training - leaving my wallet behind is good practice for relaxing more, moving to a carefree lifestyle where I don’t worry shout all that.

• “There is so much to worry about, that we forget to worry about ourselves and then it’s too late.” Here.

• I don’t know what to do with Bluesky. I’m not really into the nature of truly keeping up with what people are thinking and doing - that’s Facebook, right? And I’m not big fan of that activity in Facebook already. So, I’ve just been posting a pictures a day there. But, really, until I can just automate it, I don’t think I will anymore. Thus far, I haven’t really found a point in the post-Twitter Twitters. This is due to the huge follower-ship I’ve built up in Twitter since, whatever, 2006/7 - that’s a big asset of mine, professionally. In comparison, the use of Mastodon and Bluesky as a publishing channel is not good. Plus, I’m increasingly tired of, like, trying. After all, my bigger project of is commiting selective infosuicide.

Relevant to your interests

• A modern-era lift and shift: Danske Bank seeks massive cost-to-income ratio improvement with the help of Infosys - As ever with outsourcing: hire cheaper labor under less restrictive labor laws…?

• 28 Questions to Ask Your Boss in Your One-on-Ones - Good things to talk about in your meetings if you can’t think of anything else.

• ‘Zero trust’ was supposed to revolutionize cybersecurity. Here’s why that hasn’t happened yet. - ’John Watts, a Gartner analyst, wrote in the firm’s annual predictions memo from last December that “moving from theory to practice with zero trust is challenging,” and that fewer than 1% of large enterprises are actually using it today…. Moreover, Watts predicted that “over 60% of organizations will embrace zero trust as a starting place for security by 2025 but more than half will fail to realize the benefits.” A report from Nathan Parde of MIT’s Lincoln Lab last May, meantime, estimated the typical zero-trust deployment will take anywhere from three to five years. That is a depressing thought, to be sure.’

• Midjourney and Adobe Stock - Using AI to make stock photography, etc.

• Mobile Industry Eyes Five Billion ‘Dormant’ Phones Sitting In Desk Drawers For Reuse Or Recycling - 5 billion phones sitting in desk drawers.

• The real story of how Facebook almost acquired Waze, but we ended up with Google - “Acquisitions are the first moment when founders and investors have diverging interests. This is the one time when you should be wary of feedback from your investors. And, of course, like any negotiation, whoever is willing to walk away will get the better deal – always have a red line and hold firm to it.”

• Beyond belt-tightening: How marketing can drive resiliency during uncertain times - What’s a little crazy is: shouldn’t this all just be the way marketing always works?

• Red Hat ends the RHEL clones’ free lunch - I still don’t understand exactly what source code is unavailable and when. If CentOS is the “upstream” code…then getting an older version of the CentOS code will get you the current version of RHEL code, righ? Is it just annoying to do…or is it actually not at all possible to build a “bug for bug” copy of RHEL now without paying or having the dev license…because the actual RHEL code is actually not in CentOS? Or the magic is all in the extra stuff Red Hat adds to the RHEL build (like drivers, config, etc.), which used to be no strings attached available, but is now not?

• How Google Reader died — and why the web misses it more than ever - “At its peak, Reader had just north of 30 million users, many of them using it every day. That’s a big number — by almost any scale other than Google’s. Google scale projects are about hundreds of millions and billions of users, and executives always seemed to regard Reader as a rounding error.”

• Fuel for Thought: The Transformation of Auto Retail – Bricks, Clicks, and People

Upcoming

Talks I’ll be giving, things I’ll be doing, places I’ll be going.

July 4th, July 11th Cloud Native for Financial Services talk series. August 21st to 24th SpringOne & VMware Explore US, in Las Vegas. Sep 6th to 7th  DevOpsDays Des Moines, speaking. Sep 13th, stackconf, Berlin. Sep 18th to 19th SHIFT in Zadar. Oct 3rd Enterprise DevOps Techron, Utrecht.

Logoff

Hey, I really just am curious to figure out the stuff above. I do not have schadenfreude tingle-dreams here, or whatever. I just like to pull apart and understand puzzles like this.

See y’all next time!