Dear Red Hat: Are you dumb?

I've had a busy week, so I didn't have time until today to read this news about Red Hat locking down RHEL sources behind a Red Hat subscription.

I repeat the title: Red Hat, are you dumb?

When Red Hat decided to turn the community CentOS distribution into a leading-edge distro instead of basically "Red Hat Enterprise Linux, but free", users like me were justifiably angered.

I don't contribute to CentOS or Red Hat development much, if at all. But I have, for over a decade, provided software and tools that were compatible with RHEL, Debian, Ubuntu, Fedora, Arch, and sometimes other more exotic distros.

I could test my stuff against CentOS Stream... or UBI... or Fedora. Those are mostly like RHEL. Or I could try linking a Red Hat Developer subscription to my test runners and build tools so I could use a licensed copy of Red Hat Enterprise Linux, because that would be required for... actually ensuring compatibility.

But that's dumb.

So at this point, I have to say: fool me once, Red Hat—shame on you. Fool me twice?

At this point I'm determining whether I want to continue supporting just Fedora, or just dropping all support for RHEL and RHEL-like distributions on my open source projects. It's not worth the hassle if I'm not even sure projects like Rocky or Alma Linux can fill in the gap left by CentOS's demise for users like me.

And no, please don't post "but you can use your Red Hat Developer Subscription!" I don't have to with Debian. Or Ubuntu. Or Arch. Or... you get the point.

Update: Just wanted to point out two official statements, one from Alma Linux, another from Rocky Linux.

Further reading

linux
red hat
open source

Comments

It pains me to write this, because there are many people within Red Hat who are amazing folks who I respect greatly. And Red Hat still supports and promotes a ton of great OSS work. I am just mystified by the direction they've taken their bread-and-butter Linux distribution and the community around it.

Also, see more context around the announcement from: Alma Linux, Rocky Linux

Yep, I worked at IBM for 4 years after the company I had been working for was acquired by them. The moment I read about them having acquired Red Hat, I knew that there would be some sort of “interesting” insanity before too long. By the way, I left IBM the moment the final retention bonus check appeared in my account.

It's not greed to try to support your shareholders who financed your over $30B acquisition. But it's also not a wise move yet again by IBM. Awaiting what the other evil empire (Oracle) does here with their RHEL clone....

Oh yes it IS greed indeed, a LOT of it! This whole thing has quite obviously been motivated by nothing but pure greed. Some beancounters at Big Blue have figured that if they'll try and force everyone into their (rather ridiculously overpriced) subscriptions then they'll earn $$$. Those with higher-than-room-temperature IQ and a sound mind however probably knew that this would backfire spectacularly.

It’s always “greed” when someone else volunteers their money and risks losing it or gaining a return on it.

When we invest our own money it’s a normal reward for taking risk.

Guess that’s how the dirty judgement human spirit operates, always judging one another. What hypocrites. We’ve been given this wonderful system of volunteer capitalism but if you look around and see others with more capital than we have something must be wrong with the system.

Yes... Ubuntu also has a subscription for some security updates...

For versions past their published public LTS window of 5 years. Right now Ubuntu 18.04 is just fell off support and would need these commercial subscription if you're not ready to upgrade to 20.04 (or 22.04).

Plus, LTS versions are really much less important in the Debian/Ubuntu world where dist-upgrades are relatively trivial in comparison to monolithic versions in EL.

I'm very disappointed but nor surprised. RH was early to the game and made good moves to become a PROFITABLE Linux company through authority over time. But this is also why I don't USE Red Hat, the management makes weird decisions sometimes that are very much user-hostile, like this, the CentOS purchase, etc. I have no doubt they'll be around for a long time, and they'll continue to do good work on OSS and make money, but as a USER I can't trust them.

Red Hat was always a company of B and C players. It's a company staffed by idiots, for idiots.

I'm as serious as a heart attack. When you compare Red Hat's "engineers" with SGI and Sun kernel engineers, well, the Red Hat boys are amateurs at best. They'd rather argue with their paying customers than actually solve problems, because most of the time, they have no clue where in that haphazardly hacked-together mess of patches upon patches the problem actually is. There can be no discussion about kernel engineering when it comes to Linux because there is no engineering to speak or write of. It's all hacking. So yeah: idiots, and whoever runs that mess in production is an idiot as well, for not knowing any better and not caring to find out if there is anything better, and how it all works.

They still publish the tags on the CentOS stream git repo. This is a non-story.

CentOS stream is beta software. It's not deemed stable, and is not bug to bug compatible with RHEL. What Alma and Rocky are bug-for-bug, drop-in replacements for RHEL, and they are being used by pretty serious infrastructures maintained by people who know what they are doing and need no support.

This move dynamites a lot of production systems compromised of hundreds, if not thousands of instances which run 24/7/365 without any support from RedHat.

If you've ever had to audit the CVEs on Rocky, you wouldn't say that. To Rocky's credit, they've long admitted they have a long way to go. Their web site and OVAL are not their focus. They are far, far worse than what CentOS was prior. Alma is in the same boat.

We've had to blacklist Rocky at our partners and subs for this reason. It's not Rocky's fault. It's people overselling their CVE tracking. Again, Rocky has been honest they have a long way to go in being able to track CVEs. And now with Red Hat cutting off the Source RPMs, other than Stream, I don't know if there is any reason.

BTW, please stop saying things like unstable/experimental. Facebook and Twitter were the big pushers to get Red Hat Engineering to release Stream publicly under the CentOS(TM) trademark. Yes, Stream has always been around, but only for internal, major accounts and partners.

Stream is all backported customer bugfixes and all security patches. Normally RHEL only gets high/important/critical security patches and nothing else, until the next Update Y+1 (RHEL X.Y+1) comes out.

E.g., if there is a memory leak in a package in RHEL 8 -- currently Update 8 (RHEL 8.8) -- reported by a Red Hat customer, and Red Hat reproduces it, and backports the fix to the RHEL package, and builds it as a hotfix, Stream will also take the patch, build it, and push it through the full IHV/ISV unit, integration and regression test suite. RHEL won't get it, not until Update 9 (RHEL 8.9).

REAL WORLD: I literally ran into a serious memory leak taking down RHEL7.7 & 7.8 systems on our Data Lake cluster, one of our most important systems at a top 25 US Bank. There were 2 issues that were found at 2 different customers, and I had to wait until RHEL 7.9, and keep us on an older, pre-RHEL 7.9 package. Had I had access to Stream, I would have gotten both customer hotfixes instead of having to wait a year.

Think about it this way. That same bug that you found in 7.9 was in Stream long before it was pushed down to RHEL (and it made it all the way through "testing"). So in reality you would have probably had the bug even earlier if you were on Stream.

There is a reason why we say its not for Production use. Yes, you get the fixes faster, but you also get the bugs faster too. That's why its "free" because you just became the beta tester.

Just a small correction. Stream is not beta, its alpha software. We build the RHEL Betas based upon the CentOS stream. So what comes before Beta? Yep, Alpha.

I've been a huge supporter & user of CentOS for nearly 2 decades. I've also been a supporter of Rocky linux since they were announced.. But I no longer use Fedora/CentOS in my home labs. I've switched over to Ubuntu.

I don't even provide software to others, but as someone who depends on reliability for the companies I work for we've started using far less Red Hat software.

My new parent company has even switched to Oracle linux over RHEL because of their crazy behavior. So even people who used to use direct RHEL have switched over to Oracle which I dislike. But there's nothing I can do.

I honestly cannot believe people are holding up Oracle as an option here. I could go through the last 15 years of that mess! Oracle even admits they break a lot when you corner them on support tickets.

You should change your title to 'Dear IBM: Are you dump?"

I was taken by surprise as well, this was not discussed internally (at least not that I was aware). Someone already noticed Jeff's post and called it fallout. This personally pisses me of the wrong way.

A reminder that free software is not "gratis" but "libre" and you can always charge for it.

How bad is this from a marketing point of view, that's not for me to understand since I don't work at RedHat... That's a reminder that corporation behind free/open source projects its always dangerous compared with community driven projects like Debian

"Many people believe that the spirit of the GNU Project is that you should not charge money for distributing copies of software, or that you should charge as little as possible—just enough to cover the cost. This is a misunderstanding."

https://www.gnu.org/philosophy/selling.html

I don’t think gratis is the problem.

The problem is even if you are a paying customer (or using the free developer account), redistributing the software can result in them dropping you as a customer or even lawsuits per their agreement.

I agree on most everything you stated. However ...

I wouldn't use LTS as an argument against Stream. They are a lot more alike than not. E.g., there is this catch-22 now ...

LTS fanboy (then): "I cannot believe Red Hat hasn't patched this customer bugfix or security hole that is already upstream! Even Fedora has it!"

LTS fanboy (now): "I cannot believe Stream is backporting and patching all bugs and security holes immediately! That's unstable!"

RHEL is for when you only want high/important/critical security fixes, and nothing else. You wait for the next update X.Y+1 for the roll-up of all customer bugfixes and security patches.

Stream is for when you want all customer bugfixes and all security fixes as they exit the Red Hat build & unit-integration-regression test system.that feed all RHEL packages as well, but are not held back.

Can't have it both ways. But I agree it's wrong for Red Hat to cut off the SRPMS. But I wouldn't use LTS as an example against Stream. Stream stands on its own very, very well, right down to the backporting and kABI of the RHEL release. It just doesn't wait for the next update to release bugfixes and lower risk security patches.

Your answer is kinda dishonest about the fact that fanbois are whining regardless of what RedHat does. No, enterprise-grade distros really do NOT want bleeding-edge (i.e. barely tested) bugfixes, because they use those servers for production and would lose big if they encountered problems after minor version upgrades. Thus no, they don't want "all customer bugfixes and all security fixes as they exit the Red Hat build & unit-integration-regression test system" and now some of them are left with no choice but to switch to some completely different distro.

At DEC we had different types of releases. Many of our customers *never* installed a "major release" (2.0, 3.0, 4.0, etc) but only instated "big fix releases" with little or no new functionality. We also had "new hardware support" releases, with *no* bug fixes in them (since customers paid for bug fixes) and customers wanted their software "bug for bug compatible".

Huh? Stream is *backported* fixes to *stable* RHEL releases. They just go out as they pass the IHV/ISV test suite, instead of Red Hat 'holding back' until the next Update Y+1 (RHEL X.Y+1). E.g., when a customer gets a backported bugfix, it is also patched into Stream 8, but not the current RHEL Update 8 (RHEL 8.8). It won't go into RHEL 8 until Update 9 (RHEL 8.9). Please don't give me the 'barely tested' BS. It is literally a backported customer bugfix.

You better write "IBM are you dumb?" . IBM ... when Idiots Become Managers is between my favorite ones (ex IBMer here).

I'd more than lost faith along the way with RH (Linux user since 2000) once the hard sell began for RHEL vs. CentOS began long before Stream, and Fedora was just always some bastard cousin of theirs to me. The IBM acquisition just signaled doom, where anyone that's been around this industry any amount of time knows no good will come from that. IBM will never innovate, they'll simply squeeze the rock harder to bleed it for pension payments.

Full time Ubuntu user since 2006, later Arch since 2018. I only do Cent/RH when a vendor product requires only it.

Given the effect the decision by IBM to cut access to the source has on the market, which effectively considers RH clones as public infrastructure, why hasn't the USA Federal Trade Commission stepped in, especially given the lock in through the OEM agreements with Microsoft & RH?

For example as with the Telecom industry attempt to move away from the Network Neutrality model in 2006.
https://itheresies.blogspot.com/2006_07_01_archive.html
https://www.ftc.gov/news-events/news/press-releases/2006/08/ftc-chairma…

When you consider how many business, organisations, governmental services & just people use services hosted on CENTOS clones.

The main problem is that OEMs test & even validate server/workstation/desktop/laptop hardware for both Microsoft & RedHat OSs on the OEM provided hardware, under agreements which MAY have caveats that effect competition.
Currently you can get around this by when you purchase, lease or collocate OEM hardware originally purchased with the NO-Operating-System option or more likely second hand, but if the hardware has been tested with Red Hat Enterprise Linux it should work as expected under CENTOS clones.

It opens the market to other providers as does Telecom Network Neutrality.
IBM's decision to limit source access under any licence limiting redistribution significantly changes the market and should be investigated by the FTC and other competition monitoring agencies in the EU & worldwide.

From
https://lwn.net/Articles/936242/
""First of all, I fully acknowledge the effort the Red Hat, along with hardware vendors, has put into creating a stable Linux kernel & the distribution upon that platform. Red Hat & even IBM has been a major contributor to the open source & free licensed software community.
However ANY business decision that when implemented would significantly impact the market as a whole rightly deserves the full attention of Antitrust authorities, especially if the effect is world wide. ( The EU needs to look into this ASAP ).
CentOS ( released 14 May 2004 ) & Scientific Linux were created by government agencies, under the terms of the GPL & other open source licences, not based on Debian sources or other Linux distributions but based upon Red Hat Enterprise Linux sources because of the major issues of hardware comparability with off the shelf server hardware.
As I have pointed out "The main problem is that OEMs test & even validate server/workstation/desktop/laptop hardware for both Microsoft & RedHat OSs on the OEM provided hardware, under agreements which MAY have caveats that effect competition."
The issue is ongoing as new processors & hardware is being released constantly & will become an even more significant problem with the increasing deployment of newer APUs & other application acceleration hardware rolled into upcoming CPUs, motherboards & IO cards.
For the same hardware compatibility reason CentOS became the de facto distribution in countless VPS, cloud services, businesses & governmental organisations, who choose to support their deployments either internally or through third parties.
Even direct business competitors, such as Oracle's Linux distribution, chose to base off of CentOS exactly for the same reason.
There is now a significant market based upon both providing software & services upon that platform.
IBM's decision to limit the distribution of the sources by customer licences DIRECTLY impacts that existing market.
There has been a distinct advantage to Red Hat from the ubiquity of CentOS widespread deployment. It has remained the leading paid provider of services for the Linux platform & also collects and deploys many patches for fixes to its own Red Hat Enterprise distribution contributed by other companies & users of CentOS based distributions. These advantages were so significant that when the original CentOS maintainers began having difficulty keeping up with Red Hat patches, they folded the project into Red Hat itself.
Later Red Hat's subsequent removal of the stable distribution of CentOS rebadging it as CentOS Stream led to the inevitable creation of forks of the original CentOS kept tracked to Red Hat's Enterprise Linux Release.
https://itheresies.blogspot.com/2005_04_01_archive.html
"No vendor or open source software developer can block development for any substantial period of time without the risk of the development being taken over by a descendant of the same project -- it's called evolution."
Which, IMHO is a bigger risk to IBM's bottom line future profits from Red Hat. If the users of CentOS clones cannot access the same source then they will be forced to join together with upstream open source developers, vendors of all sorts & even antitrust authorities to force open equitable hardware vendor information access & create another single source distribution outside of the control of Red Hat & IBM.
As for the software I have written over the last three decades on behalf of my former employers, almost all of it has just been used internally although plenty of patches I have have added were contributed back to upstream open source projects under my former employers discretion.
Upstreaming is almost always easier than internal fork maintaining.""

From Cory Doctorow's enshittification article. Feels like this applies here too:

Here is how platforms die: first, they are good to their users; then they abuse their users to make things better for their business customers; finally, they abuse those business customers to claw back all the value for themselves. Then, they die.

Linux has became too valuable and big companies are going to do all they can to monopolize it. IBM with RH, Microsoft Github / Linux Foundation, Oracle Linux, etc. I agree that IBM management is terrible and destinated to irrelevance if they continue with their short term thinking.

And now I'm thinking about (gulp) Ansible. Surely the powers that be at IBM wouldn't... Oh. Right .

2 things…
1) why is an article about tuning asahi Linux on Mac M1 cpus listed as relevant?
2) it looks to me like RedHat has pulled the blog posting, but have not put up anything (yet?) acknowledging that they screwed up, are backtracking etc.

So, I wonder what lawyer who got promoted to management approved this strategy, and will then attempt to label it "Open Source", "green", and while you are at it misusing terms that mean something legally, "organic".

Looking at the open letter from Redhat https://www.redhat.com/en/blog/red-hats-commitment-open-source-response…

I mean, they are not wrong. Rocky and Alma linux are taking their work, repacking it giving it to enterprise users for use in enterprise situations without giving anything back to the upstream project. Unless I'm missing something, that's a very anti-opensource practice. When you look at it from Redhat's perspective, I can understand the frustration.

"Taking their work". That's were you went wrong with your argument. Mike is also wrong when he made the same argument "We don’t simply take upstream packages and rebuild them."

Even as a Hatter, I will admit that a majority of the work that goes into RHEL is "not our work". RHEL is based upon thousands of packages, a majority of which Red Hat has never contributed to. So yes, in a lot of cases, we are taking an upstream package and rebuilding it without contributing anything back to the project. Yes, we employee a lot of people that are here to help "make Open Source better" by contributing back to Open Source. We do that because our customers are technically paying us to do that. We make a vast fortune off of RHEL, so of course we are going to try and improve it to keep the money rolling in, otherwise why would customers continue to pay us?

The true blowback from this source decision is that it shows our true nature. We are just another profit driven company and we do things truly for our own benefit, not the community. That's really just a side effect of our own ambitions because we have to abide by the licenses also. In truth we don't want any competition. We develop on Open Source software (and also reap the benefits of the community) but believe that we are the only ones that should profit from it.

When someone else comes along and wants to do the same thing then we have to protect our fiefdom. We say things like "but they aren't contributing back!". Which is technically wrong, as I have seen bug reports, patches, etc.. from both people associated with Alma and Rocky (and lots from Oracle). It may not be to the scale that Red Hat does, but then again, we are several magnitudes larger than Alma or Rocky and they are extremely new (and were only formed because of our own blunder). I imagine both of them are the same as Red Hat was when it was young. They may not be huge committers now, but give them time to learn and grow, and the community will benefit greatly from their involvement (in commits, the community already benefits greatly from their builds).

Phoronix seems to have a plague of RH/IBM shills repeating their false claim that downstream distributions are "freeloaders" from RH's hard work.

And I assert that it's a false claim because unless Red Hat pays a royalty to every single copyright holder of every single piece of open-source code shipped in RHEL, then their behaviour is no different.

I've found that "freeloader" mentality quite pervasive in RH's professional services teams for a while. Years ago, I told one RH would be a 100 million dollar company if not for CentOS after he told me RH would be a 10 billion dollar company. Told him most of today's RHEL admins cut their teeth on CentOS of old. As a SME in the server world, I can no longer recommend RH as a vendor after the last several years of engaging their professional services teams with the problems I've seen. And I agree with you, Emily, RH is a freeloader of many open sources works too.

This is the price that Linux operating systems end up paying for not having a basic ABI supported within the community as a whole.

The kernel developers go out of their way to make certain the kernel interface maintains compatibility, but the rest of the operating system might as well be a jungle of incompatible software. Until that question is addressed in some measure, the insanity of being caught up in competing interests with each corporation making incompatible releases will never end.

It's actually that very same 'jungle' that gives Linux its vigor. I say this without in any way trying to negate your basic point...

The PROBLEMS that people use Linux to solve aren't all similar, and thus there's a range of solutions that each target a different local maximum of fitness.

I have (recently) been involved in linux-based solutions that ranged from giant systemd-based enterprise systems, through special-task configurations that trace their lineage back to bsd /etc/rc setups, all the way down to single-static-binary initramfses deployed on disposable nodes (hazardous environment; they often don't survive, even though we'd love it if they did).

Linux isn't a distro. It's one component in building anything from a smartwatch through a supercomputer. Or a Marscopter...

The problem with redhat is that they are trying to make money out of a commodity. They should try to make money based on services and products that run on top of Rhel, not on Rhel itself. Doing the latter is an old business model.
Keeping the source public will ensure cooperation with the open source community and increase quality based on which they can innovate. This is a typical case of a company trying to make more money out of an old product because they simply can't innovate in other areas.

The timing of this is what is funny to me. I have my own theory and I seriously doubt its because of other companies selling support for their clones. Oracle has been doing it since 2006, and Rocky (via CIQ) has been since 2021. There were numerous companies out there that also sold support for CentOS for decades. So I don't think the selling of support is the issue. Except for Oracle, the others are no different than the remora is to the shark.

So what is the issue. Well Mike mentioned on reddit that it had to do with one of the rebuilders and "bad faith".

My theory? Its about personnel. To be honest, things have been bad at Red Hat for a while. Its been going down hill since the IBM acquisition and IBM is really starting to assert their influence. The Red Hat culture that we all loved is mostly dead and gone. The last year has been particularly bad (which then led to the layoffs). I see very talented people leave daily. If you go check LinkedIn though, you can see where some of them are going.

CIQ is small company currently, but they are building up their teams with some people that have years (and in some cases decades) of talent in Red Hat products. If the internal rumors are correct, I think Red Hat is pretty teed off about a few people that left to go there (especially one in particular that left at the timeframe that Mike mentions), so really this is just retaliation for what they see as "poaching".