passthejoe — the blog

Goodbye CentOS Stream 8. I guess we didn't click.

Writer’s note on March 3, 2024: I think I took this post off my CentOS blog at some point because it might have been too “mean,” or maybe just too “critical.” I felt bad. Being the bull in Red Hat’s CentOS china shop wasn’t a fun role. Since this post was first written, CentOS 9 has been released and feeding into RHEL 9 for some time. I think the way it still goes is that development happens in Stream 9 and then moves to RHEL 9. But critical security patches begin in RHEL 9 and eventually make their way to Stream 9. I’m not sure if there is a regular cadence, but the evidence of patches — especially to Stream’s Linux kernel — really only exists in the changelogs on commits in the CentOS Stream Gitlab.

As I’ve shouted into the Cheeto bag, all I wanted for CentOS — the project — was for it to be run like Fedora. Like the four F’s — maybe a little less “first,” but just as much/many “friends,” “freedom” and “features.” I asked on the Fediverse if there is an active CentOS Stream community somewhere. I suppose there are SIGs. https://CentOS.org is pretty dead. The mailing lists are thin. Nothing much on Reddit. Very little on Fedora Discussion.

At the time, I thought I was advocating for users. I was just pissing off (vocal) Red Hatters. Who’s advocating for CentOS users now? I’d love to find those people, or person. I think Red Hat did a pretty good job of killing the CentOS user community. I’m not sure if that’s what they wanted to do. To the best of my knowledge, it happened.

It does trouble me that Fedora is so good. It is excellent technically, and it encourages the participation of a wonderful community. I think about all the stuff that has happened at Red Hat in the past few years: The troubles with the end of CentOS Linux, the beginnings of CentOS Stream and (possibly worst of all) the restrictions on RHEL source code and patch announcements. Is it enough to put me off of Fedora? It’s rough. I do love the distro and the community. But are we right, as Fedora users, to “help” make a RHEL that’s not anywhere near the open-source exemplar that it was in years’ past? I can’t answer.

Writer’s note on May 9, 2022: I wrote most of this post in January 2022, but held off on publishing due to my respect for Rich Bowen, then the community manager for CentOS. Maybe I was a little hot-headed. But things got a bit worse for me in terms of things not improving in Stream 8 and Red Hatters who weren’t Rich pretty much telling me to shut the fuck up. Long story short, I moved to Fedora Silverblue and have been there for a few months. But I felt it was time to publish the post. I added bits, mainly Red Hatter Carl George’s Twitter thread basically outlining the shortcomings in the Stream 8 workflow that prompted me to begin complaining about the distro’s processes in the first place.

I thank Rich for his understanding and patience, but now it’s time for this post to appear.

I’ve started to see the promised improvements in security patches for CentOS Stream 9, which I don’t think is technically a beta release, though RHEL 9 still is a beta at the time of this writing (May 2022). So all is not lost, and I may very well give CentOS Stream 9 a try. I’m running it in a VM right now, as a matter of fact.

Before this gets tl;dr, here are the highlights:

What I learned is that Red Hat proper doesn’t really know how to deal with users who aren’t paying customers. They are contemptuous of those who aren’t spending with them. Fair play, but DON’T HALF-ASS A MAJOR OPEN-SOURCE PROJECT as a result. Just say you’re uninterested and stop doing it. They did that with CentOS Linux. I imagine that when CentOS Stream 9 is well underway and has a better workflow from a security standpoint, Stream 8 will fade into irrelevance. Or that won’t happen. Who knows?

The original post (with some new bits)

I stopped running CentOS Stream 8.

I can see myself returning to CentOS Stream 9 after RHEL 9’s release, but I’m done with Stream 8.

In large part my decision to stop running CentOS Stream 8 was motivated by CentOS and Red Hat culture. It’s pretty obvious: I don’t fit. Also, Red Hat — despite its dominance in the paid-Linux space ­— is a little broken from a cultural standpoint.

The situation with the CentOS Stream 8 kernel has puzzled me ever since I began running the system. Was the kernel adequately patched, often vulnerable, or do kernel patches not matter as much as people say they do?

I really don’t know, but as I would post on Twitter about the quixotic processes at work on the Stream 8 kernel, I got a lot of pushback.

This is how I felt the responses broke down:

20%: “Trust us, we are doing this right, even though it doesn’t look that way”

40%: “CentOS 8 Stream’s processes are kind of messed up, and we will get everything right in CentOS Stream 9”

40%: “Please shut up. We like to handle things quietly, and you are not quiet.”

My contention was — and is — that CentOS is fighting for its life. It is bleeding users to Alma, Rocky, Ubuntu, Debian and just about every other server-friendly distribution.

Red Hat’s decision to “replace” CentOS Linux 8 with Stream 8, a system with opaque processes and a upstream “lead” over RHEL on most packages but definitely not the most important one — the Linux kernel — was playing with fire.

The more I looked at updates in Stream 8, the more I saw that just about every patch that had to do with security was not just days but weeks behind RHEL. And there is no errata, no e-mail, NOTHING to tell users what’s going on. Just wait for the patches and trust us seems to be the way.

An interlude in which Carl George berates me one month, then de facto agrees with me in another

I don’t know if I’d call it surprise, confirmation, grudging admission, or something else, but one of the Red Hatters who was very unhappy with me questioning Red Hat’s motivations or processes, finally did tweet pretty much what I have been saying since I figured it out. It’s Red Hat’s Carl George, who blocked me on Twitter over this very issue:

https://twitter.com/carlwgeorge/status/1515064280185552897

https://twitter.com/carlwgeorge/status/1515064522163343369

https://twitter.com/carlwgeorge/status/1515065010166374402

https://twitter.com/carlwgeorge/status/1515065570844024835

(Note: I think the above tweets can only be seen by “approved followers,” of which I am not one)

I’d have preferred that Carl George had not blocked me on Twitter. I’d have preferred that the legendary Johnny CentOS had not told me to piss off.

But those things did happen.

I’d also prefer that Carl George not soft-pedal the fact that the “least common” packages that fall under his “Scenario 3” in the tweets above are all the packages that generally get critical security patches, including the LINUX KERNEL.

Red Hat is a strange place. It’s seems paternalistic in its way. What I got was: “Don’t question, don’t complain, trust but don’t verify, you’re lucky to get these free bits, stay quiet.”

I’ve tweeted about security issues in other distros — primarily Debian — before. And while I’ve gotten fanboy apologists telling me I’m wrong to question why a web browser hasn’t been updated in half a year (a situation that has since been rectified), I never got project members aiming to bully me into silence. Only with Red Hat.

From a whisper to a Stream

Killing off the traditional downstream CentOS and only offering Stream put a heavy, self-imposed burden on Red Hat. With old CentOS, users could rely on the RHEL security advisories because the patches were the same as RHEL’s. But now much of the time (who really knows?) Stream 8 updates first, except for those times when RHEL does.

Should you equip a production server with a Linux distro that can’t — or won’t — publish security alerts that appear on Linux Weekly News? For me the answer is no.

Red Hat has to get this right. And NOW. CentOS Stream 8 should be fixed. Eventually CentOS Stream 9 will have a full EPEL, and RHEL 9 will be released, and I have to hope that all will be right when those things happen.

Seen from the outside, Red Hat has a dysfunctional culture, especially when it comes to CentOS. From what I can infer, a lot of Red Hatters would rather put the CentOS Project in a burlap sack with a bunch of heavy rocks and throw it into a deep lake.

Yet CentOS Stream 8 had enough juice to both happen in the first place as well as be positioned as the upstream to RHEL, even though it’s not an upstream at all for things that matter — like security patches for critical packages.

I don’t work for Red Hat. I am not a Linux system developer. I am not a packager. I’m just a user who writes a bit. I have no skin in the game.

What bothered me more than anything was that the few of us who brought up the issues with CentOS Stream updates pretty much stood alone.

When Red Hat decided to end the traditional downstream CentOS Linux, there was a sea of pitchforks.

But when its “replacement,” CentOS Stream 8, fell behind in security patches, those former pitchfork-holders fell silent.

I’m pretty sure they all just moved on. Smart move.

I hope things change, but right now I can’t recommend CentOS Stream 8 — for use on servers or desktops. If you want to stay in the Enterprise Linux world, “real” RHEL, or free downstreams Rocky and Alma are the way to go.

In particular, Rocky and Alma are bending over backward to create community, develop infrastructure and give users what they want. For me that’s speedy security updates.

Outside of the EL world, there are Debian, Fedora, SUSE and Ubuntu.

And I do think Stream 9 will be better. We’ll have to wait (for RHEL 9) and see.

Then there’s Fedora. Hey, that’s a Red Hat product in some way. How do they get it so right?

Fedora probably had growing pains aplenty in the years after free Red Hat Linux was hammered into paid RHEL. Now Fedora has a clear purpose, solid technical processes and an excellent security record. It also embraces its contributors and users — and wants everybody to be involved.

That’s where I went. I decided to give Fedora Silverblue a try. It’s a whole new way to build a Linux distribution, with an immutable base, Flatpaks for most GUI applications and Toolbox containers for CLI development. So far it’s working great. And I don’t have to worry about security because I’m getting new kernels all the time, and I can go to LWN.net — or follow Fedora’s Package Announce mailing list to find out what’s going on.

Silverblue is new and novel — kind of like Stream 8, just more so.

CentOS says it’s looking for community. I suspect their idea of community is a bunch of companies like Facebook with seas of developers, some of whom are eager to contribute back to RHEL via Stream. I hope it works out for them. I’m pretty sure that pushing back on users who find serious issues is not the way to build that community.

#centos_stream_8 #rhel #red_hat #centos #centos_stream