Power of the Red Hat Ecosystem

After what seems like a lifetime I have had to dive back into the world of Linux distributions. Digital Ocean, my trusted service provider, decided to stop supporting FreeBSD (my response tweet).

I will write about my explorations of finding alternatives.

Disclaimer: I currently work for a coopetitor of Red Hat. These thoughts are my own and do not represent my employer.

I have been using FreeBSD and OpenBSD in some form for years now. They have served my various needs admirably. But always at home or in hobby projects. My day job is all about Linux, containers, and Kubernetes. After the chaos that is the public cloud and container world, FreeBSD always felt like an oasis. I am extremely disappointed that I've been forced to change my happy FreeBSD setup on Digital Ocean.

My first thought was to try another old friend, Linode. But they don't support FreeBSD. There's Vultr that comes recommended by some in the FreeBSD community. That's definitely an option.

What if I went back to using Linux for my hobbies? Would that be too much? Why throw away the baby with the bath water? But I am certainly considering it.

The reason I am thinking of returning to Linux is that I spend a large part of my day in that ecosystem anyways. Might as well spend a couple weekends here or there for my hobbies, too. Reduce the cognitive load. Throw away my oasis.

The FreeBSD ecosystem is amazing. Even now, as I am on the cusp of leaving it, I believe that it is much better than the Linux ecosystem. There is a beauty that cannot be described; it must be experienced. I am not taking this decision lightly.

Let's assume I do switch back. What does that look like?

I'm immersed in Ubuntu at work. It was my first Linux distribution that I installed in 2006 on my desktop. Over the years Canonical and the Debian and Ubuntu communities have made it one of the best distributions out there. But the more time I spend with it, the less attractive it is. Canonical's ideas are always innovative. Some are way ahead of their time. They have had to abandon some technologies for practical reasons. And that's why it is a difficult beast to tame.

The beauty of FreeBSD is that it remains consistent year after year. Moving from version 12 to 13 was no big deal. I didn't have to relearn key components. The same cannot be said of Ubuntu. Every two years the LTS version looks vastly different. One has to spend significant time figuring out what impact an upgrade will have. It's hard enough to do at work, where I get paid to deal with this hassle, let alone bringing it home.

An alternative to Ubuntu is Debian. It remains pretty consistent release after release. I don't mind "old" packages either. As long as the operating system is secure and the applications have the features and capabilities I need, I don't mind what the version number says. My main use case is running websites in the cloud; I don't do anything fancy.

Recently I've picked up openSUSE for my Raspberry Pi 3. It has been a mixed experience. I have deep respect for the amazing engineering culture in the SUSE ecosystem. Years ago I worked with SLES at work and I found it much better to use than CentOS which I used prior. But openSUSE has had its issues. I still advocate for it with enthusiasm. Unfortunately, Digital Ocean does not support it. There have been back-and-forth arguments between the openSUSE folks and Digital Ocean folks over who is to blame but for a user like me the end result is I cannot run it. I did spin it up on Linode and it worked well enough. There are some reasons why I'm hesitant to leave Digital Ocean, which I will not discuss here, and that leaves openSUSE out as an option.

As the post's title has already indicated, I turned to the Red Hat ecosystem. It has been a while since I've seriously looked at any Red Hat distribution. I have used Amazon Linux, a fork, for years in AWS. Otherwise, owning to who I work for, I have not used RHEL, CentOS, or siblings in years.

I tried Fedora, CentOS, Alma Linux, Oracle Linux, and Rocky Linux on my Raspberry Pis some months ago. They all had their quirks but overall not a bad experience. I ended up using openSUSE and Ubuntu full time.

This time I set myself up with a few requirements to guide my decision. I want to use containers, if I can, so I can always use the latest nginx, for example. I do not want to use Kubernetes because it does not make sense when running a handful of static websites. That means the host operating system (OS) does not matter as much as the base OS of the containers. Any one of the RHEL upstreams or clones will work.

I'll take a detour here and talk about CentOS Stream. When Red Hat first announced their decision to pivot to Stream, I read up on both sides of the argument. My conclusion was that it is in Red Hat's and their partners' best interest to make Stream a real thing. Since I don't use CentOS or any other Red Hat Enterprise Linux (RHEL) clone for serious work, it didn't affect me like it must have affected countless others. That Alma and Rocky emerged in the aftermath testifies to the popularity of RHEL clones and the resiliency of the RHEL ecosystem.

For my hobby use cases, CentOS Stream is an excellent choice. But is it a good choice for the host OS or the container OS? I think either is fine. I don't expect Stream will brick after an upgrade. Some features may move around but I can deal with it. At least, that's what I think right now.

Once again Digital Ocean's (DO) support policy restricts my choices. They don't support Alma Linux, and while Alma produces an image I can run on DO, I'm not inclined. I want my service provider to give me a well integrated cloud experience. I don't want to hack my way around. That leaves Rocky Linux as the only viable option for a RHEL clone.

There has been a lot of reddit drama on whether Rocky is better than Alma or vice versa. This sort of drama is why I moved away from Linux to FreeBSD in the first place. I want to be part of a community that focuses on delivering value rather than slinging epithets and mud. I've kind of soured on Rocky Linux and sort of sided with Team Alma. But not to the extent that I would not use Rocky; I will when I must.

Fedora is another choice. I'm afraid of the maintenance bandwidth though. With FreeBSD's stability I didn't have to touch it that often. Ever since I automated pretty much everything with Ansible, I manually touch my FreeBSD box even less. Going back to the world-shaking changes between versions, also demonstrated by Ubuntu LTS, but this time every six months, is not appetizing.

I think this is where I may be wrong. Fedora in 2022 is not the same Fedora when I was a full time Linux user.

Software Collections, introduced in RHEL 6 and 7 (I think), seemed like a great idea. It just felt too complicated to grok. It was a way to "containerize" some applications without modern containerization techniques. This is where FreeBSD ports win hands down because they can provide newest versions of applications because of how ports work. In modern systems, Snap and Flatpak are the next iteration, somewhere between Software Collections and OCI containers.

Fedora has the concept of modules now and so does RHEL (AppStream). This is a better idea and implementation. It took me a while to truly understand its benefits. I had a similarly hard time with Snaps but when I "got" it, it made sense. Users like me need a stable (read: consistent experience) base OS that can run applications at their release cadence. No longer do we want to wait two or three years to get an application update just because the vendor released the next version of their OS. Modules make it easy to get newer versions of some applications. It's not a perfect implementation though, because the vendor (Fedora or Red Hat) decides what applications they want to deliver this way. But it's the right step forward.

Let's pause here and try to distill where I stand now. I need a consistent base OS with easily upgradeable applications. I want to run containers because they make it easier to run newer versions of applications. But with modules I can maybe forego containers?

I'm thinking that Rocky Linux or maybe even CentOS Stream could be my host OS. I could run custom Fedora, Alma, Rocky, CentOS Stream, or even Red Hat UBI containers with the applications I need. Or I could skip containers and run modules instead.

That's the power of the Red Hat ecosystem. The last time I was heavily involved in this ecosystem I went back to Debian or Ubuntu because they had versatility and consistency in the right balance. Over the years it looks like Red Hat has taken that mantle. It would be very hard to overcome this advantage. It's funny that I haven't even talked about alternatives like Fedora Silverblue or Fedora CoreOS. It looks like Red Hat's ecosystem is poised to dominate Linux for many years. Competitors like Canonical and SUSE will need to step up.

I haven't made a final decision yet. I don't know if I'll continue using FreeBSD until DO forces me to delete it. I don't know if I'll migrate to Linux before then. I don't know which Linux OS or OSes I will choose. But I have learned in this exploration that Red Hat is emerging as the powerhouse it was always meant to be or maybe already is and I didn't pay more attention.