[BlueOnyx:26355] Re: Redhat sources - current state of affairs

[Michael Stauber]

Hi all,

Three weeks ago Christoph asked:

> Wondering if this will be indirectly affecting BX via Alma/Rocky:
> 
> https://www.redhat.com/en/blog/furthering-evolution-centos-stream


Since then a lot has happened and there is now a clearer picture of what 
the future will hold for RHEL clones.

In general it can be said that RedHat shot itself in the foot, reloaded 
and let loose another blast.

Below I'll take some time to summarize the state of affairs in detail, 
but let me give you the gist already:


TL;DR: It's all good. Better than ever, actually.


What happened:
===============

RedHat decided to put the RHEL sources behind the paywall of a RedHat 
subscription, whose terms prohibit usage of said sources in your own 
distributions.

So *if* you're a RHEL subscriber, you get the sources (if you want). But 
if you exercise that right and RHEL doesn't like you to? Then you might 
loose your status as a client.

In how far this is legal (as the RHEL sources contain repackaged Open 
Source codes of various provenance and released under different licenses).

Even if it's legal, it sure violates the spirit of open source and that 
didn't sit well with a lot of people. Even within RedHat/Fedora/CentOS 
itself and certainly not with rebuilders or users of rebuilds.

And if that wasn't bad enough, RedHat's VP Mike McGrath pured oil into 
the fire, calling all rebuilders and users of such "freeloaders" and "a 
real threat to open source":

"Simply rebuilding code, without adding value or changing it in any way, 
represents a real threat to open source companies everywhere. This is a 
real threat to open source, and one that has the potential to revert 
open source back into a hobbyist- and hackers-only activity." -- Mike 
McGrath

Source: 
https://www.redhat.com/en/blog/red-hats-commitment-open-source-response-gitcentosorg-changes


Why?
====

RedHat's vendetta against "rebuilders" (or "freeloaders" as they called 
'em) seems to be particularly aimed against RockyLinux and in special 
against Gregory Kurtzer (co-founder of CentOS and founder of 
RockyLinux). Gregory Kurtzer is also CEO of the company CIQ and chairs 
the Rocky Enterprise Software Foundation (RESF). This is a much, much 
"tighter" control than for example CloudLinux exercises over AlmaLinux, 
which they founded, funded and then basically "sent it on its way".

RedHat apparently has qualms that CIQ offers "commercial support" for 
RockyLinux, yet they have no issues with Oracle being a full blown 
global business or that AlmaLinux is more or less backed by the company 
CloudLinux. Likewise: There is Amazon and their Amazon Linux. Another 
corporate entity of "freeloaders", yet they haven't been singled out by 
name by RedHat either.

So why the hate against Gregory Kurtzer? Simply put: We don't know, but 
this feud might stem back to the CentOS days and only got reinvigorated 
when RedHat lost out on a NASA contract to RockyLinux just days prior to 
their knee-jerk reaction.

See:

https://twitter.com/geerlingguy/status/1674070940575997952
https://sam.gov/opp/2e0365ce1e3c4c179b50fb15573d68e4/view


Reactions:
===========

The reactions in the Open Source community so far has been pretty 
interesting. There have been rants and raves, but the net result was 
rather productive and goal-oriented:

Very well known open source gurus such as Jeff Geerling (Ansible), 
Michael DeHaan (also Ansible) and Graham Leggett (Apache Software 
Foundation and basically "Mr. Mod_Proxy") as well as Jim Jagielski (Head 
of OSPO at Salesforce) were the first prominent people to come forward 
and bash RedHat:

Jeff Geerling basically dropped RHEL support for his "Ansible Playbook", 
but will continue to support it on RHEL clones.

Here are quotes from the others I mentioned above:

"Hi Redhat. [...] You’ve been taking my work for about 25 years, and 
happily consuming it downstream. You’ve never paid me, but I’ve 
encouraged my clients to pay you. Perhaps it’s time we discuss how you 
will compensate us for our work" -- Graham Leggett

Tweet URL: https://twitter.com/minfrin/status/1673751642951262211?s=20

Also: "Redhat has literally taken my code, and given me nothing. They 
are downstream from me. I didn’t have a problem with Redhat being 
downstream from me, until Redhat had a problem with others being 
downstream from them." -- Graham Leggett

Tweet URL: https://twitter.com/minfrin/status/1674068070464208896?s=20

"The amount of _my_ code that Red Hat has taken is kinda substantial. 
I've never been paid nor compensated for it by them. Of course, I am 
lucky enough that I am able to volunteer my time and work, but if they 
start complaining about freeloaders, then "people in glass houses..." -- 
Jim Jagielski

Tweet URL: https://twitter.com/jimjag/status/1674075880543473664?s=20


Ansible:
=========

You might have noticed that in the list above I mentioned two well known 
and important Ansible contributors. What is Ansible? Ansible is an open 
source IT automation tool that automates provisioning, configuration 
management, application deployment, orchestration, and many other manual 
IT processes.

Simply put: You can't understate what a cash cow Ansible is and always 
has been for RedHat. And that's why they also bought it 
lock-stock-and-barrel some time ago. And *then* they started pissing off 
key developers and contributors. /facepalm

Even Markus Glantz, who still is a Principal Solution Architect at 
RedHat and also working on Ansible (among other things). And Markus 
Glantz recently posted this interesting article on LinkedIn:

https://www.linkedin.com/pulse/secret-behind-fedora-centos-rhel-magnus-glantz/

This insight into the inner workings or Fedora Rawhide -> Fedora ELN -> 
RHEL/CentOS Stream might just have contained the "missing links" that 
the rebuilders needed to get a new perspective and "options".

For that he might not get the "employee of the year"-award this time 
around, though.


Rebuilder Reactions:
=====================

I'll break this down by projects:


RockyLinux:
===========

They were the first to present a "novel" approach to long term fix their 
inability to directly pull sources from RHEL.

They remodeled their cloud based build system into using two (new) forms 
of obtaining sources: For anything not Kernel related they can poll the 
UBI images, that RedHat still releases for virtualization stacks such as 
Docker, Flatpack and similar. For kernel sources they rent a virtual 
RHEL instance at a cloud based ISP and therefore (via subletting) have a 
valid RHEL subscription. Then they DNF-download the SRPMs they need and 
spool the VPS down and delete it, ending their contract and contractual 
obligations.

If that is legal is for the courts to settle.

Even if not: RHEL would shoot itself in the foot again by going after 
cloud providers who offer RHEL instances and pay them well for it in return.

But TBH: RockyLinux's approach is a bit audacious and smells fishy.


SuSE Linux:
============

Why do I mention them? They weren't offering a RHEL clone, right? Well, 
SURPRISE: They just announced they will fund a foundation with 10 
million USD for the next 10 years to offer a RHEL clone of their own:

https://www.suse.com/news/SUSE-Preserves-Choice-in-Enterprise-Linux/

We'll have to see how that plays out, but it's an interesting and 
welcome move.


AlmaLinux:
===========

I guess the guys at AlmaLinux have taken the information that Markus 
Glantz provided to the next level and announced this:

https://almalinux.org/blog/future-of-almalinux/

In essence: AlmaLinux will now take the CentOS Stream sources *and* user 
provided patches and will incorporate them into AlmaLinux. In the past 
rebuilds tried to be bug-for-bug as well as ABI/binary compatible.

That also means they were (by nature) always lagging behind a little. 
AlmaLinux now ditches the bug-for-bug compatibility and will retain 
ABI/binary compatibility. So something compiled on RHEL or other RHEL 
clone will still work, but AlmaLinux may have more bugs fixed than 
"upstream" or other rebuilders.

Likewise: CloudLinux (the corporation that founded AlmaLinux) seems to 
have rebased their commercial "CloudLinux" RHEL clone on AlmaLinux already.


EuroLinux:
===========

This is a lesser known RHEL fork, mostly known in Europe and interesting 
for those who can't or won't (for legal reasons and "big brother") want 
to tie themselves directly into an US based "upstream". What they'll be 
doing is everyone's best guess, but I suspect they will base their 
rebuilds off AlmaLinux or copy their approach at least.


Oracle:
=======

The only real "big commercial player" in the pond responded with some 
comedy value, offering disgruntled RedHat clients (and developers!) a 
new and stable home without the constant moving of goal posts. How they 
plan to obtain sources for their own RHEL rebuild is everyone's best 
guess, though. RedHat certainly doesn't want to tests its lawyers 
against theirs anyway.


Conclusion:
============

It's all good. RHEL clones aren't going away. To the contrary: It 
spawned the birth of new RHEL clone and this has also furthered the 
cooperation among rebuilders. Also: I love the approach by the AlmaLinux 
guys, as this really has the potential to improve their already great 
RHEL clone even more.


Personal thoughts:
===================

The reactions of the whole Linux-ecosystem to this newest brain fart of 
IBM's and RedHat's corporate leadership was quite something. The general 
consensus is that this was plain and simply corporate greed at work. And 
the community reacted in a (mostly) professional and refreshing manner, 
providing solutions, keeping and expanding the eco-system and adding new 
building blocks and possibilities.

I've also been somewhat surprised by the casual "muggers"-approach that 
RockyLinux decided to pursue. My respect for them dropped by a few 
notches as a result of that.

And for what it's worth: Markus Glantz also (and quite tongue-in-cheek) 
pointed out a real vulnerability in RedHat's Fedora -> CentOS -> RHEL 
development chain in one of his comments under his own LinkedIn posts:

https://www.linkedin.com/feed/update/urn:li:ugcPost:7080625554265096193?commentUrn=urn%3Ali%3Acomment%3A%28ugcPost%3A7080625554265096193%2C7080634122691317761%29&replyUrn=urn%3Ali%3Acomment%3A%28ugcPost%3A7080625554265096193%2C7080905500728393728%29

"[CentOS] Board membership is an individual role. Directors do not act 
on behalf of their employer, with the important exception of the Red Hat 
Liaison"

It would be the height of irony if some time in the future the board 
members of the CentOS government board might vote someone or "something" 
off the board or off the agenda. I don't see this happening in the close 
future, but one never knows what might come. :p

-- 
With best regards

Michael Stauber
-------------- next part --------------
A non-text attachment was scrubbed...
Name: future-of-rebuilds.png
Type: image/png
Size: 139309 bytes
Desc: not available
URL: <http://mail.blueonyx.it/pipermail/blueonyx/attachments/20230714/74707959/attachment.png>