Lessons for Developers from 20 Years of Linux Kernel Work – InApps Technology 2022

Greg Kroah-Hartman, the Linux Foundation fellow currently responsible for stable Linux kernel releases shared the lessons he’s learned as a kernel developer that are applicable to other developers at this year’s Linux App Summit.

Kroah-Hartman has been helping develop the Linux kernel for over 20 years — “way too long,” he jokes — but reminded the audience that he has done some userspace work and maintains some small Linux userspace packages. He started by showing how he could succinctly distill the essence of the talk into a single five-word slide:

“Don’t make your users mad.”

“Let’s go into a little more detail, though,” he adds with a laugh. “I’ll try to explain why you don’t want to make users mad, and what happens when you do.”

“So, all this stuff comes down to a pretty simple thing: who are you writing code for?”

It’s important because what you should do depends on why you’re doing it, he explains, starting with the easiest scenario. If you’re writing code for yourself, “Throw it over the wall and have fun.”

But if you’re writing it for others, “Now you have to care about users.”

“Within reason,” he adds, sharing a corner-case example from what he calls the infamous XKCD cartoon illustrating how users can complain about even the most essential of fixes — like “the CPU no longer overheats when you hold down spacebar.”

“Everything I say here is ‘within reason’, because there are always some users you can never please, no matter what. And those people you can safely ignore.”

Kroah-Hartman explains that’s one of Linus Torvalds’ most deeply-held convictions: don’t break userspace.

“Other operating systems have this rule as well — it’s a very solid rule — because we always want you to upgrade. And we want you to upgrade without worrying about it. We don’t want you to feel scared. If you see a new release, and we say, ‘Hey, this fixes a bunch of problems,’ we don’t want you to feel worried about taking that. That’s really really important — especially with security.”

This leads to what Kroah-Hartman called Greg’s rule #1: “Users will not update if they are afraid you will break their current system.”

“We’ve learned this the hard way…”

Evolving with Users

This obviously informs the way that changes ultimately get made. “The trick is you just support everything until there are no users,” Kroah-Hartman explained. “Figuring out when there are and are not users is up to you.”

Kroah-Hartman told a remarkable story about the time where the Linux kernel developers realized one particular architecture was only being used by exactly two machines in existence. “One of them broke, and the other one — we actually paid the developer to replace it, because we didn’t want to support that thing anymore.”

“But otherwise, we have to keep supporting users. Because we made that contract. We made that contract that we gave them a functionality. We cannot remove it.”

He points out that Linux can still support 25-year-old binaries, so “People think Linux has stagnated. But in reality, we’ve created new functionalities right next to the old functionalities, that you can do different things — you can take advantage of stuff.”

He provided another very prominent example of an application evolving slowly that a few notice: KDE.

In the Real World

Another piece of advice for application developers: Don’t take away things that work. “If somebody is using this functionality, keep it there. It’s that simple… Don’t think by removing it, you’re doing them a favor.”

When he turned his attention to library developers, his first slide simply read: “I pity you.”

“You never know if an API is really useful, until you have too many people using it to ever be able to change it.”—Greg Kroah-Hartman

“This is the hardest job ever,” he said with a laugh. “I really, really feel sorry for you… It’s one of the hardest things to do.”

“You can never know if your API actually works until you have a lot of people using it. And by the time you have a lot of people using it, then you realize all the problems in it — and then you can’t change it.” He said, laughing. “Getting it right the first time is almost impossible.”

There is one alternative — but it still involves honoring the users who are still using that old library that you want to deprecate. If you really need to make a fresh start — “do that as a whole new library,” he advised.

“And then you have no users! And then you don’t know if anybody’s using your API, you don’t know if you got it right, until you have users, and then it’s broken — and the cycle continues.”

He also offered an important tip for developers who actually want their users to switch over to that new library. “You need to document it really well — because if you don’t Stack Overflow is going to try to do it.” He laughs knowingly, remembering a flood of bad drivers and submissions that were apparently inspired by a 10-year-old Stack Overflow post. “You don’t want Stack Overflow to be your primary documentation. Provide good documentation.”

But then he added seriously, “If you violate those things, you cause people to lose trust. If I rebuild and see ‘deprecated’ all over the place or APIs changed, it’s just horrible. You don’t want to do that. If you violate that trust, you won’t have that user anymore.”

If you do make a change, make sure there truly is a compelling reason. “You have to provide enough reason and enough goodness to force somebody to take the time to learn to do something else. That’s very rare.”

His example of this was systemd, which unified a variety of service configurations and initialization processes. “They did it right. They provided all the functionality, they solved a real problem that was there. They unified all these existing tools and problems in such a way that it was just so much better to use, and it provided enough impetus that everybody was willing to do the work to modify their own stuff and move to the new model.

“It worked. People still complain about it, but it worked. Everybody switched… It works well. It solves a real problem.

“That was an example of how you can provide a compelling reason to move on — and make the change.”

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Real.