Out on the bleeding edge with Fedora Silverblue

I recently bought a new laptop for myself, and like all computing systems I have, installed a Linux flavor outright. I don’t really dual boot and never keep Windows – which is to say nothing about the Windows platform or even the merits of dual booting; it’s just a preference and I just haven’t found the time to boot a flavor of BSD.

Fedora Silverblue had just come out of beta and I thought – yea OK I’ll bite. What’s all this flatpak and modularity and oh yea we bought CoreOS too so here’s rpm-ostree.

At first I wanted to be very judicous of what I installed on the actual OS through rpm-ostree. I started with just Ansible – which I’ll talk about why later – and jq – you know, just to make things all pretty.

Then, make my development environment completely in a container, because that’s the thing to do with a OS that’s it seems containers are pretty legit technology.

But, by the end of it, I had whole build chains and all sorts of -devel packages which is the complete opposite of what I’d wanted. So here’s my story. Personal log, star date – whenever, my journey out the the bleeding edge of Fedora.

img of rpm ostree

What had happened really, was that the container tool-chain – namely buildah and podman – had failed. Containers – that I’d built through ansible – seemed to have no file systems. The symptom of which was that nothing was in the PATH. I could not ls or /usr/bin/ls or cat or whatever in teh fuck.

Alright – so there’s some bug introduced by some update or maybe something I’d over-layed on the OS or really any number of things that you just have to go looking for. I’m a software developer after all. People have paid me to do this.

So I went looking; I could build perfectly good containers with the same steps manually. Same step even. A container built from just the single command ‘buildah from …’ would not work when built in ansible (specifically the ansible shell module).

Now at this point I could have just written a Dockerfile or a buildah script instead of trying to build it with Ansible. It was maybe 5 or so days after the initial image and I mean why bother with the trouble right? But boy, I just love templating files.

What’s more though is, a lot of containers are just dependent on environment variables but I’m building a dev environment. Not a highly configurable web-app with some mess of ‘-e VAR -e VAR -e VAR’ but, what looks like, a complete OS with all sorts of etc files. And I know I want ansible on my base OS to template it’s etc files. So I gotta dig in my heels and get to work.

Alright – debug mode. Let’s play 5 questions.
Query: How’s this fake ass filesystem work anyhow? Grrrr.
Answer: It’s got to be real at some point.

Query: Yes! That’s right. OK where do I find that? That place that’s real.
Answer: Hm.. Well.. Configs may point to something probably. It is real, but it’s also temporary so… Also docs likely have something. Go RTFM dude.

Statement: Been a few days. It’s getting deep. All debug output I can generate says they’re the same exact image, just with different ids. I know they’re different somehow, and I’m almost sure it’s something to do with user environments and the fact that ansible is actually running all sorts of different independent shells – different environments – and that abstraction has probably leaked.

Why don’t I just backup a few commits in the rpm-ostree?

Wait.
Stop the madness.
There’s got to be a better way.

I have this mindset – and here, I hope, is the main point of this blog – that everything digital is so ephemeral, and more importantly, repeatable (re-createable?) that I don’t give a fork to just reformat this entire drive and start all over. Just wipe it clean and do it all over again.

After a ‘come to Jesus’ moment I’m back to square one.

But that’s not really true is it? Square one. I learned a bit on the way, and you know, bugs exist, #DealWithIt. Only this time, let’s pivot – as the kewl kids say – and just and install anything I need to get this mother going.

Oh buildah broken? Build from source.
What’s that – ansible no good? Git clone && make.
I need libassuan-devel. Cool! Gimme that rpm pl0x!

And you know what? It works now. I can make solid images all day through buildah and ansible. I had to make both of them and override one, but as this guy ranted, customization is the power of Linux, not it’s detriment. That’s just the configuration of the OS I run at the moment. And if I want to repeat it, I can write an ansible playbook backwards, in my sleep, upside down at this point.

I shouldn’t be afraid to re-image and build it all again just to see if that may work. Because it’s all ephemeral; it’s all transitory; it’s all just about to fail in some new and novel way. One day, those bugs will be fixed and I can re-image all over again and get back to a nice clean ostree. Until then, I’ve got make, cmake, gcc and golang which by a wild guess I would say can build 60% of projects on github.

Having to debug and read documentation is just a part of this software game. And it’s not easy or always fun. And there’s always some learning curve. But I got through it.

Along the way I got distracted and read some interesting topics at https://discussion.fedoraproject.org/ and even became a member. Signed up and even voted on an discussion.