I love Podman.
While Podman purports to be a way to test and troubleshoot Pods – “the smallest deployable units of computing that can be created and managed in Kubernetes” – where its real value lies for me and my coworkers is as a non-root, daemonless, drop-in replacement for Docker. We can run containers on our laptops and our servers without needing root access or the overhead of the Docker daemon.
It works so well, you could probably replace Docker with Podman and people wouldn’t even notice…
This morning, I saw a tweet by Dan Walsh of Red Hat, linking to an article he wrote on the details of containers/storage:
I did deep dive into the details of containers/storage, Content you did not know you wanted to know. 👍You probably want to bookmark this blog for future reference on where your container images are stored. @kubernetesio @redhat @openshift @coreos https://t.co/4yLNe8LNQW
— Daniel Walsh (@rhatdan) January 24, 2019
This was something I’d been looking for! Buildah, Podman, Skopeo – these are all great tools for working with containers sans Docker, but it was unclear to me how they all worked together with regards to the container images they each had access to. The article cleared all that up, and it re-primed my interest in playing around with Podman again.
(I’ve been so focused on OKD (formerly OpenShift Origin) at $WORK that I’d not built or run a container by hand in a while.)
Apparently, though, Podman had different ideas:
$ podman ps
ERRO[0000] 'overlay' is not supported over xfs at "/home/chris/.local/share/containers/storage/overlay" error creating libpod runtime: kernel does not support overlay fs: 'overlay' is not supported over xfs at "/home/chris/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver
I called shenanigans on that… I’ve been using overlay2 with Podman and Docker for – years? – now, with the latest version of Fedora. The kernel dang-well does support it!
Weirdly, I could pull an image once, if I wiped out /home/chris/.local/share/containers/storage, and would get the error, but it would work. Every subsequent command would fail though, even just podman ps:
$ podman pull centos:latest
ERRO[0000] 'overlay' is not supported over xfs at "/home/chris/.local/share/containers/storage/overlay"
Getting image source signatures Copying blob sha256:a02a4930cb5d36f3290eb84f4bfa30668ef2e9fe3a1fb73ec015fc58b9958b17 71.68 MiB / 71.68 MiB [====================================================] 6s Copying config sha256:1e1148e4cc2c148c6890a18e3b2d2dde41a6745ceb4e5fe94a923d811bf82ddb 2.13 KiB / 2.13 KiB [======================================================] 0s Writing manifest to image destination Storing signatures 1e1148e4cc2c148c6890a18e3b2d2dde41a6745ceb4e5fe94a923d811bf82ddb
$ podman images
ERRO[0000] 'overlay' is not supported over xfs at "/home/chris/.local/share/containers/storage/overlay" Could not get runtime: kernel does not support overlay fs: 'overlay' is not supported over xfs at "/home/chris/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver
Knowing that I’d had Podman working before, I double-checked all the things I could think of that might have been an issue.
I recently partitioned some free space and mounted it to containers/storage so it wouldn’t fill up the rest of my home directory. Since I just setup the filesystem (xfs) I checked that -ftype=1 was set. Older versions of CentOS and RHEL did not default to that, and that setting is required for Overlay to work. Perhaps I forgot to do that?
No, it’s definitely set:
$ xfs_info /home/chris/.local/share/containers/storage/ | grep ftype
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
Then I checked the SELinux permissions.
No, not because “it’s always SELinux”. Come on, now…
I checked the context because I’d recently mounted the partition to contianers/storage, I wanted to be sure the context was correct. This was an issue I’d run into at $WORK, when we mounted large partitions to /var/lib/docker, and the Docker daemon failed to work due to incorrect context.
In this case, they appeared correct, but I checked just to be sure:
$ ls -ldZ ~/.local/share/containers/storage/
drwxr-xr-x. 9 chris chris system_u:object_r:data_home_t:s0 150 Jan 25 10:10 /home/chris/.local/share/containers/storage/
$ matchpathcon ~/.local/share/containers/storage/
/home/chris/.local/share/containers/storage unconfined_u:object_r:data_home_t:s0
$ matchpathcon -V ~/.local/share/containers/storage/
/home/chris/.local/share/containers/storage verified.
After pulling out all the hair I don’t, as a bald man, have, I tried dnf reinstall podman … with no luck.
Finally, I decided this was past my ability to fix on my own, and to open an issue in the Podman GitHub repo. I decided to double-check for existing issues, I found this:
Yes, and I think that’s a duplicate of containers/libpod#2158
If your ~/.config/containers/storage.conf is using the camel case format, then try switching to the lower case format, remove ~/.local/share/containers, and retry fedora-toolbox. See the above libpod issue about the formats.
https://github.com/debarshiray/fedora-toolbox/issues/42#issuecomment-457269578
Well, dang. The storage.conf in my homedir was all camel-case-y:
RunRoot = "/run/user/1000"
GraphRoot = "/home/chris/.local/share/containers/storage"
GraphDriverName = "overlay"
GraphDriverOptions = ["overlay.mount_program=/usr/bin/fuse-overlayfs"]
And that’s not at ALL what Dan’s looked like in the article this morning. For one thing, his had sections…
[storage]
# Default Storage Driver driver = "overlay"
So, it looks like containers/libpod#2158 was the culprit. I was using an old config file, and because it’s the right thing to do (if unhelpful in this case) dnf update did not replace the config file when I upgraded packages recently.
So, time to get a new one. First, though, since so many things use containers/storage, it’s unlikely that it’s a Podman config file. dnf didn’t seem to know what the $HOME/.config/containers/storage.conf file in my homedirectory belonged to (or more likley, I don’t know how to ask it properly…), but it did tell me that the global /etc/containers/storage.conf belonged to the containers-common package:
$ dnf whatprovides /home/chris/.config/containers/storage.conf
Error: No Matches found
$ dnf whatprovides $HOME/.config/containers/storage.conf
Error: No Matches found
$ dnf whatprovides /etc/containers/storage.conf
containers-common-1:0.1.34-1.dev.gite96a9b0.fc29.x86_64 : Configuration files for working
: with image signatures
Repo : @System
Matched from:
Filename : /etc/containers/storage.conf
Since I hadn’t really done any special customizations, I just went ahead and removed the storage.conf file in my home directory, and reinstalled containers-common and was provided with a shiny, new, package-fresh configuration file:
[storage]
driver = "overlay"
runroot = "/run/user/1000"
graphroot = "/home/chris/.local/share/containers/storage"
[storage.options]
mount_program = "/usr/bin/fuse-overlayfs"
And with the correct config file, Podman was much happier, and I could get back to building and running container images without root!
$ podman pull docker.io/centos:latest && podman images
Trying to pull docker.io/centos:latest…Getting image source signatures
Copying blob a02a4930cb5d: 71.68 MiB / 71.68 MiB [==========================] 6s
Copying config 1e1148e4cc2c: 2.13 KiB / 2.13 KiB [==========================] 0s
Writing manifest to image destination
Storing signatures
1e1148e4cc2c148c6890a18e3b2d2dde41a6745ceb4e5fe94a923d811bf82ddb
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/library/centos latest 1e1148e4cc2c 7 weeks ago 210 MB