PODMAN: ‘overlay’ is not supported over xfs

I *love* Podman.

While Podman purports to be a way to test and troubleshoot Pods – “the smallest deployable units of computing that can be created and managed in Kubernetes” – where its real value lies for me and my coworkers is as a non-root, daemonless, drop-in replacement for Docker. We can run containers on our laptops and our servers without needing root access or the overhead of the Docker daemon.

It works so well, you could probably replace Docker with Podman and people wouldn’t even notice

This morning, I saw a tweet by Dan Walsh of Red Hat, linking to an article he wrote on the details of containers/storage:

I did deep dive into the details of containers/storage, Content you did not know you wanted to know. 👍You probably want to bookmark this blog for future reference on where your container images are stored. @kubernetesio @redhat @openshift @coreos https://t.co/4yLNe8LNQW— Daniel Walsh (@rhatdan) January 24, 2019

https://twitter.com/rhatdan/status/1088499288865423360

This was something I’d been looking for! Buildah, Podman, Skopeo – these are all great tools for working with containers sans Docker, but it was unclear to me how they all worked together with regards to the container images they each had access to. The article cleared all that up, and it re-primed my interest in playing around with Podman again.

(I’ve been so focused on OKD (formerly OpenShift Origin) at $WORK that I’d not built or run a container by hand in a while.)

Apparently, though, Podman had different ideas:

$ podman ps ERRO[0000] 'overlay' is not supported over xfs at "/home/chris/.local/share/containers/storage/overlay"  error creating libpod runtime: kernel does not support overlay fs: 'overlay' is not supported over xfs at "/home/chris/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

I called shenanigans on that… I’ve been using overlay2 with Podman and Docker for – years? – now, with the latest version of Fedora. The kernel dang-well *does* support it!

Weirdly, I could pull an image once, if I wiped out /home/chris/.local/share/containers/storage, and would get the error, but it would work. Every subsequent command would fail though, even just podman ps:

$ podman pull centos:latest
ERRO[0000] 'overlay' is not supported over xfs at "/home/chris/.local/share/containers/storage/overlay"
Getting image source signatures Copying blob sha256:a02a4930cb5d36f3290eb84f4bfa30668ef2e9fe3a1fb73ec015fc58b9958b17 71.68 MiB / 71.68 MiB [====================================================] 6s Copying config sha256:1e1148e4cc2c148c6890a18e3b2d2dde41a6745ceb4e5fe94a923d811bf82ddb 2.13 KiB / 2.13 KiB [======================================================] 0s Writing manifest to image destination Storing signatures 1e1148e4cc2c148c6890a18e3b2d2dde41a6745ceb4e5fe94a923d811bf82ddb

$ podman images ERRO[0000] 'overlay' is not supported over xfs at "/home/chris/.local/share/containers/storage/overlay" Could not get runtime: kernel does not support overlay fs: 'overlay' is not supported over xfs at "/home/chris/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

Knowing that I’d had Podman working before, I double-checked all the things I could think of that might have been an issue.

I recently partitioned some free space and mounted it to containers/storage so it wouldn’t fill up the rest of my home directory. Since I *just* setup the filesystem (xfs) I checked that -ftype=1 was set. Older versions of CentOS and RHEL did not default to that, and that setting is required for Overlay to work. Perhaps I forgot to do that?

No, it’s definitely set:

[[email protected] ~]$ xfs_info /home/chris/.local/share/containers/storage/ | grep ftype
naming =version 2 bsize=4096 ascii-ci=0, ftype=1

Then I checked the SELinux permissions.

No, not because “it’s always SELinux”. Come on, now…

I checked the context because I’d recently mounted the partition to contianers/storage, I wanted to be sure the context was correct. This was an issue I’d run into at $WORK, when we mounted large partitions to /var/lib/docker, and the Docker daemon failed to work due to incorrect context.

In this case, they appeared correct, but I checked just to be sure:

[[email protected] ~]$ ls -ldZ ~/.local/share/containers/storage/
drwxr-xr-x. 9 chris chris system_u:object_r:data_home_t:s0 150 Jan 25 10:10 /home/chris/.local/share/containers/storage/

[[email protected] ~]$ matchpathcon ~/.local/share/containers/storage/
/home/chris/.local/share/containers/storage unconfined_u:object_r:data_home_t:s0

[[email protected] ~]$ matchpathcon -V ~/.local/share/containers/storage/
/home/chris/.local/share/containers/storage verified.

After pulling out all the hair I don’t, as a bald man, have, I tried dnf reinstall podman … with no luck.

Finally, I decided this was past my ability to fix on my own, and to open an issue in the Podman GitHub repo. I decided to double-check for existing issues, I found this:

Yes, and I think that’s a duplicate of containers/libpod#2158

If your ~/.config/containers/storage.conf is using the camel case format, then try switching to the lower case format, remove ~/.local/share/containers, and retry fedora-toolbox. See the above libpod issue about the formats.

https://github.com/debarshiray/fedora-toolbox/issues/42#issuecomment-457269578

Well, dang. The storage.conf in my homedir was all camel-case-y:

RunRoot = "/run/user/1000"
GraphRoot = "/home/chris/.local/share/containers/storage"
GraphDriverName = "overlay"
GraphDriverOptions = ["overlay.mount_program=/usr/bin/fuse-overlayfs"]

And that’s not at ALL what Dan’s looked like in the article this morning. For one thing, his had sections…

[storage]
# Default Storage Driver driver = "overlay"

...

So, it looks like containers/libpod#2158 was the culprit. I was using an old config file, and because it’s the right thing to do (if unhelpful in this case) dnf update did not replace the config file when I upgraded packages recently.

So, time to get a new one. First, though, since so many things use containers/storage, it’s unlikely that it’s a Podman config file. dnf didn’t seem to know what the $HOME/.config/containers/storage.conf file in my homedirectory belonged to (or more likley, I don’t know how to ask it properly…), but it did tell me that the global /etc/containers/storage.conf belonged to the containers-common package:

[[email protected] ~]$ dnf whatprovides /home/chris/.config/containers/storage.conf
Error: No Matches found

[[email protected] ~]$ dnf whatprovides $HOME/.config/containers/storage.conf Error: No Matches found

[[email protected] ~]$ dnf whatprovides /etc/containers/storage.conf
containers-common-1:0.1.34-1.dev.gite96a9b0.fc29.x86_64 : Configuration files for working
: with image signatures
Repo : @System
Matched from:
Filename : /etc/containers/storage.conf

Since I hadn’t really done any special customizations, I just went ahead and removed the storage.conf file in my home directory, and reinstalled containers-common and was provided with a shiny, new, package-fresh configuration file:

[storage]
driver = "overlay"
runroot = "/run/user/1000"
graphroot = "/home/chris/.local/share/containers/storage"
[storage.options]
mount_program = "/usr/bin/fuse-overlayfs"

And with the correct config file, Podman was much happier, and I could get back to building and running container images without root!

[[email protected] ~]$ podman pull docker.io/centos:latest && podman images
Trying to pull docker.io/centos:latest…Getting image source signatures
Copying blob a02a4930cb5d: 71.68 MiB / 71.68 MiB [==========================] 6s
Copying config 1e1148e4cc2c: 2.13 KiB / 2.13 KiB [==========================] 0s
Writing manifest to image destination
Storing signatures
1e1148e4cc2c148c6890a18e3b2d2dde41a6745ceb4e5fe94a923d811bf82ddb
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/library/centos latest 1e1148e4cc2c 7 weeks ago 210 MB

#NoBigFatDaemon

OpenShift Error – Failed to build: MatchNodeSelector (6)

OpenShift Error – Failed to build: MatchNodeSelector (6)

Brought to you by the Home Office1 department of jolly old $WORK:

I recently upgraded our OpenShift Origin dev cluster from version 3.7 to 3.9, and for the most part things went smoothly.  Version 3.9 brings with it a new version of Kubernetes, a whole host of bugfixes, and other goodies.  A little while after the upgrade, however, one of the engineers doing testing for us noticed that he was unable to deploy any applications, noting:

Nothing is running. They all have events that say:

12:38:58 PM Warning Failed Scheduling 0/6 nodes are available: 6 MatchNodeSelector.

I’m unsure if this was a consequence of the actual upgrade or something else, but I was able to confirm with my own test application.  Each build failed with the same error.

After diving in a bit and doing some research, I found that there was a defaultNodeSelector set in projectConfig section of the the master-config.yml:

projectConfig:
     defaultNodeSelector: node-role.kubernetes.io/compute=true

The defaultNodeSelector is does exactly what it says: it sets the default node selector (think: label) for each project. This means all pods without a nodeSelector will be deployed onto an OpenShift node with labels that match the defaultNodeSelector.

Unfortunately for us, none of our nodes had a label that matched node-role.kubernetes.io/compute=true. During our initial install of OpenShift Origin 3.7, we used the suggested example labels in the Configuring Node Host Labels section of the Advanced Install documentation, eg: region=primary for our standard nodes, and region=infra for infrastructure nodes, with the intention that we’d change the region for nodes in other datacenters when we deployed them, or that we’d add extra labels to define special nodes (for compliance, etc).

I was able to verify the labels we did have applied to our nodes with the oc get nodes --show-labels command.

[[email protected] Projects] $ oc get nodes --show-labels
NAME STATUS AGE VERSION LABELS
master-dev-01    Ready    54d    v1.9.1+a0ce1bc657     beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,datacenter=lab,kubernetes.io/hostname=master-dev-01,node-role.kubernetes.io/master=true,openshift-infra=apiserver,region=primary
node-dev-01      Ready    11d    v1.9.1+a0ce1bc657     beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,datacenter=lab,kubernetes.io/hostname=node-dev-01,region=infra
node-dev-02      Ready    11d    v1.9.1+a0ce1bc657     beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,datacenter=lab,kubernetes.io/hostname=node-dev-02,region=infra
node-dev-03      Ready    11d    v1.9.1+a0ce1bc657     beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,datacenter=lab,kubernetes.io/hostname=node-dev-03,region=primary
node-dev-04      Ready    11d    v1.9.1+a0ce1bc657     beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,datacenter=lab,kubernetes.io/hostname=node-dev-04,region=primary
node-dev-05      Ready    11d    v1.9.1+a0ce1bc657     beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,datacenter=lab,kubernetes.io/hostname=node-dev-05,region=primary

Some posts I found regarding the defaultNodeSelector suggested setting it to a blank string, but I decided I’d rather go with the region=primary label so we don’t accidentally get pods deployed onto new nodes that we want to spin up in the future. (Disclaimer: I am not 100% sure that’s how the empty string works – I need to do further research.)

After changing the master-config.yml file to use our chosen value, it was just a matter of restarting the Origin master service:

systemctl restart origin-master-controllers origin-master-api

With that done, I was able to kick off a new deploy, and watch as pods were scheduled onto the nodes with the region=primary label.

I am a little uneasy following all of this. I never was able to find out what caused the change in the master-config.yml. That value was unlikely to be set already (though I cannot say for sure), and there’s nowhere in the OpenShift-Ansible playbook referencing that value. One possibility is that the master-config.yml was replaced during the upgrade with a default from the Origin master container image2 , and then updated by the Ansible playbooks.

One of the drawbacks to the otherwise excellent OpenShift-Ansible advanced install process is that it’s not conducive to configuration management for the config files, as it generates new ones. I suppose one should use the actual Ansible playbooks as your configuration management – that’s what would be done with a standard Ansible-managed host – but it feels different somehow. Maybe that’s just me.

Finally, a semi-related point that was brought up by all of this is the need to have some better, more descriptive labels. region=primary works for now, but we’ll be better off in the long run with labels that reflect more about the hosts themselves. Chalk that up to just getting a dev cluster up and running. Now we know what we need for production.


1. See what I did there? …yeah, ok, it’s a stretch.

2. We’re running a fully containerized install of Origin on RHEL Atomic hosts. The install process copies files from inside the container to the host filesystem, and then mounts those files into the container so they can be managed like a traditional host.

ERROR: could not find an available, non-overlapping IPv4 address pool

ERROR: could not find an available, non-overlapping IPv4 address pool

Creating network “” with the default driver
ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

This appears to be related to the number of IP addresses reserved for Docker networks. On my laptop, it looks like 31 is the magic number (a /27 subnet, perhaps?).

docker-compose run --rm test
Creating network "test_default" with the default driver
ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

# Check the network list
# Don't count the header
docker network ls -q | wc -l
31

# Remove an unneeded network
docker network rm 173fff3fa69b
173fff3fa69b

# retry
docker-compose run --rm test
Creating network "test_default" with the default driver
"It Worked!"

I definitely need to dig in more and figure out what is actually happening under the covers, but for now, quick fix.