5. Skopeo - Moving & Sharing
5. Skopeo - Moving & Sharing ź“ė Ø
In this step, we are going to do a couple of simple exercises with Skopeo to give you a feel for what it can do. Skopeo doesn't need to interact with the local container storage (.local/share/containers), it can move directly between registries, between container engine storage, or even directories.
Remotely Inspecting Images
First, lets start with the use case that kicked off the Skopeo project. Sometimes, it's really convenient to inspect an image remotely before pulling it down to the local cache. This allows us to inspect the meta-data of the image and see if we really want to use it, without synchronizing it to the local image cache:
skopeo inspect docker://registry.fedoraproject.org/fedora
"Name": "registry.fedoraproject.org/fedora",
"Digest": "sha256:1972716109b1c906120061063bd4cb50a46c2138d95002ccb90126928d98e013",
"RepoTags": [
"34-aarch64",
...
"40"
],
"Created": "2023-08-04T06:50:36Z",
"DockerVersion": "1.10.1",
"Labels": {
"license": "MIT",
"name": "fedora",
"vendor": "Fedora Project",
"version": "38"
},
"Architecture": "amd64",
"Os": "linux",
"Layers": [
"sha256:18ca996a454fc86375a6ea7ad01157a6b39e28c32460d36eb1479d42334e57ad"
],
"Env": [
"DISTTAG=f38container",
"FGC=f38",
"container=oci"
]
}
We can easily see the "Architecture" and "Os" meta-data which tells us a lot about the image. We can also see the labels, which are consumed by most container engines, and passed to the runtime to be constructed as environment variables. By comparison, here's how to see this meta-data in a running container:
podman run --name meta-data-container -id registry.fedoraproject.org/fedora bash
podman inspect meta-data-container
# Trying to pull registry.fedoraproject.org/fedora:latest...
# Getting image source signatures
# Copying blob 18ca996a454f done
# Copying config 72c9e45642 done
# Writing manifest to image destination
# Storing signatures
# a83dd382c16d658770e61c0019fc2d9f9adfab2385bcf0cd408962ee88ca5ed9
[
{
"Id": "a83dd382c16d658770e61c0019fc2d9f9adfab2385bcf0cd408962ee88ca5ed9",
"Created": "2023-09-12T08:39:14.415149571Z",
"Path": "bash",
"Args": [
"bash"
],
"State": {
"OciVersion": "1.0.2-dev",
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6502,
"ConmonPid": 6491,
"ExitCode": 0,
"Error": "",
"StartedAt": "2023-09-12T08:39:14.734913968Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Health": {
"Status": "",
"FailingStreak": 0,
"Log": null
},
"CgroupPath": "/machine.slice/libpod-a83dd382c16d658770e61c0019fc2d9f9adfab2385bcf0cd408962ee88ca5ed9.scope",
"CheckpointedAt": "0001-01-01T00:00:00Z",
"RestoredAt": "0001-01-01T00:00:00Z"
},
"Image": "72c9e456423548988a55fa920bb35c194d568ca1959ffcc7316c02e2f60ea0ff",
"ImageName": "registry.fedoraproject.org/fedora:latest",
"Rootfs": "",
"Pod": "",
"ResolvConfPath": "/run/containers/storage/overlay-containers/a83dd382c16d658770e61c0019fc2d9f9adfab2385bcf0cd408962ee88ca5ed9/userdata/resolv.conf",
"HostnamePath": "/run/containers/storage/overlay-containers/a83dd382c16d658770e61c0019fc2d9f9adfab2385bcf0cd408962ee88ca5ed9/userdata/hostname",
"HostsPath": "/run/containers/storage/overlay-containers/a83dd382c16d658770e61c0019fc2d9f9adfab2385bcf0cd408962ee88ca5ed9/userdata/hosts",
"StaticDir": "/var/lib/containers/storage/overlay-containers/a83dd382c16d658770e61c0019fc2d9f9adfab2385bcf0cd408962ee88ca5ed9/userdata",
"OCIConfigPath": "/var/lib/containers/storage/overlay-containers/a83dd382c16d658770e61c0019fc2d9f9adfab2385bcf0cd408962ee88ca5ed9/userdata/config.json",
"OCIRuntime": "runc",
"ConmonPidFile": "/run/containers/storage/overlay-containers/a83dd382c16d658770e61c0019fc2d9f9adfab2385bcf0cd408962ee88ca5ed9/userdata/conmon.pid",
"PidFile": "/run/containers/storage/overlay-containers/a83dd382c16d658770e61c0019fc2d9f9adfab2385bcf0cd408962ee88ca5ed9/userdata/pidfile",
"Name": "meta-data-container",
"RestartCount": 0,
"Driver": "overlay",
"MountLabel": "system_u:object_r:container_file_t:s0:c298,c877",
"ProcessLabel": "system_u:system_r:container_t:s0:c298,c877",
"AppArmorProfile": "",
"EffectiveCaps": [
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_NET_BIND_SERVICE",
"CAP_NET_RAW",
"CAP_SETFCAP",
"CAP_SETGID",
"CAP_SETPCAP",
"CAP_SETUID",
"CAP_SYS_CHROOT"
],
"BoundingCaps": [
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_NET_BIND_SERVICE",
"CAP_NET_RAW",
"CAP_SETFCAP",
"CAP_SETGID",
"CAP_SETPCAP",
"CAP_SETUID",
"CAP_SYS_CHROOT"
],
"ExecIDs": [],
"GraphDriver": {
"Name": "overlay",
"Data": {
"LowerDir": "/var/lib/containers/storage/overlay/5eb2e729c16708697e84f3cf2648d1ff3f2ce56f2f1eceb0289b226bc96b061a/diff",
"MergedDir": "/var/lib/containers/storage/overlay/b51b49a949e02b75a36a818eb456cd9c982c4b6da2b755116f1df753d2256268/merged",
"UpperDir": "/var/lib/containers/storage/overlay/b51b49a949e02b75a36a818eb456cd9c982c4b6da2b755116f1df753d2256268/diff",
"WorkDir": "/var/lib/containers/storage/overlay/b51b49a949e02b75a36a818eb456cd9c982c4b6da2b755116f1df753d2256268/work"
}
},
"Mounts": [],
"Dependencies": [],
"NetworkSettings": {
"EndpointID": "",
"Gateway": "10.88.0.1",
"IPAddress": "10.88.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "56:d6:14:2f:2c:74",
"Bridge": "",
"SandboxID": "",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/run/netns/netns-7439927c-bfc3-af01-5d98-b54389861757",
"Networks": {
"podman": {
"EndpointID": "",
"Gateway": "10.88.0.1",
"IPAddress": "10.88.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "56:d6:14:2f:2c:74",
"NetworkID": "podman",
"DriverOpts": null,
"IPAMConfig": null,
"Links": null,
"Aliases": [
"a83dd382c16d"
]
}
}
},
"Namespace": "",
"IsInfra": false,
"IsService": false,
"Config": {
"Hostname": "a83dd382c16d",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": true,
"StdinOnce": false,
"Env": [
"container=oci",
"DISTTAG=f38container",
"FGC=f38",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm",
"HOME=/root",
"HOSTNAME=a83dd382c16d"
],
"Cmd": [
"bash"
],
"Image": "registry.fedoraproject.org/fedora:latest",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": "",
"OnBuild": null,
"Labels": {
"license": "MIT",
"name": "fedora",
"vendor": "Fedora Project",
"version": "38"
},
"Annotations": {
"io.container.manager": "libpod",
"io.kubernetes.cri-o.Created": "2023-09-12T08:39:14.415149571Z",
"io.kubernetes.cri-o.TTY": "false",
"io.podman.annotations.autoremove": "FALSE",
"io.podman.annotations.init": "FALSE",
"io.podman.annotations.privileged": "FALSE",
"io.podman.annotations.publish-all": "FALSE",
"org.opencontainers.image.stopSignal": "15"
},
"StopSignal": 15,
"HealthcheckOnFailureAction": "none",
"CreateCommand": [
"podman",
"run",
"--name",
"meta-data-container",
"-id",
"registry.fedoraproject.org/fedora",
"bash"
],
"Umask": "0022",
"Timeout": 0,
"StopTimeout": 10,
"Passwd": true
},
"HostConfig": {
"Binds": [],
"CgroupManager": "systemd",
"CgroupMode": "host",
"ContainerIDFile": "",
"LogConfig": {
"Type": "k8s-file",
"Config": null,
"Path": "/var/lib/containers/storage/overlay-containers/a83dd382c16d658770e61c0019fc2d9f9adfab2385bcf0cd408962ee88ca5ed9/userdata/ctr.log",
"Tag": "",
"Size": "0B"
},
"NetworkMode": "bridge",
"PortBindings": {},
"RestartPolicy": {
"Name": "",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": [],
"CapDrop": [
"CAP_AUDIT_WRITE",
"CAP_MKNOD"
],
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": [],
"GroupAdd": [],
"IpcMode": "shareable",
"Cgroup": "",
"Cgroups": "default",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "private",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [],
"Tmpfs": {},
"UTSMode": "private",
"UsernsMode": "",
"ShmSize": 65536000,
"Runtime": "oci",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": 0,
"OomKillDisable": false,
"PidsLimit": 2048,
"Ulimits": [
{
"Name": "RLIMIT_NOFILE",
"Soft": 1048576,
"Hard": 1048576
},
{
"Name": "RLIMIT_NPROC",
"Soft": 4194304,
"Hard": 4194304
}
],
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"CgroupConf": null
}
}
]
Pulling Images
Like, Podman, Skopeo can be used to pull images down into the local container storage:
skopeo copy docker://registry.fedoraproject.org/fedora containers-storage:fedora
# INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
# Getting image source signatures
# Copying blob 18ca996a454f skipped: already exists
# Copying config 72c9e45642 done
# Writing manifest to image destination
# Storing signatures
But, it can also be used to pull them into a local directory:
skopeo copy docker://registry.fedoraproject.org/fedora dir:$HOME/fedora-skopeo
# Getting image source signatures
#
# Copying blob 18ca996a454f done
# Copying config 72c9e45642 done
# Writing manifest to image destination
# Storing signatures
This has the advantage of not being mapped into our container storage. This can be convenient for security analysis:
ls -alh ~/fedora-skopeo
# total 71M
# drwxr-xr-x. 2 root root 186 Sep 12 08:41 .
# dr-xr-x---. 6 root root 183 Sep 12 08:41 ..
# -rw-r--r--. 1 root root 71M Sep 12 08:41 18ca996a454fc86375a6ea7ad01157a6b39e28c32460d36eb1479d42334e57ad
# -rw-r--r--. 1 root root 1.3K Sep 12 08:41 72c9e456423548988a55fa920bb35c194d568ca1959ffcc7316c02e2f60ea0ff
# -rw-r--r--. 1 root root 429 Sep 12 08:41 manifest.json
# -rw-r--r--. 1 root root 33 Sep 12 08:41 version
The Config and Image Layers are there, but remember we need to rely on a Graph Driver in a Container Engine to map them into a RootFS.
Moving Between Container Storage (Podman & Docker)
First, let's do a little hack to install Docker CE side by side with Podman on RHEL 8. Don't do this on a production system as this will overwrite the version of runc provided by Red Hat:
yes|sudo rpm -ivh --nodeps --force https://download.docker.com/linux/centos/8/x86_64/stable/Packages/containerd.io-1.3.7-3.1.el8.x86_64.rpm
# Retrieving https://download.docker.com/linux/centos/8/x86_64/stable/Packages/containerd.io-1.3.7-3.1.el8.x86_64.rpm
# warning: /var/tmp/rpm-tmp.6uT8NZ: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY
# Verifying... ################################# [100%]
# Preparing... ################################# [100%]
# Updating / installing...
# 1:containerd.io-1.3.7-3.1.el8 ################################# [100%]
sudo yum install -y docker-ce --nobest
Now, enable the Docker CE service:
sudo systemctl enable --now docker
# Created symlink /etc/systemd/system/multi-user.target.wants/docker.service ā /usr/lib/systemd/system/docker.service.
Now that we have Docker and Podman installed side by side with the Docker daemon running, lets copy an image from Podman to Docker. Since we have the image stored locally in .local/share/containers, it's trivial to copy it to /var/lib/docker using the daemon:
skopeo copy containers-storage:registry.fedoraproject.org/fedora docker-daemon:registry.fedoraproject.org/fedora:latest
# INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
# Getting image source signatures
# Copying blob 5eb2e729c167 done
# Copying config 72c9e45642 done
# Writing manifest to image destination
# Storing signatures
Verify that the repository is now in the Docker CE cache:
docker images | grep registry.fedoraproject.org
# registry.fedoraproject.org/fedora latest 72c9e4564235 5 weeks ago 190MB
This can be useful when testing and getting comfortable with other OCI complaint tools like Podman, Buildah, and Skopeo. Sometimes, you aren't quite ready to let go of what you know so having them side by side can be useful. Remember though, this isn't supported because it replaces the runc provided by Red Hat.
Moving Between Container Registries
Finally, lets copy from one registry to another. I have set up a writeable repository under my username (fatherlinux) on quay.io. To do this, you have to use the credentials provided below. Notice, that we use the "--dest-creds" option to authenticate. We can also use the "--source-cred" option to pull from a registry which requires authentication. This tool is very flexible. Designed by engineers, for engineers.
skopeo copy docker://registry.fedoraproject.org/fedora docker://quay.io/fatherlinux/fedora --dest-creds fatherlinux+fedora:5R4YX2LHHVB682OX232TMFSBGFT350IV70SBLDKU46LAFIY6HEGN4OYGJ2SCD4HI
# Getting image source signatures
# Copying blob 18ca996a454f skipped: already exists
# Copying config 72c9e45642 done
# Writing manifest to image destination
# Storing signatures
This command just synchronized the fedora repository from the Fedora Registry to Quay.io without ever caching it in the local container storage. Very cool right?
Finally, exit the "rhel" user because we need root for the next lab:
exit
Conclusion
You have a new tool in your tool belt for sharing and moving containers. Hopefully, you find other uses for Skopeo.