Updating Docker with Puppet on CentOS 7

I have a lot of CentOS 7.1 hosts that are under config management with Puppet. These CentOS hosts run some Docker containers to provide services to Mac devices.

I’ve been on the Docker bandwagon (ship?) for a while now so when these hosts were created and put into use, the latest and greatest version of Docker was 1.5.

Recently Docker sent out an email alert me that they will soon be changing their Docker Hub so that docker versions prior to 1.7 will no longer be able to push and pull from their hub.

Dear Docker Hub user
Last spring Docker released Engine version 1.6 and Registry version 2. These introduced an updated push / pull protocol featuring faster image transfers, simplified image definition, and improved security. The Docker community has aggressively adopted them, and as a result over 99% of Docker Hub usage is based on these newer version. As a result we are deprecating support on Docker Hub for clients version 1.5 and earlier.

* On November 19, 2015 Docker clients version 1.5 and earlier will not be able to push images to Docker Hub. They will still be able to pull images. And of course the repositories are fully accessible via newer versions of the Docker client.

* On December 7, 2015, pulls via clients 1.5 and earlier will be disabled. Only version 1.6 or later will be supported.

Handling this migration is simple; all that you need to do is upgrade your Docker client to version 1.6 or later. Please be sure to upgrade any clients that are pushing or pulling from your repository, including those on development machines, product servers, or that are part of CI and CD workflows.

If you have any questions, please do not hesitate to contact us.

Best regards,

The Docker Hub Team

So I thought I better upgrade the Docker binary on my hosts.

I manage Docker with a pretty basic puppet manifest that looks a bit like this

So I updated the package ensure section of the manifest to include the version that I want my hosts to use, in this case its 1.7.1-115.el7

However when I ran this on a host I found that after the Docker binary had been updated, my containers which were started with the –restart-always tag did not automatically start back up.

Even trying to stop them and start them again with docker stop <container name> docker start <container name> failed

Further digging found error messages like this:

err="Cannot start container netboot_server: [8] System error: Unit docker-a643761a31b77c798057b9036f8bc4c3802e4831608dcb3956255729f414ece4.scope already exists." statusCode=500

So thats kind of weird.

Telling systemctl to stop that unit and restarting docker fixed the issue so I knew it wasn’t the end of the world

So running the following brought my containers back online:

systemctl stop docker-a643761a31b77c798057b9036f8bc4c3802e4831608dcb3956255729f414ece4.scope

systemctl restart docker

But thats not really nice, and how do we make sure that we get the right name of the docker container?

Well its pretty easy to get the name of the docker units by doing something like:

systemctl list-units --type=scope | awk -F " " '/docker/ {print $1}'

In my case this gave me two containers which is what I have running on my hosts.

So now I just needed to add a pipe to head to get the first result:

systemctl list-units --type=scope | awk -F " " '/docker/ {print $1}' | head -n1

And for the second result I pipe it to tail to get the last result

systemctl list-units --type=scope | awk -F " " '/docker/ {print $1}' | tail -1

Then to put it all together as a single command to run

systemctl stop $(systemctl list-units --type=scope | awk -F " " '/docker/ {print $1}' | head -n1); systemctl stop $(systemctl list-units --type=scope | awk -F " " '/docker/ {print $1}' | tail -1); systemctl restart docker

So now we have a command we can run to stop our stuck docker containers and restart the docker daemon and bring our containers back up.

To add this into our puppet manifest, I created an exec and set it to subscribe to the Package[‘docker’]

So the final manifest looks like this (Note that I had to escape the quotes in the puppet command so i used single quotes for everything and escaped them with backslashes)

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s