Changing credentials on Docker Swarm Services

This post will give you and overview on how docker performs the authentication into image registries, and how changes to the credentials alters the workflow when multiple nodes are involved.

The first part of this post gives and overview of what docker is, where docker images are stored and what docker login does. If you already know some basics on how Docker and Docker Swarm works, you can skip directly here.

Docker basics

Docker has a client-server architecture and most of the docker commands are communicating with the docker daemon via some REST-ish API.

Running docker -H tcp://1.2.4.5:2375 run --rm nginx will contact the docker daemon on the 1.2.4.5 host and will instruct it to download and run the nginx image.

Docker swarm

Docker Swarm is a container orchestrator that allows you to run docker containers across multiple nodes. It allows you also to abstract from the nodes that are running the Swarm, you decide the desired state and Docker Swarm will distribute the load across the nodes forming the cluster.

Docker image registry

Docker images are stored into a "registry" and images can be uploaded and downloaded from/to it by the docker daemon. Most of the registries require authentication.

Docker login

To authenticate a docker client into a docker image registry you can run:

docker login

This command will try to authenticate us into hub.docker.com (the commercial docker registry), asking us username and password. you can also login into other registries (offered by other companies or hosting our own registries) by specifying the registry host name as third parameter.

As example docker login registry.gitlab.com will try authenticate us into the Gitlab registry.

By default credentials are saved into $HOME/.docker/config.json on the computer running the docker login command (there are also other storages).

To notice that docker login is a "client" command, and does not communicate with the docker daemon.

Downloading private images

What happens when you create/update a Docker Swarm service using the --with-registry-auth option (or deploy a Docker Stack with the same option)?

The docker client will:

1. Collect a list of docker images to download
2. Read the authentication informations from the credentials-store for the images to download (for instance reading the $HOME/.docker/config.json from the computer where the client is installed)
3. Create (or update) the service in the Swarm RAFT state. The state information include also the authentication information on how to download the image.

In this way when the container will be scheduled on some node, the docker daemon will read the RAFT state and forward the stored credentials to that node, allowing it do download autonomously the docker image.

Changing credentials

If you update your credentials (as example change your password), the credentials stored in the Swarm RAFT state are not correct anymore and new nodes will not be able to download images.

As a solution, you should re-authenticate into the image registry using the docker login command and update the credentials stored into the Swarm RAFT state by running:

docker login

docker service update SERVICE_NAME --with-registry-auth

This command will leave the service as it is, it will just update the credentials.

The same strategy can be used for the Docker Stack, but is more risky as it could trigger unwanted container updates since Docker in some cases might try to download a newer image (if you update a docker image tag) or re-run services that are configured as restart: never.

Other things to consider

Which services should I update?

If you have a lot of services (maybe using different registries) it might be difficult to keep track of which credentials should be updated.

Services can be labeled (using the --label or --label-add options), and later you can update only the services having a specific label.

docker login -u "me" -p "xxxxxx" registry.gitlab.com

for i in $(docker service ls --filter=label=gitlab-registry); do docker service update "$i" --with-registry-auth -d; done

This script will update the credentials for all the services having the gitlab-registry label.

Note: It is not safe to provide the password as commandline argument, find here some alternatives.

Expiring credentials

Some docker registries (AWS or Azure) have credentials that expire automatically after a specified amount of time. Also in this case the credentials stored in the Swarm RAFT state will be not correct anymore and new nodes will not be able to download images.

As a solution, it is possible to setup a cronjob to perform periodically the update.

There is a trap!

If you have a DockerHub account (the commercial docker registry), and you did login into it using the docker login command, and you created/updated a service or stack using the --with-registry-auth option, those credentials WILL be used to retrieve the public images (that could have been downloaded anonymously otherwise). This means that if you change credentials on DockerHub, your nodes will not be able to download public images.

To mitigate this issue make sure to not use the --with-registry-auth option when creating or updating a service that uses public images. Unfortunately if you are deploying a Docker Stack that contains mixed services (with private and public images), the --with-registry-auth is global and can not be specified per single service. In alternative you can use the same labeling strategy described before and update also the credentials for public services.