Like any sound long-running system, such as a generator or your long-running containers, there's always some maintenance that you should consider (and should be scheduled, too!). Without this maintenance, you might find the performance degrades over time or experience outright failures. In this post, we will explore some of these maintenance tasks you should consider to get a better handle on the docker engine you are running and ensure you're getting the most out of it.

The topics we'll cover include a detailed description of the Docker engine and the routine maintenance commands you should be running to keep it clean. We'll also discuss effectively managing container updates for your containerized workloads. This is crucial as it ensures you're always secure with the latest updates to your runtimes, enhancing the overall security of your system. Finally, we'll introduce tools to help you monitor your Docker containers effectively. This will give you better observability over the Docker engine, allowing you to diagnose issues before they impact your workloads.

Docker Engine Clean-up

According to Docker themselves, the "Docker Engine is an open source containerization technology for building and containerizing your applications". When you run Docker on a machine, you run this Docker Engine application in the background. It's where all your containers live and run through their whole container lifecycle, and it includes features like sandboxing containers, setting up virtual networking, and managing storage access. As a developer, you're likely familiar with accessing the engine through applications like the Docker/Docker-Compose CLI or the Docker desktop application. These applications interact with the Docker Engine API to execute your desired actions, such as pulling, stopping, and starting containers.

The Docker engine is a crucial part of your container operations, so cleaning up this environment is essential to ensure it runs at its best performance. You must build a lovely table with a clean woodworking shop!

Docker Prune

When you operate Docker, you constantly stop and start your containers, building new images, each with its layers that could change with each interaction. You may have some resources in your Docker engine, including Docker images and containers that are no longer running, that you can clean up using the Docker prune command.

❯ docker system prune -h Flag shorthand -h has been deprecated, please use --help Usage: docker system prune [OPTIONS] Remove unused data Options: -a, --all Remove all unused images not just dangling ones --filter filter Provide filter values (e.g. "label=<key>=<value>") -f, --force Do not prompt for confirmation --volumes Prune anonymous volumes

For developers of containers, this command will ensure all the images you create while developing a runtime container are now orphaned and can save you gigabytes sometimes, depending on how long it has been since you last ran the command. You may want to consider

Consider this command for operators of docker containers for production workloads. Constantly starting and stopping containers with new tags, you can end up with orphaned container images on a host, taking up disk space and often costing if you run on a cloud service. Consider running this command at the end of your container deployment automation to clean up after successfully deploying a workload.

Log Rotation

All your running containers produce logs, so it's essential to maintain them to ensure you write only a little or trim for your application requirements. If you do not consider this, your service could fail due to the inability to write logs or fill up disk space. Storing too many logs with sensitive details can also be a security risk you should consider. If you need more extended log storage, ship these logs to somewhere central so you can secure the storage and search there.

With your Docker Engine, you can also control how you do log rotation on your system. This is done by editing the log-driver property in the daemon.json file. Docker is set up by default to use local logging and write files onto the host. This config can be seen below:

{ "log-driver": "local" }

 

For each log driver, you can pass in options to control how logging is handled, including the maximum size of files and how many files are saved before old ones are rotated out and deleted. This can provide a high level of flexibility for your application needs to store the right amount of information:

{ "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3", "labels": "production_status", "env": "os,customer" } }

 

For any changes made to the daemon.json file, don't forget to save the file and restart the Docker Engine service to apply these changes to the Docker Engine.

If you want to control the logging at a container level instead of the Docker Engine level, you can pass in the logging driver when executing the container. You can also configure logging driver options, just like the daemon configuration.

docker run -it --log-driver json-file alpine bash

 

Container Update Management

When running many Docker containers, you need to consider and manage whether or not these containers should be updated. This is to ensure that your container runtimes are using the latest versions and functions in their apps and to ensure you get all of the relevant security updates and patches to the underlying OS used by the containers or for the applications packages or external dependencies. Suppose you are a software developer or operator with long-running containers. This management is critical to ensure your environment is current with the latest container tags for your workloads. It is recommended that tooling be set up to improve the observability of the container versions being run so they can be better managed.

Watchtower

This open-source project contains a container that runs an automated script to check for container tag updates. This monitoring is done by inspecting the Docker Engine socket directly, allowing it to react quickly to container start-up events on the Docker Engine. You can set up this container to automatically update it to the latest version by triggering a restart or notifying you that you should perform an update manually.

The watchtower container is also easy to run! Just run it in the background to monitor your containers without further input. If you want to be more specific about the containers being updated, you can use the following command to get Watchtower to only update containers with the label autoupdate=true; you can add --label-enable autoupdate=true to the Watchtower command:

docker run \ -d \ --name watchtower \ --restart=unless-stopped \ -v /var/run/docker.sock:/var/run/docker.sock \ containrrr/watchtower --label-enable autoupdate=true --cleanup --interval 30

 

Just Script It!

You can achieve this with a simple bash script if you don't want to run another container. This is not recommended as it is cumbersome, and you will need to update the script to match your specific needs if or when they change. Below, you can find the script using the Docker CLI to fetch the required information on running containers, inspecting the container images and then looking for the new container image if it's available:

#!/bin/bash # Get running container IDs container_ids=$(docker ps -q) # Iterate over each container for container_id in $container_ids; do     # Get container image name and ID     container_image=$(docker inspect --format='{{.Config.Image}}' $container_id)     container_name=$(docker inspect --format='{{.Name}}' $container_id)     # Check if there's a newer image available     newer_image=$(docker images --format "{{.Repository}}:{{.Tag}}" $container_image | grep -F "$container_image")     if [ -n "$newer_image" ]; then         echo "Newer image available for $container_name ($container_id)"         echo "Restarting container..."         docker restart $container_id     else         echo "No newer image available for $container_name ($container_id)"     fi done

 

You can trigger this script manually, but you can also set this up to run via CRON in the background. Add the following line to your crontab file, and it will execute the script every 5 minutes.

*/5 * * * * /path/to/docker_autorestart.sh >/dev/null 2>&1

 

Container Monitoring

Stopping and starting containers is just the first step in working with them. Once you run them, you will want to monitor them to see how they operate with your applications. This monitoring can include inspecting configuration, viewing logs and metrics, or accessing the container directly via the terminal. Below, you'll find some options available to monitor your Docker Engine at multiple levels and some strategies you can adopt to make monitoring easier in the long term.

Docker CLI

This is the CLI you may be most familiar with when interacting with the Docker Engine. This CLI comes with every Docker installation and will interact directly with the Docker socket on a host or the Docker API endpoint, depending on how you have set up your Docker Engine installation. All of these configurations are controlled using the following config file on your current user who is using the Docker CLI. As this CLI interacts directly with the local or remote installations of the Docker Engine, it can access all of the information available in the Docker Engine, from the containers running to internal system data.

For most docker commands used to query information about the Docker Engine, you can use the --filter flag to get just the information you need by the container's name, label or other information on each runtime. With this CLI, you can always pipe this data into files or other tools you may already be using to further query or format the data to your needs or even schedule the data querying with a CRON or orchestration tool.

Docker Desktop

This is also a Docker product that comes with installing Docker on Windows and Mac OS. This graphical UI allows for the visual display and interaction of your Docker Engine. This tool is focused mainly on developers who are building containers to be run somewhere else. The tool has useful debugging features to quickly access container logs and debug images, such as the image layers or the packages installed in each image.

Portainer

Portainer provides a sleek, modern UI to view all the critical information on the containers running on your docker host. This connection is facilitated through a direct socket connection via a volume mount or an HTTP connection to the docker API. The UI provides information for each container, such as logs, configuration, and management actions like restarting, deleting, or editing containers.

If you are running Docker on multiple hosts, Portainer also makes it easy to view and manage each host's Docker environment through a single pane of glass, saving you time. Portainer also provides a scalable way to give others access to the Docker environment through user provisioning and resource RBAC. Note that these are available through the paid version of the app.

Labelling Containers

When running many containers within a Docker Engine, it can become difficult to find the containers you are concerned with. Often, as an operator of containers, you may ask what runtimes I am running, what package versions have vulnerabilities, who owned them, etc.

If you use containers from others like Ubuntu or Alpine, they already contain many helpful labels. You can also use these labels to filter search to find specific containers easily. This labelling can be done when building your containers to add important information from build or start-up time, such as:

• Commit to the project
• The date that it was built
• Dependency version installed
• OS version installed

You can define these labels when building your containers using the Dockerfile with the following format and the LABEL key to add your key-value pairs.

ARG COMMIT_SHA FROM <base_image> # Set labels LABEL NODE_VERSION="20" LABEL PYTHON_VERSION="3.13" LABEL COMMIT_SHA="${COMMIT_SHA}" # Other instructions in your Dockerfile

 

These labels can also be applied to your containers when they are started to add further context for the reason and environment when CICD or a user requests that the container be started manually.

docker run --label "environment=production" --label "owner=JohnDoe" <image_name>

 

Now that you have some containers defined with some labels, you should also inspect them better to understand the current state of your container fleet. For any running containers on a host with a Docker Engine, you can view all of the labels on a specific container using the following command.

docker inspect --format='{{json .Config.Labels}}' <container_id_or_name>

 

Or search all containers on a Docker Engine that matches your filter label.

docker ps --filter "label=environment=production"

 

Wrapping up

Docker is a potent tool that developers and software operators can use to leverage their time to achieve more and manage resources more efficiently. However, with just a few minor changes and possible automation, we can improve how we interact with the Docker engine to improve our performance further and get the most out of the technology.

In this post, you learnt some of these tools and strategies for managing your Docker Engine installation, including what the Docker Engine is and how you should clean it up regularly, how you can effectively manage updates for containers running in your Docker environment, and finally, some simple changes or applications you can add to improve your Docker Engine observability on both the engine and your runtimes.

If you are looking for more posts talking about Docker, Homelab or servers and how we can automate all of that with code, consider checking out the following posts:

• Running Your Own Docker Container Registry Made Easy
• 7 Apps for your Homelab in 2024
• Lessons From My Homelab

 

Connect Further

• 🌩 If you are in the market for some homelab or home server products, check our store's products.
• 📖 You can also find some of our other blog posts.
• 📶 Connect with our social media accounts on Twitter, Facebook, and Instagram.