DevOps

Easy Container Cleanup in Cron + Docker Environments

By nature, Docker containers are temporary. Yet as anyone who runs containers via Cron can tell you, it’s not always so great at cleaning up after itself. In today’s article, we are going to explore two flags that make it easier to run Dockerized Cron jobs.

Before we get too far into this article, let’s explore how we would traditionally create and clean up a Docker container.

Creating a Simple Hello World Container

For this article, we will use the official “Hello World” Docker container. Let’s jump right in by downloading this container and running it with the docker run command.

$ docker run hello-world  
Unable to find image 'hello-world:latest' locally  
latest: Pulling from library/hello-world  
ca4f61b1923c: Pull complete  
Digest: sha256:445b2fe9afea8b4aa0b2f27fe49dd6ad130dfe7a8fd0832be5de99625dad47cd  
Status: Downloaded newer image for hello-world:latest  

Hello from Docker!

This container is simple — it prints a hello message from Docker and quickly exists. In theory, this container was “temporary.” If we run the docker ps command, it no longer appears.

$ docker ps  
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

The above shows no running containers. But what happens if we add the -a flag?

$ docker ps -a  
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES  
cc7edd5e71f8        hello-world         "/hello"            3 minutes ago       Exited (0) 3 minutes ago                       sad_goldberg

The above docker ps -a output shows that we still have a hello-world container.

Why didn’t we see this in our first example? By default, the docker ps command only shows running containers. The -a flag tells the docker ps command to list all containers regardless of status.

So what does this tell us? It tells us that while the hello-world container was running temporarily, the container itself is not actually temporary. It still exists even though it is not running.

In fact, we can even restart that container using the docker start command.

$ docker start -a sad_goldberg  

Hello from Docker!

So how would we clean up this container? We remove it of course.

Removing a Container

To remove our example container, we can execute the docker rm command.

$ docker rm sad_goldberg  
sad_goldberg

$ docker ps -a  
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

We can see from the above that our container is gone, thus making it once again “temporary.”

Why Does This Matter?

While the above information is useful, one might ask, “Why does this matter?” If you recall the beginning of this article, I mentioned running Docker containers as Cron jobs. Some environments will use Cron to launch scheduled Docker containers.

Using Docker with Cron gives users the benefits of Dockerizing an application while still using the application within a simple scheduled tasks environment.

A common problem that occurs in those environments is too many containers. Let’s take a look at a simple Cron + Docker example.

# Run hello-world every minute

* * * * * /usr/local/bin/docker run hello-world

The above Cron job will start a new hello-world container every minute. What’s problematic is that it will create a new container every minute. This means after 100 minutes of saving this crontab, we will have 100 hello-world containers. This is a sure-fire way to create fun and interesting production issues.

Like anything, Docker containers take up resources. Running containers will use memory, CPU, and disk space. Non-running containers take up resources as well. Specifically, non-running containers will use disk space.

While our example container is fairly small, a real world container is likely to be much larger. Over time, creating new containers over and over again will start to use up the available disk space on the host.

When all the disk space on a host is consumed, bad things happen.

How Do We Solve This?

There are a few ways to solve this issue. One is to leverage a container managment platform such as Kuberentes. However, in many cases even with things like Minikube, that might be a bit of overkill.

Another simple solution, is to leverage the –rm flag.

Using the –rm flag

The --rm flag is an option passed on execution of the docker run command. This flag tells Docker to automatically remove the container after execution. To get a better understanding, let’s go ahead and run another hello-world container, this time with the --rm flag specified.

$ docker run --rm hello-world  

Hello from Docker!

The execution and output of the container didn’t change, but let’s look at docker ps -a once again.

$ docker ps -a  
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

This time when we look at our container list, we see nothing. This shows exactly how the --rm flag can be useful. Useful, by removing the container once it’s complete.

Using this flag, we can solve our problem of creating too many containers. However, there is another common issue that many Cron + Docker environments run into: the same task running twice.

In some cases, it doesn’t matter if two copies of a Cron job are running at the same time. However, more often than not, Cron jobs are not generally designed to be executed more than once at a time. This can sometimes show itself as issues with resource utilization (ie, CPU or Memory usage). Or in cases such as Database maintenance, duplicate jobs create very troublesome issues.

To solve this, we can combine the --rm flag with another flag.

Eliminating duplicates with the –name flag

In a previous article, I mentioned using the --name flag to ensure only one instance of a container runs at a time. This is a very viable solution to our Cron example as well.

Before exploring how this can solve our issue, let’s take a second to get a refresher on how the --name flag works.

$ docker run --name example hello-world     

Hello from Docker!

By adding the --name flag to our docker run command, we are giving our hello-world container a name. The name in our example was example. If we run a docker ps at the time of execution, we can see this name in use.

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                  PORTS               NAMES  
7392326a5dc7        hello-world         "/hello"            2 seconds ago       Up Less than a second                       example

What happens when the Docker run command is executed again?

$ docker run --name example hello-world  
docker: Error response from daemon: Conflict. The container name "/example" is already in use by container "eb3c32c077aac8d7589eac90f62db22ef60d78c0abd46c48871923838818db52". You have to remove (or rename) that container to be able to reuse that name.  
See 'docker run --help'.

When we execute the docker run command a second time, we receive an error stating our example name is in use. While this is a great way to solve duplicate containers running, any subsequent run of our crontab will result in the above error.

However, if we combine the --rm flag with the --name flag, we get the best of both worlds.

$ docker run --rm --name example hello-world  

Hello from Docker!

In the above example, our container was created with the example name just as before. However, when the container was finished executing, it was removed. We can see this again if we look at the following docker ps (running in a while loop).

$ while  
while> do  
while> sleep 1  
while> docker ps -a  
while> done  
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES  
2e109d270d19        hello-world         "/hello"            1 second ago        Created                                 example  
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                  PORTS               NAMES  
2e109d270d19        hello-world         "/hello"            2 seconds ago       Up Less than a second                       example  
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

While the container was running, no other container could use the example name. However, since it was removed after execution, a new container can use that same name, solving our issue of duplicate jobs running at once.

Summary

In today’s article, we first learned the basics of creating and removing a container. We also learned how Docker can be a bit messy and leave non-running containers around.

Most importantly, what we showed today is that using both the --rm and the --name flags together can be powerful. This is especially true for those using Cron + Docker to create temporary containers — containers that need to exist while running but go away afterward.

Published on System Code Geeks with permission by Ben Cane, partner at our SCG program. See the original article here: Easy Container Cleanup in Cron + Docker Environments

Opinions expressed by System Code Geeks contributors are their own.

Ben Cane

Benjamin Cane is a systems architect in the financial services industry. He writes about Linux systems administration on his blog and has recently published his first book, Red Hat Enterprise Linux Troubleshooting Guide.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Back to top button