Container Orchestration with Docker Machine, Docker Stack, and Docker Swarm

An Overview of Docker tools

Docker:
- Build, ship, publish, download, and run docker images.
- When we say “Docker”, we typically mean the Docker engine, the core of the docker platform. It consists of a docker daemon, a Rest API for interacting with the daemon, and a command line interface (CLI) that talks to the daemon through the Rest API.
Docker Compose:
- Define and run multiple containers linked together on a single host.
- Useful for setting up development and testing workflows.
Docker Machine:
- Tool for provisioning and managing docker hosts (virtual hosts running docker engine).
- It automatically creates hosts, installs Docker Engine on them, then configures the docker clients.
- You can use Machine to create Docker hosts on your local machine using a virtualization software like VirtualBox or VMWare Fusion.
- Docker machine also supports various cloud providers like AWS, Azure, Digital Ocean, Google Compute Engine, OpenStack, RackSpace etc.
Docker Swarm:
- A swarm is a group of docker hosts linked together into a cluster.
- The swarm cluster consists of a swarm manager and a set of workers.
- You interact with the cluster by executing commands on the swarm manager.
- With swarm, you can deploy and scale your applications to multiple hosts.
- Swarm helps with managing, scaling, networking, service discovery, and load balancing between the nodes in the cluster.
Docker Stack:
- Define and run multiple containers on a swarm cluster.
- Just like docker-compose helps you define and run multi-container applications on a single host, docker-stack helps you define and run multi-container applications on a swarm cluster.

Creating docker hosts, initializing a swarm cluster, and deploying a real-world containerized app

Let’s do some hands-on to get acquainted with docker machine, swarm, and docker stack. In this article, we’ll learn how to create docker hosts on our local machine using docker-machine, initialize a swarm cluster, and deploy a multi-container app on the cluster using docker-stack.

STEP 1: Prepare the Application that will be deployed to swarm cluster

We’ll be deploying a simple Golang App that contains a Rest API to display the “Quote of the day”. It fetches the quote from a public API hosted at http://quotes.rest/ and caches the result in Redis so that it can return the result immediately from the cache when you hit the API next time.

You can find the application on the following Github repository.

https://github.com/callicoder/go-docker-swarm

Dockerfile

Following is the dockerfile that is used to build the image of the Go app:

# Dockerfile References: https://docs.docker.com/engine/reference/builder/

# Start from golang:1.12-alpine base image
FROM golang:1.12-alpine

# The latest alpine images don't have some tools like (`git` and `bash`).
# Adding git, bash and openssh to the image
RUN apk update && apk upgrade && \
    apk add --no-cache bash git openssh

# Add Maintainer Info
LABEL maintainer="Rajeev Singh <rajeevhub@gmail.com>"

# Set the Current Working Directory inside the container
WORKDIR /app

# Copy go mod and sum files
COPY go.mod go.sum ./

# Download all dependencies. Dependencies will be cached if the go.mod and go.sum files are not changed
RUN go mod download

# Copy the source from the current directory to the Working Directory inside the container
COPY . .

# Build the Go app
RUN go build -o main .

# Expose port 8080 to the outside world
EXPOSE 8080

# Run the executable
CMD ["./main"]

You can build and publish the image to docker registry using the following commands -

# Build the image
$ docker build -t go-docker-swarm .

# Tag the image
$ docker tag go-docker-swarm callicoder/swarm-demo-app:1.0.0

# Login to docker with your docker id
$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don\'t have a Docker ID, head over to https://hub.docker.com to create one.
Username (callicoder): callicoder
Password:
Login Succeeded

# Push the image to docker hub
$ docker push callicoder/swarm-demo-app:1.0.0

I’ve already built and published the docker image for our demo app to the docker hub at callicoder/swarm-demo-app:1.0.0

STEP 2: Define application’s services in a yaml file

The application consists of an API service and a Redis cache. We define all the application’s services in a yaml file called docker-stack.yml -

version: '3.7'

# Define services
services:
  # App Service
  app:
    # Configuration for building the docker image for the service
    image: callicoder/swarm-demo-app:1.0.0
    ports:
      - "8080:8080" # Forward the exposed port 8080 on the container to port 8080 on the host machine
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: "0.2"
          memory: 50M
      restart_policy:
        condition: on-failure  
    environment: # Pass environment variables to the service
      REDIS_URL: redis:6379    
    depends_on:
     - redis
    networks: # Networks to join (Services on the same network can communicate with each other using their name)
      - webnet

  # Redis Service   
  redis:
    image: redis:alpine
    ports:
      - "6379:6379"
    networks:
      - webnet
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints: [node.role == manager]
        
networks:
  webnet:
    driver: overlay
    attachable: true

STEP 3: Create VMs using docker-machine

Let’s create docker hosts on our local machine using docker-machine command.

$ docker-machine create --driver virtualbox swarm-vm1
$ docker-machine create --driver virtualbox swarm-vm2
$ docker-machine create --driver virtualbox swarm-vm3

We’re using virtualbox driver for creating the VMs. Make sure that you have virtualbox installed in your system.

docker-machine allows you to create docker VMs on various cloud providers as well like AWS, google cloud etc. For example, Here is how you can create a docker VM on AWS using docker-machine:
$ docker-machine create --driver amazonec2 \ 
--amazonec2-access-key <ACCESS_KEY> \ 
--amazonec2-secret-key <ACCESS_SECRET> \ 
--amazonec2-region "us-east-1" \ 
--amazonec2-instance-type "t2.micro" \ 
aws-sandbox

List VMs

After creating all the VMs, you can list them using the following command -

$ docker-machine ls
NAME        ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER     ERRORS
swarm-vm1   -        virtualbox   Running   tcp://192.168.99.100:2376           v18.09.0   
swarm-vm2   -        virtualbox   Running   tcp://192.168.99.101:2376           v18.09.0   
swarm-vm3   -        virtualbox   Running   tcp://192.168.99.102:2376           v18.09.0

STEP 4: Create a Swarm cluster

Initialize a Swarm cluster

Let’s initialize a swarm cluster by executing docker swarm init command on the first VM.

$ docker-machine ssh swarm-vm1 "docker swarm init --advertise-addr 192.168.99.100"
Swarm initialized: current node (j5obt23bbdcvphmgncd97q5r6) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-073v8uw449vvkwngz6g0mtgivv50dbbqzt2o9f9jmwvwrkihoj-6glz90svgl4t21z1nnio0oce2 192.168.99.100:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

The first machine which initializes a swarm becomes the swarm manager. Other VMs can join the cluster using docker swarm join command.

Add Workers to the Swarm cluster

Let’s add other two VMs as workers to the swarm cluster -

$ docker-machine ssh swarm-vm2 "docker swarm join --token SWMTKN-1-073v8uw449vvkwngz6g0mtgivv50dbbqzt2o9f9jmwvwrkihoj-6glz90svgl4t21z1nnio0oce2 192.168.99.100:2377"
This node joined a swarm as a worker.

$ docker-machine ssh swarm-vm3 "docker swarm join --token SWMTKN-1-073v8uw449vvkwngz6g0mtgivv50dbbqzt2o9f9jmwvwrkihoj-6glz90svgl4t21z1nnio0oce2 192.168.99.100:2377"
This node joined a swarm as a worker.

List all nodes in the swarm cluster

You can now list all the nodes that are part of the cluster by executing docker node ls command on the swarm manager -

$ docker-machine ssh swarm-vm1 "docker node ls"
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
j5obt23bbdcvphmgncd97q5r6 *   swarm-vm1           Ready               Active              Leader              18.09.0
yb7bab6suhxsfuqlwivi090ir     swarm-vm2           Ready               Active                                  18.09.0
tqmisvt9nu8v8o7yn26rh9cox     swarm-vm3           Ready               Active                                  18.09.0

Configure the current shell to talk to the docker daemon on the swarm manager

So far, we’ve been using docker-machine ssh to execute commands on the swarm manager. Instead of doing that every time, you can also configure your current shell to talk to the docker daemon on the swarm manager directly -

$  docker-machine env swarm-vm1
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.100:2376"
export DOCKER_CERT_PATH="/Users/callicoder/.docker/machine/machines/swarm-vm1"
export DOCKER_MACHINE_NAME="swarm-vm1"
# Run this command to configure your shell: 
# eval $(docker-machine env swarm-vm1)

$ eval $(docker-machine env swarm-vm1)

That’s it. Your current shell is now configured to talk to the swarm manager’s docker daemon directly. You can run docker-machine ls command to verify that swarm-vm1 is the currently active VM -

$ docker-machine ls

NAME        ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER     ERRORS
swarm-vm1   *        virtualbox   Running   tcp://192.168.99.100:2376           v18.09.0   
swarm-vm2   -        virtualbox   Running   tcp://192.168.99.101:2376           v18.09.0   
swarm-vm3   -        virtualbox   Running   tcp://192.168.99.102:2376           v18.09.0

The asterisk in the ACTIVE column indicates that swarm-vm1 is the active machine. You can also get the currently active machine like this -

$ docker-machine active
swarm-vm1

STEP 4: Deploy the app on the Swarm cluster using docker stack

Finally, Let’s deploy the application to the swarm cluster using docker stack -

$ docker stack deploy -c docker-stack.yml swarm-stack
Creating network swarm-stack_webnet
Creating service swarm-stack_app
Creating service swarm-stack_redis

That’s it, the application is now running. You can hit the API by typing the following command -

$ curl http://192.168.99.100:8080
2cf29ce35cee: Welcome! Please hit the `/qod` API to get the quote of the day.

$ curl http://192.168.99.100:8080/qod
f689ae553c6b: If I work as hard as I can, I wonder how much I can do in a day?

List all the services in the stack

$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                             PORTS
mtbekecn7dd2        swarm-stack_app     replicated          3/3                 callicoder/swarm-demo-app:1.0.0   *:8080->8080/tcp
09be7or5dr1o        swarm-stack_redis   replicated          1/1                 redis:alpine                      *:6379->6379/tcp

Check out service logs

$ docker service logs swarm-stack_app
swarm-stack_app.2.yi2ie8euinry@swarm-vm3    | 2019/02/03 06:59:13 Starting Server
swarm-stack_app.2.yi2ie8euinry@swarm-vm3    | 2019/02/03 06:59:20 Cache miss for date  2019-02-03
swarm-stack_app.2.yi2ie8euinry@swarm-vm3    | 2019/02/03 06:59:21 Quote API Returned:  200 OK
swarm-stack_app.2.yi2ie8euinry@swarm-vm3    | 2019/02/03 07:01:15 Cache Hit for date  2019-02-03
swarm-stack_app.2.yi2ie8euinry@swarm-vm3    | 2019/02/03 07:02:04 Cache Hit for date  2019-02-03
swarm-stack_app.2.yi2ie8euinry@swarm-vm3    | 2019/02/03 07:02:11 Cache Hit for date  2019-02-03

List all the tasks in the stack

Tasks are instances of services running in the stack. Our app service has three replicas, one on every VM. The redis server however is running only on the manager -

$ docker stack ps swarm-stack
ID                  NAME                    IMAGE                             NODE                DESIRED STATE       CURRENT STATE           ERROR                       PORTS
nccbholn9pu8        swarm-stack_app.1       callicoder/swarm-demo-app:1.0.0   swarm-vm2           Running             Running 7 minutes ago                               
qrdt9ba3xrtk        swarm-stack_redis.1     redis:alpine                      swarm-vm1           Running             Running 7 minutes ago                               
yi2ie8euinry        swarm-stack_app.2       callicoder/swarm-demo-app:1.0.0   swarm-vm3           Running             Running 7 minutes ago                               
ovu9ngd15ep4        swarm-stack_app.3       callicoder/swarm-demo-app:1.0.0   swarm-vm1           Running             Running 7 minutes ago

Tear down the stack

You can tear down the stack with the following command -

$ docker stack rm swarm-stack
Removing service swarm-stack_app
Removing service swarm-stack_redis
Removing network swarm-stack_webnet

Conclusion

That’s all folks. You can find the complete code for this article on Github:

https://github.com/callicoder/go-docker-swarm

Check out the following guides on official docker website to learn more:

Container Orchestration with Docker Machine, Docker Stack, and Docker Swarm

An Overview of Docker tools

Creating docker hosts, initializing a swarm cluster, and deploying a real-world containerized app

STEP 1: Prepare the Application that will be deployed to swarm cluster

STEP 2: Define application’s services in a yaml file

STEP 3: Create VMs using docker-machine

List VMs

STEP 4: Create a Swarm cluster

Initialize a Swarm cluster

Add Workers to the Swarm cluster

List all nodes in the swarm cluster

Configure the current shell to talk to the docker daemon on the swarm manager

STEP 4: Deploy the app on the Swarm cluster using docker stack

List all the services in the stack

List all the tasks in the stack

Tear down the stack

Conclusion

Deploying a full-stack Spring boot, Mysql, and React app on Kubernetes with Persistent Volumes and Secrets

8 Software engineering principles to live by

Deploying a stateless Go app with Redis on Kubernetes

Deploying a containerized Go app on Kubernetes

Stock Span Problem

Java CompletableFuture Tutorial with Examples

Spring Boot + Spring Security + JWT + MySQL + React Full Stack Polling App - Part 1

Building a chat application with Spring Boot and WebSocket

Spring Boot, MySQL, JPA, Hibernate Restful CRUD API Tutorial

Building a Restful CRUD API with Node.js, Express and MongoDB

An Overview of Docker tools

Creating docker hosts, initializing a swarm cluster, and deploying a real-world containerized app

STEP 1: Prepare the Application that will be deployed to swarm cluster

STEP 2: Define application’s services in a yaml file

STEP 3: Create VMs using docker-machine

List VMs

STEP 4: Create a Swarm cluster

Initialize a Swarm cluster

Add Workers to the Swarm cluster

List all nodes in the swarm cluster

Configure the current shell to talk to the docker daemon on the swarm manager

STEP 4: Deploy the app on the Swarm cluster using docker stack

List all the services in the stack

List all the tasks in the stack

Tear down the stack

Conclusion

Share on social media