Tuto pour le déploiement des sites avec Docker

How to deploy on docker on BR infra

Note: all of this tutorial is particularly tricky, don't hesitate to contact a BRman to get some help.

Requirements

• Basics of git (videos here or the beginning of this awesome online book) ;
• Basics of containers (see a really good explanation here) and of docker-compose (see the getting-started here) ;
• Basics of web (see an introduction on MDN) ;
• Basics of gitlab-ci (reading the getting-started is recommended) ;

Existing architecture

A physical server called endalcher deploys and runs your container. It uses the following services to connect your service to the Internet:

• docker-compose to deploy the correct environment for your containers ;
• traefik as a reverse proxy to serve your site behind your endpoint, it is configured here ;
• nginx to serve your site to the outside world (as all the binets sites).

Deployment

To deploy a service you need to trigger a pipeline of br/docker-services. This pipeline has the rights to deploy on endalcher, and will push a docker-compose.yaml file. The pipeline will have to choose which docker-compose.yaml file to use, for that matter you will need to correctly set the variable PROJECT : the ci runner will deploy the file named docker-compose.${PROJECT}.yaml. You can also pass any parameter needed in the compose file.

In order to create the correct docker-compose.${PROJECT}.yaml file, you will need to create a Merge Request on br/docker-services, or send a docker-compose.yaml file to a BRman.

Triggering a pipeline is particularly adapted for CI jobs. For example, you could write a ci job in your repo that triggers the pipeline of br/docker-services.

Example

You want to deploy the QDJ site with docker, and bind it to the server name qdj.binets.fr. Your site is running on flask and uses a postgresql database.

Pre-existant configuration

You have a little Dockerfile:

# Base image: python3
FROM python:3
WORKDIR /usr/src/app

# Install dependacies
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

# Copy application files
COPY . .

EXPOSE 5000/tcp

# the command runs the migrations and then launches gunicorn to serve the app
CMD python -m flask db upgrade && gunicorn -b :5000 qdj:app

You have the following docker-compose.yaml:

version: '3.1'

services:
  db:
    # Use a small postgres image, as we don't need anything on this container
    image: postgres:alpine
    environment:
      POSTGRES_USER: qdj
      POSTGRES_PASSWORD: somepassword
      POSTGRES_DB: qdj
    # store the data on a persistent file
    volumes:
      - "qdj-data:/var/lib/postgresql/data"
    restart: always

  app:
    # Use the image from Dockerfile
    image: ${DOCKER_IMAGE}
    environment:
      FLASK_APP: qdj.py
      FLASK_ENV: production
      FLASK_DEBUG: 0
      DATABASE_URI: "postgresql://qdj:somepassword@db:5432/qdj"
      SECRET_KEY:
    # Tells to docker to start app after db (otherwise flask crashes)
    depends_on:
      - db
    # Always restart on crash
    restart: always
    
volumes:
  qdj-data:

Production docker-compose.yaml

We will adapt your docker-compose.yaml file in order to bind your site to the hostname :

1. First let's connect your app container to the global network. We won't bind the db container to the global network to protect it from intruders. Network graph in endalcher We will add network configuration to the docker-compose.yaml:

   • to enable the network web, we add at the bottom of the file:

     networks:
       web:
         external: true
     

   • to connect app to the network web, we add to the app configuration:

     networks:
       - web
       - default
     

2. Then we can add labels to tell traefik to match the hostname to a packet going to app:

   labels:
       - "traefik.docker.network=web"
       - "traefik.frontend.rule=Host:qdj.binets.fr"
       - "traefik.port=5000"
   

This gives us the following docker-compose.qdj.yaml:

version: '3.1'

services:

  db:
    image: postgres:alpine
    restart: always
    environment:
      POSTGRES_USER: qdj
      POSTGRES_PASSWORD: qdjpw
      POSTGRES_DB: qdj
    volumes:
      - "qdj-data:/var/lib/postgresql/data"

  app:
    image: ${DOCKER_IMAGE}
    environment:
      FLASK_APP: qdj.py
      FLASK_ENV: production
      FLASK_DEBUG: 0
      DATABASE_URI: "postgresql://qdj:qdjpw@db:5432/qdj"
      SECRET_KEY:
    depends_on:
      - db
    restart: always
    networks:
      - web
      - default
    labels:
      - "traefik.docker.network=web"
      - "traefik.frontend.rule=Host:qdj.binets.fr"
      - "traefik.port=5000"

volumes:
  qdj-data:

networks:
  web:
    external: true

This file goes into the br/docker-services repository, it will be used by the triggered pipeline to create your containers.

Triggering pipelines with Gitlab CI

In this pipeline, we will need to add 2 jobs:

• one to build the docker image using the Dockerfile
• one to trigger the br/docker-services pipeline to create the containers.

To build the docker image, we will use the template do so:

# Build docker image and upload to registry
docker:build:
  stage: build
  only:
    - master
    - stable
  image: docker:stable
  services:
    - docker:dind
  variables:
    DOCKER_HOST: tcp://docker:2375/
    DOCKER_DRIVER: overlay2
    # For disabling TLS in docker-in-docker (not needed)
    DOCKER_TLS_CERTDIR: ''
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build --pull --build-arg version=$CI_PIPELINE_IID -t $IMAGE_TAG -f deployment/Dockerfile .
    - docker push $IMAGE_TAG

The second job is rather tricky. We will use curl to trigger the pipeline, using Gitlab Pipeline API. This call is secured by a trigger token, which should be saved into your project variables as $TRIGGER_TOKEN. This is what the job looks like:

docker:deploy:
  stage: deploy
  when: manual
  script:
    - apt-get update && apt-get install -y curl
    - curl --request POST --form token=$TRIGGER_TOKEN --form ref=master --form variables[PROJECT]=qdj --form variables[DOCKER_IMAGE]=$IMAGE_TAG https://gitlab.binets.fr/api/v4/projects/560/trigger/pipeline

Pay attention to this syntax because debugging this api call takes a lot of time...

If you want to add variables that will be used in your docker-compose.yaml, add other arguments to curl like --form variables[WELCOME_MESSAGE]="Hello World!".

You can see an example of a CI using this in the br/qdj repo.

Note: for BR project, you can use the trigger syntax in the API call since you don't need autorisation.

History

• In 2017, the BR16 set up a Kubernetes cluster. They managed to set up a workflow to deploy seamlessly from gitlab on that cluster. However, the configuration was hard to maintain and the BR17 was not able to keep it going.
• In 2018, the BR17 (Oliver Facklam + Hadrien Renaud) set up a small docker service on a computer from the Salle Informatiques

Developping a site for BR docker deployement

Docker allows for most technologies to be abstracted into a container and thus most - if not all - sites can be deployed in docker. However, here is a bit of advice to make deployement less painful :

• Before using a particular external service (something that should run in a seperate container), check that a well supported docker image exists and with a BRman that there are no other constraints (for example, using a MinIO will go through a seperate dedicated server that already exists rather than a container).
• Use a postgres database to get access to an admin website for your server.