Docker | Create an extensible build environment

by Ralph Featured Docker, Jupyter

TL;DR

The complete code for the post is here.

General Information

Building a Docker image mainly means creating a Dockefile and specifying all the required components to install and configure.

So a Dockerfile contains at least many operating system commands for installing and configuring software and packages.

Keeping all commands in one file (Dockerfile) can lead to a confusing structure.

In this post I describe and explain an extensible structure for a Docker project so that you can reuse components for other components.

General Structure

The basic idea behind the structure is: split all installation instructions into separate installation scripts and call them individually in the dockerfile.

In the end the structure will look something like this (with some additional collaborators to be added and described later)

RUN bash /tmp/${SCRIPT_UBUNTU}
RUN bash /tmp/${SCRIPT_ADD_USER}
RUN bash /tmp/${SCRIPT_NODEJS}
RUN bash /tmp/${SCRIPT_JULIA}
RUN bash /tmp/${SCRIPT_ANACONDA}
RUN cat ${SCRIPT_ANACONDA_USER} | su user
RUN bash /tmp/${SCRIPT_CLEANUP}

For example, preparing the Ubuntu image by installing basic commands is transfered into the script 01_ubuntu_sh

Hint: There are hardly any restrictions on the choice of names (variables and files/scripts). For the scripts, I use numbering to express order.

The script contains this code:

apt-get update                                       
apt-get install --yes apt-utils
apt-get install --yes build-essential lsb-release curl sudo vim python3-pip

echo "root:root" | chpasswd

Since we will be working with several scripts, an exchange of information is necessary. For example, one script installs a package and the other needs the installation folder.

We will therefore store information needed by multiple scripts in a separate file: the environment file environment

#--------------------------------------------------------------------------------------
USR_NAME=user
USR_HOME=/home/user

GRP_NAME=work

#--------------------------------------------------------------------------------------
ANACONDA=anaconda3
ANACONDA_HOME=/opt/$ANACONDA
ANACONDA_INSTALLER=/tmp/installer_${ANACONDA}.sh

JULIA=julia
JULIA_HOME=/opt/${JULIA}-1.7.2
JULIA_INSTALLER=/tmp/installer_${JULIA}.tgz

And each installation script must start with a preamble to use this environment file:

#--------------------------------------------------------------------------------------
# get environment variables
. environment

Preparing Dockerfile and Build Environment

When building an image, Docker needs all files and scripts to run inside the image. Since we created the installation scripts outside of the image, we need to copy them into the image (run run them). This also applies to the environment file environment.

File copying is done by the Docker ADD statement.

First we need our environment file in the image, so let’s copy this:

#======================================================================================
FROM ubuntu:latest as builder

#--------------------------------------------------------------------------------------
ADD environment environment

Each block to install one required software looks like this. To be flexible, we use variable for the script names.

ARG SCRIPT_UBUNTU=01_ubuntu.sh
ADD ${SCRIPT_UBUNTU}    /tmp/${SCRIPT_UBUNTU}
RUN bash tmp/${SCRIPT_UBUNTU}

Note: We can’t run the script directly because the run bit may not be set. So we will use bash to run the text file as a script.

As an add-on, we will be using Docker’s multi-stage builds. So here is the final code:

#--------------------------------------------------------------------------------------
# UBUNTU CORE
#--------------------------------------------------------------------------------------
FROM builder as ubuntu_core
ARG SCRIPT_UBUNTU=01_ubuntu.sh
ADD ${SCRIPT_UBUNTU}    /tmp/${SCRIPT_UBUNTU}
RUN bash

Final results

Dockerfile

Here is the final Dockerfile:

#==========================================================================================
FROM ubuntu:latest as builder

#------------------------------------------------------------------------------------------
# 
#------------------------------------------------------------------------------------------
ADD environment environment
#------------------------------------------------------------------------------------------
# UBUNTU CORE
#------------------------------------------------------------------------------------------
FROM builder as ubuntu_core
ARG SCRIPT_UBUNTU=01_ubuntu.sh
ADD ${SCRIPT_UBUNTU}    /tmp/${SCRIPT_UBUNTU}
RUN bash                /tmp/${SCRIPT_UBUNTU}

#------------------------------------------------------------------------------------------
# LOCAL USER
#------------------------------------------------------------------------------------------
ARG SCRIPT_ADD_USER=02_add_user.sh
ADD ${SCRIPT_ADD_USER}  /tmp/${SCRIPT_ADD_USER}
RUN bash /tmp/${SCRIPT_ADD_USER}

#------------------------------------------------------------------------------------------
# NODEJS
#-----------------------------------------------------------------------------------------
FROM ubuntu_core as with_nodejs

ARG SCRIPT_NODEJS=10_nodejs.sh
ADD ${SCRIPT_NODEJS}    /tmp/${SCRIPT_NODEJS}
RUN bash                /tmp/${SCRIPT_NODEJS}

#--------------------------------------------------------------------------------------------------
# JULIA
#--------------------------------------------------------------------------------------------------
FROM with_nodejs as with_julia

ARG SCRIPT_JULIA=11_julia.sh
ADD ${SCRIPT_JULIA} /tmp/${SCRIPT_JULIA}
RUN bash /tmp/${SCRIPT_JULIA}

#---------------------------------------------------------------------------------------------
# ANACONDA3  with Julia Extensions
#---------------------------------------------------------------------------------------------
FROM with_julia as with_anaconda

ARG SCRIPT_ANACONDA=21_anaconda3.sh
ADD ${SCRIPT_ANACONDA} /tmp/${SCRIPT_ANACONDA}
RUN bash /tmp/${SCRIPT_ANACONDA}

#---------------------------------------------------------------------------------------------
#
#---------------------------------------------------------------------------------------------
FROM with_anaconda as with_anaconda_user
ARG SCRIPT_ANACONDA_USER=22_anaconda3_as_user.sh
ADD ${SCRIPT_ANACONDA_USER} /tmp/${SCRIPT_ANACONDA_USER}
#RUN cat ${SCRIPT_ANACONDA_USER} | su user

#---------------------------------------------------------------------------------------------
#
#---------------------------------------------------------------------------------------------
FROM with_anaconda_user as with_cleanup

ARG SCRIPT_CLEANUP=99_cleanup.sh
ADD ${SCRIPT_CLEANUP} /tmp/${SCRIPT_CLEANUP}
RUN bash /tmp/${SCRIPT_CLEANUP}

#=============================================================================================
USER    user
WORKDIR /home/user

#
CMD ["bash"]

Makefile

HERE := ${CURDIR}

CONTAINER := playground_docker

default:
	cat Makefile

build:
	docker build -t ${CONTAINER} .

clean:
	docker_rmi_all
	
run:
	docker run  -it --rm  -p 127.0.0.1:8888:8888 -v ${HERE}:/src:rw -v ${HERE}/notebooks:/notebooks:rw --name ${CONTAINER} ${CONTAINER}

notebook:
	docker run  -it --rm  -p 127.0.0.1:8888:8888 -v ${HERE}:/src:rw -v ${HERE}/notebooks:/notebooks:rw --name ${CONTAINER} ${CONTAINER} bash .local/bin/run_jupyter

Installation scripts

01_ubuntu.sh

#--------------------------------------------------------------------------------------------------
# get environment variables
. environment

#--------------------------------------------------------------------------------------------------
export DEBIAN_FRONTEND=noninteractive
export TZ='Europe/Berlin'

echo $TZ > /etc/timezone 

apt-get update                                       
apt-get install --yes apt-utils

#
apt-get -y install tzdata

rm /etc/localtime
ln -snf /usr/share/zoneinfo/$TZ /etc/localtime
dpkg-reconfigure -f noninteractive tzdata

#
apt-get install --yes build-essential lsb-release curl sudo vim python3-pip

#
echo "root:password" | chpasswd