{"id":8693,"date":"2022-03-06T17:31:48","date_gmt":"2022-03-06T16:31:48","guid":{"rendered":"https:\/\/via-internet.de\/blog\/?p=8693"},"modified":"2022-03-06T17:33:50","modified_gmt":"2022-03-06T16:33:50","slug":"docker-create-an-extensible-build-environment","status":"publish","type":"post","link":"https:\/\/via-internet.de\/blog\/2022\/03\/06\/docker-create-an-extensible-build-environment\/","title":{"rendered":"Docker | Create an extensible build environment"},"content":{"rendered":"\n

TL;DR<\/h2>\n\n\n\n

The complete code for the post is here<\/a>.<\/p>\n\n\n\n

General Information<\/h2>\n\n\n\n

Building a Docker image mainly means creating a Dockefile and specifying all the required components to install and configure.<\/p>\n\n\n\n

So a Dockerfile contains at least many operating system commands for installing and configuring software and packages.<\/p>\n\n\n\n

Keeping all commands in one file (Dockerfile) can lead to a confusing structure.<\/p>\n\n\n\n

In this post I describe and explain an extensible structure for a Docker project so that you can reuse components for other components.<\/p>\n\n\n\n

General Structure<\/h2>\n\n\n\n

The basic idea behind the structure is: split all installation instructions into separate installation scripts and call them individually in the dockerfile.<\/p>\n\n\n\n

In the end the structure will look something like this (with some additional collaborators to be added and described later)<\/p>\n\n\n\n

RUN bash \/tmp\/${SCRIPT_UBUNTU}\nRUN bash \/tmp\/${SCRIPT_ADD_USER}\nRUN bash \/tmp\/${SCRIPT_NODEJS}\nRUN bash \/tmp\/${SCRIPT_JULIA}\nRUN bash \/tmp\/${SCRIPT_ANACONDA}\nRUN cat ${SCRIPT_ANACONDA_USER} | su user\nRUN bash \/tmp\/${SCRIPT_CLEANUP}<\/pre>\n\n\n\n
For example, preparing the Ubuntu image by installing basic commands is transfered into the script 01_ubuntu_sh<\/code><\/p>\n\n\n\n
Hint<\/strong>: There are hardly any restrictions on the choice of names (variables and files\/scripts). For the scripts, I use numbering to express order.<\/p>\n\n\n\n
The script contains this code:<\/p>\n\n\n\n
apt-get update                                       \napt-get install --yes apt-utils\napt-get install --yes build-essential lsb-release curl sudo vim python3-pip\n\necho \"root:root\" | chpasswd<\/pre>\n\n\n\n
Since we will be working with several scripts, an exchange of information is necessary. For example, one script installs a package and the other needs the installation folder.<\/p>\n\n\n\n
We will therefore store information needed by multiple scripts in a separate file: the environment file environment<\/code><\/p>\n\n\n\n
#--------------------------------------------------------------------------------------\nUSR_NAME=user\nUSR_HOME=\/home\/user\n\nGRP_NAME=work\n\n#--------------------------------------------------------------------------------------\nANACONDA=anaconda3\nANACONDA_HOME=\/opt\/$ANACONDA\nANACONDA_INSTALLER=\/tmp\/installer_${ANACONDA}.sh\n\nJULIA=julia\nJULIA_HOME=\/opt\/${JULIA}-1.7.2\nJULIA_INSTALLER=\/tmp\/installer_${JULIA}.tgz<\/pre>\n\n\n\n
And each installation script must start with a preamble to use this environment file:<\/p>\n\n\n\n
#--------------------------------------------------------------------------------------\n# get environment variables\n. environment<\/pre>\n\n\n\n
Preparing Dockerfile and Build Environment<\/h2>\n\n\n\n
When building an image, Docker needs all files and scripts to run inside the image. Since we created the installation scripts outside of the image, we need to copy them into the image (run run them). This also applies to the environment file environment<\/code>.<\/p>\n\n\n\n
File copying is done by the Docker ADD statement.<\/p>\n\n\n\n
First we need our environment<\/code> file in the image, so let’s copy this:<\/p>\n\n\n\n
#======================================================================================\nFROM ubuntu:latest as builder\n\n#--------------------------------------------------------------------------------------\nADD environment environment<\/pre>\n\n\n\n
 Each block to install one required software looks like this. To be flexible, we use variable for the script names.<\/p>\n\n\n\n
ARG SCRIPT_UBUNTU=01_ubuntu.sh\nADD ${SCRIPT_UBUNTU}    \/tmp\/${SCRIPT_UBUNTU}\nRUN bash tmp\/${SCRIPT_UBUNTU}<\/pre>\n\n\n\n
Note: We can’t run the script directly because the run bit may not be set. So we will use bash to run the text file as a script.<\/p>\n\n\n\n
As an add-on, we will be using Docker’s multi-stage builds. So here is the final code:<\/p>\n\n\n\n
#--------------------------------------------------------------------------------------\n# UBUNTU CORE\n#--------------------------------------------------------------------------------------\nFROM builder as ubuntu_core\nARG SCRIPT_UBUNTU=01_ubuntu.sh\nADD ${SCRIPT_UBUNTU}    \/tmp\/${SCRIPT_UBUNTU}\nRUN bash <\/pre>\n\n\n\n
Final results<\/h2>\n\n\n\n
Dockerfile<\/h3>\n\n\n\n
Here is the final Dockerfile:<\/p>\n\n\n\n
#==========================================================================================\nFROM ubuntu:latest as builder\n\n#------------------------------------------------------------------------------------------\n# \n#------------------------------------------------------------------------------------------\nADD environment environment\n#------------------------------------------------------------------------------------------\n# UBUNTU CORE\n#------------------------------------------------------------------------------------------\nFROM builder as ubuntu_core\nARG SCRIPT_UBUNTU=01_ubuntu.sh\nADD ${SCRIPT_UBUNTU}    \/tmp\/${SCRIPT_UBUNTU}\nRUN bash                \/tmp\/${SCRIPT_UBUNTU}\n\n#------------------------------------------------------------------------------------------\n# LOCAL USER\n#------------------------------------------------------------------------------------------\nARG SCRIPT_ADD_USER=02_add_user.sh\nADD ${SCRIPT_ADD_USER}  \/tmp\/${SCRIPT_ADD_USER}\nRUN bash \/tmp\/${SCRIPT_ADD_USER}\n\n#------------------------------------------------------------------------------------------\n# NODEJS\n#-----------------------------------------------------------------------------------------\nFROM ubuntu_core as with_nodejs\n\nARG SCRIPT_NODEJS=10_nodejs.sh\nADD ${SCRIPT_NODEJS}    \/tmp\/${SCRIPT_NODEJS}\nRUN bash                \/tmp\/${SCRIPT_NODEJS}\n\n#--------------------------------------------------------------------------------------------------\n# JULIA\n#--------------------------------------------------------------------------------------------------\nFROM with_nodejs as with_julia\n\nARG SCRIPT_JULIA=11_julia.sh\nADD ${SCRIPT_JULIA} \/tmp\/${SCRIPT_JULIA}\nRUN bash \/tmp\/${SCRIPT_JULIA}\n\n#---------------------------------------------------------------------------------------------\n# ANACONDA3  with Julia Extensions\n#---------------------------------------------------------------------------------------------\nFROM with_julia as with_anaconda\n\nARG SCRIPT_ANACONDA=21_anaconda3.sh\nADD ${SCRIPT_ANACONDA} \/tmp\/${SCRIPT_ANACONDA}\nRUN bash \/tmp\/${SCRIPT_ANACONDA}\n\n#---------------------------------------------------------------------------------------------\n#\n#---------------------------------------------------------------------------------------------\nFROM with_anaconda as with_anaconda_user\nARG SCRIPT_ANACONDA_USER=22_anaconda3_as_user.sh\nADD ${SCRIPT_ANACONDA_USER} \/tmp\/${SCRIPT_ANACONDA_USER}\n#RUN cat ${SCRIPT_ANACONDA_USER} | su user\n\n#---------------------------------------------------------------------------------------------\n#\n#---------------------------------------------------------------------------------------------\nFROM with_anaconda_user as with_cleanup\n\nARG SCRIPT_CLEANUP=99_cleanup.sh\nADD ${SCRIPT_CLEANUP} \/tmp\/${SCRIPT_CLEANUP}\nRUN bash \/tmp\/${SCRIPT_CLEANUP}\n\n#=============================================================================================\nUSER    user\nWORKDIR \/home\/user\n\n#\nCMD [\"bash\"]\n<\/pre>\n\n\n\n
Makefile<\/h3>\n\n\n\n
HERE := ${CURDIR}\n\nCONTAINER := playground_docker\n\ndefault:\n\tcat Makefile\n\nbuild:\n\tdocker build -t ${CONTAINER} .\n\nclean:\n\tdocker_rmi_all\n\t\nrun:\n\tdocker run  -it --rm  -p 127.0.0.1:8888:8888 -v ${HERE}:\/src:rw -v ${HERE}\/notebooks:\/notebooks:rw --name ${CONTAINER} ${CONTAINER}\n\nnotebook:\n\tdocker run  -it --rm  -p 127.0.0.1:8888:8888 -v ${HERE}:\/src:rw -v ${HERE}\/notebooks:\/notebooks:rw --name ${CONTAINER} ${CONTAINER} bash .local\/bin\/run_jupyter<\/pre>\n\n\n\n
Installation scripts<\/h3>\n\n\n\n
01_ubuntu.sh<\/h3>\n\n\n\n
#--------------------------------------------------------------------------------------------------\n# get environment variables\n. environment\n\n#--------------------------------------------------------------------------------------------------\nexport DEBIAN_FRONTEND=noninteractive\nexport TZ='Europe\/Berlin'\n\necho $TZ > \/etc\/timezone \n\napt-get update                                       \napt-get install --yes apt-utils\n\n#\napt-get -y install tzdata\n\nrm \/etc\/localtime\nln -snf \/usr\/share\/zoneinfo\/$TZ \/etc\/localtime\ndpkg-reconfigure -f noninteractive tzdata\n\n#\napt-get install --yes build-essential lsb-release curl sudo vim python3-pip\n\n#\necho \"root:password\" | chpasswd<\/pre>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
TL;DR The complete code for the post is here. General Information Building a Docker image mainly means creating a Dockefile and specifying all the required components to install and configure. So a Dockerfile contains at least many operating system commands for installing and configuring software and packages. Keeping all commands in one file (Dockerfile) can lead to a confusing structure. In this post I describe and explain an extensible structure for a Docker project so that you can reuse components for other components. General Structure The basic idea behind the structure is: split all installation instructions into separate installation scripts and […]<\/p>\n","protected":false},"author":1,"featured_media":8707,"comment_status":"closed","ping_status":"closed","sticky":true,"template":"","format":"standard","meta":{"_crdt_document":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[22,41],"tags":[],"class_list":["post-8693","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-docker","category-jupyter"],"jetpack_featured_media_url":"https:\/\/via-internet.de\/blog\/wp-content\/uploads\/2022\/03\/Bildschirmfoto-2022-03-06-um-17.22.29.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/posts\/8693","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/comments?post=8693"}],"version-history":[{"count":9,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/posts\/8693\/revisions"}],"predecessor-version":[{"id":8706,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/posts\/8693\/revisions\/8706"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/media\/8707"}],"wp:attachment":[{"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/media?parent=8693"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/categories?post=8693"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/tags?post=8693"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}