Skip to the content.

width:1000px

udocker

Part 1 - Introduction

https://github.com/indigo-dc/udocker

Mario David david@lip.pt, Jorge Gomes jorge@lip.pt

width:150px width:1200px


Scientific Computing Challenges I

Running applications across infrastructures often requires considerable effort

bg right:40% w:500px


Scientific Computing Challenges II

bg right:40% w:500px


Why using containers for applications I

Encapsulation:

Efficiency:


Why using containers for applications II

Reproducibility:

Maintainability:

bg right:40% w:500px


udocker - origin


Containers for batch processing - I


Containers for batch processing - II


udocker: Introduction - I


udocker: Introduction - II

bg right:40% w:500px


udocker advantages: deployment I


udocker advantages: deployment II


udocker advantages: execution I


udocker advantages: execution II


udocker: Command Line Interface

udocker is mainly a run-time to execute docker containers:

         
clone export help images import
inspect install load login logout
mkrepo name protect ps pull
rm rmi rmname search setup
showconf unprotect verify version create
run save      

By design udocker does not have container creation functionality. Containers can be created with other tools.


udocker: How does it work…


Programing languages and OS


Components


udocker in 4 steps - I

Step 1 - Installation:


udocker in 4 steps - II

Step 2 - Get container images:


udocker in 4 steps - III

Step 3 - Create from images:

Step 4 - Execute containers:


udocker in 4 steps - IV

The steps to fetch and execute containers are important.

The created container can be run as many times as you wish.


udocker is an integration tool

w:1100px


udocker: pull - Images I


udocker: pull - Images II

bg right:50% w:650px


udocker: Create containers - I


udocker: Create containers - II

w:1100px


udocker: Run container

w:1100px


udocker: Execution engines I


udocker: Execution engines II

Mode Base Description
P1 PRoot PTRACE accelerated (with SECCOMP filtering): DEFAULT
P2 PRoot PTRACE non-accelerated (without SECCOMP filtering)
R1 runC rootless unprivileged using user namespaces
R2 runC rootless unprivileged using user namespaces + P1
R3 runC rootless unprivileged using user namespaces + P2
F1 Fakechroot with loader as argument and LD_LIBRARY_PATH
F2 Fakechroot with modified loader, loader as argument and LD_LIBRARY_PATH
F3 Fakechroot modified loader and ELF headers of binaries + libs changed
F4 Fakechroot modified loader and ELF headers dynamically changed
S1 Singularity where locally installed using chroot or user namespaces

Selection in terms of performance

Mode Base Description
P1 PRoot System call intensive applications may suffer degradation
P2 PRoot Same limitations as P1 apply. All system calls are traced causing higher overheads than P1
R1 runC Same performance as namespace based applications
R2 runC Only for software installation and similar. Same performance as P1
R3 runC Only for software installation and similar. Same performance as P2
F1 Fakechroot All Fn modes have similar performance during execution. Frequently the Fn modes are the fastest
F2 Fakechroot Same as F1
F3 Fakechroot Same as F1. Setup can be very slow
F4 Fakechroot Same as F1. Setup can be very slow
S1 Singularity Similar to Rn

Selection in terms of interoperability I

Mode Base Description
P1 PRoot PTRACE + SECCOMP requires kernel >= 3.5. Can fall back to P2 if SECCOMP is unavailable
P2 PRoot Runs across a wide range of kernels even old ones. Can run with kernels and libraries that would fail with kernel too old
R1 runC User namespace limitations apply
R2 runC User namespace limitations apply. Same limitations as P1 also apply, this is a nested mode P1 over R
R3 runC User namespace limitations apply. Same limitations as P2 also apply, this is a nested mode P2 over R

Selection in terms of interoperability II

Mode Base Description
F1 Fakechroot May escape and load host libraries. Requires shared library compiled against same libc as in container
F2 Fakechroot Same as F1
F3 Fakechroot Requires shared library compiled against same libc as in container. Binary executables and libraries get tied to the user HOME pathname
F4 Fakechroot Same as F3. Executables and libraries can be compiled or added dynamically
S1 Singularity Not part of udocker must already exist on the system, may use user namespaces or chroot

udocker: Running applications …


udocker & Lattice QCD

OpenQCD is a very advanced code to run lattice simulations

Scaling performance as a function of the cores for the computation of application of the Dirac operator to a spinor field.

Using OpenMPI, udocker in P1 mode

bg right:40% w:550px


udocker & udocker & Molecular dynamics

Gromacs is widely used both in biochemical and non-biochemical systems.

In this comparison Gromacs was run using CUDA and OpenMP

bg right:50% w:700px


udocker & Phenomenology

MasterCode connects several complex codes. Hard to deploy. Scanning through large parameter spaces. High Throughput Computing.

C++, Fortran, many authors, legacy code. Performance Degradation (udocker in P1 mode)

Environment Compiling Running
HOST 0% 0%
DOCKER 10% 1.0%
udocker 7% 1.3%
VirtualBox 15% 1.6%
KVM 5% 2.6%

Thank you!

Questions ?

udocker@lip.pt

width:150px width:1200px


Backup slides


Other container technologies