Lightweight Container Integration into Workflow Systems:
A First Look at Singularity and Makeflow
Author/Presenter
Event Type
Workshop
TimeMonday, November 13th12:20pm -
12:30pm
Location501
DescriptionTraditionally, HPC centers have relied on professional
system administrators to install, upgrade, and manage
operating systems and application software suitable for
a particular site. This relieves end users of the burden
of managing these details, but also limits users’
freedom to deploy software environments specialized for
their task. Container technology addresses this
limitation by allowing the end user to create and deploy
custom software environments for a task. However, when
used at large scale with a workflow management system in
the most obvious way, container images can result in
considerable waste in both filesystem space and network
transfers. To address this, we consider two methods of
composing software and data elements into a container –
stacked and flat composition – and use these to create
an efficient architecture for integrating containers
into workflow systems. We demonstrate these techniques
using the Singularity lightweight container environment
and the Makeflow workflow system, demonstrating that
flat composition results in reduced storage consumption
and network transfers, on both synthetic and
bioinformatics workflow.
Author/Presenter
