HSF Packaging Group Meeting #31, January 30, 2019
Participants: Graeme Stewart, Chris Burr, Guilherme Amadio, Ben Morgan, Marco Clemencic, Patrick Gartung, Chris Green, Geri Ganis, Pere Mato, Henry Schreiner, Ben Couturier.
- Group plans for 2019
- Primary: provide meaningful, testable, solutions based on the HEP use cases identified last year
- Updates from each project every few months
- Longer term: CHEP2019 paper? Updating technical note (now ~3yrs old)
- Ongoing projects:
- LCG-like stack with Spack (Graeme and Javier)
- ATLAS/FCC as first clients
- Now building Gaudi dependencies
- Tidying hep-spack package repo, some to upstream spack, use as community repo as holding place
- HSF spack-config repo for sharing
- Migrating from Homebrew to Spack to better support from-source builds and to benefit from
work of this group
- Want to also support end user installs as currently no CVMFS, though Docker/Singularity images published
- FNAL Spack/SpackDev
- Chris: Getting back to it, need more feedback on MVP
- Working on LArSoft stack, which is more complex in terms of dependencies
- Looking at adding spack-dev as a spack extension to ease use
- Patrick: working with buildcaches, found a bug that’s been reported and fix submitted. Caused by compiler path outside spack.
- LHCb: Nix/Conda, presentations today
- Portage: tutorial from Guilherme at last meeting
- EasyBuild maybe also worth looking at?
- Chris Burr: Compute Canada using Nix as base system, plus EB on top.
- Now in phase of working rather than discussing, so propose move to a baseline of 1 meeting per month
- Can have additional meetings if people want/have topics to discuss. Just let Graeme and Ben know.
- HOW2019 Workshop
- Software Tools session where packaging will give an update
- PM that is language agnostic, Multi-Platform/Architecture
- Installs binaries (no install from source support)
- Very popular in academic/data science
- Very simple to install, no root/sudo, any location supported
- Graeme: is the shell function for packages? No, just to assist in setting up “environments”
- Conda environment:
- Just a folder, switch between via “activate/deactivate” commands
- Create with “conda create”
- Then install packages in environment with “conda install”
- Packages come from “channels”:
- Anaconda Distribution (OSS/Commercial, mostly Python/R)
- Can include proprietary software like Intel/CUDA
- GUI for managing environments
- Anaconda.org: Can set up yourself, but limited storage (3-5GB)
- Any HTTP(S) can be used as a channel, provided correct structure is present
- Community Channel: Conda Forge
- GitHub org plus various CI providers for builds
- One Git repo per package
- Lots of automation
- Specialized “forges”: Bioconda, AstroConda (AstroPy seem to be setting one up)
- Can specify priorities for channels to over/underride packages
- Package “recipe” is a folder containing a file
meta.yaml (uses Jinja2 templates)
- Defined with cross-compilation in mind (build/host/run dependencies)
- Builds done with a shell script
- Build/Install environment, tested by installed to later environment
- Lots of checks to ensure relocatability
- Used to rely on system compilers (effectively CentOS5, rely on ABI versioning)
- Now more maintainable, with Anaconda 5, conda-forge migration (gcc 7)
- Now uses “Core Dependency Trees” (CDTs)
- Repackages CentOS6 package into a sysroot of conda’s pseudo-cross compilers
- Chosen because of lack of resources, possible better performance from host libraries
- All conda packages must be relocatable
- RPATH tricks
- Build/Install into very long (255 char) path
- Sed/replace on install on both text/binaries (latter appends null bytes)
- Conda in HEP?
- Huge number already using for analysis (and some experiments, Xenon1T, NEXT)
- Lack of ROOT inconvenient (was a Netherlands e-Science one, but no longer maintained), but already have cling in conda-forge
- However, now have ROOT 6.16 with gcc7 and Python 2.7/3.6/3.7
- Though need to use CC/CXX env vars to access compiler
- Currently conda-forge doesn’t support unreleased versions, and struggle with CI resources
- Builds of conda package in ROOT Nightlies to be added
- Conda very young, but lots of progress recently, especially on compilers and binary compatibility
- Does support build variants, version pinning, exporting/importing environments
- Pere: How are versions controlled?
- Can specify version when building, as long as binary compatibility is specified
- Don’t have to rebuild everything if versions are pinned
- Patrick: How is ABI compatibility managed?
- Relies on package maintainer and upstream looking after this and being honest about it
- Graeme: Is Conda now capable of managing multiple environments?
- Yes, just activate/deactivate as needed
- Patrick: very similar to Spack’s environments (Spack may even been inspired by Conda)
- Ben C: How are binaries similar to Python wheels?
- Similar, just rely on a few expected system packages (derived from CentOS5, but patches on top to allow, e.g. C++11)
- Chris: nothing limits how “deep” Conda can go, so could build down to libc if needed
- Graeme: How are dependencies specified?
- In the
meta.yaml file under
- Can also track features in this way like
avx for example
- Graeme: are variants combinable? Yes
Nix (Chris Burr)