HSF Packaging Group Meeting #29, November 7, 2018
Participants: Graeme Stewart, Pere Mato, Patrick Gartung, Ben Morgan, James Amundson, Emil Obreshkov, Guilherme Amadio, Martin Ritter, Chris Green, Oana Boeriu
- Next HSF workshop is going to be 18-22 March 2019 at JLab
- Will be held jointly with WLCG and OSG
- More information to follow, but save the date
SFT Spack Update
- SFT group looking at other approaches to stack building, Spack continues as most promising tool/approach.
- New features coming (upstream) that help
- Fuzzy concretization
- Synergy with HEP/Science communities
- Goal of LCG/Spack
- Fairly complete current LCG release, built with Spack
- Use this as “base” for
- Production runtime
- Developer buildenv
- Initially targeting CentOS7 + GCC 8.2
- Package sets:
- Gaudi+Externals (broad, but not huge, number)
- FCC Software (already using Spack, so this exercises Spack Chains, lower chain being Gaudi)
- ATLAS externals (using ATLAS’s CMake recipes)
- Graeme has CentOS7 Docker image
- Add recent Spack develop branch, plus patches for Chain, ROOT recipe, from current upstream PRs
- Using Docker on a macOS laptop, mounting Spack from main laptop filesystem
- Different mount points for build/install
- Spack-Config setup under HSF github
- Not without issues, but mostly related to Docker/macOS
- Problem of building ncurses/termcap on APFS filesystem
- Needed to use disk image that is explicitly case sensitive in this case.
- After that, went well
- GCC 8 bootstrap from system compiler fine
- Minor issues with tar/fifos, possible due to docker/permissions
- Still need RELAX, but with that will complete Gaudi externals layer
- Apart from issues noted, very happy with ease of use and package availability.
- Jim: have preliminary work to query system for availability of system packages. Would it be better to set up a user other than root in the Docker image? Graeme: yes, just did the easiest thing to get things working.
- Jim will send Graeme the system query work.
- Patrick: Spack have some Dockerfiles for various platforms. Some issues with binary caches though (absolute links in bzip2 for example)
- Pere: Would like to get the prototype to the point that experiments wouldn’t have to care whether packages were built from LCGCMake or Spack.
- Patrick: Only major difference in binary tarballs would be rpaths in binaries. Using rpaths from binaries would require following spack’s install layout.
- Graeme: Views would help here
- Quick recap: based on CMake ExternalProject, uses single to define package lists and versions.
- Delivery process is split to:
- Build nodes running LCGCmake, pushes binaries to EOS
- Install on CVMFS release nodes
- Publication CVMFS
- Views are used to provide single rooted view (bin/ lib/ include/ so on) of coherent set of packages/views
- Activated by single setup script
- Non-CVMFS releases?
- For local installs, networkless, don’t have matching os/compiler
- Now have direct
lcgcmake command that can provide local installs on demand
- Supports reuse of binaries when possible, build-from-source otherwise, or mix.
- Simple to use, just clone lcgcmake repo and run command.
- Provides subcommands for tasks and queries (e.g. install, show)
- Can install packages in one go, or one-by-one
- Various options to install different versions etc
- Example use for GeantV Externals
- Relatively large set: ROOT, Geant4, VecGeom, devtools (and all their deps)
- Can use
lcgcmake install GeantV-externals
- Development can then proceed natively
- Lcgcmake currently tested on most common platforms (Ubuntu,SL,CentOS,macOS) but some prerequisites needed.
- Longer term, and in general for all packaging, have to think about what those prerequisites are.
- Use and feedback is welcome!
- Ben: is GeantV-externals a “metapackage”? Pere: effectively yes, uses a CMake custom target as the implementation.
- Patrick: If there are rpms, can there be post-install scripts? Pere: yes, can have postinstall either in rpms or via extra steps for the CMake ExternalProject.
- Anyone going to SC2018?
- No one from SFT/FNAL at least directly involved in Packaging work (Marc Paterno from FNAL is going). Possibly some
participants from CERN OpenLab.
- Noted that we have good ongoing contacts with Spack community
- Patrick: Packaging meeting at HSF meeting at JLAB? Graeme: Yes, could do so, even a hackathon?
- Next meeting: Provisionally 21st November https://indico.cern.ch/event/772263/
. Guilherme: Can give update on Portage/Docker.