Rucio - WebDAV/HTTP-based transfer orchestration

Description

Rucio is a data management system for modern large-scale scientific experiments. It allows experiments to deal with vast amounts of data in a scalable, modular, and flexible way. For example, the ATLAS experiment orchestrated an Exabyte of data transfer and processing and is growing rapidly. Many exciting experiments have adopted Rucio or are actively evaluating it. To orchestrate these global transfers between data-centres, Rucio supports pluggable external transfer tools dedicated to large-scale experiments. However, especially for smaller communities the deployment and operation of such an external transfer tool involves a significant amount of operations and resources. For this reason we are seeking a motivated student to design, implement, and test a lightweight WebDAV/HTTP-based transfer tool that supports the Rucio transfertool API natively and internally. If you are interested in large-scale data management, networked systems, and open protocols then come and join our open development team and contribute to Github repository!

Task idea

Development of a plugin module for the Rucio transfer system, which can enable third-party copy of files using the WebDAV/HTTP standard.

Expected results

Objective 1 - Setup a Rucio development environment

Objective 2 - Design an architecture for a lightweight WebDAV/HTTP-based transfer tool

Objective 3 - Implement the lightweight WebDAV/HTTP-based transfer tool

Objective 4 - Report

Requirements

Mentors

Corresponding Project

Participating Organizations