During the latest years, a substantial amount of effort has been put into making it possible to perform analysis of High Energy Physics data with modern big data technologies from the Hadoop ecosystem, such as Apache Spark, an open-source software framework for large-scale data processing. One of the most important challenges to this effort has been the need to effectively read data located in custom storage systems from within popular big data engines.
At CERN the vast majority of physics and infrastructure data reside in a system named EOS. EOS is a disk-based, low-latency storage service with a highly scalable hierarchical namespace, which enables data access via the XRootD protocol.
In order to read files in the Hadoop ecosystem from EOS, the “Hadoop-XRootD Connector” was created. This Java-based library connects to the XRootD Client of EOS via the Java Native Interface Framework (JNI) and is capable of reading files directly, without the need to import or export files to HDFS.
The project aims to address the existing feature requirements of the “Hadoop-XRootD Connector” as well as expanding and optimizing the existing codebase of the library in order to make it production-ready. A successful outcome will allow researchers at CERN to perform analysis over PBs of physics and infrastructure data with Apache Spark and other popular big data technologies and, in addition, enable users, institutions, and citizen scientists outside CERN to easily access and analyze PBs of physics data in the Hadoop ecosystem via the EOS-based CERN Open Data Project.
A production-ready connection library between the EOS Storage Service of CERN and the Hadoop Ecosystem, working in a similar fashion as other existing connection libraries to popular file systems such as Amazon S3.
JAVA, JNI, C++, Spark, Hadoop