Interfacing PODIO with Julia

About Me

I am Soumil Baldota in my sophomore year in college with a few years of development experience in C++ and Python. For me Julia was a new language but I got acclimated quickly and have had a good time contributing to HSF organization. I am thankful to my mentors and the organization.

Community Bonding

In the community bonding period, with the help of Thomas and Benedikt, I setup my development environment along with local install of podio. To get familiar with podio and the contributing workflow. I worked on some minor issues.

Issues raised:

Pull Requests:

Coding Period

Following the Community Bonding Period I began to work on a prototype for the Interface which described how the generated code would look like. The prototype was created for classes with one to one relations, one to many relations and vector members. Then I made unittests for the prototype and added them to github actions for continuous integration. The prototype can be found at the below link Prototype

After the prototype was created. I added an attribute to store the Julia type of a data member described in C++ to the MemberVariable Class and a function to get the Julia type in generator_utils.py, this was a necessary step to feed the jinja2 templates with the appropriate Julia type for code generation. After Julia type was added I made some unittests for the same and added them to test_MemberParser.py.

Following the addition of the attributes necessary for building the includes required to feed new templates that will generate Julia code. There is actual preprocessing required for building these includes required by our Constructor and the MutableStruct templates (covered further in the blog). This preprocessing is done by the preprocess_for_julia function. This functions fills our includes dictionary passed to the templates.

The equivalent of C++ classes in Julia are the Mutable Structs, these help by providing structure to our data and allow us to set the type of the data after construction which allows us to use our dataypes, by calling our Constructor. These constructors are named after the types themselves in our case, in order to abstract information from the users regarding the existence of Mutable Structs which are not interacted with, by the users. When two mutually dependent Mutable Structs are passed their Julia Types using Constructors, A Mutually Recursive/Inclusive Type Declaration occurs. This can be solved easily using forward declarations in C++ but it requires the use of either Parametric types or Abstract Types in Julia (long standing issue).

For the purpose of code generation and a cleaner approach we chose to use the parametric method. This required the list of parameters needed by each Constructor. This data is processed by the get_julia_params

The Parametric Types are used by the Datatypes, which have one to one and one to many relations and are not used by the Components as they do not have any Cyclic Relations that could cause an infinite inclusion loop, Similarly Types with pure Vector Members also do not need a Parametric Type.

The Datatypes described by the yaml input are allowed to have a namespaces. These namespaces have been implemented with the help of modules and Submodules in Julia. So for to emulate the namespaces we have grouped the code into modules, such that each Type is wrapped in its own module. For Example the MCParticle Constructor would be a part of the MCParticleModule. the modules generated in parent namespaces required the collection of children in each namespace this is handled by the function get_namespace_dict.

Usage Of EDM4hep

Usage

edm4hep

	include("edm4hep.jl")
	edm4hep.MCParticle()

	# alternatively

	include("edm4hep.jl")
	using .edm4hep: MCParticle

Using a specific datatype

	# to include the MCParticleCollection / MCParticle directly.

	include("MCParticleCollection.jl") 

	# or to just use the MCParticle alone.

	include("MCParticle.jl")
	using .MCParticleModule: MCParticle

	mcp1 = MCParticle()
	mcp1.PDG = 2212
	mcp1.mass = 0.938
	mcp1.momentumAtEndpoint = [0.0,0.0,7000.0]
	mcp1.generatorStatus=3

	mcp2 = MCParticle()
	mcp2.PDG = 1
	mcp2.mass = 0.0
	mcp2.generatorStatus=3
	mcp2.momentumAtEndpoint = [0.750,-1.569,32.191]
	push!(mcp2.parents,mcp1)

	mcpc = MCParticleCollection()
	push!(mcpc, mcp1)
	push!(mcpc, mcp2)

For more Usage Examples look at unittests for Julia

Templates

MutableStuct.jl.jinja2 template is used for generation of Structs which use the includes_jl and params_jl dictionaries created while preprocessing.
Constructor.jl.jinja2 template is used for creating corresponding Constructors which initialize the structs with default values and bring the datatypes into their own namespace/module.
JuliaCollection.jl.jinja2 template is used for creating Collection of the given Datatype.
ParentModule.jl.jinja2 template is used to create a parent namespace.

Testing

unittests were added to test the functionality of the following in the generated code:

Namespaces
Collections
Cyclic Dependency
Vector Members
Relations

The unit test suite covering the Julia code generation of the example datamodel was added to the test setup with the help of the mentors. This setup is also run in the CI workflows if a suitable Julia version is detected.