APS Data Management System

The APS Data Management system consists of tools to help automate the transfer of data between acquisition devices, computing resources, and data storage systems, and distribute data to users.
Overview

The APS Data Management system is designed to help streamline processing files created during data collection, and eases the process of electronic access to data for users. The main design goal for the system is to alleviate tedious data management tasks for beamline staff, while ensuring the integrity and security of data. By simplifing remote electronic access to data for users, it allows them to transfer or analyze their data shortly after their beam time.

Software engineers in the XSD Scientific Software Engineering & Data Management group and beamline scientists at the APS have been working closely with the Globus Services team and the Argonne Leadership Computing Facility (ALCF) to implement and deploy data management tools that integrate with beamline data workflows, and large data storage systems. These tools help automate the transfer of data between acquisition devices, computing resources, and data storage systems. Ownership and access permissions are maintained based on an experiment’s user group. A metadata catalog allows beamline staff to populate experiment conditions and information for access via a web portal. Users can download data at their home institutions using Globus Online. Short-term data storage is available within the APS; larger storage systems for longer-term storage are hosted by the ALCF (see diagram). These resources are now available at a number of XSD beamlines: 1-ID, 6-ID, 7-ID, 8-ID, 11-ID, 33-ID, and 34-ID.

In the coming years, these tools will be deployed at other APS beamlines. Based on feedback, these resources will be improved with new features and capabilities.
Distribution

https://subversion.xray.aps.anl.gov/DataManagement/

If you would like to deploy data management tools at your beamline, please contact someone from the SDM group.
Acknowledgements

This work is primarily developed and maintained by Sinisa Veseli of the XSD Scientific Software Engineering & Data Management group. Many helped to make this work possible – APS/XSD: Jon Almer, Francesco De Carlo, Barbara Frosik, Arthur Glowacki, Doga Gursoy, Peter Kenesei, Faisal Khan, Wenjun Liu, Suresh Narayanan, Jun-Sang Park, Jon Tischler, Stefan Vogt, and Ruqing Xu; APS/AES – Brian Pruitt, Brian Robinson, Giampiero Sciutto, Ken Sidorowicz, and Dave Wallis; CELS/ALCF: William Allcock and Mike Papka; Globus Services: Rachana Ananthakrishnan and Ian Foster.

Work at the APS is supported and maintained by the XSD Scientific Software Engineering & Data Management group (XSD-SDM), the AES Information Technology (AES-IT) group, and many XSD beamline staff with funding from DE-AC02-06CH11357 and initially from the Tao of Fusion LDRD. The Extrepid and Petrel systems are supported by the Computing, Environment, and Life Sciences (CELS) directorate and the Argonne Leadership Computing Facility (ALCF) with funding from the Department of Energy, Office of Science.