Showroom

“What can we use Maestro Core for?”

Overview

Workflow applications using Maestro Core offer and demand CDOs to the Maestro Pool, a conceptual entity that represents the set of resources contributed to Maestro, in particular the set of offered CDOs. One typical use of Maestro Core is bypassing persistent storage for application coupling, by way of the Maestro Pool. The Pool Manager is a Maestro Core-produced application that handles networking, transport scheduling and propagates pool events for inspection.

CDOs are basically Maestro Core currency, they contain data and metadata, including user-defined metadata plus data semantics such as data layout information and other information on data usage the transfer scheduler may take advantage of.

Maestro core components overview

Events allow for higher-level control of the Maestro-enabled workflow, and interfacing with many useful components. In particular, they allow the implementation of data-driven Workflow Managers or more generally Execution Frameworks, meaning they can schedule jobs based on CDO availability and location which is indeed notified by events. As an example of another useful workflow components, staging and archiving CDOs of choice may be implemented by users via dedicated components, archiving relying especially on events to be notified of the availability of (possibly cherry-picked) CDOs for archiving purposes.

In addition to data management, Maestro Core facilitates memory management, allowing to abstract away memory layer consideration as well as providing features such as tiling. CDO location is expressed as a so-called Memory Object, which can mean a certain number of layers (DRAM, GDRAM, NVRAM, PFS, etc.) and associated access methods (pointer, path, object ID, etc.), which is then used by Maestro Core to handle the transport between layers and nodes.

CDOs may convey a wide range of data and memory semantics. In particular, data layout information lets Maestro make the transformations between a given offered CDO in a given data layout L1 (say row-major) to the demanded CDO on the consumer side that may have a layout L2 (say column-major). This allows user to abstract away data layout, especially if tools or source-to-source substitution fills the data layout attribute (semi) automatically. Data layouts may be distributed and Maestro Core handle the redistribution complying with producer and consumer layout and distribution scheme requirements.

Demos

A few demonstrators are shipped with Maestro Core, which are compiled and run through

make check

  • Local multi-threaded setup demo_mvp_d3_2: with a few parameters such as number of producer, consumer and archiver threads.

  • Multi-application setup check_pm_interlock: The interlock demo is a minimal workflow with Pool Manager and two applications, exchanging a CDO under the Pool Manager supervision, using filesystem and object store transport, RDMA is used if any transport unavailabilities.

  • Logger sample application simple_telemetry_listener

  • Archiver sample application simple_archiver