Users’ Stories

“How can we do […] with Maestro Core?”

User metadata

Users may define their own metadata, using a YAML schema:

schema-name: My schema that represents my own metadata
schema-version: 1
schema-namespace: ".maestro.my-namespace."

maestro-attributes:

  - key: "my-key"
    type: int()
    required: True
    documentation: This attribute is an integer that is required,
                   meaning if the user does not set a value,
                   Maestro Core will issue a warning.
  ...

The schema syntax may be checked against Maestro Core meta-schema with a dedicated tool such as yamale

yamale -s ./attributes/maestro-schema-schema.yaml \
           --strict my_schema.yaml

Then the schema may be imported to Maestro Core via environment variable

MSTRO_SCHEMA_LIST="my_schema.yaml, ..."

making it available for attributes to be set to CDOs, similarly to core attributes.

int value = X;
mstro_cdo_attribute_set(handle,
                        ".maestro.my-namespace.my-key",
                        &value, ...);

Cherry-pick CDOs of interest based on metadata

Selector by key-value comparison or regexp

mstro_cdo_selector selector=NULL;
mstro_cdo_selector_create(...,
                          "(.maestro.my-namespace.my-key <= VALUE)",
                          &selector);

and then use Maestro Core events subscriptions.

Seamless C<->Fortran applications transfers

Maestro performs layout transformations implicitly provided the layout attributes specified on the producer side and on the consumer side differ. Here, we want the C-side MSTRO_ATTR_CORE_CDO_LAYOUT_ORDER to be set to row-major, and column-major on the Fortran-side.

int64_t patt = ROWMAJ; // 0 for row-major, 1 for column-major
mstro_cdo_attribute_set(cdo,
                        MSTRO_ATTR_CORE_CDO_LAYOUT_ORDER,
                        &patt, ...);

To that we need to also add attributes that Maestro Core will need internally to properly define the arrays, namely element size and array dimensions.

int64_t dimsz[NDIMS] = {N, M};
int64_t ndims = NDIMS;
int64_t elsz = sizeof(double);
mstro_cdo_attribute_set(cdo,
                        MSTRO_ATTR_CORE_CDO_LAYOUT_DIMS_SIZE,
                        dimsz, ...);
mstro_cdo_attribute_set(cdo,
                        MSTRO_ATTR_CORE_CDO_LAYOUT_ELEMENT_SIZE,
                        &elsz, ...);
mstro_cdo_attribute_set(cdo,
                        MSTRO_ATTR_CORE_CDO_LAYOUT_NDIMS,
                        &ndims, ...);

Of course before sealing we need a size and a pointer for the upcoming data transfer

mstro_cdo_attribute_set(cdo,
                        MSTRO_ATTR_CORE_CDO_RAW_PTR,
                        data, ...);
mstro_cdo_attribute_set(cdo,
                        MSTRO_ATTR_CORE_CDO_SCOPE_LOCAL_SIZE,
                        &size, ...);

In that setup, both producer and consumer need to specify their layout attributes, that is the producer the actual data layout, and the consumer the layout wanted. Maestro Core makes the correspondance between producer and consumer CDO thanks to their identical name, and transparently performs the transformation on the DEMANDer, and ensures the CDO is in the requested layout when the DEMAND returns.

Not let producers withdraw early

Events allow for a consumer to enquire about CDO it has no knowledge about, but might be interested in its content, depending on metadata.

mstro_subscribe(selector,
                MSTRO_POOL_EVENT_OFFER,
                true, &subscription));

takes a boolean flag for acknowledgment request, meaning the event producer is stalled by the Pool Manager until it receives an explicit from the subscriber via

mstro_subscription_ack(subscription, event);

Effectively, if we zoom in on an offer event that a consumer has subscribed to with request for acknowledgment

which means the consumer here, upon receiving the offer event, may declare and require the CDO associated with the event, before it may be withdrawn, because the producer component cannot proceed, in that the mstro_cdo_offer is not returning before receiving the offer_ack.

Making sure an app is up and running with Maestro core

CDOs may also be used as semaphores in a sense. An app may notify its readiness with an empty CDO

mstro_cdo_declare(CDO_READY_NAME, MSTRO_ATTR_DEFAULT, &cdo_ready);
mstro_cdo_offer(cdo_ready);

And typically withdraw and dispose at the very end, when it is effectively not ready to work anymore. On the other side, apps willing to check its readiness would

mstro_cdo_declare(CDO_READY_NAME, MSTRO_ATTR_DEFAULT, &cdo_ready);
mstro_cdo_require(cdo_ready);
mstro_cdo_demand(cdo_ready);
mstro_cdo_dispose(cdo_ready);

When demand returns, the probing app is certain the sender is ready.