Skip to content
Open
500 changes: 500 additions & 0 deletions design_agreements.md

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions doc/appendices/glossary.rst
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,10 @@ Glossary
All other layers in the hierarchy are defined by the combination of the configuration of the framework job and the layers defined in the input data.
In *art*, the data-layer hierarchy is `Run`\ -\ `Subrun`\ -\ `Event`.

Computational graph
The portion of the workflow graph that performs the computatoins of the reconstruction or simulation job.
This does not include the portion of the workflow graph that performs input and output operations.

Concrete data product
An object managed by the framework and for which provenance information is recorded.

Expand Down
28 changes: 14 additions & 14 deletions doc/appendices/requirements.rst
Original file line number Diff line number Diff line change
Expand Up @@ -546,7 +546,7 @@ Conceptual Requirements

The framework shall support executing programs configured by composing configurations of separate components.

- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification`
- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification`

.. req:: Graceful shutdown of framework program
:collapse:
Expand Down Expand Up @@ -662,7 +662,7 @@ Supporting Requirements

The framework shall provide an option to persist the configuration of each framework execution to the output of that execution.

- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification`
- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification`

.. req:: Record execution environment
:collapse:
Expand Down Expand Up @@ -738,7 +738,7 @@ Supporting Requirements

The framework shall allow algorithms to use the same parallelism mechanisms the framework uses to schedule the execution of algorithms.

- See :numref:`ch_subsystem_design/task_management:Task management`
- See :numref:`ch_technical_design/task_management:Task management`

.. req:: Logging resource usage per algorithm invocation
:collapse:
Expand Down Expand Up @@ -770,7 +770,7 @@ Supporting Requirements

The framework shall be able to report the global memory use of the framework program at user-specified points in time.

- See :numref:`ch_subsystem_design/monitoring:Monitoring and reporting`
- See :numref:`ch_technical_design/monitoring:Monitoring and reporting`

.. req:: Elapsed time information
:collapse:
Expand All @@ -784,7 +784,7 @@ Supporting Requirements

The framework shall have an option to provide elapsed time information for each algorithm executed in a framework program.

- See :numref:`ch_subsystem_design/monitoring:Monitoring and reporting`
- See :numref:`ch_technical_design/monitoring:Monitoring and reporting`

.. req:: Framework-independent message logging
:collapse:
Expand All @@ -796,7 +796,7 @@ Supporting Requirements

The framework shall support a logging solution that is usable in an algorithm without that algorithm explicitly relying on the framework.

- See :numref:`ch_subsystem_design/logging:Logging`
- See :numref:`ch_technical_design/logging:Logging`

.. req:: Independence from unique hardware characteristics
:collapse:
Expand All @@ -820,7 +820,7 @@ Supporting Requirements

The framework shall validate an algorithm's configuration against specifications provided at registration time.

- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification`
- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification`

.. req:: Algorithm configuration schema availability
:collapse:
Expand All @@ -832,7 +832,7 @@ Supporting Requirements

The framework shall have an option to emit an algorithm's configuration schema in human-readable form.

- See :numref:`ch_subsystem_design/configuration:Semantic structure of the configuration`
- See :numref:`ch_technical_design/configuration:Semantic structure of the configuration`

.. req:: Emit message describing data flow of program
:collapse:
Expand All @@ -855,7 +855,7 @@ Supporting Requirements

The framework shall validate the configuration of each algorithm before that algorithm processes data.

- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification`, :numref:`ch_subsystem_design/configuration:Semantic structure of the configuration`.
- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification`, :numref:`ch_technical_design/configuration:Semantic structure of the configuration`.

.. req:: I/O backend for ROOT
:collapse:
Expand Down Expand Up @@ -898,7 +898,7 @@ Supporting Requirements

The framework shall provide the ability to compare two configurations.

- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification`
- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification`

.. req:: User-selectable list of recordable execution environment components
:collapse:
Expand Down Expand Up @@ -1033,7 +1033,7 @@ Supporting Requirements

The framework shall provide a command-line interface that allows the setting of configuration parameters.

- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification`
- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification`

.. req:: Support local configuration changes
:collapse:
Expand All @@ -1044,7 +1044,7 @@ Supporting Requirements

The framework shall support the use of local configuration changes with respect to a separate complete configuration to modify the execution of a program.

- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification`
- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification`

.. req:: Configuration tracing
:collapse:
Expand All @@ -1055,7 +1055,7 @@ Supporting Requirements

The framework configuration system shall have an option to provide diagnostic information for an evaluated configuration, including origins of final parameter values.

- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification`
- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification`

.. req:: Configuration language single point of maintenance
:collapse:
Expand All @@ -1067,7 +1067,7 @@ Supporting Requirements

The language used for configuring a framework program shall include features for maintaining hierarchical configurations from a single point of maintenance.

- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification`
- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification`

.. req:: Enable identification of data sets containing chunked data products
:collapse:
Expand Down
4 changes: 2 additions & 2 deletions doc/ch_conceptual_design/hofs/filters.rst
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ A predicate expression can be evaluated on a higher-level data cell than the dat
For example, suppose none of the :product:`GoodHits` data products in a given `Spill` were suitable for processing.
It is possible to create a filter that would reject all :product:`GoodHits` data products from that `Spill` even though the predicate itself interrogated only the `Spill` information and not the lower-level good-hits information from the `APA`.

The supported grammar of the predicate expression is discussed in :numref:`ch_subsystem_design/task_management:Task Management`.
The supported grammar of the predicate expression is discussed in :numref:`ch_technical_design/task_management:Task Management`.

Registration interface
^^^^^^^^^^^^^^^^^^^^^^
Expand All @@ -47,7 +47,7 @@ The following example shows the complete registration for histogramming the filt
auto h_resource = m.resource<histogramming>();

m.observe(histogram_hits, concurrency::serial)
.input_family(product_query{.suffix = "GoodHits", .layer = "APA"},
.input_family(product_selector{.suffix = "GoodHits", .layer = "APA"},
h_resource->make<TH1F>(...))
.when("high_energy"); // <= predicate expression within the when(...) call
}
Expand Down
2 changes: 1 addition & 1 deletion doc/ch_conceptual_design/hofs/observers.rst
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ It uses the :cpp:`resource<histogramming>` interface to provide access to a puta

m.observe(histogram_hits, concurrency::serial)
.input_family(
product_query{.suffix = "GoodHits", .layer = "APA"},
product_selector{.suffix = "GoodHits", .layer = "APA"},
h_resource->make<TH1F>(...)
);
}
Expand Down
2 changes: 1 addition & 1 deletion doc/ch_conceptual_design/hofs/partitioned_folds.rst
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ The :mathfunc:`fold(sum_energies)` node in :numref:`workflow` would be represent
concurrency::unlimited // <= Allowed concurrency
)
.input_family(
product_query{.suffix = "GoodHits", .layer = "APA"}
product_selector{.suffix = "GoodHits", .layer = "APA"}
)
.output_product_suffixes("TotalHitEnergy");
}
Expand Down
2 changes: 1 addition & 1 deletion doc/ch_conceptual_design/hofs/partitioned_unfolds.rst
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ The class is then registered with Phlex via:
concurrency::unlimited // <= Allowed concurrency
)
.input_family(
product_query{.suffix = "SimDepos", .layer = "Spill"}
product_selector{.suffix = "SimDepos", .layer = "Spill"}
)
.output_product_suffixes("Waveforms");
}
Expand Down
2 changes: 1 addition & 1 deletion doc/ch_conceptual_design/hofs/predicates.rst
Original file line number Diff line number Diff line change
Expand Up @@ -47,5 +47,5 @@ The registration for the predicate node in :numref:`workflow` would look like:
PHLEX_REGISTER_ALGORITHMS(m)
{
m.predicate("high_energy", high_energy, concurrency::unlimited)
.input_family(product_query{.suffix = "GoodHits", .layer = "APA"});
.input_family(product_selector{.suffix = "GoodHits", .layer = "APA"});
}
10 changes: 5 additions & 5 deletions doc/ch_conceptual_design/hofs/transforms.rst
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ To illustrate the different ways a transform's algorithm can be registered with
PHLEX_REGISTER_ALGORITHMS(m)
{
m.transform("hit_finder", find_hits, concurrency::unlimited)
.input_family(product_query{.suffix = "Waveforms", .layer = "APA"});
.input_family(product_selector{.suffix = "Waveforms", .layer = "APA"});
}

**Transform with one argument (user-specified output product name)**
Expand All @@ -72,7 +72,7 @@ To illustrate the different ways a transform's algorithm can be registered with
PHLEX_REGISTER_ALGORITHMS(m)
{
m.transform("hit_finder", find_hits, concurrency::unlimited)
.input_family(product_query{.suffix = "Waveforms", .layer = "APA"})
.input_family(product_selector{.suffix = "Waveforms", .layer = "APA"})
.output_product_suffixes("GoodHits");
}

Expand All @@ -86,8 +86,8 @@ To illustrate the different ways a transform's algorithm can be registered with
{
m.transform("vertex_maker", make_vertices, concurrency::unlimited)
.input_family(
product_query{.suffix = "Geometry", .layer = "Job"},
product_query{.suffix = "GoodTracks", .layer = "APA"}
product_selector{.suffix = "Geometry", .layer = "Job"},
product_selector{.suffix = "GoodTracks", .layer = "APA"}
)
.output_product_suffixes("Vertices");
}
Expand All @@ -99,6 +99,6 @@ To illustrate the different ways a transform's algorithm can be registered with
PHLEX_REGISTER_ALGORITHMS(m)
{
m.transform("hit_counter", count_good_hits, concurrency::unlimited)
.input_family(product_query{.suffix = "GoodHits", .layer = "APA"})
.input_family(product_selector{.suffix = "GoodHits", .layer = "APA"})
.output_product_suffixes("NumGoodHits", "NumAllHits"); // <= One name per tuple slot
}
2 changes: 1 addition & 1 deletion doc/ch_conceptual_design/hofs/windows.rst
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ The :mathfunc:`window(make_tracks)` node in :numref:`workflow` would be represen
"APA", // <= Output data layer
concurrency::unlimited // <= Allowed concurrency
)
.input_family(product_query{.suffix = "GoodHits", .layer = "APA"})
.input_family(product_selector{.suffix = "GoodHits", .layer = "APA"})
.output_product_suffixes("GoodTracks");
}

Expand Down
2 changes: 1 addition & 1 deletion doc/ch_conceptual_design/program_configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@ To support reproducibility of data, a Phlex program is configured by *only one*
These documents may be written in the following languages: Jsonnet, YAML, JSON (as a strict subset of Jsonnet), or FHiCL :need:`DUNE 72`.
A given configuration can be written in any of these languages, but no configuration document may be written in more than one language.

The other details and rules by which a Phlex program is configured largely do not inform the conceptual design and are described in :numref:`ch_subsystem_design/configuration:Configuration`.
The other details and rules by which a Phlex program is configured largely do not inform the conceptual design and are described in :numref:`ch_technical_design/configuration:Configuration`.
26 changes: 13 additions & 13 deletions doc/ch_conceptual_design/registration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ This can be achieved by in terms of the C++ *registration stanza*:
concurrency::unlimited // 4. Allowed CPU concurrency
)
.input_family( // 5. Specification of input data-product family (see text)
product_query{.creator = "calibrate_wires", .suffix = "", .layer = "spill"}
product_selector{.creator = "calibrate_wires", .suffix = "", .layer = "spill"}
)
.output_product_suffixes(
"GoodHits" // 6. Specification of output data product from find_hits
Expand Down Expand Up @@ -64,7 +64,7 @@ Specifically, in the registration stanza above, we have the following:

:cpp:`input_family(...)`
5. The specification of the input family :math:`\ifamily{a}{\text{input}}` requires (a) the specification of the creator of each data product that serve as input family elements :need:`DUNE 65`, and (b) the label of the data layer in which each input data product is found.
In the registration code above, this is achieved by the construction of a :cpp:`product_query` with the appropriate arguments.
In the registration code above, this is achieved by the construction of a :cpp:`product_selector` with the appropriate arguments.
The example shows finding products created by a node named "calibrate_wires", with an empty product suffix, and in the layer labeled "spill".
Because the product suffix is empty, the :cpp:`.suffix` field could have been omitted altogether.

Expand Down Expand Up @@ -95,7 +95,7 @@ The block, however, must contain a registration statement to execute an algorith
Algorithms with Multiple Input Data Products
--------------------------------------------

The registration example given above in :numref:`ch_conceptual_design/registration:Framework Registration` creates an output family by applying a one-parameter algorithm :cpp:`find_hits` to each element of the input family, as specified by :cpp:`input_family(product_query{.suffix = "Waveforms", .layer = "APA"})`.
The registration example given above in :numref:`ch_conceptual_design/registration:Framework Registration` creates an output family by applying a one-parameter algorithm :cpp:`find_hits` to each element of the input family, as specified by :cpp:`input_family(product_selector{.suffix = "Waveforms", .layer = "APA"})`.
In many cases, however, the algorithm will require more than one data product.
Consider another algorithm :cpp:`find_hits_subtract_pedestals`, which forms hits by first subtracting pedestal values from the waveforms, both of which are presented to the algorithm as data products from the `APA`.
The interface of the algorithm and its registration would look like:
Expand All @@ -111,8 +111,8 @@ The interface of the algorithm and its registration would look like:
{
m.transform("find_hits", find_hits_subtract_pedestals, concurrency::unlimited)
.input_family(
product_query{.suffix = "Waveforms", .layer = "APA"},
product_query{.suffix = "Pedestals", .layer = "APA"}
product_selector{.suffix = "Waveforms", .layer = "APA"},
product_selector{.suffix = "Pedestals", .layer = "APA"}
)
.output_product_suffixes("GoodHits");
}
Expand Down Expand Up @@ -141,8 +141,8 @@ This would be expressed in C++ as:
{
m.transform("vertex_maker", make_vertices, concurrency::unlimited)
.input_family(
product_query{.suffix = "GoodTracks", .layer = "APA"},
product_query{.suffix = "Geometry", .layer = "Job"}
product_selector{.suffix = "GoodTracks", .layer = "APA"},
product_selector{.suffix = "Geometry", .layer = "Job"}
)
.output_product_suffixes("Vertices");
}
Expand All @@ -153,15 +153,15 @@ Phlex supports such uses cases :need:`DUNE 113`, even if the specified data laye
For example, suppose an algorithm needed to access a data product from a `Spill`, and it also required a calibration offset provided from an external database table :need:`DUNE 35`.
Instead of providing a separate mechanism for handling calibration constants, a separate layer could be invented (e.g. `Calibration`) whose data cells corresponded to intervals of validity.
So long as a relation can be defined between specific `Spill` data cells and specific `Calibration` data cells, the framework can use that relation to form the input family of `Spill`\ -\ `Calibration` data-product pairs that are presented to the algorithm.
How the relation between data cells is defined is referred to as *data marshaling*, and it is described further in :numref:`ch_subsystem_design/task_management:Data-Marshaling`.
How the relation between data cells is defined is referred to as *data marshaling*, and it is described further in :numref:`ch_technical_design/task_management:Data-Marshaling`.

Data Products from Adjacent Data Cells
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In some cases, it may be necessary to simultaneously access data products from adjacent data-products sets :need:`DUNE 91`, where *adjacency* is defined by the user :need:`DUNE 92`.
The notion of adjacency can be critical for (e.g.) time-windowed processing (see :numref:`ch_conceptual_design/hofs/windows:Windows`), where the details of the "next" time bin are needed to accurately calculate properties of the "current" time bin.

Supporting the processing of adjacent data cells is described further in :numref:`ch_subsystem_design/task_management:Data-Marshaling`.
Supporting the processing of adjacent data cells is described further in :numref:`ch_technical_design/task_management:Data-Marshaling`.

Accessing Configuration Information
-----------------------------------
Expand All @@ -176,13 +176,13 @@ To do this, an additional argument (e.g. :cpp:`config`) is passed to the registr
auto selected_data_layer = config.get<std::string>("data_layer");

m.transform("hit_finder", find_hits, concurrency::unlimited)
.input_family(product_query{.suffix = "Waveforms", .layer = selected_data_layer})
.input_family(product_selector{.suffix = "Waveforms", .layer = selected_data_layer})
.output_product_suffixes("GoodHits");
}

.. note::

As discussed in :numref:`ch_subsystem_design/configuration:Programmatic access to the configuration`, the registration code will have access only to the configuration relevant to the algorithm being registered, and to certain framework-level configuration such as debug level, verbosity, or parallelization options.
As discussed in :numref:`ch_technical_design/configuration:Programmatic access to the configuration`, the registration code will have access only to the configuration relevant to the algorithm being registered, and to certain framework-level configuration such as debug level, verbosity, or parallelization options.

Except for the specification of :cpp:`find_hits` as the algorithm to be invoked, and :cpp:`transform` as the HOF, all other pieces of information may be provided through the configuration.

Expand All @@ -208,7 +208,7 @@ By specifying a lambda expression that takes a :cpp:`phlex::handle<waveforms>` o
[](phlex::handle<waveforms> ws) { return find_hits_debug(*ws, ws.id().number()); },
concurrency::unlimited
)
.input_family(product_query{.suffix = "Waveforms", .layer = "APA"})
.input_family(product_selector{.suffix = "Waveforms", .layer = "APA"})
.output_product_suffixes("GoodHits");
}

Expand Down Expand Up @@ -237,7 +237,7 @@ For example, the :cpp:`find_hits` algorithm author could have instead created a

m.make<hit_finder>(sigma_threshold) // <= Make framework-owned instance of hit_finder
.transform("hit_finder", &hit_finder::find, concurrency::unlimited)
.input_family(product_query{.suffix = "Waveforms", .layer = selected_data_scope})
.input_family(product_selector{.suffix = "Waveforms", .layer = selected_data_scope})
.output_product_suffixes("GoodHits");
}

Expand Down
Loading
Loading