diff --git a/design_agreements.md b/design_agreements.md new file mode 100644 index 0000000..838ffed --- /dev/null +++ b/design_agreements.md @@ -0,0 +1,500 @@ +# Agreements and Design Decisions + +This file records design agreements and agreement-like decisions identified from: + +- GitHub issues at [Framework-R-D/phlex issues](https://github.com/Framework-R-D/phlex/issues) +- GitHub organization discussions at [Framework-R-D discussions](https://github.com/orgs/Framework-R-D/discussions) + +The agreements are organized below by Phlex subsystem, following the subsystem structure documented in [`phlex-design`](https://github.com/Framework-R-D/phlex-design), especially: + +- [Framework Subsystems](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/framework_subsystems.rst) +- [Program Startup](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/program_startup.rst) +- [Data Product Management](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/data_product_management.rst) +- [Metadata Management](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/metadata_management.rst) +- [IO](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/io.rst) +- [Resource Management](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/resource_management.rst) +- [Configuration](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/configuration.rst) +- [Algorithm Registration](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/registration.rst) + +Some entries are stronger than others: + +- Strong: explicit agreement or decision language in the thread +- Partial: meeting notes or converging design direction, but with visible caveats +- Weak/Not included: brainstorming without clear agreement + +**This is a snapshot as of 27 May 2026.** + +## Task Management + +This subsystem is described in [`framework_subsystems.rst`](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/framework_subsystems.rst) as providing the runtime scheduler and executor, workflow-graph types, and computational, provider, translator, preserver, and driver nodes. + +### Translator placement and behavior in the graph + +Primary sources: + +- [#404](https://github.com/Framework-R-D/phlex/issues/404) +- [#535](https://github.com/orgs/Framework-R-D/discussions/535) + +Strength: Strong to Partial + +Agreements and decisions identified: + +From [#404](https://github.com/Framework-R-D/phlex/issues/404): + +1. A translator is a node whose input is of one concrete data-product type and whose output is of a different concrete data-product type, both of which model the same data-product concept. +2. A translator will be automatically inserted into the graph between two nodes whose data-product types are concept-compatible but not identical. +3. The computational graph does not include providers, preservers, or the framework driver. +4. The computational graph can be disjoint. +5. Translator nodes belong to the computational graph: there are no translators that precede a provider; there are no translators that follow a preserver. + +From [#535](https://github.com/orgs/Framework-R-D/discussions/535): + +6. From the flow-graph point of view, translators are treated as first-class nodes, while being mostly irrelevant or invisible to the user. +7. Translator nodes for a given data representation conversion should be singular by construction and should not be duplicated. +8. Translators are not configured directly by the framework user; they must be discovered by the framework based on needs implied by the user configuration. +9. The framework should be able to discover translators and select which ones belong in the graph without loading every discovered translator DLL. +10. The translator metadata required for graph building should be generated at translator plugin build time. + +Important caveat from [#535](https://github.com/orgs/Framework-R-D/discussions/535): + +- There is explicit disagreement on whether an explicit provider must specify the destination layer for its output products, or whether the layer may be deduced when unambiguous. + +### Drivers and runtime hierarchy handling + +Primary sources: + +- [#395](https://github.com/Framework-R-D/phlex/issues/395) +- [#505](https://github.com/orgs/Framework-R-D/discussions/505) + +Strength: Strong + +Agreements and decisions identified: + +From [#395](https://github.com/Framework-R-D/phlex/issues/395): + +1. The hierarchy will be declared to the framework before processing any data-cell indices. +2. Different policies for supporting hierarchies will be supported: + - a fixed hierarchy may be declared for the entire job + - the global hierarchy may be updated based on hierarchies stored in input files in the middle of the job +3. Whenever the global hierarchy is established or updated, the framework validates it against what the computational graph requires. +4. Validation may involve creating more index-set nodes, provider nodes, preserver nodes, and translator nodes. +5. The job shall gracefully end when one of the input files contains a hierarchy that does not meet the hierarchy constraints of the computational graph. + +From [#505](https://github.com/orgs/Framework-R-D/discussions/505): + +6. The semantic distinction between framework drivers and data-product providers will be retained. +7. A data-cell index iterator that supplies one data-cell index at a time is sufficient for v0.3.0. +8. Range-based processing of multiple data cells at one time is not needed for v0.3.0. +9. The data-layer hierarchy will not be known until the input file is opened. +10. Phlex will implement a driver interface that enables opening an input file. +11. After opening the file, FORM will provide metadata used by the framework to synthesize a provider node for each required data product. +12. After v0.3.0, expected future needs include: + - updating processed hierarchy when opening another input file + - a mode where external data management declares file hierarchy in advance + - support for multiple concurrently open input files for multithreading or overlay use cases + +## Data Modeling + +This subsystem is described in [`framework_subsystems.rst`](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/framework_subsystems.rst) as providing mechanisms by which users define data-product types and the tools used to translate between in-memory representations of data-product concepts. + +### Data-product concepts and concrete representations + +Primary sources: + +- [#404](https://github.com/Framework-R-D/phlex/issues/404) +- [#265](https://github.com/Framework-R-D/phlex/issues/265) + +Strength: Strong to Partial + +Agreements and notes identified: + +From [#404](https://github.com/Framework-R-D/phlex/issues/404): + +1. A data-product concept can be modeled by multiple concrete data-product types. +2. A concrete data-product type may model more than one data-product concept. +3. The specification of input data products must support data-product concepts. This may include an optional explicit specification of the data-product concept. + +## Data-Product Management + +This subsystem is described in [`framework_subsystems.rst`](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/framework_subsystems.rst) and [`data_product_management.rst`](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/data_product_management.rst) as covering the organization and identification of data products, data cells, data-product lookup, families, layers, and lifetime management. + +### `product_query` semantics and lookup policy + +Primary sources: + +- [#391](https://github.com/Framework-R-D/phlex/issues/391) +- [#278](https://github.com/Framework-R-D/phlex/issues/278) +- [#506](https://github.com/orgs/Framework-R-D/discussions/506) + +Strength: Strong + +Agreements and findings identified: + +From [#391](https://github.com/Framework-R-D/phlex/issues/391): + +1. Only the data-product concept will be unconditionally required, and that could be calculated by the framework. +2. It is a framework-graph initialization error if the `product_query` object resolves to multiple data products. +3. Ambiguities can be resolved by specifying additional fields. +4. This implies that a working Phlex job may fail once a user adds another data product whose specification matches an existing `product_query`. + +Open questions noted in [#391](https://github.com/Framework-R-D/phlex/issues/391): + +- whether input files should be schema-validated +- whether a summary file or database describing a file set should be supported +- what utilities are needed to inspect available file contents +- what algorithm-configuration options should exist when an expected product is absent +- when input files should be opened during graph formation + +From [#278](https://github.com/Framework-R-D/phlex/issues/278): + +5. Duplicate specifications, as resolved from `product_query` objects, of the same input data product will be allowed. + +From [#506](https://github.com/orgs/Framework-R-D/discussions/506): + +6. A `product_query` with an absent layer specification implies that any data products matching the query will be presented to the node. +7. The data layers of data products processed by an algorithm do not necessarily need to be known before forming the data-flow graph. +8. If the fields in the `product_query` are sufficient to unambiguously connect one node to another, the data layer does not need to be known ahead of time. +9. The user must specify a product's data layer, or equivalent disambiguating information using other product-query fields, to guarantee that only products associated with that layer will be processed. + +### Data-layer semantics for transforms + +Primary source: + +- [#398](https://github.com/Framework-R-D/phlex/issues/398) + +Strength: Strong + +Agreement identified: + +1. If a transform algorithm receives input data products from unrelated data layers, the data product created by the algorithm will be associated with every input data layer. + +### Identifier restrictions in product specification + +Primary source: + +- [#343](https://github.com/Framework-R-D/phlex/issues/343) + +Strength: Strong + +Agreement identified: + +1. Phlex will enforce the restriction that the following identifiers used in specifying a data product must be representable as a valid C++ variable name: + - plugin name + - node name + - suffix + - stage name + - data-layer name element + +## Metadata Management + +This subsystem is described in [`framework_subsystems.rst`](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/framework_subsystems.rst) and [`metadata_management.rst`](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/metadata_management.rst) as covering the association and retrieval of metadata for jobs, data families, data cells, data products, and related entities. + +### Handle-accessible provenance and metadata + +Primary sources: + +- [#508](https://github.com/orgs/Framework-R-D/discussions/508) +- [#265](https://github.com/Framework-R-D/phlex/issues/265) + +Strength: Strong to Partial + +Agreements identified about handle behavior from [#508](https://github.com/orgs/Framework-R-D/discussions/508): + +1. A data-product handle must provide access to one and only one data product. +2. A constructed handle object is always valid; access to the data product is guaranteed upon successful construction. +3. Handles should not be default-constructible. +4. It is always safe to dereference a handle. +5. It is an error if a handle cannot be constructed when access to a data product is requested. + +Agreements identified about data accessible through a handle from [#508](https://github.com/orgs/Framework-R-D/discussions/508): + +6. The data product itself should be accessible. +7. The index of the data cell to which the product belongs should be accessible. +8. The data-product concept modeled by the concrete data-product type should be accessible. +9. The creator name should be accessible, including: + - algorithm name + - module name in which the algorithm was registered +10. The product suffix should be accessible. +11. The phase name or stage name should be accessible. +12. Users should not explicitly invoke anything that creates a handle; the framework creates handles as needed. + +## IO + +This subsystem is described in [`framework_subsystems.rst`](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/framework_subsystems.rst) and [`io.rst`](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/io.rst) as providing mechanisms by which data products and metadata are read and written to persistent storage, through IO plugins and backends such as FORM. + +### FORM integration and IO-backed provider synthesis + +Primary sources: + +- [#505](https://github.com/orgs/Framework-R-D/discussions/505) +- [#404](https://github.com/Framework-R-D/phlex/issues/404) + +Strength: Strong + +Agreements and decisions identified: + +From [#505](https://github.com/orgs/Framework-R-D/discussions/505): + +1. Phlex v0.3.0 will support reading one FORM file as input. +2. Phlex will enable registration of providers that return type-erased data. +3. After opening an input file, FORM will provide metadata used by the framework to synthesize a provider node for each required data product. +4. `std::type_info` will be used to request from FORM a data product with a specific type. +5. An unrequited data product will have a provider created for it. +6. For v0.3.0, an exception will be raised if more than one data product in the FORM file matches the specified product query, including its `std::type_info`. +7. Phlex or FORM must be able to generate a reproducible name for a type if no product suffix is associated with the data product. + +From [#404](https://github.com/Framework-R-D/phlex/issues/404): + +8. The data-product concept must be persisted in some form to a file through Phlex's IO backend interface, such as FORM. + +Important caveat from [#505](https://github.com/orgs/Framework-R-D/discussions/505): + +- The use of `std::type_info` is called out as potentially problematic across process boundaries. This looks like a `v0.3.0` decision rather than a forever-settled design. + +## Resource Management + +This subsystem is described in [`framework_subsystems.rst`](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/framework_subsystems.rst) and [`resource_management.rst`](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/resource_management.rst) as handling framework program resource requests and runtime resource coordination. + +### Replacement for global-state framework services + +Primary source: + +- [#515](https://github.com/orgs/Framework-R-D/discussions/515) + +Strength: Partial + +Agreement-like design direction identified: + +1. Token-based or resource-limited modeling is the intended direction for replacing framework services that rely on global state. +2. For best parallel performance, functions that require a limited resource should be kept small and fast. +3. In the histogram example discussed, modeling the consuming step as an observer using a limited resource may be preferable to modeling it as a fold over a persistent data product. + +This thread reads as converging design exploration rather than a finalized, fully settled interface. + +## Framework Application and Program Startup + +This area is described in [`framework_subsystems.rst`](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/framework_subsystems.rst) and more specifically in [`program_startup.rst`](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/program_startup.rst), which covers configuration ingestion, source and module loading, graph assembly, provider matching, translator insertion, and preserver creation. + +### Graph assembly and startup-time matching + +Primary source: + +- [#506](https://github.com/orgs/Framework-R-D/discussions/506) + +Strength: Strong + +Agreements and findings identified: + +1. The program startup design will connect output ports and input ports by an edge when the input-port `product_query` matches the data product of the output port. +2. The framework must be able to detect cycles among computational nodes. +3. It may be easier to form the graph in Python than in C++. +4. Construction of TBB flow-graph nodes may be delayed until graph-formation mechanics are largely complete. +5. User-friendly error messages must be provided for misconfigurations. + +Important caveat from [#506](https://github.com/orgs/Framework-R-D/discussions/506): + +- The terminology around computational node, pure node, and IO node is explicitly not yet settled. + +## Configuration + +This subsystem is described in [`framework_subsystems.rst`](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/framework_subsystems.rst) and [`configuration.rst`](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/configuration.rst) as handling configuration ingestion, validation, CLI interaction, evaluated configuration semantics, and programmatic access to configuration. + +### Configuration-specific discussion status + +Reviewed but not counted as a settled agreement: + +- [#522](https://github.com/orgs/Framework-R-D/discussions/522) + +Assessment: + +- No substantive design agreement was recorded in-thread. +- The discussion mostly points back to comments in [#506](https://github.com/orgs/Framework-R-D/discussions/506). + +## Registration + +This subsystem is described in [`framework_subsystems.rst`](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/framework_subsystems.rst) and [`registration.rst`](https://github.com/Framework-R-D/phlex-design/blob/main/doc/ch_subsystem_design/registration.rst) as providing the coupling between user-specified code and the framework, including the registration of algorithms, data-product types, providers, drivers, and plugins. + +### Providers and registration policy + +Primary sources: + +- [#397](https://github.com/Framework-R-D/phlex/issues/397) +- [#505](https://github.com/orgs/Framework-R-D/discussions/505) + +Strength: Strong + +Agreements identified: + +From [#397](https://github.com/Framework-R-D/phlex/issues/397): + +1. Providers attached to an input source will be automatically registered for any data products that cannot be provided by an explicit provider or a computational node. +2. Users must be allowed to indicate a precedence for looking up data products from different stages. + +From [#505](https://github.com/orgs/Framework-R-D/discussions/505): + +3. Phlex will enable registration of providers that return type-erased data. + +## Discussions Reviewed But Not Counted As Settled Agreements + +### Discussion #519: Data product support + +Source: [#519](https://github.com/orgs/Framework-R-D/discussions/519) + +Assessment: + +- Mostly references to slides and open-ended proposals. +- No clear in-thread consensus or decision text was identified. + +### Discussion #594: Asynchronous fanin/fanout + +Source: [#594](https://github.com/orgs/Framework-R-D/discussions/594) + +Assessment: + +- This is a proposal and problem statement from one participant. +- No design agreement or cross-participant consensus was identified. + +## Consolidated Remembered Agreement-Bearing Sources + +### Issues + +- [#404](https://github.com/Framework-R-D/phlex/issues/404) +- [#398](https://github.com/Framework-R-D/phlex/issues/398) +- [#397](https://github.com/Framework-R-D/phlex/issues/397) +- [#395](https://github.com/Framework-R-D/phlex/issues/395) +- [#391](https://github.com/Framework-R-D/phlex/issues/391) +- [#343](https://github.com/Framework-R-D/phlex/issues/343) +- [#278](https://github.com/Framework-R-D/phlex/issues/278) +- [#265](https://github.com/Framework-R-D/phlex/issues/265) as partial + +### Discussions + +- [#535](https://github.com/orgs/Framework-R-D/discussions/535) +- [#505](https://github.com/orgs/Framework-R-D/discussions/505) +- [#506](https://github.com/orgs/Framework-R-D/discussions/506) +- [#508](https://github.com/orgs/Framework-R-D/discussions/508) +- [#515](https://github.com/orgs/Framework-R-D/discussions/515) as partial + +## Contradictions and Tensions in the Current Collection + +The current record contains at least one real tension and one apparent contradiction. +There is a third tension that may be simple to resolve but needs to be addressed. + +### Data-Product Management: Layer-less `product_query` semantics + +Sources: + +- [#391](https://github.com/Framework-R-D/phlex/issues/391) +- [#506](https://github.com/orgs/Framework-R-D/discussions/506) + +Statements in tension: + +From [#391](https://github.com/Framework-R-D/phlex/issues/391): + +- It is a framework-graph initialization error if the `product_query` object resolves to multiple data products. + +From [#506](https://github.com/orgs/Framework-R-D/discussions/506): + +- A `product_query` with an absent layer specification implies that any data products that match the query will be presented to the node. + +Why this conflicts: + +- One source says multiple matches are an ambiguity error. +- The other says that, at least in the absent-layer case, multiple matches are intentionally accepted and delivered. + +Possible reconciliation: + +- [#391](https://github.com/Framework-R-D/phlex/issues/391) may apply only in contexts that require exactly one resolved data product. +- [#506](https://github.com/orgs/Framework-R-D/discussions/506) may represent newer or more specific graph-connection semantics. + +As written, though, these are contradictory. + +### Task Management and Program Startup: Need to know hierarchy or layers before graph formation + +Sources: + +- [#395](https://github.com/Framework-R-D/phlex/issues/395) +- [#506](https://github.com/orgs/Framework-R-D/discussions/506) + +Statements in tension: + +From [#395](https://github.com/Framework-R-D/phlex/issues/395): + +- The hierarchy will be declared to the framework before processing any data-cell indices. +- The framework validates the hierarchy against graph requirements whenever established or updated. + +From [#506](https://github.com/orgs/Framework-R-D/discussions/506): + +- Data layers of products processed by an algorithm do not necessarily need to be known before forming the graph. +- If `product_query` fields are otherwise sufficient, the data layer does not need to be known ahead of time. + +Why this conflicts: + +- One source assumes up-front hierarchy establishment and validation. +- The other says graph formation need not know layers ahead of time. + +Possible reconciliation: + +- [#395](https://github.com/Framework-R-D/phlex/issues/395) may describe runtime processing hierarchy and validation. +- [#506](https://github.com/orgs/Framework-R-D/discussions/506) may describe minimal information required for edge formation. + +Even with that possible reconciliation, the current statements point in different directions and should be treated as a live design tension. + +## Non-Contradictions or Compatible Decisions + +### IO: `std::type_info` for FORM in v0.3.0 + +Source: [#505](https://github.com/orgs/Framework-R-D/discussions/505) + +Assessment: + +- The thread records `std::type_info` as the chosen decision for v0.3.0. +- Concerns about ABI or cross-process boundaries are future-risk notes, not a rejection of the current choice. + +### Task Management: Translators as graph entities + +Sources: + +- [#404](https://github.com/Framework-R-D/phlex/issues/404) +- [#535](https://github.com/orgs/Framework-R-D/discussions/535) + +Assessment: + +- These are aligned. +- Both sources place translators inside the graph and treat them as graph-level entities. + +### Task Management and Data Modeling: Translator auto-insertion and translator discovery + +Sources: + +- [#404](https://github.com/Framework-R-D/phlex/issues/404) +- [#535](https://github.com/orgs/Framework-R-D/discussions/535) + +Assessment: + +- These are complementary. +- Auto-insertion addresses graph semantics. +- Discovery without loading all DLLs addresses implementation strategy. + +### Metadata Management: Product name to product suffix + +Source discussion: [#508](https://github.com/orgs/Framework-R-D/discussions/508) + +Related issue context: [#446](https://github.com/Framework-R-D/phlex/issues/446) + +Assessment: + +- No contradiction has been identified from the material reviewed. +- This looks like normal evolution toward a clearer naming choice. + +## Summary of the Most Important Current Tensions + +The clearest contradiction in the collected record is: + +1. [#391](https://github.com/Framework-R-D/phlex/issues/391): multiple matches are an ambiguity error +2. [#506](https://github.com/orgs/Framework-R-D/discussions/506): absent-layer queries imply all matching products are presented + +The next strongest tension is: + +1. [#395](https://github.com/Framework-R-D/phlex/issues/395): hierarchy declared and validated up front +2. [#506](https://github.com/orgs/Framework-R-D/discussions/506): layers need not be known before graph formation diff --git a/doc/appendices/glossary.rst b/doc/appendices/glossary.rst index 543ab53..6380f8a 100644 --- a/doc/appendices/glossary.rst +++ b/doc/appendices/glossary.rst @@ -39,6 +39,10 @@ Glossary All other layers in the hierarchy are defined by the combination of the configuration of the framework job and the layers defined in the input data. In *art*, the data-layer hierarchy is `Run`\ -\ `Subrun`\ -\ `Event`. + Computational graph + The portion of the workflow graph that performs the computatoins of the reconstruction or simulation job. + This does not include the portion of the workflow graph that performs input and output operations. + Concrete data product An object managed by the framework and for which provenance information is recorded. diff --git a/doc/appendices/requirements.rst b/doc/appendices/requirements.rst index 3175070..ff69d51 100644 --- a/doc/appendices/requirements.rst +++ b/doc/appendices/requirements.rst @@ -546,7 +546,7 @@ Conceptual Requirements The framework shall support executing programs configured by composing configurations of separate components. -- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification` +- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification` .. req:: Graceful shutdown of framework program :collapse: @@ -662,7 +662,7 @@ Supporting Requirements The framework shall provide an option to persist the configuration of each framework execution to the output of that execution. -- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification` +- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification` .. req:: Record execution environment :collapse: @@ -738,7 +738,7 @@ Supporting Requirements The framework shall allow algorithms to use the same parallelism mechanisms the framework uses to schedule the execution of algorithms. -- See :numref:`ch_subsystem_design/task_management:Task management` +- See :numref:`ch_technical_design/task_management:Task management` .. req:: Logging resource usage per algorithm invocation :collapse: @@ -770,7 +770,7 @@ Supporting Requirements The framework shall be able to report the global memory use of the framework program at user-specified points in time. -- See :numref:`ch_subsystem_design/monitoring:Monitoring and reporting` +- See :numref:`ch_technical_design/monitoring:Monitoring and reporting` .. req:: Elapsed time information :collapse: @@ -784,7 +784,7 @@ Supporting Requirements The framework shall have an option to provide elapsed time information for each algorithm executed in a framework program. -- See :numref:`ch_subsystem_design/monitoring:Monitoring and reporting` +- See :numref:`ch_technical_design/monitoring:Monitoring and reporting` .. req:: Framework-independent message logging :collapse: @@ -796,7 +796,7 @@ Supporting Requirements The framework shall support a logging solution that is usable in an algorithm without that algorithm explicitly relying on the framework. -- See :numref:`ch_subsystem_design/logging:Logging` +- See :numref:`ch_technical_design/logging:Logging` .. req:: Independence from unique hardware characteristics :collapse: @@ -820,7 +820,7 @@ Supporting Requirements The framework shall validate an algorithm's configuration against specifications provided at registration time. -- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification` +- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification` .. req:: Algorithm configuration schema availability :collapse: @@ -832,7 +832,7 @@ Supporting Requirements The framework shall have an option to emit an algorithm's configuration schema in human-readable form. -- See :numref:`ch_subsystem_design/configuration:Semantic structure of the configuration` +- See :numref:`ch_technical_design/configuration:Semantic structure of the configuration` .. req:: Emit message describing data flow of program :collapse: @@ -855,7 +855,7 @@ Supporting Requirements The framework shall validate the configuration of each algorithm before that algorithm processes data. -- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification`, :numref:`ch_subsystem_design/configuration:Semantic structure of the configuration`. +- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification`, :numref:`ch_technical_design/configuration:Semantic structure of the configuration`. .. req:: I/O backend for ROOT :collapse: @@ -898,7 +898,7 @@ Supporting Requirements The framework shall provide the ability to compare two configurations. -- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification` +- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification` .. req:: User-selectable list of recordable execution environment components :collapse: @@ -1033,7 +1033,7 @@ Supporting Requirements The framework shall provide a command-line interface that allows the setting of configuration parameters. -- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification` +- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification` .. req:: Support local configuration changes :collapse: @@ -1044,7 +1044,7 @@ Supporting Requirements The framework shall support the use of local configuration changes with respect to a separate complete configuration to modify the execution of a program. -- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification` +- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification` .. req:: Configuration tracing :collapse: @@ -1055,7 +1055,7 @@ Supporting Requirements The framework configuration system shall have an option to provide diagnostic information for an evaluated configuration, including origins of final parameter values. -- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification` +- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification` .. req:: Configuration language single point of maintenance :collapse: @@ -1067,7 +1067,7 @@ Supporting Requirements The language used for configuring a framework program shall include features for maintaining hierarchical configurations from a single point of maintenance. -- See :numref:`ch_subsystem_design/configuration:Mechanics of Configuration Specification` +- See :numref:`ch_technical_design/configuration:Mechanics of Configuration Specification` .. req:: Enable identification of data sets containing chunked data products :collapse: diff --git a/doc/ch_conceptual_design/hofs/filters.rst b/doc/ch_conceptual_design/hofs/filters.rst index 97ecffa..6876a06 100644 --- a/doc/ch_conceptual_design/hofs/filters.rst +++ b/doc/ch_conceptual_design/hofs/filters.rst @@ -30,7 +30,7 @@ A predicate expression can be evaluated on a higher-level data cell than the dat For example, suppose none of the :product:`GoodHits` data products in a given `Spill` were suitable for processing. It is possible to create a filter that would reject all :product:`GoodHits` data products from that `Spill` even though the predicate itself interrogated only the `Spill` information and not the lower-level good-hits information from the `APA`. -The supported grammar of the predicate expression is discussed in :numref:`ch_subsystem_design/task_management:Task Management`. +The supported grammar of the predicate expression is discussed in :numref:`ch_technical_design/task_management:Task Management`. Registration interface ^^^^^^^^^^^^^^^^^^^^^^ @@ -47,7 +47,7 @@ The following example shows the complete registration for histogramming the filt auto h_resource = m.resource(); m.observe(histogram_hits, concurrency::serial) - .input_family(product_query{.suffix = "GoodHits", .layer = "APA"}, + .input_family(product_selector{.suffix = "GoodHits", .layer = "APA"}, h_resource->make(...)) .when("high_energy"); // <= predicate expression within the when(...) call } diff --git a/doc/ch_conceptual_design/hofs/observers.rst b/doc/ch_conceptual_design/hofs/observers.rst index 3d13d19..34b3280 100644 --- a/doc/ch_conceptual_design/hofs/observers.rst +++ b/doc/ch_conceptual_design/hofs/observers.rst @@ -45,7 +45,7 @@ It uses the :cpp:`resource` interface to provide access to a puta m.observe(histogram_hits, concurrency::serial) .input_family( - product_query{.suffix = "GoodHits", .layer = "APA"}, + product_selector{.suffix = "GoodHits", .layer = "APA"}, h_resource->make(...) ); } diff --git a/doc/ch_conceptual_design/hofs/partitioned_folds.rst b/doc/ch_conceptual_design/hofs/partitioned_folds.rst index c61b1ef..e00eda5 100644 --- a/doc/ch_conceptual_design/hofs/partitioned_folds.rst +++ b/doc/ch_conceptual_design/hofs/partitioned_folds.rst @@ -137,7 +137,7 @@ The :mathfunc:`fold(sum_energies)` node in :numref:`workflow` would be represent concurrency::unlimited // <= Allowed concurrency ) .input_family( - product_query{.suffix = "GoodHits", .layer = "APA"} + product_selector{.suffix = "GoodHits", .layer = "APA"} ) .output_product_suffixes("TotalHitEnergy"); } diff --git a/doc/ch_conceptual_design/hofs/partitioned_unfolds.rst b/doc/ch_conceptual_design/hofs/partitioned_unfolds.rst index f54fb0e..86bd86f 100644 --- a/doc/ch_conceptual_design/hofs/partitioned_unfolds.rst +++ b/doc/ch_conceptual_design/hofs/partitioned_unfolds.rst @@ -141,7 +141,7 @@ The class is then registered with Phlex via: concurrency::unlimited // <= Allowed concurrency ) .input_family( - product_query{.suffix = "SimDepos", .layer = "Spill"} + product_selector{.suffix = "SimDepos", .layer = "Spill"} ) .output_product_suffixes("Waveforms"); } diff --git a/doc/ch_conceptual_design/hofs/predicates.rst b/doc/ch_conceptual_design/hofs/predicates.rst index 1aec86a..452305e 100644 --- a/doc/ch_conceptual_design/hofs/predicates.rst +++ b/doc/ch_conceptual_design/hofs/predicates.rst @@ -47,5 +47,5 @@ The registration for the predicate node in :numref:`workflow` would look like: PHLEX_REGISTER_ALGORITHMS(m) { m.predicate("high_energy", high_energy, concurrency::unlimited) - .input_family(product_query{.suffix = "GoodHits", .layer = "APA"}); + .input_family(product_selector{.suffix = "GoodHits", .layer = "APA"}); } diff --git a/doc/ch_conceptual_design/hofs/transforms.rst b/doc/ch_conceptual_design/hofs/transforms.rst index 2fa39a8..886bf8c 100644 --- a/doc/ch_conceptual_design/hofs/transforms.rst +++ b/doc/ch_conceptual_design/hofs/transforms.rst @@ -60,7 +60,7 @@ To illustrate the different ways a transform's algorithm can be registered with PHLEX_REGISTER_ALGORITHMS(m) { m.transform("hit_finder", find_hits, concurrency::unlimited) - .input_family(product_query{.suffix = "Waveforms", .layer = "APA"}); + .input_family(product_selector{.suffix = "Waveforms", .layer = "APA"}); } **Transform with one argument (user-specified output product name)** @@ -72,7 +72,7 @@ To illustrate the different ways a transform's algorithm can be registered with PHLEX_REGISTER_ALGORITHMS(m) { m.transform("hit_finder", find_hits, concurrency::unlimited) - .input_family(product_query{.suffix = "Waveforms", .layer = "APA"}) + .input_family(product_selector{.suffix = "Waveforms", .layer = "APA"}) .output_product_suffixes("GoodHits"); } @@ -86,8 +86,8 @@ To illustrate the different ways a transform's algorithm can be registered with { m.transform("vertex_maker", make_vertices, concurrency::unlimited) .input_family( - product_query{.suffix = "Geometry", .layer = "Job"}, - product_query{.suffix = "GoodTracks", .layer = "APA"} + product_selector{.suffix = "Geometry", .layer = "Job"}, + product_selector{.suffix = "GoodTracks", .layer = "APA"} ) .output_product_suffixes("Vertices"); } @@ -99,6 +99,6 @@ To illustrate the different ways a transform's algorithm can be registered with PHLEX_REGISTER_ALGORITHMS(m) { m.transform("hit_counter", count_good_hits, concurrency::unlimited) - .input_family(product_query{.suffix = "GoodHits", .layer = "APA"}) + .input_family(product_selector{.suffix = "GoodHits", .layer = "APA"}) .output_product_suffixes("NumGoodHits", "NumAllHits"); // <= One name per tuple slot } diff --git a/doc/ch_conceptual_design/hofs/windows.rst b/doc/ch_conceptual_design/hofs/windows.rst index 8870ea5..46dce73 100644 --- a/doc/ch_conceptual_design/hofs/windows.rst +++ b/doc/ch_conceptual_design/hofs/windows.rst @@ -105,7 +105,7 @@ The :mathfunc:`window(make_tracks)` node in :numref:`workflow` would be represen "APA", // <= Output data layer concurrency::unlimited // <= Allowed concurrency ) - .input_family(product_query{.suffix = "GoodHits", .layer = "APA"}) + .input_family(product_selector{.suffix = "GoodHits", .layer = "APA"}) .output_product_suffixes("GoodTracks"); } diff --git a/doc/ch_conceptual_design/program_configuration.rst b/doc/ch_conceptual_design/program_configuration.rst index 52895ac..f813427 100644 --- a/doc/ch_conceptual_design/program_configuration.rst +++ b/doc/ch_conceptual_design/program_configuration.rst @@ -7,4 +7,4 @@ To support reproducibility of data, a Phlex program is configured by *only one* These documents may be written in the following languages: Jsonnet, YAML, JSON (as a strict subset of Jsonnet), or FHiCL :need:`DUNE 72`. A given configuration can be written in any of these languages, but no configuration document may be written in more than one language. -The other details and rules by which a Phlex program is configured largely do not inform the conceptual design and are described in :numref:`ch_subsystem_design/configuration:Configuration`. +The other details and rules by which a Phlex program is configured largely do not inform the conceptual design and are described in :numref:`ch_technical_design/configuration:Configuration`. diff --git a/doc/ch_conceptual_design/registration.rst b/doc/ch_conceptual_design/registration.rst index 4a90e78..a312b35 100644 --- a/doc/ch_conceptual_design/registration.rst +++ b/doc/ch_conceptual_design/registration.rst @@ -29,7 +29,7 @@ This can be achieved by in terms of the C++ *registration stanza*: concurrency::unlimited // 4. Allowed CPU concurrency ) .input_family( // 5. Specification of input data-product family (see text) - product_query{.creator = "calibrate_wires", .suffix = "", .layer = "spill"} + product_selector{.creator = "calibrate_wires", .suffix = "", .layer = "spill"} ) .output_product_suffixes( "GoodHits" // 6. Specification of output data product from find_hits @@ -64,7 +64,7 @@ Specifically, in the registration stanza above, we have the following: :cpp:`input_family(...)` 5. The specification of the input family :math:`\ifamily{a}{\text{input}}` requires (a) the specification of the creator of each data product that serve as input family elements :need:`DUNE 65`, and (b) the label of the data layer in which each input data product is found. - In the registration code above, this is achieved by the construction of a :cpp:`product_query` with the appropriate arguments. + In the registration code above, this is achieved by the construction of a :cpp:`product_selector` with the appropriate arguments. The example shows finding products created by a node named "calibrate_wires", with an empty product suffix, and in the layer labeled "spill". Because the product suffix is empty, the :cpp:`.suffix` field could have been omitted altogether. @@ -95,7 +95,7 @@ The block, however, must contain a registration statement to execute an algorith Algorithms with Multiple Input Data Products -------------------------------------------- -The registration example given above in :numref:`ch_conceptual_design/registration:Framework Registration` creates an output family by applying a one-parameter algorithm :cpp:`find_hits` to each element of the input family, as specified by :cpp:`input_family(product_query{.suffix = "Waveforms", .layer = "APA"})`. +The registration example given above in :numref:`ch_conceptual_design/registration:Framework Registration` creates an output family by applying a one-parameter algorithm :cpp:`find_hits` to each element of the input family, as specified by :cpp:`input_family(product_selector{.suffix = "Waveforms", .layer = "APA"})`. In many cases, however, the algorithm will require more than one data product. Consider another algorithm :cpp:`find_hits_subtract_pedestals`, which forms hits by first subtracting pedestal values from the waveforms, both of which are presented to the algorithm as data products from the `APA`. The interface of the algorithm and its registration would look like: @@ -111,8 +111,8 @@ The interface of the algorithm and its registration would look like: { m.transform("find_hits", find_hits_subtract_pedestals, concurrency::unlimited) .input_family( - product_query{.suffix = "Waveforms", .layer = "APA"}, - product_query{.suffix = "Pedestals", .layer = "APA"} + product_selector{.suffix = "Waveforms", .layer = "APA"}, + product_selector{.suffix = "Pedestals", .layer = "APA"} ) .output_product_suffixes("GoodHits"); } @@ -141,8 +141,8 @@ This would be expressed in C++ as: { m.transform("vertex_maker", make_vertices, concurrency::unlimited) .input_family( - product_query{.suffix = "GoodTracks", .layer = "APA"}, - product_query{.suffix = "Geometry", .layer = "Job"} + product_selector{.suffix = "GoodTracks", .layer = "APA"}, + product_selector{.suffix = "Geometry", .layer = "Job"} ) .output_product_suffixes("Vertices"); } @@ -153,7 +153,7 @@ Phlex supports such uses cases :need:`DUNE 113`, even if the specified data laye For example, suppose an algorithm needed to access a data product from a `Spill`, and it also required a calibration offset provided from an external database table :need:`DUNE 35`. Instead of providing a separate mechanism for handling calibration constants, a separate layer could be invented (e.g. `Calibration`) whose data cells corresponded to intervals of validity. So long as a relation can be defined between specific `Spill` data cells and specific `Calibration` data cells, the framework can use that relation to form the input family of `Spill`\ -\ `Calibration` data-product pairs that are presented to the algorithm. -How the relation between data cells is defined is referred to as *data marshaling*, and it is described further in :numref:`ch_subsystem_design/task_management:Data-Marshaling`. +How the relation between data cells is defined is referred to as *data marshaling*, and it is described further in :numref:`ch_technical_design/task_management:Data-Marshaling`. Data Products from Adjacent Data Cells ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -161,7 +161,7 @@ Data Products from Adjacent Data Cells In some cases, it may be necessary to simultaneously access data products from adjacent data-products sets :need:`DUNE 91`, where *adjacency* is defined by the user :need:`DUNE 92`. The notion of adjacency can be critical for (e.g.) time-windowed processing (see :numref:`ch_conceptual_design/hofs/windows:Windows`), where the details of the "next" time bin are needed to accurately calculate properties of the "current" time bin. -Supporting the processing of adjacent data cells is described further in :numref:`ch_subsystem_design/task_management:Data-Marshaling`. +Supporting the processing of adjacent data cells is described further in :numref:`ch_technical_design/task_management:Data-Marshaling`. Accessing Configuration Information ----------------------------------- @@ -176,13 +176,13 @@ To do this, an additional argument (e.g. :cpp:`config`) is passed to the registr auto selected_data_layer = config.get("data_layer"); m.transform("hit_finder", find_hits, concurrency::unlimited) - .input_family(product_query{.suffix = "Waveforms", .layer = selected_data_layer}) + .input_family(product_selector{.suffix = "Waveforms", .layer = selected_data_layer}) .output_product_suffixes("GoodHits"); } .. note:: - As discussed in :numref:`ch_subsystem_design/configuration:Programmatic access to the configuration`, the registration code will have access only to the configuration relevant to the algorithm being registered, and to certain framework-level configuration such as debug level, verbosity, or parallelization options. + As discussed in :numref:`ch_technical_design/configuration:Programmatic access to the configuration`, the registration code will have access only to the configuration relevant to the algorithm being registered, and to certain framework-level configuration such as debug level, verbosity, or parallelization options. Except for the specification of :cpp:`find_hits` as the algorithm to be invoked, and :cpp:`transform` as the HOF, all other pieces of information may be provided through the configuration. @@ -208,7 +208,7 @@ By specifying a lambda expression that takes a :cpp:`phlex::handle` o [](phlex::handle ws) { return find_hits_debug(*ws, ws.id().number()); }, concurrency::unlimited ) - .input_family(product_query{.suffix = "Waveforms", .layer = "APA"}) + .input_family(product_selector{.suffix = "Waveforms", .layer = "APA"}) .output_product_suffixes("GoodHits"); } @@ -237,7 +237,7 @@ For example, the :cpp:`find_hits` algorithm author could have instead created a m.make(sigma_threshold) // <= Make framework-owned instance of hit_finder .transform("hit_finder", &hit_finder::find, concurrency::unlimited) - .input_family(product_query{.suffix = "Waveforms", .layer = selected_data_scope}) + .input_family(product_selector{.suffix = "Waveforms", .layer = selected_data_scope}) .output_product_suffixes("GoodHits"); } diff --git a/doc/ch_conceptual_design/resources.rst b/doc/ch_conceptual_design/resources.rst index 334e67f..5decffb 100644 --- a/doc/ch_conceptual_design/resources.rst +++ b/doc/ch_conceptual_design/resources.rst @@ -75,7 +75,7 @@ Such resource types have no dependency on Phlex, so that a user algorithm employ .. rubric:: Footnotes -.. [#fdetails] Details for specifying program resources are described in :numref:`ch_subsystem_design/configuration:Program resource specification`. +.. [#fdetails] Details for specifying program resources are described in :numref:`ch_technical_design/configuration:Program resource specification`. .. only:: html diff --git a/doc/ch_conceptual_design/scheduling.rst b/doc/ch_conceptual_design/scheduling.rst index f61307c..b7f1765 100644 --- a/doc/ch_conceptual_design/scheduling.rst +++ b/doc/ch_conceptual_design/scheduling.rst @@ -13,7 +13,7 @@ It is the program configuration that specifies: - any resources needed to process the data, and - any other constraints to impose on the program. -The pattern used for constructing the data-flow graph is described in detail in :numref:`ch_subsystem_design/task_management:Creating the data-flow graph`. +The pattern used for constructing the data-flow graph is described in detail in :numref:`ch_technical_design/task_management:Creating the data-flow graph`. Here it is sufficient to note that the graph is primarily driven by which data products are specified to be written to output (i.e. *precious* data products). By default, the framework will schedule for execution only those algorithms needed to preserve precious data products. This means that not necessarily every algorithm in the program configuration will be used in a general workflow. diff --git a/doc/ch_conceptual_design/supported_hofs.rst b/doc/ch_conceptual_design/supported_hofs.rst index 8da833f..6f6b5e4 100644 --- a/doc/ch_conceptual_design/supported_hofs.rst +++ b/doc/ch_conceptual_design/supported_hofs.rst @@ -28,4 +28,4 @@ The :math:`\textit{label}` operator is explicitly mentioned only for those HOFs .. rubric:: Footnotes -.. [#label] The specific rules by which the :math:`\textit{label}` operator can be used are given in :numref:`ch_subsystem_design/data_product_management:Data Product Management`. +.. [#label] The specific rules by which the :math:`\textit{label}` operator can be used are given in :numref:`ch_technical_design/data_product_management:Data Product Management`. diff --git a/doc/ch_subsystem_design/algo_description.rst b/doc/ch_subsystem_design/algo_description.rst deleted file mode 100644 index 8f020e9..0000000 --- a/doc/ch_subsystem_design/algo_description.rst +++ /dev/null @@ -1,7 +0,0 @@ -Algorithm Description -===================== - -.. todo:: - - - Providing information to the user as to what an algorithm can do and what configuration parameters it supports - - This could support describing all user-registered code diff --git a/doc/ch_subsystem_design/task_management.rst b/doc/ch_subsystem_design/task_management.rst deleted file mode 100644 index 5498e94..0000000 --- a/doc/ch_subsystem_design/task_management.rst +++ /dev/null @@ -1,47 +0,0 @@ -Task Management -=============== - -.. todo:: - - Includes scheduling of tasks as well as any relevant task priorities or concurrency constraints - - Respecting data-marshaling demands - - Relates to algorithms and data (product) sets - - Is this where the grammar goes for the predicate expression (:numref:`ch_conceptual_design/hofs/filters:Predicate Expression`)? - -The execution of computational tasks will be managed by `oneTBB Flow Graph `_. -With oneTBB, the user does not explicitly manage threads but specifies tasks that should be scheduled by oneTBB for execution using its own thread pool. - -Depending on the need, oneTBB interface may be explicitly used in algorithm code to take advantage of intra-algorithm parallelism. -Such use will integrate smoothly with Phlex's scheduled tasks as both the framework and the algorithm will be using oneTBB's shared thread pool :need:`DUNE 46`. - -Creating the Data-Flow Graph ----------------------------- - -The data-flow graph is constructed using the pattern described in the subsections below. - -Program Configuration -^^^^^^^^^^^^^^^^^^^^^ - -The Phlex program ingests a program configuration that includes a specification of all algorithms that are graph-node *candidates*. - -Required Data Products -^^^^^^^^^^^^^^^^^^^^^^ - -All configured observers and data sinks are constructed as graph nodes. -The framework records the list of data products required by each of these nodes. -A data sink can be configured to accept all data products created by the algorithms in the configuration, without specifying the individual data products created by each algorithm. - -Created Data Products -^^^^^^^^^^^^^^^^^^^^^ - -The transforms, folds, and unfolds specified in the configuration are created as graph nodes. -The framework records the list of the data products produced by each of these nodes. - -Connecting Algorithms -^^^^^^^^^^^^^^^^^^^^^ - -The data product lists recorded in :numref:`ch_subsystem_design/task_management:Required data products` and :numref:`ch_subsystem_design/task_management:Created data products` are compared. -Edges are created between the nodes in :numref:`ch_subsystem_design/task_management:Created data products` that create data products and those that match what is required by those in :numref:`ch_subsystem_design/task_management:Required data products`. -This step is iterative so that all graph-node candidates from :numref:`ch_subsystem_design/task_management:Program configuration`. - -Data-Marshaling ---------------- diff --git a/doc/ch_technical_design/algo_description.rst b/doc/ch_technical_design/algo_description.rst new file mode 100644 index 0000000..3f674f8 --- /dev/null +++ b/doc/ch_technical_design/algo_description.rst @@ -0,0 +1,152 @@ +Algorithm Description +===================== + +The Algorithm Description subsystem collects and provides information about all user-registered code (computational algorithms, providers, preservers, and translators) that can appear in the workflow graph. +This information is available both as structured data (for tools and the framework) and as human-readable text (for users). + +Responsibilities +---------------- + +The subsystem is responsible for: + +- **Providing information to the user as to what an algorithm can do** — human-readable descriptions of algorithm capabilities, behavior, and use cases. +- **Providing information about configuration parameters** — schema, defaults, validation rules, and documentation for each configurable parameter. +- **Supporting descriptions of all user-registered code** — extensible to any plugin type (computational, provider, preserver, translator) and language (C++, Python). + +The subsystem answers questions such as: + +- "What does this algorithm/node do?" +- "What are its inputs and outputs?" +- "What resources does it require?" +- "What configuration parameters does it accept, and what are their constraints?" +- "How do I configure this algorithm?" + +Descriptor Data Model +--------------------- + +The subsystem defines a canonical descriptor object for each registered algorithm, containing: + +**Identity** + - Stable algorithm identifier (module + symbol + version) + - Display name + - Category (computational/provider/preserver/translator) + - Language (C++/Python) + +**Human-readable description** + - Short summary (one-liner) + - Detailed description (what it does, capabilities, use cases) + - Author/maintainer information (optional) + +**Signature** (derived from :numref:`ch_conceptual_design/algorithms:Algorithms`) + - Ordered list of data-product parameters, each with: + - Parameter kind (``P const&``, ``P const*``, ``P``, ``phlex::handle

``) + - Concrete type name (fully qualified) + - Associated data-product concept identifier (if available) + - Parameter documentation + - Ordered list of resource parameters, each with: + - Type name + - Access mode (read-only or read-write) + - Parameter documentation + - Return type: + - Created data-product type (``T`` or ``std::unique_ptr``) + - Return documentation + +**Ownership and concurrency contracts** + - Ownership model (framework-owned; algorithm must not retain hard references) + - Concurrency safety (whether safe under configured concurrency) + - Member function const-ness (``const`` recommended; non-``const`` discouraged) + +**Configuration schema** + - Parameter list with types, defaults, and required/optional status + - Validation rules (ranges, enums, structural constraints) + - Documentation string for each parameter + - Examples of valid configurations + +**Provenance** + - Metadata to be attached to produced data products (creator, suffix, etc.) + +Descriptor Production +--------------------- + +Descriptors are produced at registration time and capture both structural information (signature, types) and human-readable metadata. + +**C++** + +- Signature information is extracted at registration time; the registration API templates already deduce types. +- Stored information includes: + - Demangled type names + - Parameter kinds (reference, pointer, value, handle) + - Return ownership model + - Resource access modes +- Human-readable metadata is captured from: + - Optional registration-time annotations (e.g., ``description(...)``, ``param_doc(...)`` helpers) + - Source code comments (if available via reflection) + +**Python** + +- Signature information is extracted via reflection and type annotations. +- Fallback mechanisms include: + - Explicit annotation objects in registration + - JIT wrappers (e.g., Numba) that expose signatures + - ``ctypes`` bindings with explicit signature metadata +- Human-readable metadata is captured from: + - Docstrings (PEP 257) + - Type hints + - Optional registration-time annotations + +Interfaces +---------- + +**Registration-time API** + +The subsystem provides builders and helpers to make it easy to attach descriptors to registered nodes, including: + +- Descriptor construction from signature information +- Helpers to add human-readable descriptions, parameter documentation, and configuration examples + +*Selector API** + +The subsystem provides queries to retrieve descriptors by: + +- Algorithm identifier or name +- Input/output type +- Data-product concept +- Configuration parameter +- Capability or keyword + +**User-facing reporting** + +The subsystem provides tools for users to discover and understand registered algorithms: + +- CLI command ``phlex describe `` — human-readable output of algorithm capabilities, signature, and configuration +- CLI command ``phlex list-algorithms [--filter ...]`` — list all registered algorithms with brief descriptions +- JSON/YAML export — for tooling and programmatic access +- Documentation generation — Sphinx integration to generate reference pages +- Configuration validation — with helpful error messages explaining constraints and valid values + +**Validation hooks** + +The subsystem validates that registered algorithms satisfy conceptual constraints: + +- At least one data-product parameter is required +- Return type must model a created data-product type (``T`` or ``std::unique_ptr``) +- Ambiguous ownership returns (bare pointers, references) are forbidden +- Non-``const`` member functions trigger warnings +- Runtime configuration is validated against the schema with clear error messages + +Integration with other subsystems +---------------------------------- + +- **Plugin Management**: Descriptors are discovered and loaded alongside modules. +- **Configuration**: Configuration schema is consumed to validate runtime configuration; the subsystem provides user-friendly validation messages. +- **Task Management**: Signature and resource information is used to schedule algorithms safely (e.g., enforcing resource write locks). +- **Data Product Management / Metadata Management**: Descriptor information is used to label produced data products and support data-product lookup. +- **Data Modeling**: Concept identifiers and translation availability information (from descriptors) inform translator insertion decisions. + +Open issues +----------- + +- **Python signature reliability**: Ensuring that Python type annotations and docstrings are reliably available and correctly interpreted. +- **Concept identifier availability**: Mapping concrete types to data-product concepts, especially for legacy code. +- **Versioning and backward compatibility**: How descriptors evolve as algorithms are updated. +- **Configuration validation error quality**: Providing clear, actionable error messages when configuration is invalid. diff --git a/doc/ch_subsystem_design/application.rst b/doc/ch_technical_design/application.rst similarity index 100% rename from doc/ch_subsystem_design/application.rst rename to doc/ch_technical_design/application.rst diff --git a/doc/ch_subsystem_design/configuration.rst b/doc/ch_technical_design/configuration.rst similarity index 100% rename from doc/ch_subsystem_design/configuration.rst rename to doc/ch_technical_design/configuration.rst diff --git a/doc/ch_subsystem_design/data_modeling.rst b/doc/ch_technical_design/data_modeling.rst similarity index 59% rename from doc/ch_subsystem_design/data_modeling.rst rename to doc/ch_technical_design/data_modeling.rst index 7285d6d..d247de9 100644 --- a/doc/ch_subsystem_design/data_modeling.rst +++ b/doc/ch_technical_design/data_modeling.rst @@ -7,3 +7,10 @@ Data Modeling - Provides a mechanism for connecting in-memory data representations with information needed by the IO backends to read-and-write the data - Responsible for ensuring the same data product is representable in the supported languages - May need to translate from one in-memory representation to another when crossing a language boundary + + +A data-product concept can be modeled by multiple concrete data-product types. +A concrete data-product type may model more than one data-product concept. + +The specification of input data products must support data-product concepts. +This may include an optional explicit specification of the data-product concept. diff --git a/doc/ch_subsystem_design/data_product_management.rst b/doc/ch_technical_design/data_product_management.rst similarity index 56% rename from doc/ch_subsystem_design/data_product_management.rst rename to doc/ch_technical_design/data_product_management.rst index 0261038..3b8ab32 100644 --- a/doc/ch_subsystem_design/data_product_management.rst +++ b/doc/ch_technical_design/data_product_management.rst @@ -15,7 +15,7 @@ Specifying Familiies -------------------- As discussed in :numref:`ch_conceptual_design/registration:Framework Registration`, the input family to a HOF is specified via a :cpp:`family(...)` clause, where the ellipsis is replaced by a *family specification*. -One of the specifications provided is :cpp:`product_query{.suffix = "good_hits", .layer = "spill"}`, which indicates that a family should be formed from data products labeled :cpp:`"good_hits"` from each :cpp:`"spill"` data layer. +One of the specifications provided is :cpp:`product_selector{.suffix = "good_hits", .layer = "spill"}`, which indicates that a family should be formed from data products labeled :cpp:`"good_hits"` from each :cpp:`"spill"` data layer. Input Family Specification -------------------------- @@ -43,6 +43,26 @@ Output family specification .. productionlist:: output_family_spec : ProductOutputSpec | ProductOutputSpec "," `output_family_spec` + + +Semantics and lookup policy for `product_selector` +---------------------------------------------- + +1. Only the data-product concept will be unconditionally required, and that could be calculated by the framework. +2. It is a framework-graph initialization error if the `product_selector` object resolves to multiple data products. +3. Ambiguities can be resolved by specifying additional fields. +4. This implies that a working Phlex job may fail once a user adds another data product whose specification matches an existing `product_selector`. + +5. Duplicate specifications, as resolved from `product_selector` objects, of the same input data product will be allowed. + + +6. A `product_selector` with an absent layer specification implies that any data products matching the query will be presented to the node. +7. The data layers of data products processed by an algorithm do not necessarily need to be known before forming the data-flow graph. +8. If the fields in the `product_selector` are sufficient to unambiguously connect one node to another, the data layer does not need to be known ahead of time. +9. The user must specify a product's data layer, or equivalent disambiguating information using other `product_selector` fields, to guarantee that only products associated with that layer will be processed. + + + .. only:: html .. rubric:: References diff --git a/doc/ch_subsystem_design/error_handler.rst b/doc/ch_technical_design/error_handler.rst similarity index 100% rename from doc/ch_subsystem_design/error_handler.rst rename to doc/ch_technical_design/error_handler.rst diff --git a/doc/ch_subsystem_design/framework_subsystems.rst b/doc/ch_technical_design/framework_subsystems.rst similarity index 100% rename from doc/ch_subsystem_design/framework_subsystems.rst rename to doc/ch_technical_design/framework_subsystems.rst diff --git a/doc/ch_subsystem_design/io.rst b/doc/ch_technical_design/io.rst similarity index 100% rename from doc/ch_subsystem_design/io.rst rename to doc/ch_technical_design/io.rst diff --git a/doc/ch_subsystem_design/logging.rst b/doc/ch_technical_design/logging.rst similarity index 100% rename from doc/ch_subsystem_design/logging.rst rename to doc/ch_technical_design/logging.rst diff --git a/doc/ch_subsystem_design/metadata_management.rst b/doc/ch_technical_design/metadata_management.rst similarity index 100% rename from doc/ch_subsystem_design/metadata_management.rst rename to doc/ch_technical_design/metadata_management.rst diff --git a/doc/ch_subsystem_design/monitoring.rst b/doc/ch_technical_design/monitoring.rst similarity index 100% rename from doc/ch_subsystem_design/monitoring.rst rename to doc/ch_technical_design/monitoring.rst diff --git a/doc/ch_subsystem_design/plugin_management.rst b/doc/ch_technical_design/plugin_management.rst similarity index 100% rename from doc/ch_subsystem_design/plugin_management.rst rename to doc/ch_technical_design/plugin_management.rst diff --git a/doc/ch_subsystem_design/program_startup.rst b/doc/ch_technical_design/program_startup.rst similarity index 100% rename from doc/ch_subsystem_design/program_startup.rst rename to doc/ch_technical_design/program_startup.rst diff --git a/doc/ch_subsystem_design/random_numbers.rst b/doc/ch_technical_design/random_numbers.rst similarity index 100% rename from doc/ch_subsystem_design/random_numbers.rst rename to doc/ch_technical_design/random_numbers.rst diff --git a/doc/ch_subsystem_design/registration.rst b/doc/ch_technical_design/registration.rst similarity index 100% rename from doc/ch_subsystem_design/registration.rst rename to doc/ch_technical_design/registration.rst diff --git a/doc/ch_subsystem_design/resource_management.rst b/doc/ch_technical_design/resource_management.rst similarity index 100% rename from doc/ch_subsystem_design/resource_management.rst rename to doc/ch_technical_design/resource_management.rst diff --git a/doc/ch_subsystem_design/role_of_data_product_concepts.rst b/doc/ch_technical_design/role_of_data_product_concepts.rst similarity index 100% rename from doc/ch_subsystem_design/role_of_data_product_concepts.rst rename to doc/ch_technical_design/role_of_data_product_concepts.rst diff --git a/doc/ch_subsystem_design/signal_handler.rst b/doc/ch_technical_design/signal_handler.rst similarity index 100% rename from doc/ch_subsystem_design/signal_handler.rst rename to doc/ch_technical_design/signal_handler.rst diff --git a/doc/ch_subsystem_design/subsystems.gv b/doc/ch_technical_design/subsystems.gv similarity index 100% rename from doc/ch_subsystem_design/subsystems.gv rename to doc/ch_technical_design/subsystems.gv diff --git a/doc/ch_technical_design/task_management.rst b/doc/ch_technical_design/task_management.rst new file mode 100644 index 0000000..31eea33 --- /dev/null +++ b/doc/ch_technical_design/task_management.rst @@ -0,0 +1,94 @@ +Task Management +=============== + +.. todo:: + - Includes scheduling of tasks as well as any relevant task priorities or concurrency constraints + - Respecting data-marshaling demands + - Relates to algorithms and data (product) sets + - Is this where the grammar goes for the predicate expression (:numref:`ch_conceptual_design/hofs/filters:Predicate Expression`)? + +The execution of computational tasks will be managed by `oneTBB Flow Graph `_. +With oneTBB, the user does not explicitly manage threads but specifies tasks that should be scheduled by oneTBB for execution using its own thread pool. + +Depending on the need, oneTBB interface may be explicitly used in algorithm code to take advantage of intra-algorithm parallelism. +Such use will integrate smoothly with Phlex's scheduled tasks as both the framework and the algorithm will be using oneTBB's shared thread pool :need:`DUNE 46`. + +Creating the Data-Flow Graph +---------------------------- + +The data-flow graph is constructed using the pattern described in the subsections below. + +Program Configuration +^^^^^^^^^^^^^^^^^^^^^ + +The Phlex program ingests a program configuration that includes a specification of all algorithms that are graph-node *candidates*. + +Required Data Products +^^^^^^^^^^^^^^^^^^^^^^ + +All configured observers and data sinks are constructed as graph nodes. +The framework records the list of data products required by each of these nodes. +A data sink can be configured to accept all data products created by the algorithms in the configuration, without specifying the individual data products created by each algorithm. + +Created Data Products +^^^^^^^^^^^^^^^^^^^^^ + +The transforms, folds, and unfolds specified in the configuration are created as graph nodes. +The framework records the list of the data products produced by each of these nodes. + +Connecting Algorithms +^^^^^^^^^^^^^^^^^^^^^ + +The data product lists recorded in :numref:`ch_technical_design/task_management:Required data products` and :numref:`ch_technical_design/task_management:Created data products` are compared. +Edges are created between the nodes in :numref:`ch_technical_design/task_management:Created data products` that create data products and those that match what is required by those in :numref:`ch_technical_design/task_management:Required data products`. +This step is iterative so that all graph-node candidates from :numref:`ch_technical_design/task_management:Program configuration`. + +Data-Marshaling +--------------- + +Translators +----------- + +A *translator* is a node whose input is of one concrete data-product type and whose output is of a different concrete data-product type, both of which model the same data-product concept. +A translator will be automatically inserted into the graph between two nodes whose data-product types are concept-compatible but not identical. +Translator nodes belong to the computational graph. +There are no translators that precede a provider. +There are no translators that follow a preserver. + +From the flow-graph point of view, translators are treated as first-class nodes, while being mostly irrelevant or invisible to the user. +Translator nodes for a given data representation conversion should be singular by construction and should not be duplicated. +Translators are not configured directly by the framework user; they must be discovered by the framework based on needs implied by the user configuration. +The framework should be able to discover translators and select which ones belong in the graph without loading every discovered translator DLL. +The translator metadata required for graph building should be generated at translator plugin build time. + + +Drivers and Runtime Hierarchy Handling +-------------------------------------- + +Phlex will implement a driver interface that enables opening an input file. + +The hierarchy of data-cell indices will be declared to the framework before processing any data-cell indices. + +Different policies for supporting hierarchies will be supported: + + - a fixed hierarchy may be declared for the entire job + - the global hierarchy may be updated based on hierarchies stored in input files in the middle of the job + +The data-layer hierarchy will not be known until the input file is opened. +Whenever the global hierarchy is established (e.g. by the first input file) or updated (e.g. by a subsequent input file), the framework validates it against what the computational graph requires. +Validation may involve creating more index-set nodes, provider nodes, preserver nodes, and translator nodes. +The job shall gracefully end when one of the input files contains a hierarchy that does not meet the hierarchy constraints of the computational graph. + +The semantic distinction between framework drivers and data-product providers will be retained. + +For release v0.3.0, a *data-cell index iterator* that supplies one data-cell index at a time is sufficient. +This is a temporary solution that will be modified in the future. +Range-based processing of multiple data cells at one time is not needed for v0.3.0. + +After v0.3.0, expected future needs include: + + - updating processed hierarchy when opening another input file + - a mode where external data management declares file hierarchy in advance + - support for multiple concurrently open input files for multithreading or overlay use cases + + diff --git a/doc/conceptual_design.rst b/doc/conceptual_design.rst index c5da2f4..16594c0 100644 --- a/doc/conceptual_design.rst +++ b/doc/conceptual_design.rst @@ -62,12 +62,19 @@ Various framework aspects are demonstrated by that figure: The histogramming resource in :numref:`workflow` enables the *observe* algorithm to fill and write histograms to a ROOT analysis file. + *computational graph* + The computational graph is the part of the data-flow graph that performs the computations of the reconstruction or simulation task of the configured workflow. + The computational graph does not include providers, preservers, or the framework driver. + The computational graph can be disjoint: a workflow may contain parts that are independent of each other. + In :numref:`workflow` the computational graph is formed by the unshaded nodes. + Note that in this workflow, the names `Spill` and `APA` are not special to the Phlex framework; they are names (hypothetically) chosen by the experiment. Each data product is also indexed, thus associating it with a particular data cell (e.g. :math:`\textit{GoodHits}_{3,5,9}` denotes the :product:`GoodHits` data product belonging to `APA` :math:`9` of `Spill` :math:`5` of `Run` :math:`3`). .. graphviz:: work-flow.gv :caption: A fictitious workflow showing how HOFs are used in a Phlex program. Each unshaded node represent a HOF bound to a user-defined algorithm. + These nodes form the computational graph of the workflow. The top part of the box shows the configured name of the node; the bottom part shows higher-order function invoked, and the name of the user-defined algorithm that is bound to it. The algorithm name is shaded in blue. Each user-defined algorithm operates on arguments received from the incoming arrows to the node: data products are passed along solid arrows; objects that provide access to resources are passed along dashed arrows. diff --git a/doc/conf.py b/doc/conf.py index 5debf2a..7bea73f 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -200,7 +200,7 @@ def watermark_blurb(scale, contents): if conceptual_design_only := os.environ.get("PHLEX_CONCEPTUAL_DESIGN_ONLY", False): tags.add("conceptual_design_only") # pyright: ignore[reportUndefinedVariable] exclude_patterns = ["supporting_design.rst", - "ch_subsystem_design/*", + "ch_technical_design/*", "appendices/deployment.rst", "appendices/platforms.rst"] suppress_warnings = ["toc.excluded"] diff --git a/doc/supporting_design.rst b/doc/supporting_design.rst index 9f992f4..f0d8a1f 100644 --- a/doc/supporting_design.rst +++ b/doc/supporting_design.rst @@ -16,25 +16,25 @@ This chapter of the document enumerates the subsystems, and describes their resp .. toctree:: :maxdepth: 2 - ch_subsystem_design/framework_subsystems - ch_subsystem_design/program_startup - ch_subsystem_design/role_of_data_product_concepts - ch_subsystem_design/application - ch_subsystem_design/task_management - ch_subsystem_design/data_modeling - ch_subsystem_design/data_product_management - ch_subsystem_design/metadata_management - ch_subsystem_design/io - ch_subsystem_design/plugin_management - ch_subsystem_design/resource_management - ch_subsystem_design/monitoring - ch_subsystem_design/registration - ch_subsystem_design/algo_description - ch_subsystem_design/error_handler - ch_subsystem_design/signal_handler - ch_subsystem_design/random_numbers - ch_subsystem_design/configuration - ch_subsystem_design/logging + ch_technical_design/framework_subsystems + ch_technical_design/program_startup + ch_technical_design/role_of_data_product_concepts + ch_technical_design/application + ch_technical_design/task_management + ch_technical_design/data_modeling + ch_technical_design/data_product_management + ch_technical_design/metadata_management + ch_technical_design/io + ch_technical_design/plugin_management + ch_technical_design/resource_management + ch_technical_design/monitoring + ch_technical_design/registration + ch_technical_design/algo_description + ch_technical_design/error_handler + ch_technical_design/signal_handler + ch_technical_design/random_numbers + ch_technical_design/configuration + ch_technical_design/logging External Subsystems ===================