Skip to content
1 change: 1 addition & 0 deletions proto/substrait/extensions/extensions.proto
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ message SimpleExtensionURN {

// The extension URN that uniquely identifies this extension. This must follow the
// format extension:<OWNER>:<ID> and serves as the "namespace" of this extension.
// The URN must be valid RFC 8141 format without the urn: prefix.
string urn = 2;
}

Expand Down
4 changes: 3 additions & 1 deletion site/docs/extensions/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,15 @@ Some kinds of primitives are so frequently extended that Substrait defines a sta
* Window Functions
* Table Functions

To extend these items, developers can create one or more YAML files that describe the properties of each of these extensions. Each YAML file must include a required `urn` field that uniquely identifies the extension. While these identifiers are URN-like but not technically URNs (they lack the `urn:` prefix), they will be referred to as `extension URNs` for clarity.
To extend these items, developers can create one or more YAML files that describe the properties of each of these extensions. Each YAML file must include a required `urn` field that uniquely identifies the extension. While these identifiers are URN-like but not technically URNs (they lack the `urn:` prefix), they will be referred to as `extension URNs` for clarity. These URNs must be valid [RFC 8141](https://www.rfc-editor.org/rfc/rfc8141.html) format without the `urn:` prefix.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor adjustment for clarity.

Suggested change
To extend these items, developers can create one or more YAML files that describe the properties of each of these extensions. Each YAML file must include a required `urn` field that uniquely identifies the extension. While these identifiers are URN-like but not technically URNs (they lack the `urn:` prefix), they will be referred to as `extension URNs` for clarity. These URNs must be valid [RFC 8141](https://www.rfc-editor.org/rfc/rfc8141.html) format without the `urn:` prefix.
To extend these items, users can create one or more YAML files that describe the properties of each of these extensions. Each YAML file must include a required `urn` field that uniquely identifies the extension. These identifiers are URN-like but not technically URNs (they are prefixed with `extension:` instead of `urn:`), and will be referred to as `extension URNs` for clarity.
Extension URNs must be valid [RFC 8141](https://www.rfc-editor.org/rfc/rfc8141.html) URNs when replacing `extension:` with `urn:`.


This extension URN uses the format `extension:<OWNER>:<ID>`, where:

- `OWNER` represents the organization or entity providing the extension and should follow [reverse domain name convention](https://en.wikipedia.org/wiki/Reverse_domain_name_notation) (e.g., `io.substrait`, `com.example`, `org.apache.arrow`) to prevent name collisions
- `ID` is the specific identifier for the extension (e.g., `functions_arithmetic`, `custom_types`)

Thus, if `extension:<OWNER>:<ID>` is our URN, then `urn:extension:<OWNER>:<ID>` must be a valid [RFC 8141 URN](https://www.rfc-editor.org/rfc/rfc8141.html).

The YAML file is constructed according to the [YAML Schema](https://github.com/substrait-io/substrait/blob/main/text/simple_extensions_schema.yaml). Each definition in the file corresponds to the YAML-based serialization of the relevant data structure. If a user only wants to extend one of these types of objects (e.g. types), a developer does not have to provide definitions for the other extension points.

A Substrait plan can reference one or more YAML files via their extension URN. In the places where these entities are referenced, they will be referenced using an extension URN + name reference. The name scheme per type works as follows:
Expand Down
2 changes: 1 addition & 1 deletion site/docs/serialization/binary_serialization.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ For simple extensions, a plan references the extension URNs associated with the

Simple extensions within a plan are split into three components: an extension URN, an extension declaration and a number of references.

* **Extension URN**: A unique identifier for the extension following the format `extension:<OWNER>:<ID>` that identifies a YAML document specifying one or more specific extensions. Declares an anchor that can be used in extension declarations.
* **Extension URN**: A unique identifier for the extension following the format `extension:<OWNER>:<ID>` that identifies a YAML document specifying one or more specific extensions. Declares an anchor that can be used in extension declarations. The URN with the `urn:` prefix added must conform to [RFC 8141](https://www.rfc-editor.org/rfc/rfc8141.html).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The URN with the urn: prefix added must conform to RFC 8141.

It's a bit weird to say this. The way we've structured them now maps to

         | <NID> | <NSS>
extension:<owner>:<id>

I guess they do conform to the RFC if we prefix urn, but the interpretation would be different technically:

   | <NID>   |   <NSS>
urn:extension:<owner>:<id>

What do you think about:

The Extension URN with the extension: replaced with urn: must conform to RFC 8141

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... I see the point you are making. However, we don't actually enforce that the structure of the <owner> part is reverse DNS. Why don't we just loosen the restriction on the urn entirely and say:

The urn is required to be a valid URN when urn: is prepended to the string. The format must conform to urn:extension:<Identifier>. The recommended format for the identifier is <Reverse-DNS-Name>:<any-valid-name>. This is consistent with the default substrait extensions and prevents name collisions.

To me, this feels more consistent with the urn spec. Maybe its just how my brain works, but saying "urn: added to the front makes it a valid URN" makes more sense to me than saying "urn: replacing extension: makes it a valid URN".

* **Extension Declaration**: A specific extension within a single YAML document. The declaration combines a reference to the associated extension URN along with a unique key identifying the specific item within that YAML document (see [Function Signature](../extensions/index.md#function-signature)). It also defines a declaration anchor. The anchor is a plan-specific unique value that the producer creates as a key to be referenced elsewhere.
* **Extension Reference**: A specific instance or use of an extension declaration within the plan body.

Expand Down
Loading