Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/src/concepts/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,11 @@ graph TB

Connections between entities forming a knowledge graph.

- 🌐 **[Webhooks](webhooks.md)**

---

Runtime-configurable connectors that authenticate external events and map payloads to your data model.
- 🔍 **[Filtering Entities](entity-filtering.md)**

---
Expand Down Expand Up @@ -130,3 +135,4 @@ Dive deeper into each concept:
- **[Entity Templates](entity-templates.md)** - Learn how to design your data model
- **[Properties](properties.md)** - Understand property types and validation
- **[Relations](relations.md)** - Connect your entities into a graph
- **[Webhooks](webhooks.md)** - Configure inbound integrations and security strategies
216 changes: 216 additions & 0 deletions docs/src/concepts/webhooks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,216 @@
---
title: Webhooks
description: Understand webhook connectors, security strategies, and dynamic mappings in IDP-Core
---

Webhooks let external systems push JSON events to IDP-Core through a generic HTTP endpoint. You configure a webhook connector at runtime, choose a security strategy, and define mappings that translate incoming payloads into entity data with JSLT expressions.

## Overview

A webhook connector combines three concerns:

- **Connector metadata** - Identifier, title, description, and enabled flag
- **Security** - How IDP-Core authenticates incoming requests
- **Mappings** - How the payload maps to an Entity Template

```mermaid
flowchart LR
S[External system] --> E[POST /webhooks/{configurationId}]
E --> H[InboundWebhookHandler]
H --> D[Security dispatcher]
D --> C[WebhookConnector]
C --> M[Dynamic mappings]
M --> T[Entity Template]
```

## Webhook Connector

A webhook connector is the runtime configuration stored by IDP-Core for one inbound integration.

| Field | Type | Description |
| --- | --- | --- |
| `identifier` | String | Stable key used in the webhook URL and management APIs |
| `title` | String | Human-readable name |
| `description` | String | Optional explanation of the connector purpose |
| `enabled` | Boolean | Enables or disables request processing |
| `mappings` | Array | One or more dynamic mapping rules |
| `security` | Object | Authentication strategy and configuration |

### Example

```json
{
"identifier": "github-repositories",
"title": "GitHub repositories",
"description": "Receives repository events from GitHub",
"enabled": true,
"mappings": [
{
"template": "github_repository",
"filter": ".action == \"created\" or .action == \"edited\"",
"entity": {
"identifier": ".repository.full_name | gsub(\"/\"; \"_\")",
"title": ".repository.name",
"properties": {
"name": ".repository.name",
"url": ".repository.html_url",
"language": ".repository.language // \"Unknown\""
},
"relations": {
"owner": ".repository.owner.login"
}
}
}
],
"security": {
"type": "HMAC_SHA256",
"config": {
"header_name": "X-Hub-Signature-256",
"secret_alias": "GITHUB_WEBHOOK_SECRET",
"prefix": "sha256="
}
}
}
```

## Dynamic Mappings

Each connector contains at least one dynamic mapping. A mapping targets one Entity Template and describes how to derive entity fields from the incoming JSON payload with a JSLT filter and entity projections.

| Field | Description |
| --- | --- |
| `template` | Target Entity Template identifier |
| `filter` | Expression that decides whether the mapping applies |
| `entity.identifier` | Expression that generates the entity identifier |
| `entity.title` | Expression that generates the entity title |
| `entity.properties` | Map of template property names to extraction expressions |
| `entity.relations` | Map of template relation names to extraction expressions |

### Validation Rules

When you create or update a connector, IDP-Core validates each mapping against the target Entity Template.

It checks that:

- The referenced template exists
- Every mapped property exists in the template
- Every required property is mapped
- Every mapped relation exists in the template
- Every required relation is mapped

This validation keeps the connector configuration aligned with the current data model.

## Security Strategies

Each connector declares one security type. IDP-Core validates the configuration at creation time and validates requests again at runtime.

| Type | Required configuration keys | Runtime behavior |
| --- | --- | --- |
| `HMAC_SHA256` | `header_name`, `secret_alias`, `prefix` | Computes the SHA-256 HMAC of the raw body and compares it with the request header |
| `STATIC_TOKEN` | `header_name`, `secret_alias` | Compares a header value with a secret loaded from the environment |
| `BASIC_AUTH` | `username`, `secret_alias` | Compares the `Authorization: Basic ...` header with the configured username and secret |
| `JWT_BEARER` | `jwks_uri` | Validates the bearer token against a JWKS endpoint |
| `NONE` | none | Skips authentication |

> [!IMPORTANT]
> Security configuration keys accept `snake_case` and `camelCase` variants for the supported fields.
> [!WARNING]
> `secret_alias` must reference an environment variable alias in `UPPER_SNAKE_CASE`. It does not store the raw secret value in the connector configuration.

### Example Security Configurations

=== "HMAC_SHA256"
```json
{
"type": "HMAC_SHA256",
"config": {
"header_name": "X-Hub-Signature-256",
"secret_alias": "GITHUB_WEBHOOK_SECRET",
"prefix": "sha256="
}
}
```

=== "STATIC_TOKEN"
```json
{
"type": "STATIC_TOKEN",
"config": {
"header_name": "X-Webhook-Token",
"secret_alias": "WEBHOOK_SHARED_TOKEN"
}
}
```

=== "BASIC_AUTH"
```json
{
"type": "BASIC_AUTH",
"config": {
"username": "webhook-user",
"secret_alias": "WEBHOOK_PASSWORD"
}
}
```

=== "JWT_BEARER"
```json
{
"type": "JWT_BEARER",
"config": {
"jwks_uri": "https://issuer.example.com/.well-known/jwks.json"
}
}
```

## Runtime Flow

The webhook runtime uses a single generic endpoint:

```text
POST /webhooks/{configurationId}
```

The request flow is:

1. IDP-Core receives the request on the generic webhook endpoint.
2. The `configurationId` resolves the stored `WebhookConnector`.
3. If the connector is disabled, IDP-Core ignores the event.
4. The security dispatcher selects the matching strategy for the connector security type.
5. The strategy validates the headers and, when needed, the raw request body.
6. After authentication, the event is accepted for downstream processing.

> [!IMPORTANT]
> The connector model, security validation, management APIs, and mapping validation are implemented now.

## Management API Methods

You manage webhook connectors through the inbound webhook management API, which exposes standard CRUD methods.

| HTTP Method | Endpoint | Purpose |
| --- | --- | --- |
| `POST` | `/api/v1/inbound-webhooks` | Create connector |
| `GET` | `/api/v1/inbound-webhooks` | List connectors |
| `GET` | `/api/v1/inbound-webhooks/{identifier}` | Get connector |
| `PUT` | `/api/v1/inbound-webhooks/{identifier}` | Update connector |
| `DELETE` | `/api/v1/inbound-webhooks/{identifier}` | Delete connector |

This separation keeps configuration management under versioned API routes while the event ingestion endpoint stays simple for external systems.

## When to Use Webhooks

Use webhooks when an external system can push JSON events over HTTP and you want to:

- Ingest updates without redeploying IDP-Core
- Reuse one generic endpoint for multiple providers
- Apply connector-specific authentication rules
- Map external payloads to your own Entity Templates at runtime

---

## Next Steps

- **[Entity Templates](entity-templates.md)** - Define the target structures that mappings reference
- **[Entities](entities.md)** - Understand the records produced by successful ingestion
- **[Relations](relations.md)** - Model links that webhook mappings can populate
- **[Data Integration](../features/data-integration.md)** - Explore the broader ingestion roadmap
74 changes: 49 additions & 25 deletions docs/src/features/data-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ The Internal Developer Platform provides flexible data integration, allowing you
Data integration in the Internal Developer Platform follows a three-step pattern:

1. **Configure a connector** - Set up a Webhook, Kafka consumer, or Pub/Sub subscription
2. **Define mappings** - Use JQ expressions to transform incoming data
2. **Define mappings** - Use JSLT expressions to transform incoming data
3. **Ingest data** - Data flows automatically, creating and updating entities

```mermaid
Expand Down Expand Up @@ -55,6 +55,19 @@ flowchart LR

Webhooks allow external systems to push data to IDP-Core via HTTP POST requests.

### Methods

| Method | Endpoint | Purpose |
| ------ | -------- | ------- |
| `POST` | `/webhooks/{configurationId}` | Receive an inbound event for the connector identified in the URL |
| `POST` | `/api/v1/inbound-webhooks` | Create a webhook connector configuration |
| `GET` | `/api/v1/inbound-webhooks` | List webhook connector configurations |
| `GET` | `/api/v1/inbound-webhooks/{identifier}` | Read one webhook connector configuration |
| `PUT` | `/api/v1/inbound-webhooks/{identifier}` | Update one webhook connector configuration |
| `DELETE` | `/api/v1/inbound-webhooks/{identifier}` | Delete one webhook connector configuration |

These HTTP routes map to the `InboundWebhookManagementController` methods for connector management.

### Webhook Configuration

```json
Expand Down Expand Up @@ -83,33 +96,37 @@ Webhooks allow external systems to push data to IDP-Core via HTTP POST requests.
}
],
"security": {
"signature_header_name": "X-Sonar-Webhook-HMAC-SHA256",
"signature_value": "your-secret-token"
"type": "HMAC_SHA256",
"config": {
"header_name": "X-Sonar-Webhook-HMAC-SHA256",
"secret_alias": "SONAR_WEBHOOK_SECRET",
"prefix": "sha256="
}
}
}
```

### Configuration Fields

| Field | Description |
| ------------- | ---------------------------- |
| `identifier` | Unique key for this webhook |
| `title` | Human-readable name |
| `description` | Purpose of the webhook |
| `enabled` | Toggle ingestion on/off |
| `mappings` | Array of mapping rules |
| `security` | Authentication configuration |
| Field | Description |
|---------------|-----------------------------------------------------------------|
| `identifier` | Unique key for this webhook |
| `title` | Human-readable name |
| `description` | Purpose of the webhook |
| `enabled` | Toggle ingestion on/off |
| `mappings` | Array of mapping rules |
| `security` | Authentication configuration using a `type` + `config` contract |

### Mapping Structure

| Field | Description |
| ------------------- | ------------------------------------------- |
| `template` | Target Entity Template identifier |
| `filter` | JQ expression to filter incoming payloads |
| `entity.identifier` | JQ expression to generate entity identifier |
| `entity.title` | JQ expression for entity title |
| `entity.properties` | Map of property names to JQ expressions |
| `entity.relations` | Map of relation names to JQ expressions |
| Field | Description |
|---------------------|-----------------------------------------------|
| `template` | Target Entity Template identifier |
| `filter` | JSLT expression to filter incoming payloads |
| `entity.identifier` | JSLT expression to generate entity identifier |
| `entity.title` | JSLT expression for entity title |
| `entity.properties` | Map of property names to JSLT expressions |
| `entity.relations` | Map of relation names to JSLT expressions |

---

Expand Down Expand Up @@ -165,9 +182,9 @@ spring:

---

## JQ Mapping Reference
## JSLT Mapping Reference

The Internal Developer Platform will use [JQ](https://jqlang.github.io/jq/) for data transformation. It will access to the entire JSON payload sent to the webhook or consumed from Kafka/Pub-Sub. Please refer to the JQ documentation for detailed usage.
The Internal Developer Platform uses [JSLT](https://github.com/schibsted/jslt) for data transformation. It accesses the entire JSON payload sent to the webhook or consumed from Kafka/Pub-Sub. Refer to the JSLT documentation for detailed usage.

---

Expand Down Expand Up @@ -201,8 +218,12 @@ Configure a webhook to receive GitHub repository events:
}
],
"security": {
"signature_header_name": "X-Hub-Signature-256",
"signature_value": "sha256=your-webhook-secret"
"type": "HMAC_SHA256",
"config": {
"header_name": "X-Hub-Signature-256",
"secret_alias": "GITHUB_WEBHOOK_SECRET",
"prefix": "sha256="
}
}
}
```
Expand All @@ -218,8 +239,11 @@ Webhooks support signature-based authentication:
```json
{
"security": {
"signature_header_name": "X-Webhook-Signature",
"signature_value": "expected-secret-or-hmac"
"type": "STATIC_TOKEN",
"config": {
"header_name": "X-Webhook-Signature",
"secret_alias": "WEBHOOK_SHARED_TOKEN"
}
}
}
```
Expand Down
2 changes: 1 addition & 1 deletion docs/src/features/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ The Internal Developer Platform provides a comprehensive set of features to buil

---

Connect to any data source through Webhooks, Kafka, or Pub/Sub. Map incoming data to entities using JQ expressions.
Connect to any data source through Webhooks, Kafka, or Pub/Sub. Map incoming data to entities using JSLT expressions.

**Status:** 🕐 Planned

Expand Down
2 changes: 1 addition & 1 deletion docs/src/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ Define your own **Entity Templates** that mirror your organization's specific ne

### Multi-Source Data Ingestion

Connect to any data source through **Webhooks**, **Kafka/Pub-Sub**, or direct API calls. Map incoming data to your entities using JQ expressions.
Connect to any data source through **Webhooks**, **Kafka/Pub-Sub**, or direct API calls. Map incoming data to your entities using JSLT expressions.

### Scorecards & Metrics

Expand Down
3 changes: 2 additions & 1 deletion docs/zensical.toml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,8 @@ nav = [
"concepts/entity-filtering.md"
]},
"concepts/properties.md",
"concepts/relations.md"
"concepts/relations.md",
"concepts/webhooks.md"
]},
{ "Features" = [
"features/index.md",
Expand Down
Loading
Loading