You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/integrations/sources/postgres.md
+22-20Lines changed: 22 additions & 20 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,7 +9,7 @@ Airbyte's certified Postgres connector offers the following features:
9
9
- All available [sync modes](https://docs.airbyte.com/cloud/core-concepts#connection-sync-modes), providing flexibility in how data is delivered to your destination.
10
10
- Reliable replication at any table size with [checkpointing](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#state--checkpointing) and chunking of database reads.
11
11
12
-
The contents below include a 'Quick Start' guide, advanced setup steps, and reference information (data type mapping, and changelogs). See [here](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting) to troubleshooting issues with the Postgres connector.
12
+
The contents below include a 'Quick Start' guide, advanced setup steps, and reference information (data type mapping, and changelogs). See [here](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting) to troubleshoot issues with the Postgres connector.
13
13
14
14
</HideInUI>
15
15
@@ -35,13 +35,13 @@ These steps create a dedicated read-only user for replicating data. Alternativel
35
35
36
36
The following commands will create a new user:
37
37
38
-
```roomsql
38
+
```sql
39
39
CREATE USER <user_name> PASSWORD 'your_password_here';
40
40
```
41
41
42
42
Now, provide this user with read-only access to relevant schemas and tables. Re-run this command for each schema you expect to replicate data from:
43
43
44
-
```roomsql
44
+
```sql
45
45
GRANT USAGE ON SCHEMA <schema_name> TO <user_name>;
46
46
GRANTSELECTON ALL TABLES IN SCHEMA <schema_name> TO <user_name>;
47
47
ALTER DEFAULT PRIVILEGES IN SCHEMA <schema_name>GRANTSELECTON TABLES TO <user_name>;
@@ -77,7 +77,7 @@ Now, click `Set up source` in the Airbyte UI. Airbyte will now test connecting t
77
77
78
78
## Advanced Configuration using CDC
79
79
80
-
Airbyte uses [logical replication](https://www.postgresql.org/docs/10/logical-replication.html) of the Postgres write-ahead log (WAL) to incrementally capture deletes using a replication plugin:
80
+
Airbyte uses [logical replication](https://www.postgresql.org/docs/current/logical-replication.html) of the Postgres write-ahead log (WAL) to incrementally capture deletes using a replication plugin:
81
81
82
82
- See [here](https://docs.airbyte.com/understanding-airbyte/cdc) to learn more on how Airbyte implements CDC.
83
83
- See [here](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#cdc-requirements) to learn more about Postgres CDC requirements and limitations.
@@ -174,10 +174,10 @@ ALTER TABLE tbl1 REPLICA IDENTITY FULL;`. Ensure that TOAST-able tables use non
174
174
2. Create the Postgres publication. You should include all tables you want to replicate as part of the publication:
175
175
176
176
```
177
-
CREATE PUBLICATION airbyte_publication FOR TABLE <tbl1, tbl2, tbl3>;`
177
+
CREATE PUBLICATION airbyte_publication FOR TABLE <tbl1, tbl2, tbl3>;
178
178
```
179
179
180
-
The publication name is customizable. Refer to the [Postgres docs](https://www.postgresql.org/docs/10/sql-alterpublication.html) if you need to add or remove tables from your publication in the future.
180
+
The publication name is customizable. Refer to the [Postgres docs](https://www.postgresql.org/docs/current/sql-alterpublication.html) if you need to add or remove tables from your publication in the future.
181
181
</FieldAnchor>
182
182
183
183
:::note
@@ -196,7 +196,7 @@ The Postgres source currently offers 3 methods of replicating updates to your de
196
196
197
197
### CDC
198
198
199
-
Airbyte uses [logical replication](https://www.postgresql.org/docs/10/logical-replication.html) of the Postgres write-ahead log (WAL) to incrementally capture deletes using a replication plugin. To learn more how Airbyte implements CDC, refer to [Change Data Capture (CDC)](https://docs.airbyte.com/understanding-airbyte/cdc/). We recommend configuring your Postgres source with CDC when:
199
+
Airbyte uses [logical replication](https://www.postgresql.org/docs/current/logical-replication.html) of the Postgres write-ahead log (WAL) to incrementally capture deletes using a replication plugin. To learn more how Airbyte implements CDC, refer to [Change Data Capture (CDC)](https://docs.airbyte.com/understanding-airbyte/cdc/). We recommend configuring your Postgres source with CDC when:
200
200
201
201
- You need a record of deletions.
202
202
- You have a very large database (500 GB or more).
@@ -243,7 +243,7 @@ If you are using SSH tunneling, as Airbyte Cloud requires encrypted communicatio
243
243
For SSH Tunnel Method, select:
244
244
245
245
-`No Tunnel` for a direct connection to the database
246
-
-`SSH Key Authentication` to use an RSA Private as your secret for establishing the SSH tunnel
246
+
-`SSH Key Authentication` to use an RSA private key as your secret for establishing the SSH tunnel
247
247
-`Password Authentication` to use a password as your secret for establishing the SSH tunnel
248
248
249
249
#### Connect via SSH Tunnel
@@ -255,7 +255,7 @@ When using an SSH tunnel, you are configuring Airbyte to connect to an intermedi
255
255
To connect to a Postgres instance via an SSH tunnel:
256
256
257
257
1. While [setting up](#step-2-create-a-new-postgres-source-in-airbyte-ui) the Postgres source connector, from the SSH tunnel dropdown, select:
258
-
- SSH Key Authentication to use a private as your secret for establishing the SSH tunnel
258
+
- SSH Key Authentication to use a private key as your secret for establishing the SSH tunnel
259
259
- Password Authentication to use a password as your secret for establishing the SSH Tunnel
260
260
2. For **SSH Tunnel Jump Server Host**, enter the hostname or IP address for the intermediate (bastion) server that Airbyte will connect to.
261
261
3. For **SSH Connection Port**, enter the port on the bastion server. The default port for SSH connections is 22.
@@ -278,14 +278,16 @@ The command produces the private key in PEM format and the public key remains in
278
278
279
279
## Configuring Entra authentication
280
280
281
-
The Airbyte source can be configured to authenticate as a Microsoft Entra service principal.
282
-
This allows Airbyte to use short-lived identity tokens to authenticate to an Azure Postgres server.
281
+
The Airbyte source can be configured to authenticate as a Microsoft Entra service principal.
282
+
This allows Airbyte to use short-lived identity tokens to authenticate to an Azure Postgres server.
283
283
Consult the Microsoft [documentation on this topic](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-azure-ad-authentication) for more detail on configuring the server and other Entra resources.
284
284
285
-
To configure the Airbyte Postgres source with Entra authentication, set the `Username` to the Entra ID, as discussed
286
-
in [Microsoft's documentation](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-configure-sign-in-azure-ad-authentication).
287
-
Set the password to a client secret for your Entra service principal.
288
-
Expand the Optional Fields and provide the Entra tenant ID and Entra client (or app) ID of the service principal.
285
+
To configure the Airbyte Postgres source with Entra authentication:
286
+
287
+
1. Set the `Username` to the Entra ID, as discussed in [Microsoft's documentation](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-configure-sign-in-azure-ad-authentication).
288
+
2. Set the password to a client secret for your Entra service principal.
289
+
3. Enable the **Entra service principal authentication** toggle.
290
+
4. Provide the **Entra tenant ID** and **Entra client (or app) ID** of the service principal.
289
291
290
292
## Limitations & Troubleshooting
291
293
@@ -358,11 +360,11 @@ According to Postgres [documentation](https://www.postgresql.org/docs/14/datatyp
358
360
| 3.7.2 | 2026-03-04 |[74294](https://github.com/airbytehq/airbyte/pull/74294)| Fix CDC bug where a replication slot can be advanced too far, losing needed WAL segments. Remove CVEs. |
359
361
| 3.7.1 | 2026-01-27 |[72396](https://github.com/airbytehq/airbyte/pull/72396)| This version causes issues discovering streams for some users. Do not use. |
360
362
| 3.7.0 | 2025-08-12 |[57511](https://github.com/airbytehq/airbyte/pull/57511)| Add configurations for Azure authentication to Azure Postgres servers. |
361
-
| 3.6.35 | 2025-06-12|[61527](https://github.com/airbytehq/airbyte/pull/61527)| Add error handling for connection issues and adopt the latest CDK version. |
362
-
| 3.6.34 | 2025-05-11|[60876](https://github.com/airbytehq/airbyte/pull/60876)| Cache CDC initial state once constructed. |
363
-
| 3.6.33 | 2025-05-11|[60214](https://github.com/airbytehq/airbyte/pull/60214)| Migrate to new Gradle flow. |
364
-
| 3.6.32 | 2025-05-8 |[59722](https://github.com/airbytehq/airbyte/pull/59722)| Consolidate gradle set up. |
365
-
| 3.6.31 | 2025-04-18|[58132](https://github.com/airbytehq/airbyte/pull/58132)| Fix vulnerabilities in dependencies. |
363
+
| 3.6.35 | 2025-06-18|[61527](https://github.com/airbytehq/airbyte/pull/61527)| Add error handling for connection issues and adopt the latest CDK version. |
364
+
| 3.6.34 | 2025-05-29|[60876](https://github.com/airbytehq/airbyte/pull/60876)| Cache CDC initial state once constructed. |
365
+
| 3.6.33 | 2025-05-13|[60214](https://github.com/airbytehq/airbyte/pull/60214)| Migrate to new Gradle flow. |
366
+
| 3.6.32 | 2025-05-08|[59722](https://github.com/airbytehq/airbyte/pull/59722)| Consolidate gradle set up. |
367
+
| 3.6.31 | 2025-04-24|[58132](https://github.com/airbytehq/airbyte/pull/58132)| Fix vulnerabilities in dependencies. |
366
368
| 3.6.30 | 2025-03-06 |[55234](https://github.com/airbytehq/airbyte/pull/55234)| Update base image version for certified DB source connectors |
367
369
| 3.6.29 | 2025-02-13 |[53649](https://github.com/airbytehq/airbyte/pull/53649)| Fix issue that column default value did not get converted |
368
370
| 3.6.28 | 2024-12-23 |[50870](https://github.com/airbytehq/airbyte/pull/50870)| Use airbyte/java-connector-base:2.0.0 |
Copy file name to clipboardExpand all lines: docs/integrations/sources/postgres/postgres-troubleshooting.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -132,7 +132,7 @@ To customize the JDBC connection beyond common options, specify additional suppo
132
132
133
133
Example: key1=value1&key2=value2&key3=value3
134
134
135
-
These parameters will be added at the end of the JDBC URL that the AirByte will use to connect to your Postgres database.
135
+
These parameters will be added at the end of the JDBC URL that Airbyte will use to connect to your Postgres database.
136
136
137
137
The connector now supports `connectTimeout` and defaults to 60 seconds. Setting connectTimeout to 0 seconds will set the timeout to the longest time available.
138
138
@@ -146,7 +146,7 @@ The Postgres connector may need some time to start processing the data in the CD
146
146
- When the connection is set up for the first time and a snapshot is needed
147
147
- When the connector has a lot of change logs to process
148
148
149
-
The connector waits for the default initial wait time of 5 minutes (300 seconds). Setting the parameter to a longer duration will result in slower syncs, while setting it to a shorter duration may cause the connector to not have enough time to create the initial snapshot or read through the change logs. The valid range is 120 seconds to 1200 seconds.
149
+
The connector waits for the default initial wait time of 20 minutes (1200 seconds). Setting the parameter to a longer duration will result in slower syncs, while setting it to a shorter duration may cause the connector to not have enough time to create the initial snapshot or read through the change logs. The valid range is 120 seconds to 2400 seconds.
150
150
151
151
If you know there are database changes to be synced, but the connector cannot read those changes, the root cause may be insufficient waiting time. In that case, you can increase the waiting time (example: set to 600 seconds) to test if it is indeed the root cause. On the other hand, if you know there are no database changes, you can decrease the wait time to speed up the zero record syncs.
0 commit comments