Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions tidb-cloud/naming-conventions-for-data-import.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,9 +119,28 @@ If the SQL file is exported through TiDB Dumpling with the default configuration

## File pattern

If the source data file of CSV or Parquet does not conform to the naming convention, you can manually map the source data file to the target table using a file name pattern. This feature does not support Aurora Snapshot and SQL data files.
If a CSV or Parquet source file does not conform to the naming convention, you can manually map the source file to a target table using a file name pattern.

In the import wizard, on the **Destination Mapping** step, deselect **Use TiDB file naming conventions for automatic mapping**, and then fill in the **Source**, **Target Database**, and **Target Table** fields. The **Source** field accepts a file name pattern that supports the `*` and `?` wildcards.
<CustomContent plan="starter,essential,dedicated">

Manual file-pattern mapping does not support Aurora Snapshot or SQL data files.

</CustomContent>
<CustomContent plan="premium">

Manual file-pattern mapping does not support SQL data files.

</CustomContent>

In the mapping step of the import wizard, deselect **Use TiDB file naming conventions for automatic mapping**, and then fill in the **Source**, **Target Database**, and **Target Table** fields. The **Source** field accepts a file name pattern relative to the source URI and supports the `*` and `?` wildcards. Add one mapping for each target table.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated

<CustomContent plan="premium">

> **Note:**
>
> For Parquet files exported from an Aurora Snapshot, manual mapping applies only the source patterns that you configure. It does not infer a complete snapshot mapping or create the target schema. Create the target databases and tables before the import, add a mapping for each target table, and verify that the pre-check scans the expected number of data files and maps each source pattern to the intended target table.

</CustomContent>

- For CSV files, see [Step 4. Import CSV files to TiDB Cloud](/tidb-cloud/import-csv-files.md#step-4-import-csv-files-to-tidb-cloud).
- For Parquet files, see [Step 4. Import Parquet files to TiDB Cloud](/tidb-cloud/import-parquet-files.md#step-4-import-parquet-files-to-tidb-cloud).
52 changes: 24 additions & 28 deletions tidb-cloud/premium/import-csv-files-premium.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ To ensure data consistency, {{{ .premium }}} allows importing CSV files into emp
>
> - To achieve better performance, it is recommended to limit the size of each compressed file to 100 MiB.
> - The Snappy compressed file must be in the [official Snappy format](https://github.com/google/snappy). Other variants of Snappy compression are not supported.
> - For uncompressed files, if you cannot update the CSV filenames according to the preceding rules in some cases (for example, the CSV file links are also used by your other programs), you can keep the filenames unchanged and use the **Mapping Settings** in [Step 4](#step-4-import-csv-files) to import your source data to a single target table.
> - For uncompressed files, if you cannot update the CSV filenames according to the preceding rules in some cases (for example, the CSV file links are also used by your other programs), you can keep the filenames unchanged and configure manual source-to-target mappings in [Step 4](#step-4-import-csv-files).

## Step 2. Create the target table schemas

Expand Down Expand Up @@ -116,29 +116,27 @@ To import the CSV files to {{{ .premium }}}, take the following steps:

4. Click **Next**.

5. In the **Source Files Mapping** section, {{{ .premium }}} scans the bucket and proposes mappings between the source files and destination tables.
5. In the **Source Files Mapping** section, specify how source files are mapped to target tables.

When a directory is specified in **Source Files URI**, the **Use [File naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) for automatic mapping** option is selected by default.

> **Note:**
>
> When a single file is specified in **Source Files URI**, the **Use [File naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) for automatic mapping** option is not displayed, and {{{ .premium }}} automatically populates the **Source** field with the file name. In this case, you only need to select the target database and table for data import.

- Leave automatic mapping enabled to apply the [file naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) to your source files and target tables. Keep **CSV** selected as the data format.
- To use automatic mapping, leave the option selected. {{{ .premium }}} applies the [file naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) to your source files and target tables.

- **Advanced options**: expand the panel to view the `Ignore compatibility checks (advanced)` toggle. Leave it disabled unless you intentionally want to bypass schema compatibility validation.
- To configure mappings manually, deselect the automatic mapping option, and then configure a mapping for each target table:

<!-- future feature -->
> **Note:**
>
> Manual mapping is coming soon. When the toggle becomes available, clear the automatic mapping option and configure the mapping manually:
>
> - **Source**: enter a filename pattern such as `TableName.01.csv`. Wildcards `*` and `?` are supported (for example, `my-data*.csv`).
> - **Target Database** and **Target Table**: choose the destination objects for the matched files.
- **Source**: enter a source file pattern relative to the **Source Files URI**. The pattern supports `*` and `?` wildcards. For example, `my-data*.csv` matches all CSV files whose names start with `my-data`.
- **Target Database** and **Target Table**: enter the destination database and table for the matched files.
- To add another mapping, click **+**.

6. Click **Next** to run the pre-check. Review the scan results and verify the source files and target tables.

6. {{{ .premium }}} automatically scans the source path. Review the scan results, check the data files found and corresponding target tables, and then click **Start Import**.
7. Click **Start Import**.

7. When the import progress shows **Completed**, check the imported tables.
8. When the import progress shows **Completed**, check the imported tables.

</div>

Expand Down Expand Up @@ -168,29 +166,27 @@ To import the CSV files to {{{ .premium }}}, take the following steps:

4. Click **Next**.

5. In the **Source Files Mapping** section, {{{ .premium }}} scans the bucket and proposes mappings between the source files and destination tables.
5. In the **Source Files Mapping** section, specify how source files are mapped to target tables.

When a directory is specified in **Source Files URI**, the **Use [File naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) for automatic mapping** option is selected by default.

> **Note:**
>
> When a single file is specified in **Source Files URI**, the **Use [File naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) for automatic mapping** option is not displayed, and {{{ .premium }}} automatically populates the **Source** field with the file name. In this case, you only need to select the target database and table for data import.

- Leave automatic mapping enabled to apply the [file naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) to your source files and target tables. Keep **CSV** selected as the data format.
- To use automatic mapping, leave the option selected. {{{ .premium }}} applies the [file naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) to your source files and target tables.

- **Advanced options**: expand the panel to view the `Ignore compatibility checks (advanced)` toggle. Leave it disabled unless you intentionally want to bypass schema compatibility validation.
- To configure mappings manually, deselect the automatic mapping option, and then configure a mapping for each target table:

<!-- future feature -->
> **Note:**
>
> Manual mapping is coming soon. When the toggle becomes available, clear the automatic mapping option and configure the mapping manually:
>
> - **Source**: enter a filename pattern such as `TableName.01.csv`. Wildcards `*` and `?` are supported (for example, `my-data*.csv`).
> - **Target Database** and **Target Table**: choose the destination objects for the matched files.
- **Source**: enter a source file pattern relative to the **Source Files URI**. The pattern supports `*` and `?` wildcards. For example, `my-data*.csv` matches all CSV files whose names start with `my-data`.
- **Target Database** and **Target Table**: enter the destination database and table for the matched files.
- To add another mapping, click **+**.

6. Click **Next** to run the pre-check. Review the scan results and verify the source files and target tables.

6. {{{ .premium }}} automatically scans the source path. Review the scan results, check the data files found and corresponding target tables, and then click **Start Import**.
7. Click **Start Import**.

7. When the import progress shows **Completed**, check the imported tables.
8. When the import progress shows **Completed**, check the imported tables.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated

</div>

Expand All @@ -209,10 +205,10 @@ If you get an importing error, do the following:

### Resolve warnings during data import

After clicking **Start Import**, if you see a warning message such as `can't find the corresponding source files`, resolve this by providing the correct source file, renaming the existing one according to [Naming Conventions for Data Import](/tidb-cloud/naming-conventions-for-data-import.md), or using **Advanced Settings** to make changes.
If the pre-check shows a warning such as `can't find the corresponding source files`, resolve it by providing the correct source file, renaming the existing one according to [Naming Conventions for Data Import](/tidb-cloud/naming-conventions-for-data-import.md), or returning to **Source Files Mapping** and configuring manual mappings.

After resolving these issues, you need to import the data again.
After resolving the issues, run the pre-check again.

### Zero rows in the imported tables

After the import progress shows **Completed**, check the imported tables. If the number of rows is zero, it means no data files matched the Bucket URI that you entered. In this case, resolve this issue by providing the correct source file, renaming the existing one according to [Naming Conventions for Data Import](/tidb-cloud/naming-conventions-for-data-import.md), or using **Advanced Settings** to make changes. After that, import those tables again.
After the import progress shows **Completed**, check the imported tables. If the number of rows is zero, verify that the **Source Files URI** and manual source patterns match the intended files. Correct the URI or mappings, and then import the tables again.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
6 changes: 4 additions & 2 deletions tidb-cloud/premium/import-from-s3-premium.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ This document describes how to import CSV files from Amazon Simple Storage Servi

- To ensure data consistency, {{{ .premium }}} allows importing CSV files into empty tables only. If the target table already contains data, import into a staging table and then copy the rows using the `INSERT ... SELECT` statement.
- During the public preview, the user interface currently supports Amazon S3 as the only storage provider. Support for additional providers will be added in future releases.
- Each import job maps a single source pattern to one destination table.
- Each source pattern maps to one destination table. You can add multiple mappings to an import job.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated

## Step 1. Prepare the CSV files

Expand Down Expand Up @@ -55,7 +55,9 @@ The wizard includes a helper link labeled **Click here to create a new one with
- Click **Test Bucket Access** to validate connectivity. <!--Todo-- Known preview issue: the button returns to the idle state without a success toast.-->

4. Click **Next** and provide the TiDB SQL username and password for the import job. Optionally, test the connection.
5. Review the automatically generated source-to-target mapping. Disable automatic mapping if you need to define custom patterns and destination tables.
5. Configure the source-to-target mapping:
- To use automatic mapping, leave **Use TiDB file naming conventions for automatic mapping** selected.
- To configure mappings manually, deselect the automatic mapping option. For each target table, enter a source file pattern relative to the **Source Files URI**, and then enter the target database and table. The source pattern supports `*` and `?` wildcards. To add another mapping, click **+**.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
6. Click **Next** to run the pre-check. Resolve any warnings about missing files or incompatible schemas.
7. Click **Start Import** to launch the job group.
8. Monitor the job statuses until they show **Completed**, then verify the imported data in TiDB Cloud.
Expand Down