Skip to content

Commit 2dde172

Browse files
Neenu1995gemini-code-assist[bot]
Neenu1995
and
gemini-code-assist[bot]
authored
docs(bigquery-jdbc): add user guide with connection property and custom endpoint reference (#13878)
b/538176465 --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
1 parent 0557e69 commit 2dde172

4 files changed

Lines changed: 1015 additions & 0 deletions

File tree

‎java-bigquery-jdbc/DEVELOPMENT.md‎

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
# BigQuery JDBC Developer & Contributor Guide
2+
3+
This guide details the architectural design, core abstractions, coding principles, and testing workflows for developers contributing to the `google-cloud-bigquery-jdbc` module.
4+
5+
---
6+
7+
## Table of Contents
8+
1. [Core Architecture & Component Map](#1-core-architecture--component-map)
9+
2. [Developer Guardrails & Rules of Engagement](#2-developer-guardrails--rules-of-engagement)
10+
3. [Build & Test Playbook](#3-build--test-playbook)
11+
- [Local Build Commands](#local-build-commands)
12+
- [Running Unit Tests](#running-unit-tests)
13+
- [Running Integration Tests](#running-integration-tests)
14+
- [Dockerized Execution](#dockerized-execution)
15+
4. [Logging Architecture & Developer Conventions](#4-logging-architecture--developer-conventions)
16+
- [Instantiating Loggers](#instantiating-loggers)
17+
- [Developer Logging Rules & Conventions](#developer-logging-rules--conventions)
18+
5. [Pre-PR Checklist](#5-pre-pr-checklist)
19+
20+
---
21+
22+
## 1. Core Architecture & Component Map
23+
24+
The driver is structured to provide high performance, zero-allocation MDC log tracing, strict JDBC compliance, and seamless execution over the Google Cloud BigQuery REST and Storage APIs.
25+
26+
```mermaid
27+
graph TD
28+
Client[Client Application / BI Tool] -->|DriverManager.getConnection| Driver[BigQueryDriver]
29+
Driver -->|Parses URI & Options| UrlUtil[BigQueryJdbcUrlUtility]
30+
Driver -->|Configures Logging| RootLogger[BigQueryJdbcRootLogger]
31+
Driver -->|Creates| Conn[BigQueryConnection]
32+
Conn -->|Dynamic Context Proxy| Proxy[BigQueryJdbcContextProxy]
33+
Proxy -->|MDC Tracing| Mdc[BigQueryJdbcMdc]
34+
Proxy -->|Delegates Exec| DirectConn[Client Session]
35+
Conn -->|Type Mapping & Coercion| Coercion[BigQueryJdbcTypeMappings & BigQueryCoercion]
36+
Conn -->|REST / Storage API| BQSDK[google-cloud-bigquery]
37+
```
38+
39+
### Key Abstractions
40+
41+
- **`BigQueryDriver`**: JDBC entry point registered with `java.sql.DriverManager`. Intercepts `jdbc:bigquery://` URLs, initializes early logger state, and instantiates `BigQueryConnection`.
42+
- **`BigQueryConnection`**: Represents an active BigQuery session, holding dataset defaults, connection configuration maps, and transaction/session state (`EnableSession=true`, `session_id`).
43+
- **`BigQueryJdbcUrlUtility`**: Parses and validates connection string parameters using a bounded LRU parse cache (`PARSE_CACHE`) to avoid heavy allocations during frequent connection creation.
44+
- **`BigQueryJdbcContextProxy`**: A dynamic proxy layer (`java.lang.reflect.Proxy`) wrapping JDBC statements, connections, and metadata. Intercepts calls to propagate ThreadLocal MDC parameters (`connectionId`) across execution threads and enforce state validation (`checkClosed()`).
45+
- **`BigQueryJdbcTypeMappings` & `BigQueryCoercion`**: Centralized mapping logic handling standard JDBC-to-BigQuery SQL type mappings (`StandardSQLTypeName`) and object coercions (`Date`, `Timestamp`, `BigDecimal`, etc.).
46+
- **`BigQueryArrowResultSet`**: Custom result set implementation accelerating large query result retrieval via the BigQuery Storage Read API gRPC stream.
47+
48+
---
49+
50+
## 2. Developer Guardrails & Rules of Engagement
51+
52+
> [!IMPORTANT]
53+
> **Adhere strictly to the following guardrails when making code changes:**
54+
55+
1. **Visibility Principle**: Always default to the most restrictive access level (`private`, package-private, or `@InternalApi`). Do **NOT** expose classes or methods as `public` unless strictly required by standard JDBC interfaces.
56+
2. **Explicit Class Imports**: Always write explicit `import` statements. Do **NOT** use wildcard star imports or inline fully qualified class names (e.g., use `import java.math.BigDecimal;` instead of `java.math.BigDecimal` inline).
57+
3. **Logger Preference**: Always prefer `BigQueryJdbcCustomLogger` over `java.util.logging.Logger`. Format strings using `String.format(...)` before logging, as `BigQueryJdbcRootLogger` evaluates `record.getMessage()` directly.
58+
4. **Exception Handling**: Always throw exceptions from the `com.google.cloud.bigquery.exception` package (`BigQueryJdbcException`, `BigQueryJdbcSqlSyntaxErrorException`, `BigQueryConversionException`).
59+
5. **No Mocking of Final JDK Classes**: Do **NOT** mock final JDK types (`BigDecimal`, `LocalDate`, `Instant`, `UUID`) with Mockito. Mocking final JDK classes is unstable and can cause JVM crashes under JDK 21+. Always construct real instances in unit tests.
60+
61+
---
62+
63+
## 3. Build & Test Playbook
64+
65+
Builds and test tasks are managed via the module [Makefile](Makefile).
66+
67+
### Local Build Commands
68+
69+
```bash
70+
# Build & install module locally
71+
make install
72+
73+
# Clean project target directory
74+
make clean
75+
76+
# Format code and check linter compliance
77+
make lint
78+
```
79+
80+
### Running Unit Tests
81+
82+
```bash
83+
# Run all unit tests
84+
make unittest
85+
86+
# Run a specific unit test class
87+
make unittest test=BigQueryPreparedStatementTest
88+
89+
# Run a specific unit test method
90+
make unittest test=BigQueryPreparedStatementTest#testSetObjectWithTemporalTypes
91+
```
92+
93+
### Running Integration Tests
94+
95+
> [!WARNING]
96+
> Integration tests connect to real GCP BigQuery resources and require valid GCP credentials.
97+
98+
```bash
99+
# Set GCP service account credentials
100+
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json
101+
102+
# Run a specific integration test
103+
make integration-test test=ITBigQueryJDBCTest#testValidServiceAccountAuthenticationOAuthPvtKey
104+
```
105+
106+
### Dockerized Execution
107+
108+
If local Java/Maven environments are not available, use the dockerized environment:
109+
110+
```bash
111+
# Start an interactive shell session inside Docker container
112+
make docker-session
113+
114+
# Run unit tests inside Docker
115+
make docker-unittest
116+
```
117+
118+
---
119+
120+
## 4. Logging Architecture & Developer Conventions
121+
122+
The driver uses a custom logging subsystem built on top of `java.util.logging`: `BigQueryJdbcCustomLogger` and `BigQueryJdbcRootLogger`.
123+
124+
### Instantiating Loggers
125+
126+
- **For Instance Components** (`BigQueryConnection`, `BigQueryStatement`, `BigQueryDatabaseMetaData`):
127+
Use `this.toString()` to include instance identity in logger output:
128+
```java
129+
private final BigQueryJdbcCustomLogger LOG = new BigQueryJdbcCustomLogger(this.toString());
130+
```
131+
- **For Static / Utility Components** (`BigQueryJdbcUrlUtility`, `BigQueryJdbcTypeMappings`):
132+
Use the class name:
133+
```java
134+
private static final BigQueryJdbcCustomLogger LOG =
135+
new BigQueryJdbcCustomLogger(BigQueryJdbcTypeMappings.class.getName());
136+
```
137+
138+
### Developer Logging Rules & Conventions
139+
140+
1. **Method Entry / Exit Tracing**:
141+
Methods at `FINER` level must log entrance and exit points:
142+
```java
143+
public ResultSet executeQuery(String sql) throws SQLException {
144+
LOG.finer("++enter++");
145+
try {
146+
// ... execution logic ...
147+
return rs;
148+
} finally {
149+
LOG.finer("++exit++");
150+
}
151+
}
152+
```
153+
2. **Format Placeholders (Zero Allocation)**:
154+
Avoid string concatenation in log calls. Use formatting placeholders or `Supplier<String>` lambdas to prevent unneeded string allocation when the log level is disabled:
155+
```java
156+
// Recommended: Use printf-style formatting
157+
LOG.fine("Executing query on dataset: %s, table: %s", datasetId, tableId);
158+
159+
// Recommended: Use supplier lambda for expensive calculations
160+
LOG.fine(() -> "Parsed properties: " + complexObject.toDebugString());
161+
```
162+
3. **Caller Inference & MDC Propagation**:
163+
- `BigQueryJdbcCustomLogger` automatically wraps log records in `BigQueryJdbcLogRecord`, which inspects the stack trace to accurately infer caller class and method names.
164+
- `BigQueryJdbcMdc` maintains `connectionId` in a `ThreadLocal` context. When logging from proxy or worker threads, always ensure MDC context is preserved or propagated via `BigQueryJdbcContextProxy`.
165+
166+
---
167+
168+
## 5. Pre-PR Checklist
169+
170+
Before submitting a Pull Request:
171+
172+
- [ ] All new classes and methods use the narrowest possible visibility scope (`private` or package-private).
173+
- [ ] No inline fully qualified names or wildcard star imports are present.
174+
- [ ] All logger instances use `BigQueryJdbcCustomLogger`.
175+
- [ ] Method entrance/exit logging (`++enter++` / `++exit++`) is included for complex internal routines.
176+
- [ ] All method changes and feature additions are covered by corresponding JUnit 5 tests.
177+
- [ ] Unit tests pass cleanly without Mockito `UnnecessaryStubbingException` warnings.
178+
- [ ] All `Statement`, `ResultSet`, or `DatabaseMetaData` objects returned by public entry points are properly wrapped via `BigQueryJdbcContextProxy.wrap()`.
179+
- [ ] Code formatting and linting pass via `make lint`.

‎java-bigquery-jdbc/README.MD‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ Java idiomatic client for [BigQuery JDBC][product-docs].
77

88
- [Product Documentation][product-docs]
99
- [Client Library Documentation][javadocs]
10+
- [Driver User Guide](docs/USER_GUIDE.md)
11+
- [Storage APIs Deep-Dive Guide](docs/STORAGE_APIS.md)
1012

1113

1214
## Quickstart

‎java-bigquery-jdbc/docs/STORAGE_APIS.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# BigQuery Storage APIs Deep Dive & Tuning Guide
2+
3+
This document provides architectural details, property matrices, activation criteria, and workload tuning scenarios for the **BigQuery Storage Read API** and **BigQuery Storage Write API** integrated into the BigQuery JDBC Driver.
4+
5+
---
6+
7+
## 1. High-Throughput Storage Read API (HTAPI)
8+
9+
The Storage Read API streams query result sets over high-speed gRPC channels using Apache Arrow format, bypassing standard REST JSON serialization for large datasets.
10+
11+
### Property Reference
12+
13+
| Property Name | Connection Parameter | Default Value | Functional Role |
14+
| :--- | :--- | :---: | :--- |
15+
| **`EnableHighThroughputAPI`** | `EnableHighThroughputAPI=true` | `false` | **Master Toggle**: Must be `true` to enable Read API evaluation. |
16+
| **`HighThroughputMinTableSize`** | `HighThroughputMinTableSize=10000` | `10000` | **Minimum Row Threshold**: Minimum total rows (`totalRows`) required. |
17+
| **`HighThroughputActivationRatio`** | `HighThroughputActivationRatio=2` | `2` | **Page Ratio Threshold**: `totalRows / MaxResults` ratio required. |
18+
| **`MaxResults`** | `MaxResults=10000` | `10000` | **Page Size**: Controls rows per page in standard REST calls. |
19+
20+
### Activation Criteria & Fallback Mechanics
21+
22+
When `EnableHighThroughputAPI=true` is set, the driver transparently switches to the Storage Read API if all of the following conditions are met:
23+
24+
1. **Master Toggle**: `EnableHighThroughputAPI=true` is set.
25+
2. **Minimum Row Threshold**: The query returns at least `HighThroughputMinTableSize` rows (default: $\ge 10,000$ rows).
26+
3. **Multiple Response Pages**: The result set spans more than one page (total rows exceed `MaxResults`). If all rows fit on page 1, standard REST is used to avoid unnecessary gRPC stream setup.
27+
4. **Activation Ratio Test**: The ratio of total rows to page size ($\frac{\text{totalRows}}{\text{MaxResults}}$) exceeds `HighThroughputActivationRatio` (default: $> 2$).
28+
29+
> [!NOTE]
30+
> **Automatic Permission Fallback**: If `EnableHighThroughputAPI=true` is set but the connecting principal lacks the `BigQuery Read Session User` IAM role, the driver catches the `PERMISSION_DENIED` status and automatically falls back to standard REST JSON pagination.
31+
32+
### Workload Scenarios Matrix
33+
34+
| Workload Scenario | `EnableHighThroughputAPI` | `HighThroughputMinTableSize` | `HighThroughputActivationRatio` | `MaxResults` | Execution Mechanism | Use Case |
35+
| :--- | :---: | :---: | :---: | :---: | :--- | :--- |
36+
| **Standard REST (Default)** | `false` | `10000` (ignored) | `2` (ignored) | `10000` | REST JSON Pagination | Small/medium queries; standard REST security policies. |
37+
| **Default Production Extractions** | `true` | `10000` | `2` | `10000` | gRPC Storage Read API (for results $> 20,000$ rows) | Standard analytical reports and ETL extracts. |
38+
| **Aggressive Streaming** | `true` | `100` | `0` | `50` | gRPC Storage Read API (for results $\ge 100$ rows) | High-speed streaming for smaller analytical datasets. |
39+
| **Bulk ETL Analytics** | `true` | `50000` | `5` | `10000` | gRPC Storage Read API (for results $> 50,000$ rows) | Large multi-gigabyte dataset extractions. |
40+
41+
---
42+
43+
## 2. Storage Write API (SWA)
44+
45+
The Storage Write API streams high-throughput bulk insertions over gRPC channels for `PreparedStatement.executeBatch()` calls.
46+
47+
### Property Reference
48+
49+
| Property Name | Connection Parameter | Default Value | Functional Role |
50+
| :--- | :--- | :---: | :--- |
51+
| **`EnableWriteAPI`** | `EnableWriteAPI=true` | `false` | **Master Toggle**: Must be `true` to enable Storage Write API streaming. |
52+
| **`SWA_ActivationRowCount`** | `SWA_ActivationRowCount=3` | `3` | **Activation Threshold**: Minimum batch size added via `addBatch()` required to trigger SWA. |
53+
| **`SWA_AppendRowCount`** | `SWA_AppendRowCount=1000` | `1000` | **Chunk Size**: Maximum rows per gRPC append payload before flushing. |
54+
55+
### Activation Criteria & Fallback Mechanics
56+
57+
When `EnableWriteAPI=true` is set, the driver evaluates the batch size during `PreparedStatement.executeBatch()`:
58+
59+
- **At or Above Threshold ($\ge \text{SWA\_ActivationRowCount}$)**: The driver opens a gRPC Storage Write stream and appends batch records in payload chunks governed by `SWA_AppendRowCount`.
60+
- **Below Threshold ($< \text{SWA\_ActivationRowCount}$)**: The driver uses standard SQL DML (`INSERT INTO ...`) to avoid gRPC stream overhead for tiny batches.
61+
62+
### Workload Scenarios Matrix
63+
64+
| Workload Scenario | `EnableWriteAPI` | `SWA_ActivationRowCount` | `SWA_AppendRowCount` | Execution Mechanism | Use Case |
65+
| :--- | :---: | :---: | :---: | :--- | :--- |
66+
| **Standard SQL DML (Default)** | `false` | `3` (ignored) | `1000` (ignored) | Concatenated REST SQL DML | Small transactional DML; standard SQL compatibility. |
67+
| **Default High-Throughput ETL** | `true` | `3` | `1000` | gRPC SWA stream (batches $\ge 3$, flushes per 1,000 rows) | Standard batch loader applications (Spring Batch, Spark). |
68+
| **Real-Time Micro-Batching** | `true` | `1` | `100` | gRPC SWA stream (batches $\ge 1$, flushes per 100 rows) | High-frequency streaming events (Kafka/Flink consumers). |
69+
| **High-Volume Bulk Ingestion** | `true` | `100` | `5000` | gRPC SWA stream (batches $\ge 100$, flushes per 5,000 rows) | Large nightly bulk ETL loading millions of records. |

0 commit comments

Comments
 (0)