feat(auth): mTLS endpoint for Regional Access Boundaries

[vverman]

Added logic to:

1. Centralize mTLS enablement logic within the auth library.
2. Based on 1. determine whether mtls or regular RAB lookup endpoint should be called.
3. Added tests for the same.

[gemini-code-assist]

Code Review

This pull request introduces centralized mTLS enablement checks and adds fallback support for SPIFFE credentials in MtlsUtils and X509Provider, alongside integrating mTLS transport initialization during regional access boundary refreshes. The review feedback suggests optimizing performance by removing redundant configuration checks and file parsing in X509Provider.getKeyStore() and GoogleCredentials.java, and improving robustness in RegionalAccessBoundary.java by replacing only the host name in the IAM credentials URL.

[lsirac]

Should we be checking GOOGLE_API_USE_MTLS_ENDPOINT?

[lsirac]

I’m not sure that cert being present == automatically use mTLS. They can be using different credentials / not using it at all. So then we’d be adding mTLS setup and calls for credentials that are not actually using it.

I think the decision should be based on the credential type, and perhaps expose some state from the credential that we can use to check if mTLS should happen for these calls.

[lsirac]

The static shared executor (DEFAULT_SHARED_EXECUTOR) initializes with corePoolSize = 5, maximumPoolSize = 5, and a bounded queue LinkedBlockingQueue<>(100). Every RegionalAccessBoundaryManager instance across all credentials in the entire JVM shares this single queue.

Under concurrent request spikes or transient network/mTLS timeouts where background refresh tasks saturate this 100-item queue, this.executor.submit(refreshTask) at line 227 throws RejectedExecutionException.

The catch block resets isRefreshing to false without calling handleRefreshFailure(e). Because handleRefreshFailure is bypassed, cooldownState remains 0. On the next outbound HTTP request by any thread in the JVM, triggerAsyncRefresh checks isRefreshing.compareAndSet(false, true) (which succeeds), and calls this.executor.submit(refreshTask) again, throwing RejectedExecutionException again. This spins the CPU continuously on every API request across the JVM without ever entering cooldown.

We recommend treating scheduling rejections (RejectedExecutionException) as retriable refresh failures so the manager enters cooldown (handleRefreshFailure), preventing CPU spin loops during queue saturation:

@@ -226,6 +226,7 @@
       try {
         this.executor.submit(refreshTask);
       } catch (Exception | Error e) {
+        handleRefreshFailure(new IOException("Failed to submit background refresh task: " + e.getMessage(), e));
         log(
             LOGGER_PROVIDER,
             Level.FINE,

[vverman]

I hadn't considered errors during submitting to an executor. Thanks for the catch, made the changes.

[lsirac]

Checking whether the base AccessToken is expired (accessToken.getExpirationTimeMillis() < clock.currentTimeMillis()) without any clock skew buffer means near-expiry tokens can pass validation locally but expire mid-flight over the network.

Furthermore, throwing IllegalArgumentException on token expiry causes triggerAsyncRefresh (line 480) to pass IllegalArgumentException to handleRefreshFailure, immediately locking RegionalAccessBoundaryManager into a 15-minute INITIAL_COOLDOWN_MILLIS state (cooldownState). Even when GoogleCredentials refreshes its base access token milliseconds later on the next request, triggerAsyncRefresh returns immediately (if (isCooldownActive()) return;), starving outbound API requests of the x-allowed-locations header for 15 minutes.

We recommend enforcing a 3-minute (180_000L) clock skew buffer before attempting network lookups, and throwing IOException instead of IllegalArgumentException so token expiration errors are cleanly distinguished from illegal arguments:

@@ -188,7 +188,7 @@
     Preconditions.checkNotNull(accessToken, "The provided access token is null.");
     if (accessToken.getExpirationTimeMillis() != null
-        && accessToken.getExpirationTimeMillis() < clock.currentTimeMillis()) {
-      throw new IllegalArgumentException("The provided access token is expired.");
+        && accessToken.getExpirationTimeMillis() <= clock.currentTimeMillis() + 180_000L) {
+      throw new IOException("The provided access token is expired or expiring within skew buffer.");
     }

[vverman]

Done!

[lsirac]

When ComputeEngineCredentials.getAccount() returns null (which occurs frequently during early GCE VM boot when network queries to metadata.google.internal experience transient timeouts), RegionalAccessBoundaryManager.refresh() throws an exception that triggerAsyncRefresh catches and passes to handleRefreshFailure.

However, in ComputeEngineCredentials and RegionalAccessBoundaryManager (line 464), transient null account responses set skipRAB.set(true). Once skipRAB is set to true, every subsequent call to refreshRegionalAccessBoundaryIfExpired returns immediately without checking or refreshing the RAB (if (skipRAB.get()) return;). This permanently disables Regional Access Boundaries for the entire lifecycle of the GCE VM credential after a single transient startup timeout.

We recommend removing the permanent skipRAB.set(true) latch on transient null account lookups so that future requests can retry once the metadata server becomes available:

@@ -461,8 +461,7 @@
     String account = credentials.getAccount();
     if (account == null) {
       log(LOGGER_PROVIDER, Level.FINE, "GCE service account email is null; skipping RAB check.");
-      skipRAB.set(true);
-      return;
+      throw new IOException("Transient null GCE service account email; cannot resolve RAB at this time.");
     }

[vverman]

I changed the logic so we don't skip when MDS returns a null response. Thanks for the insight.

[lsirac]

If canBeEnabled(config) returns true, MtlsUtils.getEndpoint upgrades the target host to .mtls.googleapis.com (e.g. iamcredentials.mtls.googleapis.com).

However, if prepareTransportFactoryIfMtlsEnabled (line 330) subsequently returns the base non-mTLS transport (OAuth2Utils.HTTP_TRANSPORT_FACTORY)—such as when a user injects a custom HttpTransportFactory, or when certificate_config.json fails to load under AUTO endpoint usage policy—RegionalAccessBoundary.refresh() attempts an HTTPS GET against *.mtls.googleapis.com using a standard non-mTLS HTTP client without client certificates.

Because mTLS endpoints reject non-certificate connections during TLS handshake, this throws javax.net.ssl.SSLHandshakeException. This failure triggers handleRefreshFailure, trapping the credential in exponential backoff cooldown loops up to 6 hours (MAX_COOLDOWN_MILLIS).

We recommend verifying that the active transport factory is actually mTLS-enabled before mutating the URL to .mtls.googleapis.com, or falling back cleanly to the standard endpoint when client certificates cannot be attached:

@@ -327,6 +327,9 @@
     if (!canBeEnabled(config)) {
       return endpoint;
     }
+    if (transportFactory == OAuth2Utils.HTTP_TRANSPORT_FACTORY) {
+      return endpoint; // Do not switch to .mtls endpoint if using standard non-mTLS transport factory
+    }

[lsirac]

In unsuccessfulResponseHandler, the retriable status check inspects only statusCode == 500 || statusCode == 502 || statusCode == 503 || statusCode == 504.

If the Regional Access Boundary lookup endpoint (iamcredentials.googleapis.com) returns HTTP 429 Too Many Requests (throttling/rate-limiting) or HTTP 408 Request Timeout, handleResponse treats it as a terminal non-retriable auth failure and throws IOException (+235). This triggers handleRefreshFailure and immediately locks the RegionalAccessBoundaryManager into a 15-minute INITIAL_COOLDOWN_MILLIS lockout (cooldownState).

We recommend explicitly including HTTP 429 (Rate Limit) and HTTP 408 (Timeout) as retriable status codes so transient throttling backoff is handled gracefully without locking out credentials:

@@ -214,6 +214,8 @@
                   return statusCode == 500
                       || statusCode == 502
                       || statusCode == 503
-                      || statusCode == 504;
+                      || statusCode == 504
+                      || statusCode == 429
+                      || statusCode == 408;
                 });

[vverman]

I believe the decision to not make 4xx requests retryable was to avoid overwhelming the lookup endpoint. As a general rule we should avoid retrying 4xx.

I think we should keep as is.

@nbayati thoughts?

[lsirac]

In MtlsHttpTransportFactory.java, the public method signature of create() has been widened from public NetHttpTransport create() to public HttpTransport create().

Under Java Binary Compatibility rules (JLS §13.4.15 and JVMS §4.3.3), return type changes alter JVM bytecode method descriptors (create()Lcom/google/api/client/http/javanet/NetHttpTransport; vs create()Lcom/google/api/client/http/HttpTransport;). Even though MtlsHttpTransportFactory is annotated @InternalApi, any existing compiled code, downstream auth wrapper, or test library calling MtlsHttpTransportFactory.create() and expecting NetHttpTransport will fail at runtime with java.lang.NoSuchMethodError when linked against this upgraded JAR without recompilation.

We recommend preserving public NetHttpTransport create() as the return type or providing a bridge/deprecated overload if NetHttpTransport specificity must be maintained for backward binary compatibility:

@@ -63,7 +63,7 @@
    * @return an {@link HttpTransport} instance based on the configuration.
    */
   @Override
-  public HttpTransport create() {
+  public NetHttpTransport create() {

[vverman]

This was covered in this discussion.

Basically this is only ever used internally and the origin i.e. MtlsHttpTransportFactory and the destination i.e. IdentityPoolCredentials both define the contract for HttpTransport only."