Vault error during forwarded RPC request

[urfavbarae]

Hello,

I'm having an issue with Vault.

I have a Vault instance installed in Cluster1 and an application running in Cluster2. My goal is to inject secrets from Vault into the application.

I already configured this successfully for an application running in the same cluster as Vault, and it works without any issues. However, when I try to do the same for an application running in a different Kubernetes cluster, it fails.

The following errors appear in the Vault logs:

2026-07-28T09:00:13.131Z [ERROR] core: forward request error: error="error during forwarding RPC request"
2026-07-28T09:00:13.131Z [ERROR] core: error during forwarded RPC request: error="rpc error: code = Canceled desc = context canceled"

Has anyone encountered this issue before or knows what could be causing it?

[dosubot]

Hi @urfavbarae! I'm Dosu and I'm helping the HashiCorp, an IBM Company team.

The error during forwarding RPC request with context canceled typically means a Vault standby node tried to forward a request to the active node, but the connection was interrupted before completion. This is usually a symptom of an underlying network, DNS, or configuration issue rather than a Vault bug itself [1].

In cross-cluster Kubernetes setups, the most common causes are:

1. cluster_addr misconfiguration — Vault's request forwarding dials the active node's cluster_addr directly over gRPC/TLS. If cluster_addr resolves to a cluster-internal address that isn't reachable from the standby nodes (or from a cross-cluster path), forwarding will fail [2].

2. DNS resolution failures — Standby nodes may fail to resolve the active node's internal DNS name, which produces the forwarded-RPC error as a downstream symptom [3].

3. Load balancer or ingress interruption — If a load balancer or ingress sits between the client and Vault, it may route the request to a standby node that then tries (and fails) to forward to the active node. Some proxies terminate long-lived gRPC connections or impose short timeouts [4].

4. TLS certificate or clock drift issues — If TLS certificates are invalid due to clock skew between clusters, RPC forwarding will fail [5].

Recommended diagnostic steps:

• Verify that the application in Cluster2 can reach the active Vault node in Cluster1 directly (not just any Vault pod).
• Check cluster_addr and api_addr in your Vault configuration — cluster_addr should be routable between all Vault nodes [6].
• Look at the active node's logs (not just the standby), since the standby only reports that forwarding failed — the real error is often on the active side [7].
• Confirm DNS resolution and TLS certificate validity across both clusters, and ensure clocks are synchronized.

Could you share your Vault version, Vault server configuration (especially cluster_addr, api_addr, and listener settings), and how the application in Cluster2 connects to Vault (e.g., through an ingress, load balancer, or direct service)? That would help narrow down the root cause.

You may also want to post this to the HashiCorp Discuss forum so the answer can help other community members running similar multi-cluster setups.

Written by @dosu, approved by heatherezell

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[urfavbarae]

Hi @dosu,

Thanks for the explanation. Could you tell me exactly what information you need from my side?

i believe that i tested all the things that you mentioned and all seems working fine also i did create a doc in medium with the steps that i followed can you check them: https://medium.com/@baraeboul123/hashicorp-vault-injecting-secrets-into-a-kubernetes-application-925ac62aec85?postPublishedType=initial

thanks,

[dosubot]

Hi @urfavbarae,

Unfortunately I wasn't able to access your Medium article — it appears to be behind authentication or blocked from automated access. Could you share the key details directly here instead? Specifically, to help diagnose the forwarding error, I'd need:

1. Vault version — e.g., vault version output (and whether it's Community Edition or Enterprise)
2. Vault server configuration — especially the listener, cluster_addr, and api_addr settings (from your Vault Helm values or config file)
3. Vault deployment mode — are you running Vault in HA mode with multiple replicas, or a single instance? (The RPC forwarding error implies standby-to-active forwarding, which only applies in HA)
4. How the app in Cluster2 reaches Vault in Cluster1 — is it going through a LoadBalancer service, an Ingress, a VPN, or direct pod-to-pod networking? What URL/address does the Vault Agent Injector or application use to connect?
5. Kubernetes auth method configuration — specifically the kubernetes_host and how you configured the token reviewer JWT for the remote cluster
6. The application's pod annotations — the Vault Agent Injector annotations on the pod in Cluster2

Also, when you see the error during forwarding RPC request errors in Vault's logs, are those coming from a standby Vault pod or the active one? If you're running a single Vault replica (not HA), then forwarding shouldn't be happening at all, which would point to a different issue.

Any of this info shared directly in the issue would help narrow things down!

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[urfavbarae]

hello this is the content of the doc

Steps:
Step 1 — Enable Kubernetes Authentication (already exist in the prod instance):

Enable the Kubernetes authentication backend if it is not already enabled: vault auth enable -path=kubernetes-dev kubernetes

Configure it:

vault write auth/kubernetes-dev/config
kubernetes_host=https://<DEV_KUBERNETES_API>"
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
token_reviewer_jwt="$DEV_TOKEN_REVIEWER_JWT" \ . disable_local_ca_jwt=true
Step 2 — Create the Token Reviewer ServiceAccount in the Dev Cluster

Create a dedicated ServiceAccount in the Dev cluster:

apiVersion: v1

kind: ServiceAccount

metadata:

name: vault-auth

namespace: kube-system

Apply it: kubectl apply -f vault-auth.yaml

Grant it permission to perform Kubernetes TokenReviews:

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

name: vault-token-reviewer

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

name: system:auth-delegator

subjects:

• kind: ServiceAccount

name: vault-auth

namespace: kube-system

Apply: kubectl apply -f vault-token-reviewer.yaml

Verify: kubectl auth can-i create tokenreviews --as=system:serviceaccount:kube-system:vault-auth

Expected result: yes
Step 3 — Get the Dev Cluster Token Reviewer JWT

Generate a token for the vault-auth ServiceAccount: kubectl -n kube-system create token vault-auth

Store it: export DEV_TOKEN_REVIEWER_JWT=""
Become a Medium member

Then configure the authentication backend in the Prod Vault:

vault write auth/kubernetes-dev/config \

kubernetes_host=https://<DEV_KUBERNETES_API>" \

kubernetes_ca_cert=@/path/to/dev/ca.crt \

token_reviewer_jwt="$DEV_TOKEN_REVIEWER_JWT" \

disable_local_ca_jwt=true

Verify the configuration: vault read auth/kubernetes-dev/config
Step 4 — Create a KV Secret

Enable KV v2 (if not already enabled): vault secrets enable -path=secret kv-v2 (already exist in the prod instance secret/ )

Create the application’s secret:

vault kv put secret/app_name/config
key1=value1
key2=value2
key3=value3 ....
Step 5 — Create a Policy (you can do this via UI):

Example policy:

path "secret/data/app-name/config" {
capabilities = ["read"]
}

Upload it: vault policy write policy-name policy.hcl
Step 6 — Create a Kubernetes Role:

Bind the Kubernetes ServiceAccount to the policy:

vault write auth/kubernetes/role/role_name
bound_service_account_names=your-sa-name
bound_service_account_namespaces=your-app-ns
policies=your-policy
ttl=24h
Step 7 — Annotate the Deployment:

Add the following annotations to the Pod template (your deployment yaml):

metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"

vault.hashicorp.com/agent-inject-vault-addr: "https://vault.orange.be/"

vault.hashicorp.com/auth-path: "auth/kubernetes-dev"

vault.hashicorp.com/role: "role-name"

vault.hashicorp.com/agent-inject-secret-config.env: "secret/data/secretpath/config"

vault.hashicorp.com/agent-inject-template-config.env: |
{{{{- with secret "secret/data/secretpath/config" -}}}}
{{{{- range $k, $v := .Data.data }}}}
{{{{$k}}="{{ $v }}"}}
{{{{- end }}}}
{{{{- end }}}}

Step 8 — Configure the Application:

The injected secret will be available at: /vault/secrets/config.env

this is diffrent for each technology for example for laravel project you need to add this commands so a env file got generated:

command:

• sh
• -c
• |
  rm -f /app/.env
  ln -s /vault/secrets/config.env /app/.env

php artisan config:clear
php artisan cache:clear

exec php artisan serve --host=0.0.0.0 --port=8000

Step 9 — Verify the Injection:

Check that the Vault Agent sidecar has been injected: kubectl describe pod
you should see: vault.hashicorp.com/agent-inject-status: injected

[urfavbarae]

hey can i get a response from the support