slack_channel, missing_scope, idle-in-transaction timeout

[sbirl]

v0.34.1 on Kubernetes.

Application Log

On the application side, the container logs have errors related to slack_channel and missing_scope like this:

"[31mERRO\u001b[0m[375119] \u001b[31mAuthSource\u001b[0m="SourceTypeAuthProvider{fec0c5db-edc8-4e70-815f-7d4f26a06a28}" \u001b[31mAuthUserID\u001b[0m=45feafcb-0c35-4c3a-aa65-837ebc146b86 \u001b[31mRequestID\u001b[0m=c7a67791-392b-43f8-81b0-415c3fceae20 \u001b[31merror\u001b[0m="input: messageLogs lookup dest DestV1{Type: builtin-@opensearch-dashboards-highlighted-field@slack@/opensearch-dashboards-highlighted-field@-channel, Args: map[slack_channel_id:GPDPYFH29]}: missing_scope" \u001b[31mhost\u001b[0m=oncall.EXAMPLE.COM \u001b[31mhttp_method\u001b[0m=POST \u001b[31mhttp_proto\u001b[0m=HTTP/1.1 \u001b[31mreferer\u001b[0m="https://oncall.EXAMPLE.COM/admin/message-logs" \u001b[31mremote_addr\u001b[0m="10.42.192.169:35528" \u001b[31muri\u001b[0m=/api/graphql \u001b[31mx_forwarded_for\u001b[0m=10.42.209.128 \u001b[31mx_forwarded_host\u001b[0m=oncall.EXAMPLE.COM"

Database Log

On the database side, we see idle-in-transaction timeout:

2026-03-18 13:06:33.728 EDT [2699513] user=goalert db=prd_goalert app=GoAlert v0.34.1 FATAL: terminating connection due to idle-in-transaction timeout

and something logged as

CompatAuthSubSlackMissingCM

Problem

The problem is we're not really sure what the cause is in order to troubleshoot further.
We know GoAlert is trying to send notifications to a specific channel in Slack, and they're not being received, but we're not sure why. There's not enough details in the logs.

---