When multiple threads with their own MetricsLoggers are spawned there is a race to assign the static EnvironmentProvider.cachedEnvironment via EnvironmentProvider.resolveEnvironment() because multiple threads can slip past the if (cachedEnvironment != null) guard before the associated CompletableFutures have resolved. This can lead to concurrent efforts to resolve ECSEnvironment via ECSEnvironment.probe(), and non-deterministic behaviour from ECSEnvironment.getLogGroupName.
Part of setting up ECSEnvironment is calling ECSEnvironment.checkAndSetFluentHost(). In that method both a per-instance property fluentBitEndpoint and a (shared singleton) Configuration config.agentEndpoint property are set. The guard for this assignment checks that config.agentEndpoint is not already set, and if it is then neither the local fluentBitEndpoint nor config.agentEndpoint are touched. When two or more threads are simultaneously resolving ECSEnvironment on a FluentBit host and no config.agentEndpoint already exists, then the first thread will pass the !config.getAgentEndpoint().isPresent() guard and set the local fluentBitEndpoint. Subsequent threads will detect that the shared config.agentEndpoint is present and will NOT set the local fluentBitEndpoint.
When ECSEnvironment.getLogGroupName() is called, if this.fluentBitEndpoint is set then an empty string log group name is returned, otherwise a full log group name. Taken together this means that when multiple ECSEnvironments are being resolved simultaneously on a FluentBit host, the first will produce an empty string log group name, while any subsequent ones will produce a full log group name. This makes the race to assign in EnvironmentProvider consequential, since any of these ECSEnvironments could theoretically end up as the authoritative reference saved in cachedEnvironment. This can (and did!) lead to inconsistent startup behaviour where sometimes LogGroupName is empty and sometimes it is populated.
Impact: on ECS + FireLens, at cold start under concurrent first Metric emissions, a task can non-deterministically cache an ECSEnvironment whose getLogGroupName() is "", which in our case drops the LogGroup default dimension for the lifetime of the JVM. I observed this; identical tasks intermittently emit metrics with vs. without the LogGroup dimension, silently breaking dashboards/alarms keyed on it.
When multiple threads with their own MetricsLoggers are spawned there is a race to assign the static EnvironmentProvider.cachedEnvironment via EnvironmentProvider.resolveEnvironment() because multiple threads can slip past the
if (cachedEnvironment != null)guard before the associated CompletableFutures have resolved. This can lead to concurrent efforts to resolve ECSEnvironment via ECSEnvironment.probe(), and non-deterministic behaviour from ECSEnvironment.getLogGroupName.Part of setting up ECSEnvironment is calling ECSEnvironment.checkAndSetFluentHost(). In that method both a per-instance property
fluentBitEndpointand a (shared singleton) Configuration config.agentEndpoint property are set. The guard for this assignment checks that config.agentEndpoint is not already set, and if it is then neither the localfluentBitEndpointnorconfig.agentEndpointare touched. When two or more threads are simultaneously resolving ECSEnvironment on a FluentBit host and noconfig.agentEndpointalready exists, then the first thread will pass the!config.getAgentEndpoint().isPresent()guard and set the localfluentBitEndpoint. Subsequent threads will detect that the sharedconfig.agentEndpointis present and will NOT set the localfluentBitEndpoint.When ECSEnvironment.getLogGroupName() is called, if
this.fluentBitEndpointis set then an empty string log group name is returned, otherwise a full log group name. Taken together this means that when multiple ECSEnvironments are being resolved simultaneously on a FluentBit host, the first will produce an empty string log group name, while any subsequent ones will produce a full log group name. This makes the race to assign in EnvironmentProvider consequential, since any of these ECSEnvironments could theoretically end up as the authoritative reference saved in cachedEnvironment. This can (and did!) lead to inconsistent startup behaviour where sometimes LogGroupName is empty and sometimes it is populated.Impact: on ECS + FireLens, at cold start under concurrent first Metric emissions, a task can non-deterministically cache an ECSEnvironment whose getLogGroupName() is "", which in our case drops the LogGroup default dimension for the lifetime of the JVM. I observed this; identical tasks intermittently emit metrics with vs. without the LogGroup dimension, silently breaking dashboards/alarms keyed on it.