The Problem
The CARLA leaderboard always had the issue that evaluations were non-deterministic (e.g.), making evaluation expensive since they require multiple seeds to combat the variance.
Since CARLA 0.9.15 the simulator itself should be deterministic, as far as I understand, yet the CARLA leaderboard 2.0 evaluations still have large variance in our experience.
I want to point out here a problem in scenario setup that introduces non-deterministic behavior into the CARLA leaderboard.
The problem arises from the build_scenarios_loop function which is executed in a seperate thread.
This function checks periodically every 1 second (after which it sleeps) whether the ego agent is near a scenario trigger and if so sets up the scenario.
Now this causes non-deterministic evaluation behavior because there is a delay of up to 1 second wall-clock time (while the thread is sleeping).
Let's say the simulation runs at 10 Hz; this means when the ego approaches a trigger, there is a 1-20 simulated frames difference.
This is exaggerated additionally by the scenario setup itself, during which the main thread continues and may tick the simulation for an arbitrary amount of time.
Why is threading used?
Now this raises the question of why the scenarios are set up in a thread in the first place. There is a good reason for that, namely that the scenario setup requires CARLA ticks during setup, for example here and every time an actor batch is created in the carla data provider (here and here.
Running CARLA ticks within the scenario setup itself is not good because that advances the simulation without the agent being able to adjust the vehicle control.
So an interleaved control flow is required in which the leaderboard will do at least one iteration, which includes calling the agent whenever the scenario setup requires a CARLA tick. The scenario runner uses the wait_for_tick() logic for that, coupled with threaded execution.
So the threading solution implemented in the leaderboard solves the control flow issue but leads to the aforementioned non-determinism, because it only ensures that at least one iteration, while the ideal solution would guarantee that exactly one iteration is executed.
(And the ideal solution should not sleep the scenario setup logic).
Proposed solution
I believe the most elegant way to solve this control flow problem without introducing non-determinism is the use of a coding concept called stackful coroutines.
What we want is a control flow that is not parallel but can switch between functions and resume at points other than the function start.
Stackful coroutines provide that feature.
Python itself supports the yield keyword, but this is not sufficient because yield can only return from the current function, but we want to return from the scenario runner to the leaderboard through many function calls.
Fortunately, Python comes with batteries; there is a library called greenlet, that supports exactly that functionality.
The solution would look like this:
scenario_manager.py
from greenlet import greenlet
...
def __init__(self, timeout, statistics_manager, runtime_timeout, debug_mode=0):
...
self._scenario_greenlet = None
...
def build_scenarios_loop(self, debug):
"""
Keep periodically trying to start the scenarios that are close to the ego vehicle
Additionally, do the same for the spawned vehicles
"""
while self._running:
self.scenario.build_scenarios(self.ego_vehicles[0], debug=debug)
self.scenario.spawn_parked_vehicles(self.ego_vehicles[0])
self.gr_main.switch()
...
def run_scenario(self):
...
# stackful coroutine for build_scenarios
self.gr_main = greenlet.getcurrent()
self._scenario_greenlet = greenlet(self.build_scenarios_loop)
....
while self._running:
self._tick_scenario()
def _tick_scenario(self):
...
self._scenario_greenlet.switch(self._debug_mode > 0)
if self._running and self.get_running_status():
CarlaDataProvider.get_world().tick()
...
def stop_scenario(self):
...
# Set running to false. The coroutine will now exit at the next iteration.
self._running = False
# The coroutine might be busy setting up a scenario. To ensure a clean exit we will let it finish setting up
# by providing the needed CARLA ticks.
if self._scenario_greenlet is not None:
self._scenario_greenlet.switch(self._debug_mode > 0)
while not self._scenario_greenlet.dead:
CarlaDataProvider.get_world().tick()
self._scenario_greenlet.switch(self._debug_mode > 0)
self._scenario_greenlet = None
...
And in scenario_manager.py and carla_data_provider.py in scenario_runner, replace the following at the places where is_runtime_init_mode is used.
if CarlaDataProvider.is_runtime_init_mode():
gr = greenlet.getcurrent()
gr.parent.switch()
which switches back to the leaderboard loop and will resume at the same place the next time the leaderboard calls self._scenario_greenlet.switch(self._debug_mode > 0), which is exactly after one CARLA tick.
Since the scenario runner is a library, it would probably be nicer to add an is_greenlet_init_mode instead of replacing is_runtime_init_mode
for compatibility reasons.
I think this solution should be the solution with the least amount of code change that ensures both determinism and that no CARLA.tick() is called outside the leaderboard control loop.
The only downside I can see right now is an added dependency (greenlet) for the leaderboard and the scenario runner, but I think that is not a big downside, since the library is available on standard pip.
Additionally, it would probably be good if the leaderboard would set the random seed of the CarlaDataProvider somewhere, otherwise the user will not be able to run different seeds anymore (CARLA data provider always starts with _random_seed = 2000)
e.g.: CarlaDataProvider.set_random_seed(arguments.traffic_manager_seed)
The Problem
The CARLA leaderboard always had the issue that evaluations were non-deterministic (e.g.), making evaluation expensive since they require multiple seeds to combat the variance.
Since CARLA 0.9.15 the simulator itself should be deterministic, as far as I understand, yet the CARLA leaderboard 2.0 evaluations still have large variance in our experience.
I want to point out here a problem in scenario setup that introduces non-deterministic behavior into the CARLA leaderboard.
The problem arises from the build_scenarios_loop function which is executed in a seperate thread.
This function checks periodically every 1 second (after which it sleeps) whether the ego agent is near a scenario trigger and if so sets up the scenario.
Now this causes non-deterministic evaluation behavior because there is a delay of up to 1 second wall-clock time (while the thread is sleeping).
Let's say the simulation runs at 10 Hz; this means when the ego approaches a trigger, there is a 1-20 simulated frames difference.
This is exaggerated additionally by the scenario setup itself, during which the main thread continues and may tick the simulation for an arbitrary amount of time.
Why is threading used?
Now this raises the question of why the scenarios are set up in a thread in the first place. There is a good reason for that, namely that the scenario setup requires CARLA ticks during setup, for example here and every time an actor batch is created in the carla data provider (here and here.
Running CARLA ticks within the scenario setup itself is not good because that advances the simulation without the agent being able to adjust the vehicle control.
So an interleaved control flow is required in which the leaderboard will do at least one iteration, which includes calling the agent whenever the scenario setup requires a CARLA tick. The scenario runner uses the wait_for_tick() logic for that, coupled with threaded execution.
So the threading solution implemented in the leaderboard solves the control flow issue but leads to the aforementioned non-determinism, because it only ensures that at least one iteration, while the ideal solution would guarantee that exactly one iteration is executed.
(And the ideal solution should not sleep the scenario setup logic).
Proposed solution
I believe the most elegant way to solve this control flow problem without introducing non-determinism is the use of a coding concept called stackful coroutines.
What we want is a control flow that is not parallel but can switch between functions and resume at points other than the function start.
Stackful coroutines provide that feature.
Python itself supports the yield keyword, but this is not sufficient because yield can only return from the current function, but we want to return from the scenario runner to the leaderboard through many function calls.
Fortunately, Python comes with batteries; there is a library called greenlet, that supports exactly that functionality.
The solution would look like this:
scenario_manager.py
And in scenario_manager.py and carla_data_provider.py in scenario_runner, replace the following at the places where is_runtime_init_mode is used.
which switches back to the leaderboard loop and will resume at the same place the next time the leaderboard calls
self._scenario_greenlet.switch(self._debug_mode > 0), which is exactly after one CARLA tick.Since the scenario runner is a library, it would probably be nicer to add an is_greenlet_init_mode instead of replacing is_runtime_init_mode
for compatibility reasons.
I think this solution should be the solution with the least amount of code change that ensures both determinism and that no CARLA.tick() is called outside the leaderboard control loop.
The only downside I can see right now is an added dependency (greenlet) for the leaderboard and the scenario runner, but I think that is not a big downside, since the library is available on standard pip.
Additionally, it would probably be good if the leaderboard would set the random seed of the CarlaDataProvider somewhere, otherwise the user will not be able to run different seeds anymore (CARLA data provider always starts with _random_seed = 2000)
e.g.:
CarlaDataProvider.set_random_seed(arguments.traffic_manager_seed)