Race Condition while clean up of pools

[manav-yb]

Hi Team,

I'm witnessing a crash of odyssey process in one of my workloads. I'm using a bit older version of odyssey. I'm attaching below the stack trace.

thread #1, name = 'odyssey', stop reason = signal SIGSEGV
frame #0: 0x00005e30f8887c6f odysseyod_backend_connect(server=0x000032bc7e014380, context="", route_params=0x0000000000000000, client=0x000032bc7da72c00) at backend.c:815:19 frame #1: 0x00005e30f888580e odysseyod_frontend_attach(client=0x000032bc7da72c00, context="", route_params=0x0000000000000000) at frontend.c:305:8
frame #2: 0x00005e30f8881b11 odysseyod_frontend_remote [inlined] od_frontend_attach_and_deploy(client=0x000032bc7da72c00, context=<unavailable>) at frontend.c:361:11 frame #3: 0x00005e30f8881afe odysseyod_frontend_remote(client=0x000032bc7da72c00) at frontend.c:2120:13
frame #4: 0x00005e30f887dd65 odysseyod_frontend(arg=0x000032bc7da72c00) at frontend.c:2756:12 frame #5: 0x00005e30f88a6912 odysseymm_scheduler_main(arg=0x000032bc7e7cf900) at scheduler.c:17:2
frame #6: 0x00005e30f88a6b77 odyssey`mm_context_runner at context.c:28:2

The top of the threads points to: storage = route->rule->storage; used in od_router_gc_cb() function.

I'm suspecting it's happening due to a race condition on how rule field of route object is been unref https://github.com/yandex/odyssey/blob/0856d66419d8cc0a3328fa92f3d463323a55cb22/sources/router.c#L313.
`od_route_unlock(route);

/* unref route rule and free route object */
od_rules_unref(route->rule);
od_route_free(route);`

It releases the lock before doing unref and freeing the route->rule and route object respectively. What if any other worker thread is accessing route->rule after it's been unref in above code after the lock has been released.

I wanted to confirm, have you guys faced such issue in past or any fixes been done in order to handle the above scenario.

Thanks!

[rkhapov]

hi!

thanks for report! seems unfamiliar for me

there is no line 815 in backend.c for now, so i need a specific version you use (odyssey --version) to determine the problem

can you please provide it?

[manav-yb]

Thanks for your response.
I'm using
./build/latest/bin/odyssey --version odyssey (git: 26399 2.17.3.0-b25_FINAL-9763-gc4b4a52ef0 release)
In my code it points to https://github.com/yugabyte/yugabyte-db/blob/5333bb10c888dae93ab1ffa3101efb80c57521c6/src/odyssey/sources/backend.c#L815

[rkhapov]

https://github.com/yugabyte/yugabyte-db/blob/5333bb10c888dae93ab1ffa3101efb80c57521c6/src/odyssey/sources/backend.c#L815

seems like you use very old version...
you can wait for upcoming 1.4 release or use version at ac95c5f (it looks stable in our prod envs)

[manav-yb]

use version at ac95c5f

I already have the changes which are part of this commit in my build. Still I'm running into the issue.

[manav-yb]

Just to update: We have one scenario in connection burst where this issue got exposed - For de-refering the rule without acquiring the locks on route.
Making below changes in od_router_gc_cb() function

+++ b/src/odyssey/sources/router.c
@@ -384,10 +384,11 @@ clean:
+       /* unref route rule */
+       od_rules_unref(route->rule);
        od_route_unlock(route);

-       /* unref route rule and free route object */
-       od_rules_unref(route->rule);
+       /* free route object */
        od_route_free(route);
        return 0;
 done:

Fixes the problem for us.

[reshke]

Just to update: We have one scenario in connection burst where this issue got exposed - For de-refering the rule without acquiring the locks on route. Making below changes in od_router_gc_cb() function

+++ b/src/odyssey/sources/router.c
@@ -384,10 +384,11 @@ clean:
+       /* unref route rule */
+       od_rules_unref(route->rule);
        od_route_unlock(route);

-       /* unref route rule and free route object */
-       od_rules_unref(route->rule);
+       /* free route object */
        od_route_free(route);
        return 0;
 done:

Fixes the problem for us.

Great! Are you willing to submit PR for this?