`get_acoustic_detections()` : cannot allocate vector of size

[PietrH]

Reports from etn client where openCPU forwards error:

> get_acoustic_detections(acoustic_project_code = "albert")
✔ Preparing : will fetch 25.95 M detections [50.2s]
Error in `req_perform_opencpu()`:
! cannot allocate vector of size 45.6 Mb
ℹ This is an error forwarded via the API.
Run `rlang::last_trace()` to see where the error occurred.

> albert <- get_acoustic_detections(animal_project_code = "2013_albertkanaal")
✔ Preparing : will fetch 5.48 M detections [32.9s]
Error in `req_perform_opencpu()`:
! cannot allocate vector of size 44.4 Mb
ℹ This is an error forwarded via the API.
Run `rlang::last_trace()` to see where the error occurred.

[PietrH]

This could be fixed simply by placing more, smaller requests by reducing page size. This will also cut down on memory usage per request.

[PietrH]

If the problem is on the client side, when allocating a raw object of 45MB, this could not be solved by instead of parsing the HTTP response as a raw, and then writing it out via writeBin(), directly writing it to file with httr2::req_perform(path = path).

https://github.com/inbo/etn/blob/e57debdf78d18495cdb8bd365ee332b44e90afa4/R/utils-api.R#L271

https://github.com/inbo/etn/blob/e57debdf78d18495cdb8bd365ee332b44e90afa4/R/utils-api.R#L297-L300

If the error takes place on the openCPU/{etnservice} side, this is where the page is loaded into memory:

etnservice/R/get_acoustic_detections_page.R

Line 259 in ad6469a

returned_page <- DBI::dbGetQuery(connection, query)

it is then mapped in memory via dplyr data wrangling, nothing but a bunch of field renaming happens really.

Because the current openCPU implementation depends on returning the page from memory, avoiding this would mean changing to the R object API: https://www.opencpu.org/api.html#api-object

• map out of memory via a database connection and write to a file
• use file session api to fetch this file as client
• client reads these files lazily and transforms/combines them

This second option is a major refactor, and probably not worth it at this time over just reducing the page size.