Using dask for grid search can lead to huge performance improvements over multiprocessing due to better use of data locality. However, this raises a question of interface. Dask is a much larger tool, with many more intricacies. For example, I've had dask hang when it's used with multithreaded numba code. I've had other problems monitoring output when dask was launched through a notebook. This was mostly a year ago, so I'm unsure if any of these problems have been resolved.
What is the simplest way to provide a fool-proof dask based clustering backend?
One solution is to just provide it as a script, but this isn't the nicest api. Ideally it could be run through a notebook, and I'd like users to be able to provide their own client. I need to investigate how plausible this is.
Using dask for grid search can lead to huge performance improvements over multiprocessing due to better use of data locality. However, this raises a question of interface. Dask is a much larger tool, with many more intricacies. For example, I've had dask hang when it's used with multithreaded numba code. I've had other problems monitoring output when dask was launched through a notebook. This was mostly a year ago, so I'm unsure if any of these problems have been resolved.
What is the simplest way to provide a fool-proof dask based clustering backend?
One solution is to just provide it as a script, but this isn't the nicest api. Ideally it could be run through a notebook, and I'd like users to be able to provide their own client. I need to investigate how plausible this is.