Replaced example folder with a simpler workflow - #126
Conversation
michelwi
left a comment
michelwi
left a comment
There was a problem hiding this comment.
It started off with some minor beauty comments but then I wen full typing-nerd.. sorry xD
Overall very instructive example and I have even learned a thing or two.. Was fully bamboozled by the variable number of outputs of the splitter job xD
|
|
||
| # Count lines, lines is a Variable meaning once it's computed its value can be accessed using .get() | ||
| # or by converting it into a string | ||
| lines = tools.WordCount(sentences).lines |
There was a problem hiding this comment.
propose to rename lines to num_lines to clarify its the number of lines in the file and not the lines itself.
| first_half = tools.Head(sentences, middle).out | ||
| tk.register_output('first_half', first_half) | ||
|
|
||
| # Tell Sisyphus that this output is needed and should be linked inside the output directory |
There was a problem hiding this comment.
move the comment up three lines before the first use of tk.register_output
Co-authored-by: michelwi <michelwi@users.noreply.github.com>
| tk.register_output('paragraph.%02i.words' % i, wc.words) | ||
|
|
||
| # All jobs inside the wc_per_paragraph list will be computed after this line | ||
| await tk.async_run(wc_per_paragraph) |
There was a problem hiding this comment.
Not sure if this can be part of the simple example, but what do you do if you have multiple experiments and you do not want to block "later" experiements that are independent of this run?
There was a problem hiding this comment.
You'll have to put everything that can run in parallel into its own coroutine and await all of them at the end like this:
await tk.async_gather(coroutine1, coroutine2, coroutine3, ...)
|
What's missing here is a simple example of having multiple tasks for one Job and a Tasks with multiple parallel threads. Any suggestions? 🤔 |
3 participants
I think the current workflow is very confusing and a bad example to get started with Sisyphus. Here is a much simpler workflow, it doesn't cover as many edge cases, but should be much easier to understand.