gh-124397: Add free-threading support for iterators.#148894
gh-124397: Add free-threading support for iterators.#148894rhettinger wants to merge 8 commits intopython:mainfrom
Conversation
rhettinger
changed the title
rhettinger
force-pushed
the
iterator_synchronization
branch
from
21fca13 to
4c2bad0
Compare
|
|
||
| import threading | ||
|
|
||
| source = range(5) |
There was a problem hiding this comment.
This doesn't seem like a very compelling example because range() you can iterate over range concurrent from multiple threads already.
Maybe something like a simple generator would be more useful as an example?
|
|
||
| .. doctest:: | ||
|
|
||
| import threading |
There was a problem hiding this comment.
Needs >>> for doctest to run. Alternatively, use .. code-block:: python
| self.iterator = iter(iterable) | ||
| self.lock = Lock() |
There was a problem hiding this comment.
Please prefix .iterator and .lock with _ so that they aren't considered part of serialize's public API.
| 'Barrier', 'BrokenBarrierError', 'Timer', 'ThreadError', | ||
| 'setprofile', 'settrace', 'local', 'stack_size', | ||
| 'excepthook', 'ExceptHookArgs', 'gettrace', 'getprofile', | ||
| 'serialize', 'synchronized', 'concurrent_tee', |
There was a problem hiding this comment.
I suggest we not add these to __all__. Doing that complicates life for anyone unfortunate enough to have done a * import while already using these names from other imports. Code reads better if people explicitly spell them out anyways, threading.serialize(...) or @threading.synchronized_iteratorare more self explanatory than the bare words. Not putting them inall` helps encourage that. Leave a comment here mentioning that they were intentionally excluded.
| def synchronized(func): | ||
| """Wrap an iterator-returning callable to make its iterators thread-safe. | ||
|
|
||
| Existing itertools and more-itertools can be wrapped so that their | ||
| iterator instances are serialized. | ||
|
|
||
| For example, itertools.count does not make thread-safe instances, | ||
| but that is easily fixed with: | ||
|
|
||
| atomic_counter = synchronized(itertools.count) | ||
|
|
||
| Can also be used as a decorator for generator functions definitions | ||
| so that the generator instances are serialized:: | ||
|
|
||
| import time | ||
|
|
||
| @synchronized |
There was a problem hiding this comment.
| def synchronized(func): | |
| """Wrap an iterator-returning callable to make its iterators thread-safe. | |
| Existing itertools and more-itertools can be wrapped so that their | |
| iterator instances are serialized. | |
| For example, itertools.count does not make thread-safe instances, | |
| but that is easily fixed with: | |
| atomic_counter = synchronized(itertools.count) | |
| Can also be used as a decorator for generator functions definitions | |
| so that the generator instances are serialized:: | |
| import time | |
| @synchronized | |
| def synchronized_iterator(func): | |
| """Wrap an iterator-returning callable to make its iterators thread-safe. | |
| Existing itertools and more-itertools can be wrapped so that their | |
| iterator instances are serialized. | |
| For example, itertools.count does not make thread-safe instances, | |
| but that is easily fixed with: | |
| atomic_counter = threading.synchronized_iterator(itertools.count) | |
| Can also be used as a decorator for generator functions definitions | |
| so that the generator instances are serialized:: | |
| import time | |
| @threading.synchronized_iterator |
synchronized is a very generic name, I suggest making its specific purpose explicit.
| ## Synchronization tools for iterators ##################### | ||
|
|
||
| class serialize: | ||
| """Wrap a non-concurrent iterator with a lock to enforce sequential access. |
There was a problem hiding this comment.
Add an example to the docstring.
Similar to my other suggestion should we name this serialize_iterator given how generic of a word serialize is?
There was a problem hiding this comment.
Consider adding a test that an iterator __next__ raising an exception behaves properly.
|
|
||
| ## Synchronization tools for iterators ##################### | ||
|
|
||
| class serialize: |
There was a problem hiding this comment.
Should this be a subclass of Iterable? (suggested by @serhiy-storchaka on the issue)
| allow reliable concurrency support to be added to ordinary iterators and | ||
| iterator-producing callables. | ||
|
|
||
| The :class:`serialize` wrapper lets multiple threads share a single iterator and |
There was a problem hiding this comment.
To bikeshed the name a topic https://discuss.python.org/t/name-for-new-itertools-object-to-make-iterators-thread-safe/90394 was created. serialize is not my favorite, but on the topic there was no strong support for any of the other names either.
4 participants
📚 Documentation preview 📚: https://cpython-previews--148894.org.readthedocs.build/