Code Sample

import pandas as pd
import time

squares = set(a**2 for a in range(100000000))
series = pd.Series(range(100))

start = time.time()
apply_result = series.apply(lambda x: x in squares)
apply_end = time.time()
isin_result = series.isin(squares)
isin_end = time.time()

assert((apply_result==isin_result).all())

print("pandas.Series.apply() took {} seconds and pandas.Series.isin() took {} seconds.".format(apply_end - start, isin_end - apply_end))

Output:

pandas.Series.apply() took 0.0044422149658203125 seconds and pandas.Series.isin() took 72.23143887519836 seconds.

Problem description

When a set is passed to pandas.Series.isin, the set is converted to a list, before being converted back to a hash table. Consequently, the run time is linear in the size of the set, which is not ideal because one of the main reasons to use a set is that membership can be tested in constant time.

Suggested improvements

The quick and dirty workaround is to use pandas.Series.apply (as in the above code sample) instead of pandas.Series.isin. I'm not familiar enough with pandas internals to know whether there are edge cases where this would fail or whether it would be a bad idea to incorporate this workaround into isin directly. I would suggest, however, that at a minimum the documentation for isin be updated to mention that a set will be converted to a list and that this has performance implications, so that users can choose an alternative approach. (I am happy to contribute the documentation if this is the preferred solution.)

Output of pd.show_versions()