Some sort of race condition in packaging.requirement

[alex]

import sys
import threading

from packaging.requirements import Requirement


N_THREADS = 32

def target():
    Requirement("x[]")

def main(argv):
    threads = []
    for i in range(N_THREADS):
        threads.append(threading.Thread(target=target))
    for t in threads:
        t.start()
    for t in threads:
        t.join()
    print("Ok")

if __name__ == "__main__":
    main(sys.argv)

This fails every once in a while. Put in a loop in your terminal for best results.

[alex]

Oh, where are my manners, here's the traceback:

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/Users/alex_gaynor/.pyenv/versions/2.7.13/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/Users/alex_gaynor/.pyenv/versions/2.7.13/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "t.py", line 10, in target
    Requirement("x[]")
  File "/Users/alex_gaynor/.virtualenvs/tempenv-420c1556839f5/lib/python2.7/site-packages/packaging/requirements.py", line 94, in __init__
    requirement_string[e.loc:e.loc + 8]))
InvalidRequirement: Invalid requirement, parse error at "''"

[dsethi]

Alex - many thanks for closing this -- I was able to use your code to replicate and test that this is indeed fixed. One followup question -- will this be part of an official release soon (so that I can get it with pip upgrade instead of doing a manual compile/install?) Thanks again!

[fofoni]

This is still biting me with packaging==21.3. I understand that the PR (#105) tried two fixes (and went with the second):

• Wrapping the parsing in a lock. I tested, and this actually works for me.
• Initializing the pyparsing engine at import time, but no synchronization at parse time. I still get pyparsing errors when I run this inside a lot of threads. Indeed, the pyparsing folks recommend never getting inside pyparsing's codebase from more than one thread at the same time (Is PyParsing thread safe / enables multithreading? pyparsing/pyparsing#89 (comment)).

I also understand that packaging 22.0 will have a tailor-made parser instead of depending on pyparsing. Will this parser have global state which must be protected by a lock, or can I assume that calling packaging.requirements.Requirement will be thread-safe?

Thanks!

[pradyunsg]

We no longer use pyparsing in main, so those race conditions aren't an issue.

[fofoni]

I'll assume the race conditions are no longer an issue? Thanks!

[pradyunsg]

Indeed, typo fixed. :)