Problem with 0.15.3 wheels on CircleCI

[jeromekelleher]

I'm using pysam on CircleCI builds and I've hit an issue with the just-released 0.15.3 binary wheel (pysam-0.15.3-cp36-cp36m-manylinux1_x86_64.whl)

I get the following error:

  File "/home/circleci/tskit/python/tests/test_vcf.py", line 40, in <module>
    import pysam
  File "/home/circleci/.local/lib/python3.6/site-packages/pysam/__init__.py", line 5, in <module>
    from pysam.libchtslib import *
  File "pysam/libchtslib.pyx", line 325, in init pysam.libchtslib
AttributeError: type object 'pysam.libchtslib.HTSFile' has no attribute '__reduce_cython__'

I'm not sure how to debug the problem, but perhaps a good short-term solution would be to delete the linux wheel from PyPI temporarily so users download and build the source packages as before?

[stolpeo]

I encountered the same error message when importing (import pysam) after installing version 0.15.3 with pip (pip install pysam). For now my solution is to switch back to version 0.15.2.

[AndreasHeger]

Thanks, I will delete the wheels until I understand what is going on.

[mvdbeek]

Any chance that HTSLIB_MODE="separate" would need to be added to CIBW_ENVIRONMENT ?

[mvdbeek]

I tried building and testing the wheel locally with cibuildwheel but it failed with

   build/temp.linux-x86_64-3.6/htslib/regidx.o: In function `regidx_parse_bed':
  /tmp/pip-req-build-izxocbib/htslib/regidx.c:288: multiple definition of `regidx_parse_bed'
  build/temp.linux-x86_64-3.6/bcftools/regidx.c.pysam.o:/tmp/pip-req-build-izxocbib/bcftools/regidx.c.pysam.c:433: first defined here
  build/temp.linux-x86_64-3.6/htslib/regidx.o: In function `regidx_parse_tab':
  /tmp/pip-req-build-izxocbib/htslib/regidx.c:313: multiple definition of `regidx_parse_tab'
  build/temp.linux-x86_64-3.6/bcftools/regidx.c.pysam.o:/tmp/pip-req-build-izxocbib/bcftools/regidx.c.pysam.c:465: first defined here
  build/temp.linux-x86_64-3.6/htslib/regidx.o: In function `regidx_seq_nregs':
  /tmp/pip-req-build-izxocbib/htslib/regidx.c:66: multiple definition of `regidx_seq_nregs'
  build/temp.linux-x86_64-3.6/bcftools/regidx.c.pysam.o:/tmp/pip-req-build-izxocbib/bcftools/regidx.c.pysam.c:89: first defined here
  build/temp.linux-x86_64-3.6/htslib/regidx.o: In function `regidx_nregs':
  /tmp/pip-req-build-izxocbib/htslib/regidx.c:75: multiple definition of `regidx_nregs'
  build/temp.linux-x86_64-3.6/bcftools/regidx.c.pysam.o:/tmp/pip-req-build-izxocbib/bcftools/regidx.c.pysam.c:98: first defined here
  build/temp.linux-x86_64-3.6/htslib/regidx.o: In function `regidx_seq_names':
  /tmp/pip-req-build-izxocbib/htslib/regidx.c:81: multiple definition of `regidx_seq_names'
  build/temp.linux-x86_64-3.6/bcftools/regidx.c.pysam.o:/tmp/pip-req-build-izxocbib/bcftools/regidx.c.pysam.c:104: first defined here
  build/temp.linux-x86_64-3.6/htslib/regidx.o: In function `regidx_insert':
  /tmp/pip-req-build-izxocbib/htslib/regidx.c:120: multiple definition of `regidx_insert'
  build/temp.linux-x86_64-3.6/bcftools/regidx.c.pysam.o:/tmp/pip-req-build-izxocbib/bcftools/regidx.c.pysam.c:183: first defined here
  build/temp.linux-x86_64-3.6/htslib/regidx.o: In function `regidx_destroy':
  /tmp/pip-req-build-izxocbib/htslib/regidx.c:224: multiple definition of `regidx_destroy'
  build/temp.linux-x86_64-3.6/bcftools/regidx.c.pysam.o:/tmp/pip-req-build-izxocbib/bcftools/regidx.c.pysam.c:278: first defined here
  build/temp.linux-x86_64-3.6/htslib/regidx.o: In function `regidx_init':
  /tmp/pip-req-build-izxocbib/htslib/regidx.c:170: multiple definition of `regidx_init'
  build/temp.linux-x86_64-3.6/bcftools/regidx.c.pysam.o:/tmp/pip-req-build-izxocbib/bcftools/regidx.c.pysam.c:224: first defined here
  build/temp.linux-x86_64-3.6/htslib/regidx.o: In function `regidx_overlap':
  /tmp/pip-req-build-izxocbib/htslib/regidx.c:247: multiple definition of `regidx_overlap'
  build/temp.linux-x86_64-3.6/bcftools/regidx.c.pysam.o:/tmp/pip-req-build-izxocbib/bcftools/regidx.c.pysam.c:368: first defined here
  collect2: error: ld returned 1 exit status
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for pysam
  Running setup.py clean for pysam
Failed to build pysam
ERROR: Failed to build one or more wheels

[AndreasHeger]

Thanks. @mvdbeek , my thinking is that the wheels should be built with the htslib code that is included with pysam - otherwise users would see failures if no htslib was installed or if it was installed, it is probably a bit of luck whether it would link correctly.

[AndreasHeger]

reading setup.py, the default is 'shared', meaning that setup.py will build a shared htslib library from the builtin htslib. With 'separate', each pysam .pyx module contains all the htslib code. This is only a fall-back when for one reason or another the pysam installation is not able to find the shared htslib.

[nsoranzo]

I have finally found the cause of this issue: cibuildwheel calls auditwheel to fix the wheels on Linux without -L . , which seems to be needed by pysam.
I am going to open a PR for cibuildwheel to allow a configurable delocate command and then update pysam's .travis.yml once that is merged.