macOS compatibility

[mlin]

macOS-specific setup steps:

• Install Docker Desktop for Mac
• Use Docker's menu bar icon to select:
  • Settings > General > Virtual Machine Options > VirtioFS (info: macOS: task input files sporadically fail to mount (zero length) #461)
  • Settings > Advanced > Allow the default Docker socket to be used
• Install Python using brew install python3 or Miniforge (the OS/Xcode edition may have difficulties with CA certificates)
• Either pip3 install miniwdl or conda install miniwdl
• In the terminal session, export TMPDIR=/tmp to let containers mount shared temporary directories (info)
• miniwdl run_self_test

Then, proceed through the Getting Started tutorial if so desired.

---

To set up for miniwdl development:

conda create miniwdl-dev pip jq coreutils
conda activate miniwdl-dev
pip3 install -r requirements.dev.txt
make qtest

(make sure which stat refers to coreutils' version)

[illusional]

Hey @mlin, I'm trying out WDL and having troubles using the run functionality on macOS when running an untar workflow. (I thought i'd drop it here in case it relates to macOS). I'm getting the error:

franklinmichael$ miniwdl run simple.wdl -i inp-local.json 
WARNING:wdl-workflow:simple:starting workflow simple (simple.wdl Ln 7 Col 1) in /Users/franklinmichael/Desktop/workflows-for-testing/03-simple/wdl/20190722_073803_simple
WARNING:wdl-worfklow:simple:issue call-untar on untar
ERROR:wdl-task:call-untar:failed to close docker-py client
Traceback (most recent call last):
  File "/anaconda3/lib/python3.7/site-packages/WDL/runtime/task.py", line 296, in _run
    assert exit_info
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/anaconda3/lib/python3.7/site-packages/WDL/runtime/task.py", line 306, in _run
    client.close()
  File "/anaconda3/lib/python3.7/site-packages/docker/client.py", line 187, in __getattr__
    raise AttributeError(' '.join(s))
AttributeError: 'DockerClient' object has no attribute 'close' In Docker SDK for Python 2.0, this method is now on the object APIClient. See the low-level API section of the documentation for more details.
ERROR:wdl-task:call-untar:task untar (tools/untar.wdl Ln 3 Col 1) failed: AssertionError
ERROR:wdl-workflow:simple:call-untar failed
ERROR:miniwdl-run:AssertionError, 

I presume this might be related to needing permission to control Docker.

The invoking user must have permission to control Docker.

I've already changed my user-specific temp directory for mixing CWLTool and Docker, and I'm running miniwdl 0.3.0, Docker Desktop community 2.0.5.0, and the Docker python module 2.5.1.

[mlin]

@illusional based on those tracebacks I suspect the installed docker-py package might need updating (pip3 install --upgrade docker or such). Can you see if that makes a difference?

(More broadly, we don't have routine testing on macOS yet, need to plow through the issues above and get the test suite going on Travis)

[illusional]

@mlin Amazing, I didn't realise how out of date my Docker was (due to a Toil dependency). miniwdl run works on my small workflow. Will try on some bigger workflows and let you know how to goes.

I'm not sure what the min version of Docker this would require, but might be worth adding a minimum in setuptools install_requires.

[mlin]

Dang, it seems that Travis' macOS environment doesn't support Docker for Mac, so we're going to have to come up with some other approach for Mac testing.

[illusional]

Hey @mlin, just wanted to report back and say that I ran miniwdl on a germline processing pipeline (3 variant callers, scattering, gathering and small expressions) and my outputs were identical to Cromwell. This is totally awesome!

[mlin]

@illusional thank you for the field report; much appreciated! @mckinsel @bkmartinjr @tjchen