Adding Hasktorch Proposal

[austinvhuang]

Added proposal for Hasktorch - "Finalize the Hasktorch Library for a 1.0 Release".

[jaspervdj]

Two comments:

1. The motivation for needing dependent types is missing from the proposal, could you add a few sentences about that?
2. The goal of GSOC is to write code, not to do research. That's why we encourage contributing to existing libraries. This seems like a separate thing from the tensorflow bindings to Haskell, which is fine, but also needs some motivation.

[austinvhuang]

Thanks for the comments @jaspervdj.

Regarding 1) - I'm not saying anything particularly novel when I say that numerical/scientific computing is one of the domains where dependent types can make a big difference in the utility of types. In the scientific computing community, one will often hear critiques of type systems along the lines of "what's the use of the type system when all the inputs and outputs are arrays?"

Even in the pre-release phase of the project, we've seen advantages to being able to write in a more declarative style - specifying the intent in the form of an equation and having the compiler impute tensor dimensions that are needed to fulfill the computation. A small example that illustrates this - tds_new and tds_resize dimension choices from context automatically here https://github.com/hasktorch/hasktorch/blob/master/examples/gradient-descent/GradientDescent.hs. In the longer term, I think there will be benefits to numerical computing far beyond dimension imputation and correctness, these are just the low hanging fruit.

Regarding 2) - the Tensorflow bindings for Haskell are great, but just as Tensorflow for python and Pytorch coexist and occupy different design spaces, the same applies here.

Although the project is motivated in part to enable new avenues for research, the scope of this project is concretely to write code for the hasktorch library - https://github.com/hasktorch/hasktorch. Note we haven't made a big push to publicize the library in the current pre-release phase, the 1.0 release and broader dissemination is what we intend to work towards to with this proposal.

[gwils]

Given this project uses dependent types as well as backpack, I think "Advanced" would be a more appropriate description of its difficulty than "Intermediate".

[austinvhuang]

Hi @gwils, thanks for the note. Currently, our use of dependent types is for basic utilitarian purposes like typed dimensions. Most intro tutorials cover this topic as the first example, which is enough to contribute.

Being an end-user of backpack isn't conceptually advanced, it just hasn't become mainstream yet. Just the other weekend Sam ran through a great intro tutorial for the other contributors (including myself), we would do the same for someone coming on board.

[cartazio]

Question:
1 which version of torch is this based on?
2 hows the linear algebra facilities compare with blis https://github.com/flame/blis?

[austinvhuang]

Hi @cartazio, version-wise - we're tracking the Pytorch repo master branch (last refresh was probably a month or two ago). We use the repo's ATen shared library with a focus on the functional-esque C core currently.

Tracking upstream changes as they arrive has been a pretty lightweight process (usually ~ an hour or less) thanks to the core C libraries having a consistent, mostly-functional API and code generating the low-level raw bindings.

It's nice to be able to do this because they're still adding functionality to the core, most recently vectorized probability functions like gamma/erf (they pulled in some contributions from the Pyro devs). We've been able to tag along and get those functions for almost free as they're being implemented by the Pytorch team.

I haven't tried Blis so I can't make an informed comment, but the torch family of libraries is well regarded as having a very ergonomic API, and has been dog-fooded for a long time to build one of the most popular neural network libraries. Although not exactly identical, the C core is pretty close to what's surfaced by the lua torch API so you can get a sense of the usage from those docs:

https://github.com/torch/torch7/blob/master/doc/maths.md
https://github.com/torch/torch7/blob/master/doc/tensor.md
https://github.com/torch/torch7/blob/master/doc/random.md

It's fairly "batteries-included" as far as these types of libraries go, more so with time.

[stites]

@cartazio - Given a 30s overview, it looks like blis is also a blas-alternative, while Torch is generic across blas- and lapack- libraries (chosen at compile-time), but also includes operations on tensors. The torch interface is seamless across GPU and CPU backends using CuBlas and Magma for the GPU-based blas and lapack libraries, respectively, and adding a CPU version of the CuDNN library in THNN.

@jaspervdj
GSOC on engineering vs research. This is very much an engineering task and very little research is required except to figure out how to make updates from the upstream repo more maintainable across a very small dev team (which is mostly settled at this point with backpack). I think more research is going into the autodiff side, in a separate repo called diffhask, but I'm less familiar with this.

The motivation for torch vs tensorflow in haskell. From my perspective working with the python variants, tensorflow is even more "batteries included" while torch relies on the host language for its API. Haskell is excellent for parallel and concurrent execution in both CPU and GPU operations (props to accelerate here) and I see "production level torch" being able to leverage this.

[jaspervdj]

Okay, I think I'm happy to merge this, but:

1. Let's move this to "Advanced" given the API design and other things involved
2. Can you use <> in the references for links, e.g. <https://github.com/> rather than https://github.com/
3. Can you send me an email with the contact details of the mentors?

[gwils]

Can you send [Jasper] an email with the contact details of the mentors?

Please send the contact details to the Haskell.org committee so that we all have them.

[jaspervdj]

Thanks!

[stites]

Some small comments on that last commit, given the feedback here:

• I swapped out the first item, which comes across as exploratory and research-based, to be more engineering-focused. The FFI bindings will be missing any python-abstractions pytorch has -- ideally we will be hitting parity with pytorch.
• 1.0 isn't really PVP compliant (and internally we haven't spoken about how we want to version). I went with "hackage release."
• Small subtlety: I swapped the order of diffhask and backprop since diffhask is incomplete for the moment. It's more ideal, since it doesn't just hold a gradient tape, but backprop is more mature so we should point students in that direction first.