Linear regression

[mbforbes]

Implement linear regression + 2 solving methods + regularization. Includes hyperparameter tuning and making graphs for speed and accuracy of various combinations.

implementation

• Matrix inversion
• GD

math break

• latex -> svg on computer, embed in markdown and ensure works (i.e. no rawgit needed)
• section: naive regression to scalar values
  • brief intro of variables
  • least squares
    • loss
    • naive least squares loss derivative w/ some algebra, show gradient
    • gradient = 0 -> closed form
  • ridge
    • loss
    • derivative
    • gradient = 0 -> closed form
  • lasso
    • loss
    • derivative

more implementation

• ridge
• lasso

coordinate descent

• math for ols
• implementation for ols
• math for lasso
• implementation for lasso

OK then make actually useful regression to 10 classes

• multi-class OLS analytic
• multi-class OLS GD
• clean up eval
• refactor coordinate descent
• multi-class OLS CD
• multi-class ridge analytic
• multi-class ridge GD

oh wait i figured out ridge CD:

• scalar ridge CD
• math for scalar ridge CD

ok back to multiclass:

• multi-class ridge CD
• multi-class lasso GD
• multi-class lasso CD

then some tooling and graphs!

• SGD
• Hyperparam tuning framework and graphs
• Graphs of accuracy and speed of above methods
• math of multiclass (can be super brief)

[mbforbes]

Update on types:

After getting excited that mypy was helping me resolve some type errors, I got a little greedy.

I spent an afternoon trying to get mypy to know about torch and its types. And it actually worked.

But then I got even greedier. I noticed that mypy would understand that things like ints aren't torch.Tensors, but it would be fine with torch.IntTensors being passed in where torch.FloatTensors were required. See:

import torch

i = torch.IntTensor()
f = torch.FloatTensor()
reveal_type(i)
reveal_type(f)

def foo(i: torch.IntTensor, f: torch.FloatTensor) -> None:
    pass

# these behave as expected
foo(i, f)
foo(1, 2)

# these typecheck OK, but I think should error
foo(i, i)
foo(f, f)

I asked about this on python/mypy#3987. After some interaction with Guido, I've gotten the idea that: (a) mypy fails because it doesn't know enough about the base classes (of torch.*Tensor), and (b) there's no easy way to introduce "gradual" annotations to a 3rd party repository; the introduction of pyi files removes all the auto-generated information gotten from the py files themselves.

This evening I tried PyCharm to see if it might be able to figure things out. I thought that maybe they were using some kind of additional proprietary magic instead of mypy. But it didn't work; not only did PyCharm make exactly the same errors (it knew ints weren't torch.*Tensors, but couldn't tell the difference between them... I'm guessing it's using mypy under the hood), but it also couldn't handle some of the newer type annotation syntax (e.g., y: Foo = bar('please make a Foo')). So, I'm happy to stick with vscode.

For types, I can try Guido's suggestion of running stubgen and see where that takes me. If it seems feasible to make a few corrections, I can try that, though I'm guessing mucking around with their insane complex types will be getting in over my head.

The more sane thing (and one perhaps I should do anyway) is to do unions of the type of torch.*Tensor and torch.cuda.*Tensor just for annotation purposes (also ignoring that, e.g., that cuda-ness of passed tensors must match—this could be enforced by doing a union of tuples for args, but is totally moot if it doesn't know about the different tensor types at all), and then move on.

[mbforbes]

This morning I was set to just go back to how I was running before: ignore the fact that mypy can't figure out IntTensors and FloatTensors aren't compatible, and accept some of the simple type checking it did give me. Hey, at the very least, it'll catch misspellings!

However, I ended up with some weird and terrible mypy bugs. Running with --quick-and-dirty as I had been resulted in an assertion error (AssertionError: De-serialization failure: TypeInfo not fixed), but only on the second+ run, and only on my bigger files (like regression.py instead of my minimal mypy example ones). Backing off to --incremental gives assertion errors that "quick and dirty" mode should be on (AssertionError: Should never get here in normal mode). I cleared the .mypy_cache/ directory a dozen times in between various fresh starts to rebuild the cache. I couldn't avoid the problem.

Yuck. I figured since I'm pointed to site-packages/ in order to get pytorch types, I wouldn't be able to get help for these issues, because they explicitly say they don't support pointing to site-packages/.

So, I tried running stubgen torch.

At first, this seemed promising! It was able to get what mypy usually would, but would write the files so that they could be read super fast on type checking.

But just a single file (out/torch/__init__.pyi) was produced. It didn't know about any of the useful types (like tensors).

Running with stubgen --recursive torch opened up a whole slew of new problems. I started interacting with another developer on python/mypy#3987. It turns out that pytorch makes python syntax errors in their C extension API (things like using reserved keywords for arguments, and mixing default and non-default arguments), and stubgen happily follows them to generate the stubs. But when mypy gets to them, it barfs because they aren't valid python.

I started doing some manual cleanup. I documented my increasingly laborious escapades in several gists (pytorch) (recursive) (manual cleanup) (numpy). It seemed that mypy wouldn't be happy to check my file and acknowledge the types (like torch.cuda.FloatTensor) until it could import, e.g., the cuda stubs without errors. That required it to import the stubs that the cuda stub depended on without errors. And down the rabbit hole we go.

There's no way I'm going to spend weeks fixing up my own personal copy of all of these stubs. I've already wasted enough time on this fool's errand.

Sadly, I'm not sure what option I have now except for turning all type checking off. I've checked out mypy at master, but I still get the assertion errors I started today with when I point mypy back at site-packages/ and run with --quick-and-dirty or --incremental. Running with no speedups takes ~30 seconds between runs, which is too slow to be useful while editing code. I could hook up to travis to run mypy, but if all I'm hoping for is to catch misspellings, there's no point, because I'll catch those before I push anyway.

As a last ditch effort, I may try to report the assertion errors I'm getting with the faster modes. If they're able to fix the bugs, great. If not, well, no static types for me.