Skip to content

SGD and LSTM #105

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
alecmocatta wants to merge 129 commits into
base: master
Choose a base branch
from
Open

SGD and LSTM #105

alecmocatta wants to merge 129 commits into from

Conversation

@alecmocatta
Copy link
Member

@alecmocatta alecmocatta commented Aug 8, 2020 •
edited
Loading

I've vendored what looks to me like a nice, well written, pure-Rust (but can optionally use OpenBLAS, Intel MKL, Netlib or Apple's Accelerate framework under the hood) library that contains basic but relatively high-quality implementations of Stochastic Gradient Descent and LSTM RNN, with Hogwild-style parallelisation.

  • Passing test reliably
  • Integrate into amadeus
    • Parallel
    • Distributed

maciejkula and others added 30 commits December 26, 2017 16:45
@maciejkula
The world was created in one day.
577e439
@maciejkula
Update manifest.
4b8d2ed
@maciejkula
Update reamde.
af83d51
@maciejkula
Spelling.
caf6fd9
@maciejkula
More spelling.
ea6ca06
@maciejkula
Hide ParameterNode fields.
ff17b0e
@maciejkula
Spelling.
a1b72b8
@maciejkula
Add scalar sum and ln nodes.
9b91786
@maciejkula
Add node reuse benchmark.
5dbbe69
@maciejkula
Easier tests, numerically.
85f3a29
@maciejkula
Merge pull request #1 from maciejkula/ci
984306c 
Add .travis.yml.
@maciejkula
Start not repeating work.
1e39796
@maciejkula
Finish adding pass marking.
149f3a0
@maciejkula
Add node reuse benchmark.
9919338
@maciejkula
Rebase on master and fix new nodes.
5e20898
@maciejkula
Add instructions on using BLAS.
00b66fb
@maciejkula
Merge pull request #2 from maciejkula/lazy
16688ca 
Implement node caching and gradient accumulation
@maciejkula
Ignore code section in docs.
99c3a11
@maciejkula
Merge pull request #3 from maciejkula/blas_optim
bbc3a54 
Add instructions on using BLAS.
@maciejkula
Bump version to 0.3.0.
b3d3636
@maciejkula
Fix travis badge in Cargo.toml.
78cb887
@maciejkula
Push 0.3.1.
5a75d4c
@maciejkula
Extract parameters of a graph from its root.
e64983b
@maciejkula
Add stack/concat node.
4cf2410
@maciejkula
Add tanh node.
e710cb5
@maciejkula
Start adding LSTM cells.
d533bc3
@maciejkula
Initial LSTM cell tests.
c7599cc
@maciejkula
Add pi prediction test.
414330c
@maciejkula
Try to fuse the LSTM nodes.
d496c44
@maciejkula
Box nodes to manage type complexity.
13ece9f
maciejkula and others added 30 commits May 27, 2018 11:41
@maciejkula
Add slice nodes.
6e2dcff
@maciejkula
Fix compile on 1.25.0.
d11a5c6
@maciejkula
Merge pull request #17 from maciejkula/slice_ops
3ffe603 
Add slice nodes.
@maciejkula
Bump version to 0.8.1.
465eeb3
@maciejkula
Make sure sparse indices are order-independent.
fcfdc78
@maciejkula
Major overhaul of optimizers to support true sync parallelism.
ab17d2d
@maciejkula
Ergonomics fixes for synchronized optimizers.
6e13189
@maciejkula
Hashmap-based sparse accumulators.
239c378
@maciejkula
Add hibitset accumulator.
6c2e0a9
@maciejkula
Use hierarchical bitsets for sparse gradient accumulation.
220bb17
@maciejkula
Slight cleanup.
cb77c9a
@maciejkula
First step to removing gradient zeroing.
c9cd44d
@maciejkula
Eliminate manual zero_gradient calls.
6845d08
@maciejkula
Working deterministic update order.
7ee4bfa
@maciejkula
Clean up optimizer implementation.
43b7817
@maciejkula
Add default impls for optimizers. Add deny_missing_docs.
629f94d
@maciejkula
Add coupled update-forget gates to LSTM.
751ab7e
@maciejkula
Better gradient accumulation in dot nodes.
7bb1b47
@maciejkula
Merge pull request #18 from maciejkula/optimizer-overhaul
dcc9e74 
Optimizer overhaul
@maciejkula
Bump to v0.9.0.
b2ac5a0
@maciejkula
Fix warnings and bump to 0.9.1.
6272ad8
@maciejkula
Add missing debug impls.
fed37e9
@maciejkula
Merge pull request #22 from maciejkula/missing_debug_impls
954b86a 
Add missing debug impls.
@alecmocatta
move into amadeus-ml
ec8cf1f
@alecmocatta
Merge 'wyrm/master'
23b9bcd
@alecmocatta
update deps & clippy clean
6334be0
@alecmocatta
fix flaky tests
28e3c2e
@alecmocatta
Merge 'origin/master' into ml
1e11c78
@alecmocatta
add ml to CI
2e626fd
@alecmocatta
fix a stack overflow in CI on windows
45e319c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@alecmocatta @maciejkula