Skip to content

Fix the rendering of latex equation for adamax op #6294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
abhinavarora merged 2 commits into from
Dec 5, 2017
Merged

Fix the rendering of latex equation for adamax op #6294

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
303777b
Using latex fraction syntax in sigmoid and logsigmoid op
Dec 5, 2017
f28f397
Fixing the rendering of the latex equations in adamax operator
Dec 5, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions paddle/operators/activation_op.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,9 @@ class SigmoidOpMaker : public framework::OpProtoAndCheckerMaker {
AddInput("X", "Input of Sigmoid operator");
AddOutput("Y", "Output of Sigmoid operator");
AddComment(R"DOC(
Sigmoid Activation Operator.
Sigmoid Activation Operator

$y = 1 / (1 + e^{-x})$
$$y = \frac{1}{1 + e^{-x}}$$

)DOC");
}
Expand All @@ -60,9 +60,9 @@ class LogSigmoidOpMaker : public framework::OpProtoAndCheckerMaker {
AddInput("X", "Input of LogSigmoid operator");
AddOutput("Y", "Output of LogSigmoid operator");
AddComment(R"DOC(
Logsigmoid Activation Operator.
Logsigmoid Activation Operator

$y = \log(1 / (1 + e^{-x}))$
$$y = \log \frac{1}{1 + e^{-x}}$$

)DOC");
}
Expand Down
10 changes: 6 additions & 4 deletions paddle/operators/adamax_op.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,10 +107,12 @@ Adam algorithm based on the infinity norm.

Adamax updates:

$$momentOut = \beta_1 * moment + (1 - \beta_1) * grad \break
infNormOut = max(\beta_2 * infNorm + \epsilon, |grad|) \break
learningRate = learningRate /(1 - \beta_1_{pow}) \break
paramOut = param - learningRate * momentPut / infNormOut$$
$$
momentOut = \beta_{1} * moment + (1 - \beta_{1}) * grad \\
infNormOut = max(\beta_{2} * infNorm + \epsilon, |grad|) \\
learningRate = \frac{learningRate}{1 - \beta_{1}^{Beta1Pow}} \\
paramOut = param - learningRate * \frac{momentOut}{infNormOut}
$$

The original paper does not have an epsilon attribute.
However, it is added here for numerical stability to prevent the
Expand Down