Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion python/paddle/fluid/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,16 @@
import executor
from executor import *

import trainer
from trainer import Trainer
from trainer import Event

import inferencer
from inferencer import Inferencer

import params
from params import Params

import io
import evaluator
import initializer
Expand Down Expand Up @@ -47,7 +57,8 @@

Tensor = LoDTensor

__all__ = framework.__all__ + executor.__all__ + concurrency.__all__ + [
__all__ = framework.__all__ + executor.__all__ + concurrency.__all__ +\
trainer.__all__ + inferencer.__all__ + params.__all__ + [
'io',
'initializer',
'layers',
Expand Down
31 changes: 31 additions & 0 deletions python/paddle/fluid/inferencer.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

__all__ = ['Inferencer', ]


class Inferencer(object):
def __init__(self, network_func, params, place=None):
# 1. we need to generate a framework.Program by calling
# network_func. Reference: fluid.program_guard in test_word2vec.py

# 2. move the default_main_program to self.program.

# 3. run the default_startup program.
self.params = params
self.place = place

def infer(self, inputs):
# run self.program
pass
39 changes: 39 additions & 0 deletions python/paddle/fluid/params.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from . import core

__all__ = ['Params', ]


class Params(object):
def __init__(self, path=None):
self.scope = core.Scope()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the relationship between a Params instance and this scope instance?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Params is a wrapper on top of scope, and provides additional functionality to save and load parameters.

We think it's not necessary, I will send a PR to remove it.

The new interface of the Trainer constructor will be:

def Trainer(object):
  def __init__(self, network_func, optimizer, param_path=None, scope=None, place=None):

The parameters of the trainer can be optional initialized by the saved parameters in param_path, or taken from scope. If nothing is provided, the parameters will be randomly initialized.

The scope argument is necessary because we want to share the scope between inferencer and trainer. One use case is GAN.

Copy link
Contributor Author

@helinwang helinwang May 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed params.py in #10354


if path:
self._load(path)

def _load(self, path):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why load is defined as a private method but save it public?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, this is not optimal. We will remove this class: #10313 (comment)

Copy link
Contributor Author

@helinwang helinwang May 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed params.py in #10354

# reference: load_persistables in io.py
pass

def save(self, path):
# reference: save_persistables in io.py
pass

def add_params(self, scope):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite understand here -- the method name is add params, but the method comment says that it takes keys from the scope. Does it really add something?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We plan to remove this class, and add a merge method to core.Scope. It takes another scope, and set all the vars in the other scope to itself. If vars with the same name is present, its own vars will be overridden.

Copy link
Contributor

@jetfuel jetfuel May 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the Scope class, can we expose the vars and make it public? That might be necessary to implement the merge function.

We could use LocalVarNames to do the iteration as well. So if we do want to hide the vars_, we can still do that.

Copy link
Contributor Author

@helinwang helinwang May 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jetfuel maybe we can implement merge in the C++? how do you think?

Copy link
Contributor Author

@helinwang helinwang May 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed params.py in #10354

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@helinwang Yes, it should be in the C++. My approach is to use the LocalVarNames and check one by one.

void Scope::merge(Scope* scope) const {
  std::vector<std::string> names = scope.LocalVarNames;
  for (auto name_it = names.begin(); name_it != names.end(); name_it ++) {
    auto it = vars_.find(*name_it);
    if (it == vars_.end()) {
      vars_[*name_it] = scope.Var(*name_it)
    }
  }
}

Copy link
Contributor Author

@helinwang helinwang May 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jetfuel Thanks! Every var in the scope argument will be put into the receiving scope, if the receiving scope already has the same name, it will be replaced. So I guess we can just do:

void Scope::merge(Scope* scope) const {
  std::vector<std::string> names = scope.LocalVarNames;
  for (auto name_it = names.begin(); name_it != names.end(); name_it ++) {
    vars_[*name_it] = scope.Var(*name_it)
  }
}

Sorry the spec was different from what we discussed yesterday (yesterday's was to not replace if already exists).

# take the keys from the scope,
# if not already exists in self.scope,
# add the key and value into self.scope.
pass
55 changes: 55 additions & 0 deletions python/paddle/fluid/trainer.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

__all__ = [
'Event',
'Trainer',
]


class Event(object):
BEGIN_EPOCH = 0
END_EPOCH = 1
BEGIN_STEP = 2
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the "step" here "iteration"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think step is more commonly used in deep learning. In the same time iteration is more often used in programming languages. Since here is the event for training, I think the user would be more familiar with "step".

END_STEP = 3

def __init__(self):
self.step = 0
self.epoch = 0
self.type = Event.BEGIN_EPOCH


class Trainer(object):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a base class, which implies that we need derived classes like MultiGPUTrainer, MultiNodeTrainer, EDLTrainer, etc, or it is a plain class and both single-node and multi-node training logics should be implemented in it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current plan is "a plain class and both single-node and multi-node training logics should be implemented in it", the logic is controlled by environment variable. Here is an example: #10316

def __init__(self, network_func, optimizer, params=None, place=None):
# 1. we need to generate a framework.Program by calling
# network_func. Reference: fluid.program_guard in
# test_word2vec.py

# 2. move the default_main_program to self.program and run the
# default_startup program on an empty core.Scope()

# 3. call self.params.add_vars with the initialized scope, it
# will add the new vars of the initialized scope into
# self.params.
self.network_func = network_func
self.optimizer = optimizer
self.params = params
self.place = place
# TODO(helin): support distributed training

def train(self, reader, num_epochs, event_handler):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite remember how do we define a pass, and epoch, with our current reader design? If a reader reads gRPC requests, it would never end and would have no pass separations, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A call to the reader will return a iterator that contains a single pass of data. Yes, if a reader reads gRPC requests, it would never end and would have no pass separations.

pass

def test(self, reader):
pass