Floating point exception

[sanjaymeena]

Docker image : docker.paddlepaddle.org/book:0.10.0rc3

I am training my own Semantic role labeler model using modified 06.semantic role code. My training data size is 24000 sentences. If i train using sample data size ~1000 sentences, i can successfully train a model without errors and save all parameters properly. However for 25k sentences i get this following error.

I have been stuck here since many days now : ( . I verified that this may not be due to wrong data as all sentences have been iterated over in previous passes.

[lcy-seso]

I can reproduce this error. I am checking it now and will provide the solution as soon as possible.

[lcy-seso]

From the last two lines of the log file, the cost is increased from about 590 to 2974, I think you encounter the gradient explosion problem. One possible solution is to use gradient clipping, but I am really sorry, currently, PaddlePaddle has a problem with it (it may not be activated even if you specify the clipping threshold), we are working on it.

Gradient explosion can be avoided by carefully tuning the training parameters, maybe it is affected by the weight initialization, the learning algorithm, the learning rate, the length of the training sample, batch size and so on. All these parameters can be tried.

[sanjaymeena]

Hi Icy-seso ,

• i had tried changing batch size, amount of training data . Still had gradient explosion. With smaller batch size it takes more passes but still happens .
• other parameters are default ones which came in demo train.py.
• I will try changing other params.

Gradient clipping seems like an important problem to resolve. Thanks !

[lcy-seso]

Thank you and very sorry for the inconvenience, we will fix this as soon as possible. Maybe you can try to reduce the learning rate, or change to another optimization algorithm, that is what I always do.

[sanjaymeena]

Thanks Icy-seso, i reduced the learning rate to 1e-2 . what other optimization algorithm would you suggest?

[lcy-seso]

• Roughly speaking, I like to use Adam and Adadelta when training a complicated RNN model.
• In PaddlePaddle, the gradient is not divided by batch size, perhaps, 1e-2 is a large learning rate.
• If you use momentum, and the learning rate is 1e-2, it is recommended to explicitly divide the batch size like this:

learning_rate = 1e-2 / batch_size

[lcy-seso]

I will check whether gradient clipping is problematic in PaddlePaddle and give you a reply. Thanks very much for your issue.

[sanjaymeena]

Thanks, I am trying now with momentum and learning rate as you mentioned.

[sanjaymeena]

Hi lcy-seso , Is there a way to get precision, recall, f1 from paddle.v2 .trainer.test or is it possible to get using some another function?

PS : I changed the learning parameter and training so far has seems to be going on. Thanks.

[lcy-seso]

You can use the chunk_evalutor to get the precision and recall. But I am sorry to find that evaluators are not included in our doc and I create the issue #2009, we are fixing it.

Maybe you can try to read the explanation of the source codes. I will also try it and give an example later.

Thanks for your question.

[sanjaymeena]

Hi Icy-Seso, one another question. I am trying to run inference on about 1000 sentences in using paddle.infer.

for element in test_datax:
        #print element
        probs = paddle.infer(
                output_layer=predict,
                parameters=parameters,
                input=element,
                field='id')
        print probs
        #assert len(probs) == len(element[0][0])
        labels_reverse = {}
        for (k, v) in label_dict.items():
            labels_reverse[v] = k
        pre_lab = [labels_reverse[i] for i in probs]

I can get the results . However inference seems very slow.
I have following questions :

• Is there some tuning required from my side?
• How to do inference on batch data?

For reference, the word embedding i use has dimension 150 compared to 32 as mentioned in the demo semantic role labeling program.

[sanjaymeena]

I have already used the shared link and done inference before. However, it works for only one input element . For more than one i am looping through all the elements and the performance is very slow . I am trying (~3000) sentences . Regards, Sanjay

…

On May 9, 2017, at 11:43 AM, Cao Ying ***@***.***> wrote: As you mentioned the SRL demo, are you running a sequence labeling task and want to get the labeling results? Maybe the labeling results is a little different in reference. Will this be helpful? https://github.com/lcy-seso/book/blob/develop/07.label_semantic_roles/train.py#L205 <https://github.com/lcy-seso/book/blob/develop/07.label_semantic_roles/train.py#L205> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1961 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE6EVYc-0i_ilHrJIiXz32rUc4QOJmXOks5r3-DkgaJpZM4NN7PA>.

[lcy-seso]

Sorry to delete the last reply because I don't find any problem in your codes. I am asking others for help. @reyoung