顿satoshis博客使用chainer实现循环神经网络.pdfVIP

Monday, June 15, 2015 Implementing Recurrent Neura using chainer! I just started to study deep learning, which is huge boom both in academia and industry. A week ago, a Japanese called Preferred Infrastructure (PFI) released new deep learning framework chainer! This framework is really great. I was able to implement a recurren with less than 100 lines of python code. Specifically, I tried new recurrent neura work (RNN) called IRNN described in recent Hintons paper A Simple Way to Initialize Recurren works of Rectified Linear Units . It was difficult to train RNN to learn such a long dependency, but IRNN overcame it initializing recurrent weights by identity matrix, and using ReLU as activation function. Awesome! In this post, I will write about my experiment of IRNN to recognize MNIST digits by putting 724 pixels to the recurren in sequential order (a experiment in the paper at section 4.2). The technique and best parameter value in the paper is: Initialize recurrent weights matrix with identity matrix Initialize other weights matrix sampled from Gaussian distribution with mean of 0 and standard deviation (std) of 0.001 Activation function is ReLU Train th work using SGD learning rate: 10^-8, gradient clip value: 1, and mini batch size is 16. But I did not use the same settings because th seems to learn fas t least on the first few epochs. I did: Initialize other weights matrix sampled from Gaussian distribution with mean of 0 and standard deviation (std) of 0.01 No gradient clip The other setting is the same as the paper. Problem is, it takes 50 mins to run each epoch (forward and backpropagate over whole dataset once) on my local environment (CPU). Perhaps, its better to buy GPU or use AWS GPU instance. Anyway, I am currently running it wtih CPU for two days so far! The results