Reducing the Dimensionality of Data with Neural Networks外文电子书籍.pdf

/cgi/content/full/313/5786/504/DC1 Supporting Online Material for Reducing the Dimensionality of Data with Neural Networks G. E. Hinton* and R. R. Salakhutdinov *To whom correspondence should be addressed. E-mail: hinton@ Published 28 July 2006, Science 313, 504 (2006) DOI: 10.1126/science.1127647 This PDF file includes: Materials and Methods Figs. S1 to S5 Matlab Code Supporting Online Material Details of the pretraining: To speed up the pretraining of each RBM, we subdivided all datasets into mini-batches, each containing 100 data vectors and updated the weights after each mini-batch. For datasets that are not divisible by the size of a minibatch, the remaining data vectors were included in the last minibatch. For all datasets, each hidden layer was pretrained for 50 passes through the entire training set. The weights were updated after each mini-batch using the averages in Eq. 1 of the paper with a learning rate of 
 . In addition, 
 times the previous update was added to each weight and 
  times the value of the weight was sub- tracted to penalize large weights. Weights were initialized with small random values sampled from a normal distribution with zero mean and standard deviation of 
 . The Matlab code we used is available at / hinton/MatlabForSciencePaper.html Details of the fine-tuning: For the fine-tuning, we used the method of conjugate gradients on larger minibatches containing 1000 data vectors. We used Carl Rasmussen’s “minimize” code (1). Three line searches were performed for each mini-batch in each epoch. To determine an adequate number of epochs and to check for overfitting, we fine-tuned each autoencoder on a fraction of the training data and tested its perf