Request for example: Recurrent neural network for predicting next value in a sequence

Question

Can anyone give me a practicale example of a recurrent neural network in (pybrain) python in order to predict the next value of a sequence ? (I\'ve read the pybrain document

Answer 1

Issam Laradji's worked for me to predict sequence of sequences, except my version of pybrain required a tuple for the UnserpervisedDataSet object:

from pybrain.tools.shortcuts import buildNetwork
from pybrain.supervised.trainers import BackpropTrainer
from pybrain.datasets import SupervisedDataSet,UnsupervisedDataSet
from pybrain.structure import LinearLayer
ds = SupervisedDataSet(21, 21)
ds.addSample(map(int,'1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6'.split()),map(int,'1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6'.split()))
ds.addSample(map(int,'1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6'.split()),map(int,'1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7'.split()))
net = buildNetwork(21, 20, 21, outclass=LinearLayer,bias=True, recurrent=True)
trainer = BackpropTrainer(net, ds)
trainer.trainEpochs(100)
ts = UnsupervisedDataSet(21,)
ts.addSample(map(int,'1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7'.split()))
[ int(round(i)) for i in net.activateOnDataset(ts)[0]]

gives:

=> [1, 2, 5, 6, 2, 4, 5, 6, 1, 2, 5, 6, 7, 1, 4, 6, 1, 2, 2, 3, 6]

To predict smaller sequences, just train it up as such, either as sub sequences or as overlapping sequences (overlapping shown here):

from pybrain.tools.shortcuts import buildNetwork
from pybrain.supervised.trainers import BackpropTrainer
from pybrain.datasets import SupervisedDataSet,UnsupervisedDataSet
from pybrain.structure import LinearLayer
ds = SupervisedDataSet(10, 11)
z = map(int,'1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6 1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6 1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7'.split())
obsLen = 10
predLen = 11
for i in xrange(len(z)):
  if i+(obsLen-1)+predLen < len(z):
    ds.addSample([z[d] for d in range(i,i+obsLen)],[z[d] for d in range(i+1,i+1+predLen)])

net = buildNetwork(10, 20, 11, outclass=LinearLayer,bias=True, recurrent=True)
trainer = BackpropTrainer(net, ds)
trainer.trainEpochs(100)
ts = UnsupervisedDataSet(10,)
ts.addSample(map(int,'1 3 5 7 2 4 6 7 1 3'.split()))
[ int(round(i)) for i in net.activateOnDataset(ts)[0]]

gives:

=> [3, 5, 6, 2, 4, 5, 6, 1, 2, 5, 6]

Not too good...

Answer 2

These steps are meant to perform what you ask for in the first part of the question.

1) Create a supervised dataset that expects a sample and a target in its arguments,

 ds = SupervisedDataSet(21, 21)
 #add samples (this can be done automatically)
 ds.addSample(map(int,'1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6'.split()),map(int,'1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6'.split()))
 ds.addSample(map(int,'1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6'.split()),map(int,'1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7'.split()))

A succeeding sample is the target or label y of its predecessor x. We put the number 21 because each sample has 21 numbers or features.

Please note that for standard notations in the second half of your question it is better to call feature1 and feature2 as sample1 and sample2 for a sequence, and let features denote the numbers in a sample.

2) Create Network, initialize trainer and run for 100 epochs

net = buildNetwork(21, 20, 21, outclass=LinearLayer,bias=True, recurrent=True)
trainer = BackpropTrainer(net, ds)
trainer.trainEpochs(100)

Make sure to set the recurrent argument as True

3) Create the test data

ts = UnsupervisedDataSet(21, 21)
#add the sample to be predicted
ts.addSample(map(int,'1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7'.split()))

We created an unsupervised dataset because of the assumption that we don't have the labels or targets.

4) Predict the test sample using the trained network

net.activateOnDataset(ts)

This should display the values of the expected fourth run.

For the second case when a sequence can have more than sample, instead of creating a supervised dataset, create a sequential one ds = SequentialDataSet(21,21). Then, everytime you get a new sequence, call ds.newSequence() and add the samples -that you call features- in that sequence using ds.addSample().

Hope this is clear-cut :)

If you wish to have the full code to save the trouble of importing the libraries, please let me know.