In the first two blog posts, we introduced Introduction to deep learning (1) --- image recognition with convolutional neural network (1) and Introduction to deep learning (1) --- image recognition with convolutional neural network (2) In this paper, we will introduce the application of long-term and short-term memory network.
Convolution neural network is a kind of feedforward neural network, which is characterized by the signal always propagating forward in it, and the input data of each neural network has no relation with the previous input data, that is, the calculation results of neural network have no relation with the sequence of different batches of input data. But in the actual application process, many problems are related to the time sequence. At this time, it needs to use the Recurrent Neural Network to deal with them.
Baidu Encyclopedia introduces the cyclic neural network as follows:

In this paper, the long-term and short-term memory network in the recurrent neural network is used as an example to predict the currency exchange rate. The structure of the long-term and short-term memory network is as follows:

Application background:

The application of this paper is the prediction of currency exchange rate. It forecasts the value of currency exchange rate in the future through the situation of currency exchange rate in the previous days.

Data Description:

The data in this example are fictitious data, which are saved in plain text file. The data download is: link: exchangeData.txt
Extraction code: iyxm
Each row in the data has a daily exchange rate value, which is arranged in the order of later dates.

Algorithm description:

In this example, the exchange rate value of the previous three days is used to predict the exchange rate value of the fourth day, so three input values are needed, which are X1,X2,X3. The neural network model in this example is as follows:

The code is as follows:

import tensorflow as tf
import numpy as np
import pandas as pd
import sys

roundT = 100
learnRateT = 0.001

argt = sys.argv[1:]
print("argt: %s" % argt)

for v in argt:
    if v.startswith("-round="):
        roundT = int(v[len("-round="):])
    if v.startswith("-learnrate="):
        learnRateT = float(v[len("-learnrate="):])

fileData = pd.read_csv('exchangeData.txt', dtype=np.float32, header=None)
wholeData = np.reshape(fileData.as_matrix(), (-1))

print("wholeData: %s" % wholeData)

cellCount = 3
unitCount = 5

testData = wholeData[-cellCount:]
print("testData: %s\n" % testData)

rowCount = wholeData.shape[0] - cellCount
print("rowCount: %d\n" % rowCount)

xData = [wholeData[i:i + cellCount] for i in range(rowCount)]
yTrainData = [wholeData[i + cellCount] for i in range(rowCount)]

print("xData: %s\n" % xData)
print("yTrainData: %s\n" % yTrainData)

x = tf.placeholder(shape=[cellCount], dtype=tf.float32)
yTrain = tf.placeholder(dtype=tf.float32)

cellT = tf.nn.rnn_cell.BasicLSTMCell(unitCount)

initState = cellT.zero_state(1, dtype=tf.float32)

h, finalState = tf.nn.dynamic_rnn(cellT, tf.reshape(x, [1, cellCount, 1]), initial_state=initState, dtype=tf.float32)
hr = tf.reshape(h, [cellCount, unitCount])

w2 = tf.Variable(tf.random_normal([unitCount, 1]), dtype=tf.float32)
b2 = tf.Variable(0.0, dtype=tf.float32)

y = tf.reduce_sum(tf.matmul(hr, w2) + b2)

loss = tf.abs(y - yTrain)

optimizer = tf.train.RMSPropOptimizer(learnRateT)
train = optimizer.minimize(loss)

sess = tf.Session()
sess.run(tf.global_variables_initializer())

for i in range(roundT):
    lossSum = 0.0
    for j in range(rowCount):
        result = sess.run([train, x, yTrain, y, h, finalState, loss], feed_dict={x: xData[j], yTrain: yTrainData[j]})
        lossSum = lossSum + float(result[len(result) - 1])
        if j == (rowCount - 1):
            print("i: %d, x: %s, yTrain: %s, y: %s, h: %s, finalState: %s, loss: %s, avgLoss: %10.10f\n" % (i, result[1], result[2], result[3], result[4], result[5], result[6], (lossSum / rowCount)))

result = sess.run([x, y], feed_dict={x: testData})
print("x: %s, y: %s\n" % (result[0], result[1]))

Operation mode:

Enter cmd and execute the program with the following command:

python lstm1.py -round=1000

After 1000 times of training, the operation results are as follows:

Training results:

The output of the last line: x: [6.3188 6.3198 6.3198], y: 6.338545, which indicates that the exchange rate value of the fourth day is predicted to be y: 6.338545 through the exchange rate value of the first three days of x: [6.3188 6.3198 6.3198].