When using Caffe training model, we often want to save the data in the training and testing process for further analysis. One method is to call the API training of Caffe in python and write code to read the result of data preservation in layer, but this method is tedious after all. In fact, when using command line training, Caffe has provided the function of saving training logs and parsing training and test data. Just add one line of command to the training command. As follows:

#!/usr/bin/env sh
set -e
CAFFEROOT=/home/meringue/Softwares/caffe-master
CIFAR10PY=/home/meringue/Documents/CaffePy/cifar10py
MODEL=$CIFAR10PY/cifar10_caffemodels/model_Alex_ST
TOOLS=$CAFFEROOT/build/tools
echo 'start training...'

GLOG_logtostderr=0 GLOG_log_dir=$MODEL/LOG/ \ # save log file
$TOOLS/caffe train \
  --solver=$MODEL/cifar10_quick_solver.prototxt $@ 

This will generate a log file in the GLOG__log_dir path. Open it and see, it's actually a lot of things that you generate when you execute the training script. But what we need is useful information such as accuracy and error. At this time, we need to use three files: extract_seconds.py, parse_log.sh and plot_training_log.py under the caffe-master/tools/extra/path. Copy these three files into your own LOG folder. First rename the previous LOG log to a file with the suffix of. log, such as model.log, at present. Under the folder, parse_log.sh model.log command can be used to parse the model.log.train and model.log.test files. Open to see something similar to the following:

It can be found that the data we need has been parsed out and can be used to draw pictures at this time. Of course, Caffe also provides the drawing function, using plot_training_log.py, you can set different parameters to draw the graph you need, this force is no longer expanded, because the code on the parameters of the description is very detailed, you can read it yourself.
In fact, now that we have these data, we can write a python function to read and draw these data. Here is my own function, which can be used to read the data in the above two log files. Some details in the code are annotated.

## read log file of training process created by Caffe
class ReadLogFile(object):
    def __init__(self,_LogFilePath):
        """
        get log file path from LogFilePath
        """
        self.LogFilePath = _LogFilePath

    def read_trainLog(self,LogName):
        """
        get iterations and training loss from training log file named LogName
        """
        LogPath = self.LogFilePath+LogName

        # load all training data
        with open(LogPath) as f:
            data = f.readlines()
            data = data[1:] # remove the first row (title) 
            data_len = len(data)

        Iters = []
        TrainLoss = []

        for row in range(data_len):
            data_row = data[row].split(' ') # splitted by ' '
            while '' in data_row: # remove 'None'
                data_row.remove('')
            Iters.append(int(data_row[0]))
            TrainLoss.append(float(data_row[2]))           
        return Iters, TrainLoss    

    def read_testLog(self,LogName):
        """
        get iterations, test accuracy and test loss from test log file named LogName
        """
        LogPath = self.LogFilePath+LogName

        with open(LogPath) as f:
            data = f.readlines()
            data = data[1:]
            data_len = len(data)

        Iters = []
        TestAccuracy = []
        TestLoss = []

        for row in range(data_len):
            data_row = data[row].split(' ')
            while '' in data_row:
                data_row.remove('')
            Iters.append(int(data_row[0]))
            TestAccuracy.append(float(data_row[2]))
            TestLoss.append(float(data_row[3]))     
        return Iters, TestAccuracy, TestLoss