2021SC@SDUSC

Learner function

This blog focuses on the most important code in the entire training or prediction code because it initializes the loading of all other models, data processing visualization, log printing, and so on.

Before introducing these classes, let's briefly summarize the five main categories of Framework:

1. Learner: The main function is called while providing an interface for others to call.
2. DataManager: Processes data and is responsible for cutting and loading data sets.
3. ModelManage: A very important class responsible for loading models, training and predicting models.
4. Visualizer: Visual classes (including latitude drop, learning curve, and so on).
5. IOManager: A class that defines how to print logs.

Each interface is described and explained separately below.

class Learner():
    def __init__(self, config):
        self.config = config
        self.IOManager = IOManager.IOManager(self)
        self.visualizer = Visualizer.Visualizer(self)
        self.dataManager = DataManager.DataManager(self)
        self.modelManager = ModelManager.ModelManager(self)

First define the initialization of each class and the declarations in the class (feel like you can leave it alone, laugh).

    def setIO(self):
        self.IOManager.initialize()
        self.IOManager.log.Info('Set IO Over.')

Initialize IO settings

    def setVisualization(self):
        self.visualizer.initialize()
        self.IOManager.log.Info('Set Visualization Over.')

The Initialization Visualization Code section is mainly about the definition of some record arrays.

    def SL_train_load_data(self):
        self.dataManager.SL_train_load_data()
        self.IOManager.log.Info('Load Data Over.')

Loading of training set data (large amount of data loading).

    def SL_test_load_data(self):
        self.dataManager.SL_test_load_data()
        self.IOManager.log.Info('Load Data Over.')

Loading of test set data (small and probably not dataloader).

    def init_model(self):
        self.modelManager.init_model()
        self.IOManager.log.Info('Init Model Over.')

Initialize the model, because there are many models to choose from, so it is still pulled out separately to make an api.

    def load_params(self):
        self.modelManager.load_params()
        self.IOManager.log.Info('Load Parameters Over.')

Load parameters, used in parameters and predictions of complex codes or as models for real use.

    def adjust_model(self):
        self.modelManager.adjust_model()
        self.IOManager.log.Info('Adjust Model Over.')

Do some statistical work on the model, such as parameter quantity, and freeze some parameters of the model.

    def init_optimizer(self):
        self.modelManager.init_optimizer()
        self.IOManager.log.Info('Init Optimizer Over.')

Initialize the optimization function.

    def def_loss_func(self):
        self.modelManager.def_loss_func()
        self.IOManager.log.Info('Define Loss Function Over.')

Initialize the loss function, we provide the loss function for cross-moisture and Focal loss.

    def train_model(self):
        self.IOManager.log.Info('Train Model Start.')
        self.IOManager.log.Info('Learn Name: {}'.format(self.config.learn_name))
        self.IOManager.log.Info('Config: {}'.format(self.config))
        self.modelManager.train()
        # self.visualizer.draw_train_test_curve()
        # self.visualizer.draw_tsne()
        # self.visualizer.draw_ROC_PRC_curve()
        self.IOManager.log.Info('Train Model Over.')

The expansion of the training function, a very complex part, will be discussed in detail later.

    def test_model(self):
        self.IOManager.log.Info('Test Model Start.')
        self.modelManager.test()
        self.IOManager.log.Info('Test Model Over.')

The expansion of the test function is simple.

    def reset_IOManager(self):
        self.IOManager = IOManager.IOManager(self)
        self.IOManager.initialize()
        self.IOManager.log.Info('Reset IOManager Over.')

In order not to reinitialize, the corresponding reset function is provided here:

    def reset_visualizer(self):
        self.visualizer = Visualizer.Visualizer(self)
        self.visualizer.initialize()
        self.IOManager.log.Info('Reset Visualizer Over.')

Visualize reset.

    def reset_dataManager(self):
        self.dataManager = DataManager.DataManager(self)
        self.dataManager.load_data()
        self.IOManager.log.Info('Reset DataManager Over.')

Data set duplication.

    def resset_modelManager(self):
        self.modelManager = ModelManager.ModelManager(self)
        self.modelManager.init_model()
        self.IOManager.log.Info('Reset ModelManager Over.')

Model duplication.