Catalogue of series articles

Near infrared spectroscopy is a cross cutting field, which requires cooperation in chemistry, computer science, bioscience and other fields. Therefore, under the guidance of (teacher Yang Huihua team of Beijing University of Posts and Telecommunications), we are preparing to open source the traditional classical algorithms such as PLS, SVM, ANN and RF, preprocessing such as SG, MSC, first-order derivative and second-order derivative, wavelength selection algorithms such as GA, and the latest deep learning algorithms such as CNN and AE, To help other professionals more easily establish near-infrared spectral models with good prediction ability and robustness.

preface

NIRS is an electromagnetic wave between visible light and mid infrared light, with a wavelength range of (1100 ∼ 2526 nm. Due to the near infrared spectral region and hydrogen containing groups (OH, NH, CH, SH) in organic molecules The frequency combination of vibration is consistent with the absorption region of frequency doubling at all levels. The characteristic information of hydrogen containing groups of organic molecules in the sample can be obtained by scanning the near-infrared spectrum of the sample, which is often used as an effective carrier to obtain the sample information. The detection method based on NIRS has the advantages of convenience, high efficiency, accuracy, low cost, on-site detection and no damage to the sample, and is widely used in many fields Various detection fields. However, near-infrared spectroscopy has some problems, such as spectral bandwidth, serious overlap, weak absorption signal, complex information analysis, etc. unlike common chemical analysis methods, it can only be used as an indirect measurement method, and can not directly analyze the content or category of the measured sample. It depends on chemometric methods, which is between the attribute value of the sample to be measured and the near-infrared spectrum data A correlation model (or Calibration Model) is established, and then the near-infrared spectra of unknown samples are predicted by the model to obtain the predicted values of various components. The existing near-infrared modeling methods mainly include classical modeling (preprocessing + wavelength screening for feature dimensionality reduction and highlighting, and then modeling by pls and svm algorithms) and deep learning methods (end to end modeling has low dependence on preprocessing, wavelength selection, etc.)

This paper mainly describes CNN's qualitative analysis and modeling method of near infrared spectroscopy

1, Data source

Open source drugs are used, with a total of 310 samples and 404 variables for each sample. They are divided into four categories according to active ingredients, Download address
The picture is as follows:

! [drug data spectrum]( https://img-blog.csdnimg.cn/7d76d4b1ed4d4f71a150298bdae3da74.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBARWNob19Db2Rl,size_20,color_FFFFFF,t_70,g_se,x_16

2, Code interpretation

1.1 reading data

#Load data 
def TableDataLoad(tp, test_ratio, start, end, seed):

    # global data_x
    data_path = './/Data//table.csv'
    Rawdata = np.loadtxt(open(data_path, 'rb'), dtype=np.float64, delimiter=',', skiprows=0)
    table_random_state = seed

    if tp =='raw':
        data_x = Rawdata[0:, start:end]
    if tp =='SG':
        SGdata_path = './/Data//TableSG.csv'
        data = np.loadtxt(open(SGdata_path, 'rb'), dtype=np.float64, delimiter=',', skiprows=0)
        data_x = data[0:, start:end]
    if tp =='SNV':
        SNVata_path = './/Data//TableSNV.csv'
        data = np.loadtxt(open(SNVata_path, 'rb'), dtype=np.float64, delimiter=',', skiprows=0)
        data_x = data[0:, start:end]
    if tp == 'MSC':
        MSCdata_path = './/Data//TableMSC.csv'
        data = np.loadtxt(open(MSCdata_path, 'rb'), dtype=np.float64, delimiter=',', skiprows=0)
        data_x = data[0:, start:end]
    data_y = Rawdata[0:, -1]
    x_col = np.linspace(7400, 10507, 404)
    plotspc(x_col, data_x[:, :], tp=0)

    x_data = np.array(data_x)
    y_data = np.array(data_y)
    
   

    print('Training set size:{}'.format(len(X_train[:,0])))
    print('Test set size:{}'.format(len(X_test[:,0])))

1.2 dividing data sets

 X_train, X_test, y_train, y_test = train_test_split(x_data, y_data, test_size=test_ratio,random_state=table_random_state)

1.3 return the training set and test set and draw spectral pictures

    data_train, data_test = ZspPocess(X_train, X_test,y_train,y_test,need=True) #for cnn :false only used in proseesing comparsion
    return data_train, data_test

! [displayed spectral curve]( https://img-blog.csdnimg.cn/2d09358a92d4476fb073c3705d57561d.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBARWNob19Db2Rl,size_20,color_FFFFFF,t_70,g_se,x_16

2.1 establishment of 1-D CNN model

class CNN3Lyaers(nn.Module):
    def __init__(self, nls):
        super(CNN3Lyaers, self).__init__()
        self.CONV1 = nn.Sequential(
            nn.Conv1d(1, 64, 21, 1),
            nn.BatchNorm1d(64),  # Homogenize the output
            nn.ReLU(),
            nn.MaxPool1d(3, 3)
        )
        self.CONV2 = nn.Sequential(
            nn.Conv1d(64, 64, 19, 1),
            nn.BatchNorm1d(64),  # Homogenize the output
            nn.ReLU(),
            nn.MaxPool1d(3, 3)
        )
        self.CONV3 = nn.Sequential(
            nn.Conv1d(64, 64, 17, 1),
            nn.BatchNorm1d(64),  # Homogenize the output
            nn.ReLU(),
            nn.MaxPool1d(3, 3),
        )
        self.fc = nn.Sequential(
            # nn.Linear(4224, nls)
            nn.Linear(384, nls)
        )

    def forward(self, x):
        x = self.CONV1(x)
        x = self.CONV2(x)
        x = self.CONV3(x)
        x = x.view(x.size(0), -1)
        # print(x.size())
        out = self.fc(x)
        out = F.softmax(out,dim=1)
        return out

2.2 training 1-D CNN model

def CNNTrain(tp, test_ratio, BATCH_SIZE, n_epochs, nls):

    # data_train, data_test = DataLoad(tp, test_ratio, 0, 404)
    data_train, data_test = TableDataLoad(tp, test_ratio, 0, 404, seed=80)
    train_loader = torch.utils.data.DataLoader(data_train, batch_size=BATCH_SIZE, shuffle=True)
    test_loader = torch.utils.data.DataLoader(data_test, batch_size=BATCH_SIZE, shuffle=True)

    store_path = ".//model//all//CNN18"

    model = CNN3Lyaers(nls=nls).to(device)
    optimizer = optim.Adam(model.parameters(),
                           lr=0.0001,weight_decay=0.0001)
    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.5, verbose=1, eps=1e-06,
                                                           patience=10)
    criterion = nn.CrossEntropyLoss().to(device)  # The loss function is a focal loss function, which is often used in multi classification problems with category imbalance
    early_stopping = EarlyStopping(patience=30, delta=1e-4, path=store_path, verbose=False)

    for epoch in range(n_epochs):
        train_acc = []
        for i, data in enumerate(train_loader):  # gives batch data, normalize x when iterate train_loader
            model.train()  # No training
            inputs, labels = data  # Both input and label are equal to data
            inputs = Variable(inputs).type(torch.FloatTensor).to(device)  # batch x
            labels = Variable(labels).type(torch.LongTensor).to(device)  # batch y
            output = model(inputs)  # cnn output
            trian_loss = criterion(output, labels)  # cross entropy loss
            optimizer.zero_grad()  # clear gradients for this training step
            trian_loss.backward()  # backpropagation, compute gradients
            optimizer.step()  # apply gradients
            _, predicted = torch.max(output.data, 1)
            y_predicted = predicted.detach().cpu().numpy()
            y_label = labels.detach().cpu().numpy()
            acc = accuracy_score(y_label, y_predicted)
            train_acc.append(acc)

        with torch.no_grad():  # No gradient
            test_acc = []
            testloss = []
            for i, data in enumerate(test_loader):
                model.eval()  # No training
                inputs, labels = data  # Both input and label are equal to data
                inputs = Variable(inputs).type(torch.FloatTensor).to(device)  # batch x
                labels = Variable(labels).type(torch.LongTensor).to(device)  # batch y
                outputs = model(inputs)  # The output is equal to the input after entering the network
                test_loss = criterion(outputs, labels)  # cross entropy loss
                _, predicted = torch.max(outputs.data,1)
                predicted = predicted.cpu().numpy()
                labels = labels.cpu().numpy()
                acc = accuracy_score(labels, predicted)
                test_acc.append(acc)
                testloss.append(test_loss.item())
                avg_loss = np.mean(testloss)

            scheduler.step(avg_loss)
            early_stopping(avg_loss, model)
            if early_stopping.early_stop:
                print(f'Early stopping! Best validation loss: {early_stopping.get_best_score()}')
                break

2.3 testing 1-D CNN model

def CNNtest(tp, test_ratio, BATCH_SIZE, nls):
    # data_train, data_test = DataLoad(tp, test_ratio, 0, 404)
    data_train, data_test = TableDataLoad(tp, test_ratio, 0, 404, seed=80)
    test_loader = torch.utils.data.DataLoader(data_test, batch_size=BATCH_SIZE, shuffle=True)

    store_path = ".//model//all//CNN18"

    model = CNN3Lyaers(nls=nls).to(device)

    model.load_state_dict(torch.load(store_path))
    test_acc = []
    for i, data in enumerate(test_loader):
        model.eval()  # No training
        inputs, labels = data  # Both input and label are equal to data
        inputs = Variable(inputs).type(torch.FloatTensor).to(device)  # batch x
        labels = Variable(labels).type(torch.LongTensor).to(device)  # batch y
        outputs = model(inputs)  # The output is equal to the input after entering the network
        _, predicted = torch.max(outputs.data, 1)
        predicted = predicted.cpu().numpy()
        labels = labels.cpu().numpy()
        acc = accuracy_score(labels, predicted)
        test_acc.append(acc)
    return np.mean(test_acc)

3 run the CNN model and view the results with

Get model accuracy

In order to take a closer look at the causes of model errors, we use the confusion matrix to look further, and the results are as follows:

summary

The complete code is available from GitHub warehouse
The code is for academic use only. If you have any questions, please contact QQ: 1427950662, wechat: Fu_siry