This chapter will train a simple CNN network on CIFAR10 dataset:

- A simple CNN network is trained based on CIFAR-10 data set.
- Save the trained model and test it.
- Train with GPU.

## CIFAR dataset

CIFAR data sets can be divided into CIFAR10 and CIFAR100. CIFAR-10 includes 10 categories and CIFAR-100 includes 100 categories.

### CIFAR-10

Features: 32x32 color image; 10 categories; 60000 images in total; 50000 training samples + 10000 test samples; 6000 images per category, 10 x 6000 = 60000;

10 categories: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck;

Tips: you don't need to download it manually. You can download it automatically by using the Dataset API in pytorch

## Experimental process

### Prepare dataset

This step is very convenient in pytorch. Pytorch has prepared common data sets for us. We only need to import them.

The dataset is in the torchvision.dataset package:

import torch import torchvision import torchvision.transforms as transforms from torch.utils.data import DataLoader import torch.nn as nn import torch.nn.functional as F import torch.optim as optim import matplotlib.pyplot as plt import numpy as np

torchvision.dataset.CFAIR10 is a class. You can manipulate a dataset by instantiating an object of this class.

Parameters:

root ---- the path saved after the data set is downloaded

train ---- training or test

Download ---- whether automatic download is required

Transform ---- to transform an image, you generally need to transform the original image with ToTensor(), Normalize()

Then, use the DataLoader class to wrap the dataset for easy reading and use, such as min_batch read, using multithreading.

# --------------------Prepare dataset------------------ # Dataset, DataLoader transform = transforms.Compose( [transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), std =(0.5, 0.5, 0.5))]) trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform) testset = torchvision.datasets.CIFAR10(root='./data',train=False, transform=transform, download=True) trainloader = DataLoader(dataset=trainset, batch_size=4, shuffle=True, num_workers=4) testloader = DataLoader(dataset=testset, batch_size=4, shuffle=True, num_workers=4)

# dataiter = iter(trainloader) images, labels = dataiter.next() imshow(torchvision.utils.make_grid(images)) # print labels print(' '.join('%5s' % classes[labels[j]] for j in range(4)))

### Define CNN network

For simplicity, the LeNet network is used to change the input channel of the first convolution layer to 3, because CIFAR-10 is a color 3-channel image.

#Define a simple network # LeNet -5 class Net(nn.Module): def __init__(self): super(Net, self).__init__() self.conv1 = nn.Conv2d(in_channels=3, out_channels=6, kernel_size=5) self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2) self.conv2 = nn.Conv2d(in_channels=6, out_channels=16, kernel_size=5) self.fc1 = nn.Linear(in_features=16 * 5 * 5,out_features=120) self.fc2 = nn.Linear(in_features=120, out_features=84) self.fc3 = nn.Linear(in_features=84, out_features=10) def forward(self, x): x = self.pool1(F.relu(self.conv1(x))) x = self.pool1(F.relu(self.conv2(x))) x = x.view(-1, 16 * 5 * 5) # reshape tensor x = F.relu(self.fc1(x)) x = F.relu(self.fc2(x)) x = self.fc3(x) return x

### Set up the optimization and iteration methods of the network and train the network

CNN network training is essentially the problem of minimizing an objective function (loss function). In mathematics, for general convex functions, optimization methods include gradient descent method, Newton method and so on. (in addition, there are heuristic search, such as genetic algorithm). For the training of neural network, the commonly used optimization method is stochastic gradient descent method SGD.

- Definition of loss function and optimization method

Cross entropy loss function

SGD random gradient descent method is used for optimization (driving quantity term)

# Definition of loss function and optimization method # Cross enterprise loss, SGD with moment criterion = nn.CrossEntropyLoss() optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

- Iterative optimization and training

Iter ----- an iteration refers to a min_ One forward+backward of batch

Epoch ----- after iterating all training data (once), it is called an epoch

There are 20 epoch s running here.

# Training network # Iterative epoch for epoch in range(20): running_loss = 0.0 for i, data in enumerate(trainloader, 0): # get the input inputs, labels = data # zeros the paramster gradients optimizer.zero_grad() # # forward + backward + optimize outputs = net(inputs) loss = criterion(outputs, labels) # Calculate loss loss.backward() # loss derivation optimizer.step() # Update parameters # print statistics running_loss += loss.item() # tensor.item() gets the value of tensor if i % 2000 == 1999: print('[%d, %5d] loss: %.3f' % (epoch + 1, i + 1, running_loss / 2000)) # The average value of loss is output every 2000 iterations running_loss = 0.0 print('Finished Training')

### Save model

# --------Save model----------- torch.save(net, './model/model_cfair10_2.pth') # Save the whole model, and the volume is relatively large # torch.save(net.state_dict(), './model/model_cfair10.pth')

### test model

import torch import torchvision import torchvision.transforms as transforms from torch.utils.data import DataLoader import torch.nn as nn import torch.nn.functional as F import torch.optim as optim import matplotlib.pyplot as plt import numpy as np from PIL import Image

CIFAR-10 contains a total of 10 categories:

CFAIR10_names = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'forg', 'horse', 'ship', 'truck']

Loading an image, RBG, must belong to one of the above categories, otherwise it cannot be recognized

# load a image image = Image.open('/xxxx/image/dog.jpg')

Make the same transformation on the image:

transform = transforms.Compose( [transforms.Resize((32, 32)), transforms.ToTensor(), transforms.Normalize( mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5) )]) image_transformed = transform(image) print(image_transformed.size())

Points needing attention

The input of CNN network is 4D Tensor (NxCxHxW), and the converted image needs to be transformed into 4D

torsor1.unsqueeze(0) can add a dimension, so the entered tensor is 1x3x32x32

# transform = transforms.Compose( [transforms.Resize((32, 32)), transforms.ToTensor(), transforms.Normalize( mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5) )]) image_transformed = transform(image) print(image_transformed.size()) class Net(nn.Module): def __init__(self): super(Net, self).__init__() self.conv1 = nn.Conv2d(in_channels=3, out_channels=6, kernel_size=5) self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2) self.conv2 = nn.Conv2d(in_channels=6, out_channels=16, kernel_size=5) self.fc1 = nn.Linear(in_features=16 * 5 * 5,out_features=120) self.fc2 = nn.Linear(in_features=120, out_features=84) self.fc3 = nn.Linear(in_features=84, out_features=10) def forward(self, x): x = self.pool1(F.relu(self.conv1(x))) x = self.pool1(F.relu(self.conv2(x))) x = x.view(-1, 16 * 5 * 5) # reshape tensor x = F.relu(self.fc1(x)) x = F.relu(self.fc2(x)) x = self.fc3(x) return x net = torch.load('./model/model_cfair10.pth') # print(net) image_transformed = image_transformed.unsqueeze(0) output = net(image_transformed) predict_value, predict_idx = torch.max(output, 1) # Find the maximum value of the specified dimension and return the maximum value and index plt.figure() plt.imshow(np.array(image)) plt.title(CFAIR10_names[predict_idx]) plt.axis('off') plt.show()

----------------------------------------------------

## GPU training and model learning rate adjustment

Question:

- After training 20 epochs with CPU model, the loss decreased to about 0.6. Later, 20 epochs were iterated based on the previous training, and it was found that the loss was between 0.5 and 0.6.
- Training on the CPU is really slow. It took more than 1h to run 20 epochs (I don't remember the specific time), which is quite long.

### Using GPU training model

- Computer configuration GPU: 1080

First, you need to install the GPU version of pytorch. The specific installation steps are available on the pytorch official website. Training with GPU requires some minor adjustments to the code.

**step1: * * in the code, first use the function in pytorch to determine whether GPU is supported

is_support = torch.cunda.is_available() if is_support: device = torch.device('cuda:0') # device = torch.device('cuda:1') else: device = torch.device('cpu')

Step 2: transfer the calculation on CPU to GPU

net = Net() net.to(device) # GPU mode needs to be added # Training network # Iterative epoch for epoch in range(20): running_loss = 0.0 for i, data in enumerate(trainloader, 0): # get the input inputs, labels = data inputs = inputs.to(device) # GPU calculation labels = labels.to(device) # GPU calculation # zeros the paramster gradients optimizer.zero_grad() # # forward + backward + optimize outputs = net(inputs) loss = criterion(outputs, labels) # Calculate loss loss.backward() # loss derivation optimizer.step() # Update parameters # print statistics running_loss += loss.item() # tensor.item() gets the value of tensor if i % 2000 == 1999: print('[%d, %5d] loss: %.3f' % (epoch + 1, i + 1, running_loss / 2000)) # The average value of loss is output every 2000 iterations running_loss = 0.0 print('Finished Training')

run, you will find that the iteration speed flies, and 20 Epoch iterations can be completed in about 10 minutes, which is very fast.

## Learning rate adjustment

- An important parameter of stochastic gradient descent SGD is learning rate_ rate

The above code uses a fixed learning rate lr=0.001,. At the beginning of the iteration, the learning rate can be larger, so the convergence speed is fast. With the increase of the number of iterations, the learning rate should be reduced to prevent loss oscillation.

For simplicity, I adjust the learning rate to lr=0.0001, and then iterate 20 epochs based on the previous model. Loss was obviously found to be 0.3, 0.2 and 0.1.

Although GPU training was used, lr was reduced to 0.0001 and loss was also reduced (training set loss). In the test, a horse is identified as deer and a bird is identified as cat. Because, to train to a suitable model, other strategies are needed, including the use of other network models.

## Evaluate the model performance on the whole test set

- Calculate Acc

import torch import torchvision import torchvision.transforms as transforms from torch.utils.data import DataLoader import torch.nn as nn import torch.nn.functional as F import torch.optim as optim import matplotlib.pyplot as plt import numpy as np from PIL import Image CFAIR10_names = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'forg', 'horse', 'ship', 'truck'] # --------------Test data set------------------------------ transform = transforms.Compose( [transforms.Resize((32, 32)), transforms.ToTensor(), transforms.Normalize( mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5) )]) testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform) testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=4) # -----------------Nettle model------------------------------- class Net(nn.Module): def __init__(self): super(Net, self).__init__() self.conv1 = nn.Conv2d(in_channels=3, out_channels=6, kernel_size=5) self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2) self.conv2 = nn.Conv2d(in_channels=6, out_channels=16, kernel_size=5) self.fc1 = nn.Linear(in_features=16 * 5 * 5,out_features=120) self.fc2 = nn.Linear(in_features=120, out_features=84) self.fc3 = nn.Linear(in_features=84, out_features=10) def forward(self, x): x = self.pool1(F.relu(self.conv1(x))) x = self.pool1(F.relu(self.conv2(x))) x = x.view(-1, 16 * 5 * 5) # reshape tensor x = F.relu(self.fc1(x)) x = F.relu(self.fc2(x)) x = self.fc3(x) return x net = torch.load('./model/model_cfair10_20.pth',map_location='cpu') # ------------Test on the entire test set------------------------------------------- correct = 0 total = 0 count = 0 with torch.no_grad(): for sample_batch in testloader: images = sample_batch[0] labels = sample_batch[1] # forward out = net(images) # _, pred = torch.max(out, 1) correct += (pred == labels).sum().item() total += labels.size(0) print('batch:{}'.format(count + 1)) count += 1 # # Acc accuracy = float(correct) / total print('Acc = {:.5f}'.format(accuracy))

Link: https://www.jianshu.com/p/e704a6f6e8d3

Source: Jianshu