1, Dataset Download
Link: https://pan.baidu.com/s/1_7blbYJc0ouCGmqe8kBnTw extraction code: c6ex copy this content and open Baidu online disk mobile App, which is more convenient to operate
2, Training model
1. Define data initialization
import torchvision.transforms as transforms
image_size=(224,224) #
data_transforms=transforms.Compose([
transforms.RandomHorizontalFlip(), #Flip horizontally according to probability (default 0.5)
transforms.Resize(image_size ), #Image size transformation, unified processing into 224 * 224
transforms.ToTensor(), #Convert image to Tensor type
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]), #Image regularization
])
1, Crop - crop
1. Random clipping: transforms.RandomCrop
2. Center clipping: transforms.CenterCrop
3. Random aspect ratio clipping transforms.RandomResizedCrop
4. Top, bottom, left and right center clipping: transforms.FiveCrop
5. Flip after cutting up, down, left and right centers: transforms.tenprop
2, Flip and Rotation
6. Flip transforms.RandomHorizontalFlip according to probability p level
7. Flip transforms.RandomVerticalFlip vertically according to probability p
8. Random rotation: transforms.RandomRotation
3, Image transformation
9.resize: transforms.Resize
10. Standardization: transforms.Normalize
11. Change to tensor: transforms.ToTensor
12. Fill: transforms.Pad
13. Modify brightness, contrast and saturation: transforms.ColorJitter
14. Grayscale image: transforms.Grayscale
15. Linear transformation: transforms.LinearTransformation()
16. Affine transformation: transforms.RandomAffine
17. Convert to grayscale according to probability p: transforms.RandomGrayscale
18. Convert data to PILImage: transforms.topiimage
19.transforms.Lambda
4, Operate on transforms to make data enhancement more flexible
20.transforms.RandomChoice(transforms)
21.transforms.RandomApply(transforms, p=0.5)
22.transforms.RandomOrder
2, Import dataset
import torchvision.datasets as datasets train_data=datasets .ImageFolder (root='fruits-360-original-size/fruits-360-original-size/Training',transform=data_transforms ) #Add the address of the data set in double quotation marks print(train_data.classes) #Output train_ Folder name in data
Output is:
be careful:
The ImageForder function can only import folders, not files.
The Classes function determines the category according to the name of the folder.
3, Processing datasets with the DataLoader function
from torch.utils.data import DataLoader batchsize=16 train_loader=DataLoader(train_data,batch_size=batchsize,shuffle=True,num_workers=1)
be careful:
1. "batch_size =" the number of samples selected for one training.
2."shuffle =" enter a Boolean value to determine whether to disrupt the data
3. "num_workers =" is used to serve multiprocessing data loading. It is used to set how many child processes are responsible for data loading. num_workers is not the bigger the better, because too many child processes will occupy CPU computing resources, slow down other CPU computing parts of the program, and increase the overall running time
Generally, it is set by gradually increasing attempts. For example, when the GPU computing utilization is saturated, it indicates that the data reading is enough to meet the computing requirements, so it is not necessary to increase the number of worker s
4, Import model
import torchvision.models as models AlexNet=models.AlexNet() model=AlexNet
be careful:
Many pre training models are built in torchvision.models, such as:
AlexNet
VGG
ResNet
SqueezeNet
DenseNet
Inception v3
ResNet18 and AlexNet can run smoothly. In addition to using the built-in model of pytoch, you can also import the model locally or define the model yourself.
5, Weight initialization
import torch.nn.init as init
for name,module in model._modules.items() :
if (name=='fc'):
init.kaiming_uniform_(module.weight,a=0,mode='fan_in')
1. Uniform distribution torch.nn.init.uniform_(tensor, a=0, b=1) obeys ~ U(a,b)
2. Positive Pacific distribution torch.nn.init.normal_(tensor, mean=0, std=1) obeys ~ N(mean,std)
3. Initialize to constant torch.nn.init.constant_(tensor, val) initialize the whole matrix to constant val
4.Xavier
5.kaiming
6, Define optimizer
import torch optimizer=torch.optim .SGD(model.parameters(),lr=0.01 ) StepLR=torch.optim.lr_scheduler .StepLR (optimizer ,step_size= 3,gamma=1 ) #Adjust LR every three times
be careful:
StepLR() is used to adjust the learning rate at equal intervals. The adjustment multiple is gamma times and the adjustment interval is step_size.
7, Call GPU
# device=torch.device("cuda:0" if torch.cuda.is_available() else"cpu") #Code when training with a GPU
if torch.cuda.device_count()>1: #Determine whether the number of GPU s in cuda is greater than one
model=nn.DataParallel(model) #If it is greater than one, call GPU in parallel and train at the same time
print(device)
be careful:
The model trained with GPU can only be tested with GPU
8, Training model
import torch.nn.functional as F
def get_num_correct(outs, label):
pass
model.to(device) #Call GPU training
for epoch in range(10):
total_loss=0
print("epoch",epoch,":***************************")
for batch in train_loader:
images,labels=batch
images=images.to(device) #Transfer pictures to GPU
labels=labels.to(device) #Transfer labels into GPU
outs=model(images)
loss=F.cross_entropy(outs,labels) #Forward propagation
optimizer.zero_grad() #The used gradient must be cleared, otherwise it will occupy memory
loss.backward() #Back propagation
optimizer .step() #parameter optimization
total_loss +=loss.item()
print('loss',total_loss )
9, Save model
torch.save(model,'FruitModelGPU.pth')
3, Test model
1. Define data initialization
import torchvision.transforms as transforms
image_size=(224,224)
data_transforms=transforms.Compose([
transforms.RandomHorizontalFlip(),
transforms.Resize(image_size ),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])
2. Import model
import torch
model=torch.load('/home/ilab/FruitModelCpu4.pth',map_location='cpu')
model=model.module
model.eval()
be careful:
Because the GPU training model is used, at this time, we use CPU for testing, so we need to add map_location = 'CPU', and add model=model.module in another line
3. Import test set (when the test set is a folder)
import torchvision.datasets as datasets test_data=datasets .ImageFolder (root='fruits-360-original-size/fruits-360-original-size/test',transform=data_transforms )
4. Test
batch = next(iter(test_data)) images, labels = batch out = model(images) print(out)
Note: when the test set is a scrambled file, the previous method will not work. You can use the following methods:
1. Define data initialization
import torchvision.transforms as transforms
image_size=(224,224)
data_transforms=transforms.Compose([
transforms.RandomHorizontalFlip(),
transforms.Resize(image_size ),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])
2. Import training set
import torchvision.datasets as datasets train_data=datasets .ImageFolder (root='/ilab/datasets/local/fruits/train',transform=data_transforms )
3. Import model
import torch
model=torch.load('/home/ilab/FruitModelCpu4.pth',map_location='cpu')
model=model.module
4. Test
from PIL import Image
import os
i=1
sum=22688
root='/ilab/datasets/local/fruits/test' #Assign the address of the test set to Root
file_list=os.listdir(root) #Returns the list of files or folders contained in the current folder
model.eval()
with open('/home/ilab/submission','w') as f:
for filename in file_list:
filepath = os.path.join(root,filename)
img=Image.open(filepath)
x=data_transforms(img).reshape((1,3,224,224))
y= model(x)
re=filename+' '+train_data.classes[int(y.argmax(dim=1))]+'\n'
f.write(re)
print(re)
4, Complete code
"""""""""""""""""""""""Define data initialization method"""""""""""""""""""""""
import torch
import torch.nn
import torchvision
import torchvision.transforms as transforms
image_size=(224,224)
data_transforms=transforms.Compose([
transforms.RandomHorizontalFlip(),
transforms.Resize(image_size ),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])
"""""""""""""""""""""""Import dataset"""""""""""""""""""""""
import torchvision.datasets as datasets
train_data=datasets .ImageFolder (root='fruits-360-original-size/fruits-360-original-size/Training',transform=data_transforms )
print(train_data.classes)
"""""""""""""""""""""make DataLoader"""""""""""""""""""""""
from torch.utils.data import DataLoader
batchsize=500
train_loader=DataLoader(train_data,batch_size=batchsize,shuffle=True,num_workers=1)
""""""""""""""""""""""Make model"""""""""""""""""""""""""""""
import torchvision.models as models
AlexNet=models.AlexNet()#(pretrained= True)
model=AlexNet
"""""""""""""""""""""Weight initialization"""""""""""""""""""""""
import torch.nn.init as init
for name,module in model._modules.items() :
if (name=='fc'):
init.kaiming_uniform_(module.weight,a=0,mode='fan_in')
"""""""""""""""""""""""Define optimizer"""""""""""""""""""""
import torch
optimizer=torch.optim .SGD(model.parameters(),lr=0.01 )
StepLR=torch.optim.lr_scheduler .StepLR (optimizer ,step_size= 3,gamma=1 ) #Adjust LR every three times
""""""""""""""""""""""call GPU"""""""""""""""""""""""""""""
# device=torch.device("cuda:0" if torch.cuda.is_available() else"cpu")
if torch.cuda.device_count()>1:
model=nn.DataParallel(model)
print(device)
"""""""""""""""""""""""""""""""train"""""""""""""""""""""""""""""""""
import torch.nn.functional as F
def get_num_correct(outs, label):
pass
model.to(device)
for epoch in range(10):
total_loss=0
print("epoch",epoch,":***************************")
for batch in train_loader:
images,labels=batch
images=images.to(device)
labels=labels.to(device)
outs=model(images)
loss=F.cross_entropy(outs,labels)
optimizer.zero_grad()
loss.backward()
optimizer .step()
total_loss +=loss.item()
print('loss',total_loss )
""""""""""""""""""""""test"""""""""""""""""""""""""
""""""""""""""""""""""Weight initialization"""""""""""""""""""""""""
import torchvision.transforms as transforms
image_size=(224,224)
data_transforms=transforms.Compose([
transforms.RandomHorizontalFlip(),
transforms.Resize(image_size ),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
"""""""""""""""""""""""Import dataset"""""""""""""""""""""""
import os
import torchvision.datasets as datasets
train_data=datasets .ImageFolder (root='/ilab/datasets/local/fruits/train',transform=data_transforms )
"""""""""""""""""""""""Import model"""""""""""""""""""""""
import torch
model=torch.load('/home/ilab/FruitModelCpu4.pth',map_location='cpu')
model=model.module
"""""""""""""""""""""""test"""""""""""""""""""""""
from PIL import Image
import os
i=1
sum=22688
root='/ilab/datasets/local/fruits/test'
file_list=os.listdir(root) #Returns the list of files or folders contained in the current folder
model.eval()
with open('/home/ilab/submission','w') as f:
for filename in file_list:
filepath = os.path.join(root,filename)
img=Image.open(filepath)
x=data_transforms(img).reshape((1,3,224,224))
y= model(x)
re=filename+' '+train_data.classes[int(y.argmax(dim=1))]+'\n'
f.write(re)
print(re)
])