AdvSemiSeg realizes semi supervised semantic segmentation by generating confrontation

Keywords: Computer Vision Deep Learning paddlepaddle

Project description

Reproduction of Paper "Adversarial Learning for Semi-Supervised Semantic Segmentation" with PaddlePaddle.

This project reproduces the classic paper "Adversarial Learning for Semi-Supervised Semantic Segmentation" in semi supervised semantic segmentation field based on PaddlePaddle, and achieves the index of thesis.

AdvSemiSeg is one of the earliest articles in the field of semi supervised semantic segmentation. Different from the classification level label data and classification level loss function commonly used in the field of weak supervision, semi supervised learning emphasizes the combination of a small amount of labeled data and a large amount of unlabeled data. Its core lies in how to better mine the supervision information of unlabeled data through labeled data, so as to improve the performance of the model and reduce labor expenditure.

Method flow

The model consists of two modules

  • Split network

𝐻 × 𝑊 × 3 - > split network - > 𝐻 × 𝑊 × 𝐶 (where 𝐶 is the number of split categories)

  • Discriminant network

The input is a class probability graph, which is obtained by dividing the network or one hot coding from the ground truth label

The output is 𝐻 × 𝑊 × In the probability diagram of 1, the pixel value 𝑝 = 1 represents from the ground truth label, and the pixel value 𝑝 = 0 represents from the segmentation network.

Training process:

Both labeled and unlabeled images are used in training

1. When using labeled images, the segmentation network is subject to both standard cross entropy loss based on ground truth label and adversarial loss based on Discriminant network. Note that the training discriminant network only uses labeled data.

2. When the unlabeled image is used, the preliminary segmentation result is obtained by the segmentation network, and then the preliminary segmentation result is sent to the discrimination network to obtain the confidence map. The confidence map is used as the supervision signal, and the preliminary segmentation prediction result is mask ed as the label. The segmentation network is trained by the self-learning method.


Reproduction description

The default configuration is ResNet101+Deeplabv2+VOC2012+1/8Label

Other training settings are consistent with the original items: 20000 iter, batch size 10;

  • Recurrence index

advSemiSeg folder Version (original version)

Thesis indexRecurrence index
69.5 miou70.4 miou

1. The settings of the model are in the corresponding code:

  • Line 204 in;
  • Line 197 in;

2. Label rate setting: – labeled ratio 0.125 indicates 1 / 8 label rate

3. It has been set under the training instruction and evaluation instruction

4. The evaluation results are stored in the txt file in results

PaddleSeg suite version:
The model is consistent with other PaddleSeg models. Label needs to be specified_ Ratio enables semi supervised training algorithms. See the PaddleSeg/semi directory for details.

Thesis indexRecurrence index
69.5 miou72.6 miou

The following training evaluation and codes can be run with one click; Fully supervised baseline and automated testing are optional and should be noted.

Dataset preparation

PASCAL VOC2012 data set is adopted as the data set, and the enhanced label provided by the original text is adopted.

#Decompress dataset
!unzip -q data/data4379/ -d data/data4379/
!unzip -q data/data117898/ -d data/data4379/pascalvoc/VOCdevkit/VOC2012/
!cp aug.txt data/data4379/pascalvoc/VOCdevkit/VOC2012/ImageSets/Segmentation/
# Data set structure
!tree -L 1 /home/aistudio/data/data4379/pascalvoc/VOCdevkit/VOC2012

Training advSemiSeg

!cd PaddleSegSemi/ && python --config configs/deeplabv2/deeplabv2_resnet101_os8_voc_semi_321x321_20k.yml --label_ratio 0.125 --num_workers 0 --use_vdl --do_eval --save_interval 1000 --save_dir deeplabv2_res101_voc_0.125_20k
## Do not use the PaddleSeg Suite
# !cd advSemiSeg/ && python --checkpoint_dir ./checkpoints/voc_semi_0_125 --labeled-ratio 0.125 --ignore-label 255 --num-classes 21 --use_vdl

Evaluate advSemiSeg

## Using the PaddleSeg Suite
!cd PaddleSegSemi/ && python --config configs/deeplabv2/deeplabv2_resnet101_os8_voc_semi_321x321_20k.yml --model_path deeplabv2_res101_voc_0.125_20k/best_model/model.pdparams 
## Do not use the PaddleSeg Suite
# !cd advSemiSeg/ && python --dataset pascal_voc --num-classes 21 --restore-from ./checkpoints/voc_semi_0_125/20000.pdparams 
2021-11-24 10:15:59 [INFO]	
------------Environment Information-------------
platform: Linux-4.13.0-36-generic-x86_64-with-debian-stretch-sid
Python: 3.7.4 (default, Aug 13 2019, 20:35:49) [GCC 7.3.0]
Paddle compiled with cuda: True
NVCC: Cuda compilation tools, release 10.1, V10.1.243
cudnn: 7.6
GPUs used: 1
GPU: ['GPU 0: Tesla V100-SXM2-16GB']
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~16.04) 7.5.0
PaddlePaddle: 2.2.0
OpenCV: 4.1.1
Namespace(data_dir='/home/aistudio/data/data4379/pascalvoc/VOCdevkit/VOC2012', data_list='./data/voc_list/val.txt', dataset='pascal_voc', gpu=0, ignore_label=255, model='Deeplabv2', num_classes=21, restore_from='./checkpoints/voc_semi_0_125/20000.pdparams', save_dir='results', save_output_images=False)
W1124 10:15:59.974874  2351] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.0, Runtime API Version: 10.1
W1124 10:15:59.974916  2351] device: 0, cuDNN Version: 7.6.
2021-11-24 10:16:03 [INFO]	No pretrained model to load, ResNet_vd will be trained from scratch.
2021-11-24 10:16:03 [INFO]	Loading pretrained model from ./checkpoints/voc_semi_0_125/20000.pdparams
2021-11-24 10:16:03 [INFO]	There are 538/538 variables loaded into DeepLabV2.
0 processd DeprecationWarning: `` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing ``, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance:
  gt = np.asarray(label[0].numpy()[:size[0],:size[1]], DeprecationWarning: `` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing ``, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance:
  output = np.asarray(np.argmax(output, axis=2),
100 processd
200 processd
300 processd
400 processd
500 processd
600 processd
700 processd
800 processd
900 processd
1000 processd
1100 processd
1200 processd
1300 processd
1400 processd
class  0 background   IU 0.93
class  1 aeroplane    IU 0.85
class  2 bicycle      IU 0.41
class  3 bird         IU 0.85
class  4 boat         IU 0.67
class  5 bottle       IU 0.78
class  6 bus          IU 0.90
class  7 car          IU 0.84
class  8 cat          IU 0.84
class  9 chair        IU 0.32
class 10 cow          IU 0.72
class 11 diningtable  IU 0.38
class 12 dog          IU 0.81
class 13 horse        IU 0.73
class 14 motorbike    IU 0.81
class 15 person       IU 0.83
class 16 pottedplant  IU 0.44
class 17 sheep        IU 0.78
class 18 sofa         IU 0.43
class 19 train        IU 0.72
class 20 tvmonitor    IU 0.71
meanIOU: 0.7040637094579905

Train fully supervised baseline (optional)

!cd advSemiSeg/ && python --dataset pascal_voc  \
                        --checkpoint-dir ./checkpoints/voc_full \
                        --ignore-label 255 \
                        --num-classes 21 

TIPC automated test (optional)

You can view the README of PaddleSegSemi in detail. You need to install the related dependencies of the logger.

  • Install autolog
    git clone
    cd AutoLog
    pip3 install -r requirements.txt
    python3 bdist_wheel
    pip3 install ./dist/auto_log-1.0.0-py3-none-any.whl
    cd ../
#Enter the PaddleSegSemi folder and run the command
%cd PaddleSegSemi/
!pip3 install -r requirements.txt
!bash test_tipc/ ./test_tipc/configs/advsemiseg_deeplabv2_res101_humanseg/train_infer_python.txt 'lite_train_lite_infer'
!bash test_tipc/ ./test_tipc/configs/advsemiseg_deeplabv2_res101_humanseg/train_infer_python.txt 'lite_train_lite_infer'


When only a part of labeled data is used in the original data set, the addition of unlabeled data improves the performance, which proves that the adaptive semi supervised learning setting proposed in this paper is very effective.

But this paper also has shortcomings. For example, the discriminator does not use image information, which is optimized in s4GAN(TPAMI,2019).

In the future, we will continue to open-source Paddle of semi supervised semantic segmentation method ~

Welcome to pay more attention

Posted by atomm on Fri, 03 Dec 2021 14:00:05 -0800