1. What is semantic segmentation?
1.1 Task:
Segmenting the image into different areas according to the category of the object is equivalent to classifying each pixel .
1.2 Application:
Self-driving car
portrait segmentation
, real-time replacement of video background,
intelligent remote
sensing, and identification of the categories of surface objects. As you can see from the segmented image on the right, the red parts belong to lakes and rivers. Intelligent remote sensing can monitor changes in surface waters in different seasons, thereby assisting agricultural production, predicting droughts and floods, etc.
Medical image analysis
1.3 Classification
Semantic segmentation vs instance segmentation vs panoramic segmentation
2. Basic ideas of semantic segmentation
2.1 Split by color
Prior knowledge: The colors inside objects are similar, and the colors at the interfaces of objects change.
Problems:
(1) Prior knowledge is not completely accurate;
(2) Different objects may have similar colors, and objects may also contain multiple colors.
2.2 Pixel-by-pixel classification
Existing problems:
low efficiency, repeated convolution calculations in overlapping areas
Solution:
reuse convolution calculations
2.3 Convolution of fully connected layers
2.4 Upsampling of prediction maps
Bilinear Interpolation Bilinear Interpolation
Convolution realizes bilinear interpolation
Transposed Convolution Transposed Convolution
2.5 Prediction and training of fully convolutional network
2.6 Upsampling based on multi-level features
Various solutions:
2.5 Context information
The content surrounding an image (also called context) can help us make more accurate judgments.
How to use contextual information during segmentation?
Solution: Add a network branch with a larger receptive field and introduce contextual information into local predictions
2.6 Atrous convolution and Deeplab series
Dilated convolution solves downsampling problem
DeepLab model
2.6 Conditional Random Field CRF
Conditional random field
energy function meaning
2.6 Spatial Pyramid Pooling ASPP
3. Summary of semantic segmentation algorithms
4. Semantic segmentation preface algorithm
4.1 SegFormer
4.2 K-Net
4.3 Mask Forms
4.4 Mask2Former
4.5 SAM
5. Evaluation of semantic segmentation models
