Semantic Image Segmentation via Deep Parsing Network

Transcription

1 Semantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*, Ping Luo, Chen Change Loy, Xiaoou Tang Multimedia Lab, The Chinese University of Hong Kong

2 Problem

3 Problem TV Background Plant Person Table

4 CNN CNN + MRF? Previous Attempts SVM SVM + MRF

5 State-of-the-arts Learned Features Pairwise Relations Joint Training - # Iterations - Fully Convolutional Network [Long et al. CVPR 2015]

6 State-of-the-arts Learned Features Pairwise Relations Joint Training # Iterations 10 DeepLab [Chen et al. ICLR 2015]

7 State-of-the-arts Learned Features Pairwise Relations Joint Training # Iterations 10 CRF as RNN [Zheng et al. ICCV 2015]

8 State-of-the-arts Learned Features Pairwise Relations Joint Training # Iterations 1 Deep Parsing Network (DPN)

9 Contributions Extend MRF to incorporate richer relationships Formulate mean field inference of high-order MRF as CNN Capable of joint training and one-pass inference

10 Revisit MRF pp ii llllllllll = ttttttttttt = 0.8 ii Energy Function min EE = UUUUUUUUUU + PPPPPPPP Unary Term UUUUUUUUUU = ln pp ii (llllllllll) ii

11 Revisit MRF ii dddddddd ii, =, = 0.8 Energy Function min EE = UUUUUUUUUU + PPPPPPPP ii Unary Term UUUUUUUUUU = ln pp ii (llllllllll) Pairwise Term ii Appearance Consistency PPPPPPPP = ccoooooo(ii) dddddddd(ii, ) ii,

12 Revisit MRF cccccccc ii; llllllllll = ttttttttttt = 0.1 Energy Function min EE = UUUUUUUUUU + PPPPPPPP ii Label Consistency Unary Term UUUUUUUUUU = ln pp ii (llllllllll) Pairwise Term ii PPPPPPPP = ccoooooo(ii) dddddddd(ii, ) ii,

13 Richer Relationships in DPN Energy Function min EE = UUUUUUUUUU + PPPPPPPP ii Unary Term UUUUUUUUUU = ln pp ii (llllllllll) Pairwise Term ii PPPPPPPP = ccoooooo(ii) dddddddd(ii, ) ii,

14 Richer Relationships in DPN Triple Penalty Pairwise Term ii PPPPPPPP = ccccssss(ii) dddddddd(ii, ) ii, (, )

15 Richer Relationships in DPN Triple Penalty Pairwise Term PPPPPPPP = ccccssss(ii) dddddddd(ii, ) ii zz 1 ii, zz 1 zz nn (,, ) zz nn

16 Richer Relationships in DPN Triple Penalty Pairwise Term PPPPPPPP = ccccssss(ii) dddddddd(ii, ; zz) ii zz 1 ii, zz zz 1 zz nn (,, ) zz nn Triple Penalty

17 Richer Relationships in DPN Mixture of Label Contexts ii Pairwise Term PPPPPPPP = ccccssss(ii) dddddddd(ii, ; zz) ii, zz ii tttttttttt 0.8 ccccssss = 0.7 bbbbbb 0.6

18 Richer Relationships in DPN Mixture of Label Contexts Pairwise Term ii PPPPPPPP = ccccssss(ii, ) dddddddd(ii, ; zz) ii, zz ii tttttttttt 0.8 pppppppppppp 0.6 ccccssss, = 0.2

19 Richer Relationships in DPN Mixture of Label Contexts Pairwise Term ii PPPPPPPP = ccccssss(ii, ) dddddddd(ii, ; zz) ii, zz ii tttttttttt 0.8 pppppppppppp 0.6 ccccssss, = 0.2

20 Richer Relationships in DPN Mixture of Label Contexts Pairwise Term ii PPPPPPPP = ccccssss(ii, ) dddddddd(ii, ; zz) ii, zz ii pppppppppppp 0.6 tttttttttt 0.8 ccccssss, = 0.8 Spatial Order

21 Richer Relationships in DPN Mixture of Label Contexts Pairwise Term ii PPPPPPPP = ccccssss(ii, ) dddddddd(ii, ; zz) ii, zz ii tttttttttt 0.8 pppppppppppp 0.6 ccccssss, = 0.2

22 Richer Relationships in DPN Mixture of Label Contexts Pairwise Term ii PPPPPPPP = ccccssss kk (ii, ) ii, kk ii dddddddd(ii, ; zz) zz tttttttttt 0.8 kk pppppppppppp 0.6 ccccssss, = 0.2 Mixture of Label Contexts

23 Solve High-order MRF as Convolution Pairwise Term ii PPPPPPPP = ccccssss kk (ii, ) dddddddd(ii, ; zz) ii, kk zz Mean Field Solver pp ii eeeeee UUUUUUUUUU ii + PPPPPPPP ii, pp

24 Solve High-order MRF as Convolution Iterative Updating Formula ii pp ii eeeeee UUUUUUUUUU ii + PPPPPPPP ii, pp Summation Convolution PPPPPPPP ii, : Different Types of Local and Global Filters

25 Deep Parsing Network Triple Penalty Label Contexts Unary Term Pairwise Term Convolution Deconvolution Local Convolution Max Pooling Min Pooling

26 Deep Parsing Network Unary Term Fine-tuned VGG-16 Network Convolution Max Pooling Deconvolution

27 Deep Parsing Network Original Image Ground Truth Unary Term

28 Deep Parsing Network Triple Penalty Label Contexts Unary Term Pairwise Term Convolution Deconvolution Local Convolution Max Pooling Min Pooling

29 Deep Parsing Network Triple Penalty PPPPPPPP = cccccccc kk dddddddd ii, ; zz dddddddd pp zz ii, ; zz pp zz ii, kk zz zz

30 Deep Parsing Network Triple Penalty dddddddd ; zz pp zz zz # classes z j j Local Conv Unary Term

31 Deep Parsing Network Original Image Ground Truth Unary Term Triple Penalty

32 Deep Parsing Network Triple Penalty Label Contexts Unary Term Pairwise Term Convolution Deconvolution Local Convolution Max Pooling Min Pooling

33 Deep Parsing Network Mixture of Label Contexts PPPPPPPP = cccccccc kk ii, dddddddd ii, ; zz pp zz ii, kk zz

34 Deep Parsing Network Mixture of Label Contexts Triple Penalty Result # classes class 1 class 1 j i i Min Pooling cccccccc kk ii, ttrrrr cccccccc kk ii, tttttt() cccccccc kk ii, ttttii() kk

35 Deep Parsing Network Mixture of Label Contexts Triple Penalty Result # classes class 2 class 2 j i i Min Pooling cccccccc kk ii, ttrrrr cccccccc kk ii, tttttt() cccccccc kk ii, ttttii() kk

36 Deep Parsing Network Mixture of Label Contexts Triple Penalty Result # classes class 3 class 3 j i i Min Pooling cccccccc kk ii, ttrrrr cccccccc kk ii, tttttt() cccccccc kk ii, ttttii() kk

37 Deep Parsing Network Mixture of Label Contexts Min Pooling # classes cccccccc kk ii, ttttii() kk

38 Deep Parsing Network Original Image Ground Truth Unary Term Triple Penalty Label Contexts

39 Deep Parsing Network Joint Tuning Triple Penalty Label Contexts Unary Term Pairwise Term

40 Deep Parsing Network Original Image Ground Truth Unary Term Triple Penalty Label Contexts Joint Tuning

41 Overall Performance (Published Results) FCN 62.2 DeepLab 73.9 CRFasRNN 74.7 BoxSup 75.2 DPN 77.5 (PASCAL VOC 2012 Challenge test set)

42 Label Contexts Learned favor penalty bkg areo bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv bkg areo bike tv train sofa bird boat bottle bus car cat chair cow table dog horse sheep mbike person plant mbike bike person

43 Label Contexts Learned penalty person : mbike chair : person favor

44 Challenging Case Original Image Ground Truth FCN DeepLab CRFasRNN DPN

45 Failure Case Original Ground Our Result Image Truth car

46 Conclusions General framework of one-pass CNN to model high-order MRF Various types of pairwise terms are formulated as local and global filters High performance and easy to be speeded up

47 Thanks! Semantic Image Segmentation via Deep Parsing Network Project Page: