You are here

Multicue Image and Video Segmentation: a Multi-layer MRF Framework

Lifetime from: 
1999
Lifetime to: 
2007
Short description: 
The human visual system is not treating different features sequentially. Instead, multiple cues are perceived simultaneously and then they are integrated by our visual system in order to explain the observations. Therefore different image features has to be handled in a parallel fashion. In this project, we attempt to develop such a model in a Markovian framework.
Description: 

The human visual system is not treating different features sequentially. Instead, multiple cues are perceived simultaneously and then they are integrated by our visual system in order to explain the observations. Therefore different image features has to be handled in a parallel fashion. In this project, we attempt to develop such a model in a Markovian framework. The model has a multi-layer structure (see image on the left showing the layers in the case of motion and color based segmentation): Each feature has its own layer, called feature layer, where an MRF model is defined using only the corresponding feature. A special layer is assigned to the combined MRF model. This layer interacts with each feature layer and provides the segmentation based on the combination of different features. Unlike previous methods, our approach doesn’t assume common boundaries for different features. The uniqueness of the proposed method is the ability to detect boundaries that are visible only in one of the features.

The proposed model consists of 3 layers. At each layer, we use a first order neighborhood system and extra inter-layer cliques. The image features are represented by multivariate Gaussian distributions. The combined layer only uses the features indirectly, through inter-layer cliques, thus the model is not fusioning the feature data directly, rather it combines label proposals coming from the individual feature layers. The combined layer model also estimates the number of classes and chose those label pairs which are actually present in the input image.

The proposed algorithm has been tested on real video sequences using motion and color features as well as on color textured images using color and texture features. Experimental results demonstrate that the proposed method is quite powerful in combining different features in order to detect boundaries visible only in one of them.

Results

First, we present some segmentation examples using color and texture features. More results are available here.

 


Segmentation results on color textured images

Below, we show some segmentation results based on combined color and motion features. On the sythetic example, we also show detected occlusion boundaries in black color.

 


Segmentation results on real video sequences

 


Comparison with [Khan & Shah, CVPR 2001]
Mother & daughter sequence
Multi-layer model
Khan & Shah, CVPR 2001

 


Result on a synthetic sequence


 

Publications: 
Kato Z, Pong T. A multi-layer MRF model for video object segmentation. In: Narayanan P J, Nayar S K, Shum H Y, editors. COMPUTER VISION - ACCV 2006, PT II. Springer Verlag; 2006. 9. p. 953-962p.
Kato Z, Pong T, Qiang SG. Unsupervised segmentation of color textured images using a multi-layer MRF model. In: , editor. ICIP 2003: IEEE International Conference on Image Processing. IEEE; 2003. 9. p. 961-964p.
Kato Z, Pong T, Qiang SG. Multicue MRF image segmentation: Combining texture and color features. In: Katsuri R, Laurendeau D, Suen CY, editors. Proceedings 16th International Conference on Pattern Recognition (ICPR 2002). IEEE Computer Society; 2002. 6. p. 660-663p.
Benedek C, Sziranyi T, Kato Z, Zerubia J. A multi-layer MRF model for object-motion detection in unregistered airborne image-pairs. In: , editor. Proceedings - 14th International Conference on Image Processing, ICIP 2007. Piscataway: IEEE; 2006. V. VI-p. 141-p. VI-144.
Citekey not found
Kategória: 
Markov Models
Segmentation