Segmentation Using Deformable Models and Shape Priors

Problem definition: Our previous segmentation framework in Siemens is based on deformable models and detection methods. Several landmarks are detected, and a reference shape is transformed (either similarity or affine) to fit landmarks. This shape is used as an initialization. Then segmentation is performed using boundary detections and constraints of smoothness. It is similar to a combination of Snakes[1] and boosted cascade method[2], and achieves very good performance in many segmentation applications[3]. However, there is no shape constraint from the training data (like Active Shape Model does[4]). Thus the system is less stable as it may result in undesirable results in some cases. I have added such shape prior constraints to improve its performance. There are some other limitations of the current framework. First the shape quality (especially in 3D) may be degenerated, which adversely affect the training performance of landmark and boundary detections. Second, the global transformation may not exactly fit the landmark positions. I also tried to solve these problems. Here are some details for my intern work in Siemens:

Geometry improvements: Implemented a Python script with C++ wrappers which supports mesh decimation, smoothing, isotropic remesher and local deformation (TPS or LSE). The script also relies on VTK, OpenMesh and OpenFlipper. After optimizing the mesh quality using this script, the training and segmentation performance is improved. The global transformation can be replaced by the local deformation method, when landmark detections are generally correct. Using local deformation can fit landmark positions better.

ASM shape constraint: Implemented a Python script of 3D Active Shape Model (the pose and shape estimation part, not the boundary detection part). This constrain certainly brings in stableness. However, this approach has some limitations: (1) shape variation is complex and cannot always be modeled by a parametric probability distribution; (2) a shape instance derived from image appearance cues (input shape) may have gross errors; and (3) local details of the input shape are difficult to preserve if they are not statistically significant in the training data.

Sparse representation based shape constraints: Proposed an approach using sparse representations to alleviate the above-mentioned problems, and implemented this idea with Python (use CVXMOD as the solver). This method is based on two assumptions: 1) the input shape can be approximately represented by a sparse linear combination of shapes from training data; 2) the input shape may contain errors or outliers, but such outliers are sparse. Our framework can jointly recover the "clean" version of the input shape and detect the outliers.

Other work: I also implemented some other tools, including (1) a semi-automatic annotation tool (using Matlab) for X-Ray image; (2) a visualization tool (using C++) which shows volume data with multiple surface meshes; (3) a visualization tool which loads a sequence of meshes and displays them like animation. Furthermore, I also participated in the project of auto align system using adaptive multi-structural atlas construction (with Ting Chen).

Papers and patents

Shaoting Zhang, Yiqiang Zhan, Maneesh Dewan, Junzhou Huang, Dimitris N. Metaxas, Xiang Sean Zhou: Towards robust and effective shape modeling: Sparse shape composition. Medical Image Analysis 16(1): 265-277 (2012)
Shaoting Zhang, Yiqiang Zhan, Maneesh Dewan, Ting Chen and Xiang Zhou: System and Methods for Robustly Learning Shape Statistics Using Sparse Representation, invention disclosure, Docket Number: 2010E20120 US.
• Ting Chen, Maneesh Dewan, Yiqiang Zhan, Shaoting Zhang and Xiang Zhou: Adaptive Multi-Structural Atlas Construction for Auto Align System, invention disclosure, Docket Number: 2010E17098 US.

2D Lung Segmentation from X-Ray

Figure 1. Demonstration of anomaly detection. There are two detection errors, which adversely affect the performance of baseline system (Procrustes analysis) and Active Shape Model.

Figure 2. Discover detail information.

3D Liver Segmentation from Low-dose CT

Figure 3. Mesh partitioning using affinity propagation. This local shape setting can learn detail information better, and is more efficient for the optimization solver.

Figure 4. 3D initialization based on landmark detection. Comparison between the baseline system (Procrustes analysis) and sparse representation.

Figure 5. 3D deformation based on boundary detection. Comparison between the baseline system (Procrustes analysis), Active Shape Model and sparse representation.


[1]. M. Kass, A. Witkin and D. Terzopoulos, Snakes: Active contour models, IJCV 1988.
[2]. P. Viola and M. Jones, Rapid Object Detection Using a Boosted Cascade of Simple Features, CVPR 2001.
[3]. Y. Zhan, M. Dewan, X. Zhou: Cross Modality Deformable Segmentation Using Hierarchical Clustering and Learning. MICCAI 2009
[4]. T. Cootes, C. Taylor, D. Cooper, and J. Graham. Active shape model - their training and application. CVIU, 1995.

Other Tools -- Visualization Interface

Figure 6. Visualization tools. It is written using C++, VTK, ITK and QT. It displays volume data with surface meshes.

Other Tools -- 2D Annotation

Figure 7. 2D annotation tools. It is written using Matlab.