AUTOMATEDANATOMICALLANDMARKDETECTION ONDISTALFEMURSURFACE USINGCONVOLUTIONALNEURALNETWORK
Dong Yang. Shaoting Zhang² Zhennan Yan? Chaowei Tan Kang Li3 Dimitris Metaxas
2Department of Computer Science University of North Carolina at Charlotte NC US CBIM Rutgers University Piscataway NJ US3 Department of Industrial and Systems Engineering Rutgers University Piscataway NJ US
ABSTRACT
Accurate localization of the anatomical landmarks on distalfemur bone in the 3D medical images is very important for knee surgery planning and biomechanics analysis. Howev-er the landmark identification process is often conductedconsuming and lacks of accuracy. In this paper an automatic manually or by using the inserted auxiliaries which is time-initial geometric landmarks on femur surface in the 3D MR localization method is proposed to determine positions ofimages. Based on the results from the convolutional neu-ral network (CNN) classifiers and shape statistics we use the narrow-band graph cut optimization to achieve the 3Dsegmentation of femur surface. Finally the anatomical land-marks are located on the femur according to the geometric cues of surface mesh. Experiments demonstrate that the pro-posed method is effective efficient and reliable to segment femur and locate the anatomical landmarks.
Index Terms-- Deep learming anatomical landmark de-ture tection convolutional neural network graph cut mesh curva-
1.INTRODUCTION
Knee joint surgery e.g. knee replacement has been one ofthe most monly performed surgeries since it was intro- duced in 1968 [1]. According to the Agency for HealthcareResearch and Quality more than 600 000 people accept knee surgeries every year in United States. By the year 2030 3.48million U.S. adults are estimated to undergo total knee re-placement [2]. Accurate localization of 3D anatomical land- marks on the distal femur bone is vital to the success of theseputer-aided surgeries. Also the anatomical landmarksare important for biomechanical studies of bones and attached muscles (e.g. joint kinematics analysis).
tion process is often conducted manually or by using the in- During knee surgery procedures the landmark localiza-serted auxiliaries such as markers metallic pins [3]. Howev-tion accuracy especially in the views of 3D medical image.landmark localization.
As a result automatic bone landmark localization was intro-methods achieve high accuracy they are ot without their w duced in the recent studies [4 5 6]. Although the automaticshortings. For example they are either dependent heav-ily on initial manual localization or likely lack of geometric distinctiveness in the prediction.
As shown in Figure 1 some landmarks that carry ge-ometrically distinct information include Lateral Peak (LP) Lateral Epicondyle (LE) Lateral Distal Point (LDP) Medial Peak (MP) Adductor Magnus Tubercle (AMT) Medial Epi-condyle (ME) and Medial Distal Point (MDP). In this study. we aim to locate these seven landmarks in the 3D medicalimages.
Fig. 1. Left: Anatomical landmarks on a distal femur (from[3]). Right: One slice of 3D knee joint MR image.
In this paper we propose a novel framework to automati-cally locate femur landmarks from the 3D MR images using the convolutional neural network (CNN) and graph cut opti-mization. During the process of localization both the globalshape and local surface curvatures are taken into considera- tion because they define the geometric features of landmark-s. In what follows we describe the methodology and experi- ments discuss the results and conclude with the future direc-tions. Our study contributes to the practical application of theer these methods cannot guarantee a high degree of localiza- 3D medical image processing by improving the accuracy of
2.METHODOLOGY
of each landmark the corresponding classifier is applied tonate along one axis is determined by the index of image slice achieve a probability distribution of 2D images. The coordi-with the highest probability. Combining results from all threeclassifiers the coordinates of each landmark are obtained.
The automatic landmark localization framework containsthree consecutive steps. Firstly we detect several landmarksto calculate rigid transformation. Secondly we transform the with distinct surface curvatures from the 3D medical imagesmean shape of femur from the training pool to the initial shape of the new mage and then the initial shape is refined withthe proper optimization to obtain the segmentation. This seg-from segmentation determine the positions of anatomical mentation step is necessary since mesh curvatures obtainedlandmarks. Finally the anatomical landmarks are localized by both of their initial positions from segmentation and localsurface geometry.
2.1. Landmark Detection
The initial landmarks are chosen manually from the meanmesh of the training pool based on the geometric character- istics such as large absolute values of curvatures.
Many methods have been applied to detect 3D landmarksin medical images [7]. Two of the most monly used meth- ods are regression with random forest [8 9] and classificationwith marginal space learning and probabilistic boosting tree [10 11]. Although these two methods are able to achievea reasonable degree of accuracy and efficiency they requireture and steerable feature which might be time consuming the subjective selection of features such as the Haar-like fea-tion. Recently researchers have introduced deep convolution- and still cannot guarantee the optimal performance of detec-al neural network (CNN) to the field of automatic feature se-lection [12 13 14 15]. In the CNN the feature map of each layer is puted by the convolution of the entire image withthe same filter h . h is non-linear function (e.g. tanh) of the weights W's and the bias terms b . Then the feature map is
(1)
Fig. 2. The visual description of our method.
Sharing the same filter at each layer would reduce the mem- ory size and improve the performance. In the particular areaformance. However most of the current applications focus of image clasification the CNN has achieved excellent per-on 2D images with few exceptions in the 3D medical imagedomain mainly due to putational plexity.
There are many benefits associated with conducting thedetection in this way. First this method tends to produce stable results since the femur position is relatively stable inMR image without obvious translation rotation or scalingproblems. Second the power of CNN is fully utilized be- cause the 3D problems are converted into 2D problems. Pre-vious studies using 3D detection methods ran on every vox- el and the surrounding area. The putation plexity isO(MNPK3). M N and P are the dimensions of 3D images.K is the average side length of the 3D image cube. Compared to the previous methods our method is better in a sense thatthe plexity is O(3MNP). The last but not the least feature selection is automatically solved without the bother of design-ing the feature space. Our method only requires the configu-ration of few system parameters for instance the number of hidden layers and nodes.
In our study we explore the use of CNN in 3D medicalimage processing in a more efficient way. We treat the local- ization problem as the binary classification. During the train-ing we convert all 3D images into three sets of 2D imagesare labeled as either positive or negative based on whether with X Y Z axes respectively. For each landmark 2D imageslabels are created with n equals to the number of initial land- they contain this landmark. Therefore a total number of 3nmarks. Accordingly 3n CNN classifiers are trained with both images and their labels. During testing the new 3D image is also sliced into 3 sets of 2D images as above. For each axis
2.2. Femur Segmentation
We follow the conventional method to generate segmentationmeshes within two steps. First the mean shape is obtained by taking the average of the training meshes and then is rigidlytransformed into the new image as the initial segmentation.We use the predicted landmarks obtained from the previous steps and Procrustes analysis to calculate parameters of rigidtransformation. Procrustes analysis provides a close solution for parameters (translation rotation scaling) of shape fitting.
Second we use the narrow band graph cut optimizationfor mesh refinement. Such method guarantees the shape de- formation within the neighborhood of initialization whichp s q nas ae poou jo spq no implies the shape constraints [16 17 18 19]. The inner andinflating the initial meshes along the normal direction re-sulting point-to-point corespondences of two bounds. More points are sampled along the line segments between the cor-built by connecting the surrounding points as shown in Fig- responding points of the inner and outer bounds. Graph isure 3. This graph method is preferred since the quantity ofsampled points is controllable. Unlike the previous method- s this optimization does not run on voxel-level and thus itperforms efciently without down-sampling. After graph cut optimization the surface mesh is refined which contains on-ly few negligible defects. The principle ponent analysis(PCA) is used for smoothing the final results.
Fig. 3. Left: Inner and outer bounding layers. Right: Graphbuilt between two layers.
Fig 4. Left: Initial mesh from rigid transformation. Mid-dle: Segmentation after graph cut optimization. Right: PCArefined mesh.
2.3. Anatomical Landmark Localization
Since the segmentation mesh has point-wise correspondence with mean mesh the anatomical landmark locations on mesh
can be inferred by searching by the indices of those points onthe mean mesh. However the points with right indices can- not guarantee to be the true landmarks. Therefore we searchfor the ultimate landmark points within the neighbourhoodof previous predicted points. The points with maximum ab- solute values of curvature are determined as the anatomicallandmarks because of landmark definition in the beginning.
segmentation mesh boxes with ther colors represent diff- Fig. 5. Final detection results in 2D and 3D views (Blue isent landmark position).
3.EXPERIMENTS
The dataset are the 3D MR images of knee joints from the Os-teoarthritis Initiative (OAI) public dataset. The OAI knee M-RI protocol provides imaging data on multiple articular struc- tures and features relevant to knee OA that balances require-mension of each volume is 384 × 384 × 160 and the res- ments for high image quality and consistency [20]. The di-olution is 0.365 × 0.365 × 0.7mm. Small noise and vague boundaries exist in the images as shown in figures. In total 50volumes of them are used in the experiment and annotated byInitially the annotation data are labelled masks with a paintbrush tool they are converted into meshes for convenience.
ou pades ane sao e ‘fuun Suungslice sets along three axes X Y Z respectively for each land-mark. In each set the slices from different volumes which contains the one of landmark are labelled as positive samplespositive because they share very smilar appearanceTh ne Their neighboring slices in the range ±2.5m are also set asative samples are evenly selected from slices except positivesamples. During sampling because positive samles is con- siderably less than negative samples more positive samplecan be achieved by rotating initial positive slices with slight angle along axes from the volume and interpolating neigh-boring slices from them which increases variance of trainingpool and robustness of detection.
In our experiment the CNN contains two convolutionallayers two max-pooling layers one hidden layer and one 1- ogistic regression layer. The first convolutional layer has 207 × 7 kernels and the second has 40 7 × 7 kemels. Each convo-size 2 × 2. The hidden layer is after the second max-pooling lutional layer is followed by a max-pooling layer with kemel
Table 1. Quantitative parison.
Landmark Mean STD Max MinAMT LP 5.19 4.64 2.43 2.23 8.14 7.57 1.89 68°0MP 4.55 2.30 7.30 1.26ME LE 4.66 4.79 2.20 2.91 9.16 8.73 0.77 0.66MPD LDP 4.13 4.86 2.32 1.70 7.08 7.06 0.42 1.61
es at the end.For putational efficiency mage slices layer and it has 500 hidden neurons. The logistic regressionare down-sampled to 64 × 64 for both training and testing.
axes as independent image slice sets. The corresponding C- For testing the input volume is also sampled in X Y ZNN classifiers runs on every slice of these sets and returnprobability distributions along each axis for each landmark. The position with the highest response in distribution is set asnates are determined by bining with results from all the landmark's coordinates along one axis. Therefore the coordi-three image sets. Given initial landmark positions and theirrelated positions in mean shape Procrustes analysis provide close solution of transformation parameters scaling rotation translation) for fitting mean shape in the volume. Next the in- ner and outer bounding layers are generated by shrinking orinflating along the vertex normal directions for 5.0rmm. 20corresponding vertices of inner and outer bounds for buildingsearched in neighborhood with maximum distance 2.Omm af- the graph. The final positions of the anatomical landmarks areter graph cut optimization.
mark localization. The average time of training one CNN Table 1 shows the evaluation results of anatomical land-rectly dependent on the quantity of landmarks. The average classifier is about 30 minutes. The total training time is di-time of testing is around 90 second 80% of which is con-MatLab and Python on a machine with a 2.66 GHz CPU and sumed by running the CNN classifiers. All programs run in4 GB memory.
4.CONCLUSION
In this paper we proposed a novel framework to locate sevenanatomical lndmarks of the distal femurbone. Our approachis automatic and it bines both global shape information and local mesh curvatures. There are several directions forfuture research work. One possible direction is to enlarge training variance and do cascade detection for higher accura-cy. This work can also be extended to locate other anatomicalimages with difference modalities. landmarks for other bones (e.g. tibia) or organs in medical
5.REFERENCES
[1] Mike D Van Manen James Nace and Michael A Mon-t *Management of primary knee osteoarthritis and in-tioners JAOA: Journal of the American Osteopathic dications for total knee arthroplasty for general practi-Associarion vol. 112 no. 11 pp. 709715 2012.[2] Hans C Dreyer Lisa A Strycker Hilary A Senesac Brian A Jewett et al. “Essential amino acid supple- Austin D Hocker Keith Smolkowski Steven N Shah mentation in patients following total knee arthroplasty " The Journal of clinical invesrigation vol. 123 no. 11 pp. 2013.[3] K Subburaj. Bhallamudi Ravi and Manish Agarwal 3d bone models reconstructed from ct scan images " “Automated identification of anatomical landmarks onComputerized Medical Imaging and Graphics vol. 33 no. 5 pp. 359368 2009.[4] Daniel Bystrov Vladimir Pekar Stewart Young Sebas-tian PM Dries Harald S Heese and Arianne M van jo sus jo ud pn n joints a in Medical Imaging. International Society forOptics and Photonics 2007 pp. 65092Z65092Z.[5] Seung-Yeob Baek Joon-Ho Wang Insub Song Kun-bone landmarks prediction on the femur using anatom- woo Lee Jehee Lee and Seungbum Koo “Automatedvol. 45 no. 2 pp. 505510 2013. ical deformation technique " Computer-Aided Design [6] Ning Xue Michael Doellinger Charles P Ho Rachel KSurowiec and Raphael Schwarz “Automatic detectionof anatomical landmarks on the knee joint using mri da- ta. ? Journal of Magnetic Resonance Imaging 2014.[7] Yiqiang Zhan Xiang Sean Zhou Zhigang Peng andtion and segmentation in whole-body medical images " Arun Krishnan "Active scheduling of organ detec-in Medical Image Conpuring and Computer-AssistedIntervenrionMICCAI 2008 pp.313321. Springer 2008.[8] Nathan Lay Neil Birkbeck Jingdan Zhang and S KevinZhou “Rapid multi-organ segmentation using contex-tion Processing in Medicl Iging. Spinger 2013 pp. t integration and discriminative models * in Informa-450462.[9] Dong Han Yaozong Gao Guorong Wu Pew-Thian Yap and Dinggang Shen *Robust anatomical landmarkdetection for mr brain image registration " in MedicalMICCAI 2014 pp. 186193. Springer 2014. Image Computing and Computer-Assisted Intervention
[10] Yefeng Zheng Matthias John Rui Liao A Nottling Janmann and Dorin Comaniciu “Automatic aorta segmen- Boese Jorg Kempfert Thomas Walther Gernot Brock-tation and valve landmark detection in c-arm ct for tran-scatheter aortic valve mplantation Medica Imaging. IEEE Transactions on vol. 31 no. 12 pp. 2012.[11] Adrian Barbu Michael Suehling Xun Xu David Li- u S Kevin Zhou and Dorin Comaniciu “Automaticin Medical Image Computing and Computer-Assisted InterventionMICCAI 2010 p. 2836. Springer 2010.[12] Yann LeCun Leon Bottou Yoshua Bengio and PatrickHaffner “Gradient-based learning applied to document recognition " Proceedings of the IEEE vol. 86 no. 11 Pp 1998.[13] Alex Krizhevsky Ilya Sutskever and Geoffrey E Hin-ton “Imagenet classification with deep convolutional neural networks " in Achvances in nearal informationprocessing systems 2012 pp. .[14] Dan Ciresan Ueli Meier and Jurgen Schmidhuber “Multi-column deep neural networks for image classi-(CVPR). 2012 IEEE Conference on. IEEE 2012 pp. fication a in Computer Vision and Partern Recognition36423649.[15] Christian Szegedy Alexander Toshev and Dumitru Er-vances in Neural Ieformation Processing Systems 2013 pp. .[16] Yefeng Zheng Dong Yang Matthias John and Dorin jo us pumd-left atrium in c-arm ct for image-guided ablation of atrial[17] Dong Yang Yefeng Zheng and Mathias John “Graph fibrillation ” 2013.spsqnmiddle pulmonary vein extraction in c-arm ct " in SPIE Medical Imaging. International Society for Optics andPhotonics 2013 pp. 86693U86693U.[18] Yuri Y Boykov and M-P Jolly “Interactive graph cuts for optimal boundary& region segmentation of objectsin nd images " in Computer Vision 2001. ICCV 2001.IEEE 2001 vol. 1 pp. 105112. Proceedings. Eighth IEEE Intermational Conference on.[19] Mustafa Gokhan Uzunbas Chao Chen Shaoting Zhang ap q uos o Kilian M Pohl Kang Li and Dimitris Metaxas “Col-formable and graphical models "in Medical Int-MICCAI 2013 pp. 157164. Springer 2013. age Conpating and Computer-Assisted Inervention
[20] CG Peterfy E Schneider and M Nevitt “The os-teoarthritis initiative: report on the design rationale forthe magnetic resonance imaging protocol for the knee " Osteourhritis and Cartilage vol. 16 no. 12 pp. 2008.
