Construction of 3D model of knee joint motion based on MRI image registration

: There is a growing demand for information and computational technology for surgeons help with surgical planning as well as prosthetics design. The two - dimensional images are registered to the three dimensional ( 3D ) model for high e ﬃ ciency. To reconstruct the 3D model of knee joint including bone structure and main soft tissue structure, the evaluation and analysis of sports injury and rehabilitation treatment are detailed in this study. Mimics 10.0 was used to reconstruct the bone structure, ligament, and meniscus according to the pulse di ﬀ usion - weighted imaging sequence ( PDWI ) and stir sequences of magnetic resonance imaging ( MRI ) . Excluding congenital malformations and diseases of the skeletal muscle system, MRI scanning was performed on bilateral knee joints. Proton weighted sequence ( PDWI sequence ) and stir pulse sequence were selected for MRI. The models were imported into Geomagic Studio 11 software for re ﬁ nement and modi ﬁ cation, and 3D registration of bone structure and main soft tissue structure was performed to construct a digital model of knee joint bone structure and accessory cartilage and ligament structure. The 3D knee joint model including bone, meniscus, and collateral ligament was established. Reconstruction and image registration based on mimics and Geomagic Studio can build a 3D model of knee joint with satisfactory morphology, which can meet the requirements of teaching, motion simulation, and biomechanical analysis.


Introduction
Knee osteoarthritis (OA) is a common chronic joint disease that causes knee pain, deformity, and even disability in elderly patients. Its main feature is joint cartilage degeneration caused by mechanical and biological factors, and then a series of pathological changes, such as joint space stenosis, osteophyte proliferation, meniscus degeneration, and ligament contracture. Accurate understanding of the biomechanical characteristics of the OA knee joint is of great significance to prevent the occurrence of OA, delay its progress, and guide clinical treatment [1,2]. Finite element analysis based on a three-dimensional (3D) model of the knee joint is an effective tool to study the biomechanical characteristics of knee OA. Previous studies are mostly based on the normal knee joint 3D model construction, because the knee joint anatomical structure did not have pathological changes, so the construction of the knee joint model is relatively easy and accurate [3]. At present, there is no literature report on the construction of a 3D model based on knee OA, which is difficult to explore the biomechanical behavior changes of knee OA through finite element analysis, to guide clinical treatment. In this study, computed tomography (CT) and magnetic resonance imaging (MRI) image data registration and fusion method were used to construct an individualized knee OA 3D anatomical model, which can accurately display the specific anatomical characteristics of knee joint bone tissue and non-bone tissue and provide accurate knee OA anatomical model for further biomechanical research [4,5].
During orthopedic surgery, 3D imaging is enormously appreciated for the exact recovery of estimation of bone poses. The MRI, CT, and positron emission tomography (PET) are the most common 3D imaging techniques having high resolution and large information about the anatomical structure [6,7]. Generation of 3D imaging is time consuming and during the intervention, it cannot be implemented. The up-to-date anatomical state is provided by the intra-interventional images during the intervention. The generation of intra-interventional images is fast so that the changes caused by the procedure are reflected. Also, less radiation dose is produced by these intra-interventional imaging techniques [8]. The existing registration is rigid as it requires the images such as MRI and CT. The 3D modalities are time consuming and also utilize extra radiation doses on patients. As the 3D model is recovered from the statistical shape model (SSM), it is beneficial to use non-rigid two-dimensional (2D)/3D registration. No CT or MR is required once the SSM is constructed. The development of a 3D reconstruction system from a sequence of X-ray images is done as this is generic and easily customizable [9][10][11]. There are four main parts of the system, initialization, feature extraction, optimization and 3D shape analysis, as presented in Figure 1.
The region of interest is extracted accurately by the segmentation technique in the module of the feature extraction. The spectral clustering is used to enhance the 2D radiograph based on affinity matrix Eigen solution. The gradient search is then generating the candidate contour on the normal contour points. This technique is quite flexible, and with the high variation, the shapes are discriminated. The procedure of knee MRI is presented in Figure 2.
The knee bone contour was automatically extracted as a feature extraction output module. The elliptical fourier descriptors are normalized to calculate the shape descriptors for the contour shape representation. A hybrid classifier then determines the optimal pose by combining k-nearest neighbors and a support vector machine. The SSM is built in the module of 3D shape from a surface mesh models dataset. From the parameters of the selected shape, the 3D model is recovered. The similarity between the 3D and 2D radiograph is measured by the novel energy function that includes the edge, region, and homogeneity in the optimization module.

Contribution
This study uses the image registration technology of Geomagic Studio, which is the image reconstruction function of mimics, and based on CT and MRI data, to reconstruct the structure of knee joint such as bone, articular cartilage, and ligament and establish a digital model of knee joint with comprehensive tissue information in morphology, which provides a perfect basic model for digital simulation research of knee joint and surgical teaching. The organization of this article is as follows. Section 2 provides an overview of the exhaustive literature survey followed by a methodology adopted in Section 3. A detailed discussion of obtained results is in Section 4. Finally, Section 5 concludes the article.

Literature review
Different from the traditional 3D CT or MRI images, the 3D model based on CT and MRI data further expands the engineering research of knee joints. Yang et al. built a 3D digital knee joint model based on MRI and CT and measured and analyzed the femoral condyle torsion angle [12]. Schmutz et al. used the 3D model of tibia derived from CT and intramedullary nail to simulate the positioning of the implant and evaluate the different designs of intramedullary nail quantitatively. Feng et al. detailed the digital model of normal knee joint bone structure and its accessory cartilage and ligament structure based on CT and MRI hybrid registration. At present, there are few studies on the 3D model of knee OA joint [13]. The literature reports mostly use normal knee joints to simulate meniscal tears, cartilage defects, and other diseases. In this study, bone structure and non-bone structure were constructed based on CT and MRI images of knee OA joint, and the spatial relationship between bone structure and soft tissue can be accurately located by fusing CT and MRI images at multiple points in the same layer. It can observe the lesion at any angle and measure the morphological parameters in 3D. It also provides a reliable model for the biomechanical study of knee OA joints. Based on the registration and fusion method of CT and MRI image data, we can construct the anatomical model of knee OA joint including a variety of anatomical structures [14,15]. However, in the whole modeling process, from image acquisition to diagnostic analysis and construction, the loss of target information is inevitable. Further research is needed. The knee joint is an important bone connection in the human motion system [16]. The establishment of a 3D computer model of the knee joint is helpful to understand the shape, mutual position, and connection mode of the femur, tibia, patella, and meniscus, which has direct guiding significance for clinical knee surgery or knee prosthesis design. At present, there have been reports on virtual simulation of knee joints at home and abroad, but there is no report on the precise integration and real reduction of the relative spatial position between bone structure and soft tissue structure [17]. The authors detail that computer-based information technologies are in high demand for surgeon help with computer-aided as well as prosthetics design [18]. The (2D) intra-operative images are registered to a 3D model. The novel non-rigid 2D/3D registration framework is described in this article for 3D surface mesh model recovery from 2D fluoroscopic X-ray images. The training dataset of 3D surface mesh models is represented by the 3D Shape Analysis. A novel, clinically relevant 2D/3D registration framework is described, and a general approach is provided to solve the medical image registration problems. In this article, the authors describe the image segmentation approach for 3D model generation from CT scan data required in orthopedics areas [19]. The 3D model of the knee CT data is created to find the optimal segmented image. The different segmentation algorithms are validated and compared, and the finite elements are analyzed. The authors present the MRI data of knee joint patients, and the kinematic simulation is presented [11]. The compressive load measured "in vivo" is derived by the finite element knee model, and the optical force measurement system is developed. Evaluation of the predicted kinematics is done by obtaining the high-quality model at fixed flexions. An accurate model of the human knee joint is established, and the knee joint finite element analysis is done; this article is based on the biomechanics theory with the help of professional modeling software [20]. It provides the 3D geometric model modeling and materialization processing method of the knee joint for the total knee joint model establishment. The global budget payments, the diagnosis-related group (DRG) inpatient DRG payment system, are implemented by the National Health Insurance Administration to decrease the financial burden of medical expenditure [21]. In this study, the model was constructed by the authors based on the exponentially weighted moving average algorithm characteristics for control chart development.

Image data acquisition
A 58-year-old female patient with knee OA had a left knee joint injury and no history of surgery. CT (Lightspeed VCT; GE, USA) and MRI (signa3.0t; GE, USA) imaging data of the left knee joint were collected. The collected data parameters are shown in Table 1. In the process of collecting the image data, the patient took the supine position, the knee brace fixed flexion 15°, and the image data were stored in DICOM format. This study has been approved by the hospital ethics committee, University of Tennessee Anthropology Department, and informed consent has been signed with patients and their families [22,23].

Image data analysis
MRI and CT tomographic data in DICOM format were imported into mimics 19.0 (material's interactive medical image control system, material, Belgium) software. Joint surgeons read the film layer by layer, and combined with the image data, the knee joint was diagnosed and analyzed. When the meniscus of the normal knee joint is in a coronal position, the outer edge of the meniscus is close to the outer edge of the tibial plateau. When the meniscus is dislocated, its coronal plane and sagittal plane present irregular boundary contour [24][25][26]. In the coronal section showing the medial collateral ligament, tangent lines were made along the medial border of the tibial plateau and the medial meniscal synovial margin, respectively. The horizontal distance between the two tangents was used as an index to judge the distance of meniscus dislocation. When the distance L ≥3 mm, it was diagnosed as meniscal subluxation as shown in Figure 3.

The 3D model construction
Generation of 3D imaging is time consuming, and during the intervention, it cannot be implemented. The up-to-date anatomical state is provided by the intra-interventional images during the intervention. Generation of intra-interventional images is fast so that the changes caused by the procedure are reflected. Reconstruct the 3D model of knee joint including bone structure and main soft tissue structure. It provide a theoretical basis for morphological study of the human knee joint, evaluation, and analysis of sports injury and rehabilitation treatment. Mimics 10.0 wase used to reconstruct the bone structure, ligament, and meniscus according to the PDWI and stir sequences of MRI. Excluding congenital malformations and diseases of the skeletal muscle system, MRI scanning was performed on bilateral knee joints. Proton weighted sequence (PDWI sequence) and stir pulse sequence were selected for MRI.

Construction of 3D model of bone structure
The DICOM format data of CT images were imported into Mimics software. The software established "masks" based on threshold segmentation of gray values. The threshold of bone and soft tissue was clearly distinguished [27,28]. The threshold was adjusted to exclude soft tissue, and edit masks layer by layer to remove the non-target bony structure produced by the similar gray value of bone edge. The "region growing" tool was used to generate the masks of the distal femur, proximal tibia, proximal fibula, patella, and fibula, respectively, and the unlabeled part of the bone mask was manually completed. The 3D model of the knee joint was constructed by region growing and masks editing.

3D model construction of non-bone structure
3D models of articular cartilage and meniscus were constructed by sagittal T2WI images, and 3D models of medial and lateral collateral ligament, anterior and posterior cruciate ligament, and patellar ligament were constructed by T2WI images of MRI cross-section. When processing MRI tomograms, Mimics software needs to manually adjust the threshold value to distinguish the tissue boundary and then manually edit the mask according to the anatomical characteristics to define the contour boundary of the target tissue [27,29].
Combined with operation strategies such as mask segmentation and Boolean, the construction cycle is effectively reduced. Finally, according to the anatomical characteristics of the knee joint, the image data were analyzed and compared layer by layer to ensure that the anatomical characteristics of the OA knee were reflected in the anatomical model.

Registration and fusion of 3D models of skeletal and non-skeletal structures
2D image feature level registration schemes based on CT and MRI were used in this study, including Mr + Mr on-board registration fusion and CT + Mr different machine registration fusion. After registration and fusion, the 3D model still retains the original 2D image coordinate relationship. To accurately assemble the 3D models of bone structure and non-bone structure based on CT and MRI, specific reference marks were selected as the registration mark points [30,31]. The feature points include in vitro marking points and anatomical characteristics of knee joints. The 28 vitamin E capsules were fixed with adhesive tape in vitro, and the center point of the capsules was selected as the registration mark in vitro as shown in Figure 4.
According to the specific bone structure of the subjects, the registration points of multiple registration images were selected and displayed on the same level with the corresponding feature points in the images to be registered as shown in Figure 5. Through this registration method of multi-point and multi-point in the same layer, the additive fusion algorithm is selected, and the software will automatically overlay the registration images.
The 3D models of bone and non-bone tissue constructed based on CT and MRI 2D image data can be assembled together to form a complete 3D model of knee OA joint. The knee joint modeling and process in this work are presented in Figure 6.
The imaging technique NMRI is scanning equipment, and it is a highly high-resolution imaging technique. The high gradient field and high strength are the characteristics. The image resolution and the signal-to-noise ratio are improved by the technique. The data are collected from the local hospital for the experiment purpose. 4 Results and discussion 4.1 Significance of image analysis before modeling for building 3D anatomical model Different from the 3D modeling process of normal knee joint, knee OA modeling should first identify the lesion location and extent of the knee joint, especially the soft tissue lesion. MRI has become the preferred non-invasive imaging method for evaluating the soft tissue lesions of knee joints. It is very important to understand the anatomical characteristics and anatomical variations of non-bony structures in MRI images of knee OA patients, thus ensuring the accuracy of the anatomical model. Accurate diagnosis and analysis can guide engineers or doctors to accurately segment the mask during modeling. However, CT has a high resolution for bony structures (including osteophytes), which can clearly distinguish the outline of bony structures. The software automation degree and reconstruction accuracy of the construction process are high. The trained engineers can build 3D models of bone structures independently. It is difficult for engineers with simple anatomical knowledge to construct the mask of complex structure when segmenting non-osseous structural masks, while experienced radiologists or joint surgeons are more suitable for constructing non-osseous structure models. When meniscus injury and cartilage wear occur in the knee joint, it shows a high signal in coronal T2 weighted image sequence and can be accurately diagnosed. Therefore, the MRI image analysis before modeling makes the modeling engineers or doctors more clear about the  injury characteristics of the meniscus and articular cartilage of the subjects, to reduce the artificial error accumulated in image segmentation and improve the image quality [32]. However, when measuring the distance of meniscus dislocation, the K-L classification of the patient's knee joint is grade III, the joint space is narrow, and the marginal osteophytes of tibiofemoral joint and patellofemoral joint are obvious. To prevent measurement error caused by periarticular osteophytes, the thickness of periarticular osteophytes should be excluded in the measurement of meniscus dislocation distance.

Fusion registration based on CT and MRI multimodal image data
In medical image analysis, registration technology is used to establish meaningful spatial correspondence between medical images with different imaging principles, which is widely used in computer-aided diagnosis and interventional therapy. The bony structure of the knee joint is obvious and invisible in CT and MRI images [33,34]. The marker points in the image are associated with the bony structure derived from CT, which makes the landmark keep a constant spatial relationship with the bony structure derived from CT. This correlation is also true for MRI-derived models. Because some anatomic features are not constant or it is difficult for individuals to label, the first choice for image registration is to mix in vitro markers and anatomical features. The results showed that the selection range of intraoperative landmarks in knee OA was wider, and osteophytes with specific shapes were easier to be marked. Moreover, the occurrence rate of gastrocnemius bone in posterolateral structure was significantly higher than that in the normal knee joint. Therefore, the gastrocnemius bone can be used as the first choice in vivo registration feature points in CT + MR heterogeneous fusion. Sagittal and cross-sectional T2WI images were registered by MR + MR in the same machine. The 3D model after fusion had ligament and articular cartilage at the same time. MRI image acquisition occurs on the same equipment at the same time, which ensures the coordination of registration data on the same machine. The selection order of registration and image to be registered has no difference in actual registration results. However, the imaging principle and scanning parameters of CT + Mr fusion are different, and the maximum, minimum, and average algorithms in the software are difficult to accurately establish the corresponding relationship of pixels. However, the method of additive superposition is used to stack the nuclear magnetic resonance images with less relative layers into the CT image sequence. In the process of image acquisition, the patient cannot completely fix the knee joint, and the slight internal soft tissue activity will cause the registration error of CT + MR heterogeneous fusion. The external vacuum thermoplastic membrane brace can effectively reduce the error caused by soft tissue activity. After registration, the bone structure model derived from CT is moved to make the contour in the section consistent with the MRI image and improve the accuracy of CT + MR combination model. An active shape model (ASM) extracts the area of interest in X-ray fluoroscopy. The ASM is the segmentation algorithm representing the global shape constraints. The search in feasible shape is constrained by the ASM. It is beneficial to avoid the segmentation of foreign occlusions. The extraction of the training data is done from the binary images onto the 2D plane at different poses. The corresponding points of each contour are then found to align the contours. In the training data set, the average contour is considered for the segmentation and then the constrained points for the segmented contour are relaxed. With a half profile length, the normal profile gradient search is used for the relaxation. The comparison of the original ASM algorithms is done with the original ASM with relaxation, and spectral clustering, with respect to the correct boundary of the femur. The RMS, its deviation, and maximal error are shown in Table 2 and presented graphically in Figure 7.
The automatic segmentation qualitative results are presented in Figure 8, it is shown in the images, the ground truth images are in green color, and the automatically segmented results are in red. The segmented and ground truth contour similarity is compared by the dice measure. The segmented femur contour as measured by the dice similarity is 0.9494 ± 0.0089 mm, and the tibia contour segmented similarity is 0.9803 ± 0.0092 mm.
After segmentation, the fusion of MRI image data is compressed for telemedicine by the predictive coding technique in which gradient edge detector (GED) is used for prediction and arithmetic encoder is applied for encoding. GED is used because it is easy to implement and also efficient as compared to other predictors. It provides a minimum entropy value of the error image that is further encoded by the encoder. Some other performance parameters such as peak signal-to-noise ratio (PSNR), similarity index (SSIM), and dissimilarity index (DSIM) of predicted images are also analyzed as shown in Table 3.
A higher value of PSNR shows the better quality of the image. It is seen that the PSNR of the predicted image is in the range of 20-35 dB. For better analysis and visualization, the PSNR values for different image samples are shown in Figure 9.
The similarity and dissimilarity index values are also presented graphically in Figure 10 for better analysis and visualization purposes.    SSIM is also a quality parameter of an image that shows the similarity between the predicted image and the original image. The range of SSIM is from 0 to 1. SSIM of almost all the images is above 0.8 similarity index. So it is clear that there is a little difference in predicted images and corresponding original images. DSIM is the opposite of SSIM means lower the DSIM more will be the similarity of the predicted image and original image.

Conclusion
Based on the method of registration and fusion of MRI image data, the anatomical model of the knee joint with various anatomical structures can be constructed. However, in the whole modeling process, from   image acquisition to diagnostic analysis, the loss of target information is inevitable. The 3D reconstruction of knee joint model based on PDWI, mimics of STIR sequence, and Geomagic Studio based on MRI still lacks accuracy due to the limitation of current imaging equipment and reconstruction software. However, it has been able to present the human knee joint structure more realistically, which will be helpful for the subsequent analysis of the mechanical properties of the knee joint. After segmentation and reconstruction, the image samples are also compressed for telemedicine applications. PSNR of the predicted image is in the range of 20-35 dB, and the SSIM of almost all the images is above 0.8 similarity index. The segmented and ground truth contour similarity is compared by the dice measure. The segmented femur contour as measured by the dice similarity is 0.9494 ± 0.0089 mm and the tibia contour segmented similarity is 0.9803 ± 0.0092 mm. In future research, the predictive GED can be utilized further for bit rate reduction that facilitates the effective transmission of the medical data.
Funding information: None.