Skip to main content
  • Original Article
  • Open access
  • Published:

Advanced 4-dimensional cone-beam computed tomography reconstruction by combining motion estimation, motion-compensated reconstruction, biomechanical modeling and deep learning


4-Dimensional cone-beam computed tomography (4D-CBCT) offers several key advantages over conventional 3D-CBCT in moving target localization/delineation, structure de-blurring, target motion tracking, treatment dose accumulation and adaptive radiation therapy. However, the use of the 4D-CBCT in current radiation therapy practices has been limited, mostly due to its sub-optimal image quality from limited angular sampling of cone-beam projections. In this study, we summarized the recent developments of 4D-CBCT reconstruction techniques for image quality improvement, and introduced our developments of a new 4D-CBCT reconstruction technique which features simultaneous motion estimation and image reconstruction (SMEIR). Based on the original SMEIR scheme, biomechanical modeling-guided SMEIR (SMEIR-Bio) was introduced to further improve the reconstruction accuracy of fine details in lung 4D-CBCTs. To improve the efficiency of reconstruction, we recently developed a U-net-based deformation-vector-field (DVF) optimization technique to leverage a population-based deep learning scheme to improve the accuracy of intra-lung DVFs (SMEIR-Unet), without explicit biomechanical modeling. Details of each of the SMEIR, SMEIR-Bio and SMEIR-Unet techniques were included in this study, along with the corresponding results comparing the reconstruction accuracy in terms of CBCT images and the DVFs. We also discussed the application prospects of the SMEIR-type techniques in image-guided radiation therapy and adaptive radiation therapy, and presented potential schemes on future developments to achieve faster and more accurate 4D-CBCT imaging.


Accurate tumor/target localization is key to safe, precise and effective radiotherapy [1]. Cone-beam computed tomography (CBCT) imaging has become a standard-of-care in a majority of the radiotherapy clinics, with its successful capture of volumetric anatomical information to guide accurate on-board target localization and setup correction [2, 3]. In principal, the 3-dimensional CBCT (3D-CBCT) technique acquires 2D cone-beam projections from varying beam angles, usually covering an angle span of at least 200° [4], to reconstruct 3D volumetric information. Compared with 2D projections, the overlaying effects have been removed by 3D-CBCT to allow better soft-tissue contrast and more accurate structure localization in 3D [2]. However, there are multiple remaining issues with the current CBCT technique. One major issue is the motion-induced imaging artifacts and blurring for motion-involved sites such as lung and liver, which may severely reduce the tumor localization accuracy [5, 6]. The conventional 3D-CBCT technique acquires cone-beam projections and reconstructs them into a single CBCT volume without considering the motion of anatomical structures across the projections. Due to the position variations of anatomical structures, combining all projections into one reconstruction will lead to motion blurriness and inaccurate target localization. To effectively suppress the motion blurriness, methods have been proposed to develop the respiratory-correlated CBCT, also known as 4-dimensional CBCT (4D-CBCT), to capture the motion in the fourth dimension (in addition to the three spatial dimensions) [7,8,9]. The key idea of 4D-CBCT is phase-sorting [10], which sub-groups the cone-beam projections into different respiratory phase bins according to their relative positioning on a nominal respiratory cycle. The relative positioning can be determined via an external landmark, such as the real-time position management system or the Anzai Belt [11, 12]; or via an internal landmark, such as the diaphragm position or the fiducial marker position [7]. More advanced signals based on Fourier transform have also been investigated to be successful [10]. Based on these positioning signals, projections were sorted into different bins. Within each phase bin, the projections are semi-static with minimal intra-phase motion. CBCTs reconstructed from these phase-sorted projections will have the motion blurriness subdued, leading to more accurate delineation and localization of anatomical structures. Stacking these phase-specific CBCT images together will also reveal the full motion trajectory of anatomical structures to guide tumor targeting and organ-of-risk sparing. Though conceptually concise and convenient to implement, the use of 4D-CBCT in clinics is currently limited. One major concern is that the phase-sorting process employed by 4D-CBCT leads to insufficient projections inside each phase bin (angular under-sampling), causing severe imaging streaking artifacts to the clinical CBCT reconstruction technique, the Feldkamp-Davis-Kress (FDK) algorithm [13, 14].

One straightforward solution is to acquire more projections to achieve sufficient angular sampling even after phase-sorting [4]. Such a strategy, however, is not practical due to the accompanying excessive imaging dose, which may induce secondary cancers [15]. Acquiring more projections also prolongs the imaging and treatment time, which may increase patient on-board position deviations [16], and add additional logistic burdens to the clinics. Many groups have tried to develop new reconstruction techniques, mostly iterative in nature, to improve the 4D-CBCT image quality from under-sampled acquisition. Many of these techniques rely on regularization techniques via metrics like total variation [17, 18], non-local means [19, 20] or wavelet frames [21] to explore the data sparsity and reduce the noises/artifacts of under-sampling. Substantial image quality improvement has been observed, and quantitatively validated via metrics like signal-to-noise ratio, root-mean-squared-error (RMSE) or universal quality index (UQI) [22]. However, these techniques are susceptible to displace or smooth out fine details and low-contrast anatomies from the reconstructed images.

Another type of CBCT reconstruction technique tries to incorporate prior information into the reconstruction process, such as the 2D-3D deformation method [23,24,25,26,27,28]. Instead of directly reconstructing CBCT images from acquired projections, the 2D-3D deformation method views the new CBCT volume as a deformation of prior CT/CBCT images, and translates the CBCT reconstruction into a deformation-vector-field (DVF) optimization problem. For each to-be-deformed CT/CBCT image, the DVFs usually compose of three matrices in 3D, each matrix with the same dimension as the to-be-deformed CT/CBCT image. The three matrices indicate the deformation along the three Cartesian directions, x, y and z, respectively [25]. The 2D-3D deformation algorithm iteratively optimizes the DVF, such that it deforms the prior image until the digitally-reconstructed-radiographs (DRRs) of the deformed image match the acquired cone-beam projections. The 2D-3D deformation technique can not only generate 4D-CBCT images, but provide DVFs to potentially allow automatic tumor localization, target tracking, dose accumulation and adaptive radiation therapy [29,30,31,32]. The incorporation of high-quality prior information also introduces more accurate Hounsfield units [30], and allows further imaging dose reduction by acquiring fewer projections for reconstruction. However, this technique cannot reconstruct non-deformation-induced intensity changes in new CBCT volumes, since the new CBCT is simplified as a purely deformed volume of the prior image [33].

In addition to these two types of techniques, motion-compensated reconstruction is another reconstruction technique to address the 4D-CBCT under-sampling issue [34,35,36,37]. The motion-compensation technique reconstructs a reference phase CBCT image through an estimated inter-phase motion model [34]. The motion model relates other phases of the 4D-CBCT to the reference phase in the form of inter-phase DVFs. Through applying the inter-phase DVFs, motion-compensated reconstruction can combine projections from all phases to achieve sufficient angular sampling. After the reference phase is reconstructed, the other phases can be derived using corresponding inverse DVFs. Unfortunately, the motion-compensation technique is limited by the inter-phase motion model accuracy, since a prior-information-driven motion model can be invalidated after motion pattern changes [25, 34, 36]. To have the model up-to-date, studies have proposed to estimate an on-board motion model directly between 4D-CBCT phase images reconstructed from under-sampled projections [35, 37]. However, the accuracy of these on-board motion models can still be impaired by the artifacts presented in the low-quality CBCT phase images.

In response to the current challenges of high-quality 4D-CBCT imaging, we have developed a simultaneous motion estimation and image reconstruction (SMEIR) technique [38,39,40], which combines total variation-based image regularization, 2D-3D deformation-driven motion model estimation, and motion-compensated reconstruction into a comprehensive reconstruction scheme. In comparison to the previous techniques, the SMEIR algorithm estimates an inter-phase motion model via 2D-3D deformation from a motion-compensated CBCT (mCBCT) and the phase-specific projections. The improved motion model is subsequently fed back into the motion-compensated reconstruction to update the mCBCT. Total variation-regularization is incorporated into the motion-compensated reconstruction to further improve the CBCT image quality. The resulting mCBCT, with improved accuracy and quality, is iteratively fed back into the 2D-3D deformation to dynamically update the motion model. The 2D-3D motion estimation and the motion-compensated image reconstruction form an iterative loop to continuously update the mCBCT image as well as the motion model until final convergence. The 4D-CBCT images at other phases are deformed from the mCBCT via the inverse DVFs simultaneously-optimized by the 2D-3D deformation algorithm. In this study, we detailed the general philosophy and workflow of the SMEIR algorithm, and introduced two new developments based on the original SMEIR algorithm: the biomechanical modeling-guided SMEIR (SMERI-Bio) and the SMEIR algorithm with artificial intelligence (AI)-driven DVF fine-tuning (SMEIR-Unet) [41]. In comparison to the original SMEIR algorithm, the SMEIR-Bio algorithm introduced biomechanical modeling to improve the intra-lung DVF accuracy to better reconstruct the fine details in lung, and better capture their motion. Similar to SMEIR-Bio, the SMEIR-Unet algorithm was developed to fine-tune intra-lung DVFs, however via a deep learning-driven approach. Compared with SMEIR-Bio, SMEIR-Unet may allow improved efficiency with reduced complexity and computational load. Corresponding reconstruction results of both digital anthropomorphic phantoms and real lung patient data were presented to illustrate the strengths and weaknesses of different methods. In the end, a review was further included to discuss the application prospects of the developed algorithms, specifically in terms of real-time image-guided radiotherapy and adaptive radiotherapy.


The original SMEIR algorithm

The SMEIR algorithm integrates the motion estimation and image reconstruction to solve both the 4D-CBCT images and the corresponding inter-phase DVFs simultaneously. Compared with the conventional sequential scheme (Fig. 1), both the 4D-CBCT images and the DVFs are iteratively and dynamically updated by SMEIR to maximize the sharing of information to improve image and DVF accuracy.

Fig. 1
figure 1

Flow-chart comparison between (a) the conventional 4D-CBCT reconstruction and DVF-derivation scheme for image-guided radiation therapy/adaptive radiation therapy applications, and (b) the proposed SMEIR scheme. Compared with the conventional scheme, the SMEIR scheme updates the 4D-CBCT images and the DVFs simultaneously and iteratively in an interleaved fashion. By SMEIR, the mCBCT is updated towards improved quality to drive more accurate DVF solution, and the more accurate DVFs subsequently improve the accuracy of the motion-compensated reconstruction for mCBCT. 4D-CBCT: 4-Dimensional cone-beam computed tomography; SMEIR: Simultaneous motion estimation and image reconstruction; DVF: Deformation-vector-field; mCBCT: Motion-compensated CBCT

The SMEIR algorithm is comprised of two major components, the motion estimation and the motion-compensated reconstruction. For motion estimation, the 2D-3D deformation technique is used, which is based on optimizing objective functions as shown in Eqs. 1 and 2:

$$ {f}_1\left({\mathrm{DVF}}^{0\to t}\right)={\left\Vert {p}^t-A{\mu}^0\left(x+{\mathrm{DVF}}^{0\to t}\right)\right\Vert}_{l_2}^2+\beta \ast E\left({\mathrm{DVF}}^{0\to t}\right) $$
$$ {f}_2\left({\mathrm{DVF}}^{t\to 0}\right)={\left\Vert {p}^0-A{\mu}^t\left(x+{\mathrm{DVF}}^{t\to 0}\right)\right\Vert}_{l_2}^2+\beta \ast E\left({\mathrm{DVF}}^{t\to 0}\right) $$

p denotes the 2D, phase-binned cone-beam projections, with the superscript indicating the respiratory phase they belong to; “ A ” denotes the projecting matrix of the projections p, which generates DRRs corresponding to the projections; “ 0 ” indicates the reference phase (motion-compensated phase); “ t ” denotes a general symbol representing each of all other phases; μ denotes the CBCT image at the phase designated by its superscript; x denotes the 3D coordinates of the CBCT image; DVF denotes the deformation-vector-field; The \( {\left\Vert \ast \right\Vert}_{l_2}^2 \) terms in Eqs. 1 and 2 calculate the sum of squared errors between the acquired projections p and the DRRs of the deformed image; E() denotes a quadratic deformation energy term defined to regularize the DVF smoothness [25]; “ β ” balances the data fidelity term and the deformation energy term [24]. In general, the 2D-3D deformation algorithm is optimizing a DVF to deform 3D images based on matching 2D projections, as its name “2D-3D” suggests.

The objective functions of Eqs. 1 and 2 are designed as an inverse-consistent deformable registration to solve a pair of forward and inverse DVFs (DVF0 → t and DVFt → 0). The two objective functions are optimized sequentially and the output of one function is input into the other for cyclic iterations. The conversion between DVF0 → t and DVFt → 0 is achieved through the following inverse-consistent constraint:

$$ {\mathrm{DVF}}^{0\to t}\circ {\mathrm{DVF}}^{t\to 0}={\mathrm{DVF}}^{0\to t}\left(x+{\mathrm{DVF}}^{t\to 0}\right)+{\mathrm{DVF}}^{t\to 0}\ (x)=0 $$
$$ {\mathrm{DVF}}^{t\to 0}\circ {\mathrm{DVF}}^{0\to t}={\mathrm{DVF}}^{t\to 0}\left(x+{\mathrm{DVF}}^{0\to t}\right)+{\mathrm{DVF}}^{0\to t}\ (x)=0 $$

Based on the DVFs optimized by 2D-3D deformation, the SMEIR algorithm applies motion-compensated reconstruction through a modified simultaneous algebraic reconstruction technique (SART) [42], as shown in Eq. 5:

$$ {\mu}_j^{0,\left(k+1\right)}={\mu}_j^{0,(k)}+\uplambda \frac{\sum_{t,n}{\mathrm{DVF}}_{jn}^{t\to 0}{}^{\circ}{\sum}_i\left[{a}_{in}\frac{p_i^t-{\sum}_n{a}_{in}{\mu}_n^{t,(k)}}{\sum_{n=1}^J{a}_{in}}\right]}{\sum_{t,n}{\mathrm{DVF}}_{jn}^{t\to 0}{}^{\circ}{\sum}_i{a}_{in}} $$

In Eq. 5, “ j ” and “ n ” denote voxels in the mCBCT image (reference phase) and the CBCT images at the other phases, respectively; “ i ” denotes the projecting ray at pixel location i within each projection; “ J ” denotes the total number of voxels intercepted by ray i; “ a ” denotes the intersection length of projecting ray i across each voxel; “ λ ” denotes the relaxation factor for SART; “ k ” indicates the iteration number. Compared with the conventional SART, the modified SART applies the voxel-wise deformation field \( {DVF}_{jn}^{t\to 0} \) to the correction term of each phase, which aligns these correction terms to the same coordinates as those of the reference image, to contribute them all towards the update of the reference image. Using all available information, the modified SART algorithm allows the reconstruction of a high-quality mCBCT with under-sampling streaking artifacts removed. After SART, we applied total-variation regularization on the motion-compensated image to further reduce the image noises and artifacts to improve its quality. The updated mCBCT, μ0, is further fed as input into the 2D-3D deformation (Eqs. 1–4) to form an iterative loop until final convergence.

Figure 2 compares the reconstructed images by different techniques. The images shown in column (a) were reconstructed by FDK using non-phase-sorted projections from all phases. Though with minimal streaking artifacts, these FDK images presented prominent motion blurriness from mixed-phase reconstruction. By reconstruction using only the phase-sorted projections, the FDK images in column (b) display reduced motion blurriness, but with amplified under-sampling streaking artifacts and noises. With the de-noising effects from total-variation regularization, the images reconstructed by ART-TV substantially reduced the imaging noises and artifacts as compared to (b), however at the cost of missing fine details (as indicated by the arrows). In contrast, the CBCT images reconstructed by the SMEIR algorithm [column (d)] not only removed the noises/artifacts, but the fine details of the images were well preserved through motion-compensated reconstruction with a high-quality motion model.

Fig. 2
figure 2

Comparison between the reconstructed images by different methods. a Non-phase-sorted FDK images using projections from all respiratory phases; b Phase-sorted FDK images using only projections from the peak-expiration phase; c Peak-expiration phase image by the algebraic reconstruction technique with total variation-regularization; d Peak-expiration phase image by simultaneous motion estimation and image reconstruction. The first row shows the reconstructed images of a digital NCAT phantom [43]. The second row shows the reconstructed images of a real lung cancer patient from clinical projections. The “Ground-truth” image of the NCAT phantom was also presented as a reference. FDK: Feldkamp-Davis-Kress; NCAT: Non-uniform rational B-spline cardiac and torso

The SMEIR-bio algorithm

Though the SMEIR algorithm has proven generally accurate in reconstructing geometrical and intensity information for 4D-CBCT images, some fine anatomical structures, which are small in size, were found not well reconstructed by SMEIR. The organ of lung includes many such structures, including vessels, small bronchioles, and small nodules. Though small in size, these fine structures can serve important landmarks for diagnosis and treatment toxicity evaluation [44]. Accurate reconstruction and presentation of these fine details can be critical to safe and accurate image-guided radiation therapy and adaptive radiation therapy, and pivotal to achieve the maximum therapeutic ratio. These fine details may also convey constructive information towards AI-driven data analysis, such as Radiomics, to predict patient-specific outcomes and treatment responses [45, 46]. It is therefore compelling to correctly reconstruct and display these fine details in lung 4D-CBCTs. However, the 2D-3D deformation technique employed by the original SMEIR algorithm is essentially an intensity-driven approach. The intensity-driven deformable registration techniques may not perform well for these fine structure regions, the deformation of which only lead to small image intensity variations. For 2D-3D deformation, the resulting intensity changes on the 2D cone-beam projections from the deformation of these fine details are even more obscure, leading to minimal changes to the objective functions of 2D-3D deformation (Eqs. 1, 2), especially for under-sampled acquisition scenarios. To improve the motion estimation accuracy of SMEIR of these fine details, we further introduced biomechanical modeling into the SMEIR algorithm (SMEIR-Bio), to solve the DVF via a physics-driven approach [41]. Biomechanical modeling-based deformable registration has been found effective, especially at solving the deformation at low-contrast regions with minimal intensity variation signals [47,48,49,50,51]. By finite element analysis, biomechanical modeling drives the deformation of organs/structures of interest to meet displacement-based or force-based boundary conditions. The elastic properties of these organs/structures are incorporated into the finite element analysis, through a material model, to drive physically-plausible deformation. With minimal dependence on image intensity differences, biomechanical modeling can potentially boost the accuracy of SMEIR in motion estimation and reconstruction of small structures.

The biomechanical modeling process has been incorporated seamlessly into the SMEIR workflow, as illustrated by Fig. 3. To perform biomechanical modeling, we segmented the lung of the mCBCT after motion-compensated reconstruction, and automatically constructed a volumetric tetrahedral mesh using the ISO2MESH and Tetgen packages [52, 53]. The lung was segmented using the automatic “snake” method, which is based on an active contour model [54]. In detail, a few “seeds” were randomly placed within the lung based on intensity-thresholding and grew to fill up the whole lung region, which automatically defined the lung boundary. On the lung boundary, we used the lung surface DVFs solved in the following SMEIR motion estimation (2D-3D deformation) step as displacement-based boundary conditions for biomechanical modeling. With high intensity contrast at the lung boundary, the 2D-3D algorithm could accurately solve lung surface DVFs to provide adequate boundary conditions [27]. With the tetrahedral lung mesh and the boundary conditions, the intra-lung DVFs were deduced via a material model on the relationship between the strain energy and the deformation [55]. Previous studies on lung biomechanical modeling adopted multiple different models, including the linear elastic model [56], the Neo-Hookean model [55], the Ogden model [57], the Marlow model [58], and the Mooney-Rivlin model [27]. Up until now there is no consensus over the most appropriate model, as different models may yield similarly accurate results by using customized material elastic parameters. In our study, we used the Mooney-Rivlin hyper-elastic material, a model usually used to describe soft tissues with relatively large deformation. Detailed information of the model can be found in previous publications [27, 32, 55]. In this study, we modeled the lung as a homogeneous organ with the same elastic parameters throughout the whole volume. Such a strategy has proved effective and efficient by previous studies [27, 47, 55, 56, 58, 59]. After finite element analysis [60], we combined the biomechanical modeling-corrected intra-lung DVFs with outside-lung DVFs, and fed the resulting DVFs into a new motion-compensated reconstruction step before assessing the convergence. If not converged, the mCBCT as well as the inter-phase DVFs would be fed back to initiate a new round of lung segmentation, 2D-3D deformation, biomechanical modeling and motion-compensated reconstruction, until the final convergence was reached.

Fig. 3
figure 3

The general flow of the biomechanical modeling-guided SMEIR scheme. 4D-CBCT: 4-Dimensional cone-beam computed tomography; DVF: Deformation-vector-field; mCBCT: Motion-compensated CBCT

The SMEIR algorithm with deep learning-driven DVF fine-tuning (SMEIR-Unet)

The success of SMEIR-Bio requires careful implementation of biomechanical modeling, which involves organ segmentation, tetrahedral mesh generation, material model/parameter assignment and finite element analysis. The additional computational workload and the complexity may reduce the efficiency of 4D-CBCT reconstruction. Recent developments of AI have found many applications in medicine, to potentially improve the accuracy and efficiency over conventional methods [61]. Multiple AI-based studies have been reported in the field of medical image registration [62,63,64,65]. Deep convolutional neural networks have been trained to learn DVFs between fixed and moving image pairs to improve deformable registration efficiency. Inspired by the potentials of deep learning, we developed a convolution neural network featuring the U-net structure (SMEIR-Unet) [66], to simplify the SMEIR-Bio workflow and accelerate the computational speed. Instead of applying biomechanical modeling to fine-tune the intra-lung DVFs, we trained a supervised network to establish a direct conversion scheme between the 2D-3D DVFs and the high-quality DVFs. The high-quality DVFs for neural network training were obtained via direct Demons registration between 4D-CT images. In detail, we simulated limited-view phase-binned cone-beam projections from the 4D-CT images and reconstructed 4D-CBCT images and the inter-phase DVFs from the projections using the original SMEIR algorithm. And we used the Demons registration algorithm to register the other phase images of 4D-CT to the reference phase image (motion-compensated phase), to derive the corresponding high-quality Demons DVFs. We paired the 2D-3D DVFs solved by the original SMEIR algorithm with the Demons DVFs, and fed them to train a population-based neural network. To focus the neural network on fine-tuning the intra-lung regions, we cropped both the 2D-3D DVFs and the Demons DVFs using the segmented lung contours at the reference phase. As shown in Fig. 4, the U-net-based convolutional neural network was constructed with two paths: the contraction path and the expansion path. The contraction path contained five blocks, with two convolution layers and a max pooling layer in each block. The expansion path similarly contained six blocks, featuring one deconvolution layer and two convolution layers per block.

Fig. 4
figure 4

Workflow of the SMEIR-Unet algorithm. Instead of feeding the DVFs into biomechanical modeling, SMEIR-Unet applies a U-net-based convolutional neural network to directly correct the output 2D-3D DVFs to boost their intra-lung accuracy. The corrected DVFs are output as the final inter-phase DVFs, and are applied in a final motion-compensated reconstruction step to generate the mCBCT at the reference phase, as well as the other 4D-CBCT phase images via the inverse DVFs. SMEIR-Unet: Simultaneous motion estimation and image reconstruction with U-net based DVF fine-tuning; DVF: Deformation-vector-field; 4D-CBCT: 4-Dimensional cone-beam computed tomography; mCBCT: Motion-compensated CBCT

For training, validation and testing, we used a 5-fold cross-validation strategy. We grouped the 11 patient-specific 4D-CT sets into 5 groups. And each time we selected 3 groups for training, 1 group for validation and 1 group for testing. The stochastic Adam optimizer is employed to minimizing a cost function defined as the mean squared error between the predicted DVFs and the true Demons DVFs [67]. A parameter-sweeping strategy was employed for learning rate optimization and finalized a learning rate of 5e-5. The batch size was set to 4. The trained U-net could be applied to convert DVFs solved by the SMEIR algorithm, voxel-by-voxel, to improve their accuracy and quality. Note that the SMEIR-Unet algorithm only fine-tunes the DVFs through the trained U-net for once before the final output, in contrast to SMEIR-Bio, which updates the DVFs using biomechanical modeling during every iteration (Figs. 3 and 4).


For evaluation, we simulated cone-beam projections from high-quality lung 4D-CT images for 4D-CBCT reconstruction, and used the original 4D-CT images as the “Ground-truth” for reference. In this study, we used a dataset including 11 lung patient 4D-CTs acquired on a 16-slice Philips Brilliance CT scanner (Philips Medical Systems, Cleveland, Ohio), which were collected in an academic medical center and not publicly assessable. All the CTs were of the same slice thickness (1.5 mm), while the intra-slice pixel resolutions ranged from 0.78 mm to 0.98 mm, and were uniformly re-sampled to 1.5 mm in our study. Thus all the CTs were of resolution 1.5 mm × 1.5 mm × 1.5 mm. The CTs were of slice dimension 512 × 512, with the slice number ranging from 190 to 270. We simulated 40 projections for each 4D-CT phase, with projection angles evenly distributed across a 360° scan angle, using the Siddon’s ray-tracing technique [68]. We reconstructed 4D-CBCT images from these projections via methods including FDK, algebraic reconstruction technique with total-variation regularization (ART-TV) [69], SMEIR, SMEIR-Bio and SMEIR-Unet for comparison.

To quantitatively assess the accuracy of the motion model (DVFs) solved by different methods, ~ 80 lung landmarks were manually identified by expert radiation oncologists on each phase of the 4D-CT images used in our study. The 3D location changes of the same landmark between different phases were used as the “gold-standard” DVF to calculate the DVF errors of different methods. We evaluated the DVF errors along each of the three Cartesian directions (X, Y, Z), and also in terms of the error vector length.


Figure 5 compares the small lung details reconstructed by different methods at the reference phase. It can be observed that the FDK image contained structures non-existent on the “Ground-truth” image. These structures were from the under-sampling artifacts and noises. For the ART-TV images, the fine details were over-smoothed and barely recognizable. The original SMEIR algorithm was similarly susceptible to the mis-match of the fine-detail regions from those of the “Ground-truth”, indicating incorrect inter-phase DVFs. In contrast, the SMEIR-Bio algorithm reconstructed small lung details to best match the “Ground-truth”, due to the efficacy of biomechanical modeling in correcting DVFs at these fine-detail regions.

Fig. 5
figure 5

Comparison between the fine details in the CBCT images reconstructed by the FDK, ART-TV, SMEIR and SMEIR-Bio techniques, by using the “Ground-truth” 4D-CT reference phase image for evaluation. All CBCT reconstructions used 40 projections per phase, which were simulated from the “Ground-truth” images. FDK: Feldkamp-Davis-Kress; ART-TV: Algebraic reconstruction technique with total variation-regularization; SMEIR: Simultaneous motion estimation and image reconstruction; CBCT: Cone-beam computed tomography; SMEIR-Bio: Biomechanical modeling-guided SMEIR

To further evaluate the accuracy of the solved inter-phase DVFs, we used the Demons algorithm to register between the reference phase and the other phases of 4D-CT to generate Demons DVFs for comparison [70]. Since the Demons algorithm was applied directly between high-quality 4D-CT images, the resulting DVFs were of high fidelity to serve as references. Figure 6 shows a comparison between the SMEIR DVF, the SMEIR-Bio DVF and the Demons DVF of three different views. It can be observed that the SMEIR DVF only displayed substantial deformation around the high contrast lung surface. Due to the lack of sufficient intensity variations inside the lung, the intra-lung DVF was not correctly derived by the original SMEIR algorithm. After incorporating biomechanical modeling, SMEIR-Bio substantially improved the DVF accuracy to match with the Demons DVF. In our study, the SMEIR-Bio DVF appeared smoother than the Demons DVF, since we modeled the lung as a homogeneous organ. Introducing heterogeneity during the construction of the biomechanical model may help to further improve the SMEIR-Bio DVF accuracy [51, 71], to potentially better match with the Demons DVF.

Fig. 6
figure 6

Comparison of DVFs solved by SMEIR, SMEIR-Bio and the Demons registration. The Demons registration was directly performed between the “Ground-truth” high-quality 4D-CT images. SMEIR: Simultaneous motion estimation and image reconstruction; SMEIR-Bio: Biomechanical modeling-guided SMEIR; DVF: Deformation-vector-field

As demonstrated in Fig. 7a, the SMEIR-Bio DVFs closely matched the SMEIR-Unet DVFs, and both were substantially improved compared to the original SMEIR DVFs (Fig. 6) in terms of the resemblance to the high-quality Demons DVFs. Due to the homogeneous material modeling, the SMEIR-Bio DVFs were smooth with more gradual spatial variations. In contrast, the SMEIR-Unet DVFs were more heterogeneous, since they were directly learned from the heterogeneous Demons DVFs and inherited similar features. However, a comparison between the reconstructed CBCT images (Fig. 7b) shows the similarity between the reconstructed CBCT images by SMEIR-Bio and SMEIR-Unet, demonstrating that both methods were capable of reconstructing accurate and high-quality CBCT images. We also computed two image quality metrics, the RMSE [39] and the UQI [33], to compare SMEIR-Bio and SMEIR-Unet to the original SMEIR algorithm, based on 10 lung region-of-interests (ROIs) focusing on fine details. For SMEIR-Bio, the average RMSE of the 10 evaluated ROIs was 0.0033 and the average UQI was 0.87. The corresponding results were 0.0035 and 0.93 for SMEIR-Unet, and 0.0039 and 0.67 for the original SMEIR method. Both the SMEIR-Bio and SMEIR-Unet methods improved the accuracy of fine detail reconstruction in lung.

Fig. 7
figure 7

Comparison of DVFs solved by the SMEIR-Bio and the SMEIR-Unet techniques (a); Comparison of the corresponding motion-compensated CBCT images reconstructed by the SMEIR-Bio and the SMEIR-Unet techniques (b). The DVFs were from the same patient case as Fig. 6, and shown at the same slice locations. SMEIR: Simultaneous motion estimation and image reconstruction; CBCT: Cone-beam computed tomography; SMEIR-Bio: Biomechanical modeling-guided SMEIR; SMEIR-Unet: SMEIR with U-net based DVF fine-tuning; DVF: Deformation-vector-field

Table 1 shows the quantitative DVF errors for the SMEIR, SMEIR-Bio and SMEIR-Unet methods. Without DVF fine-tuning from biomechanical modeling or convolutional neural network-based conversion, the SMEIR algorithm suffered from the largest DVF error, with the mean vector DVF error around 5 mm. The SMEIR-Bio and SMEIR-Unet techniques achieved appreciable reductions of the DVF errors, with the mean vector DVF error < 3.5 mm for both methods. Compared with SMEIR-Bio, which took > 5 min in building and solving the biomechanical model (including mesh generation, boundary condition assignment, finite element analysis, etc.), SMEIR-Unet only took ~ 10 s to update and fine-tune the DVF for each phase.

Table 1 Comparison of DVF errors between SMEIR, SMEIR-Bio and SMEIR-Unet techniques


4D-CBCT imaging has many potential applications and benefits in radiation therapy. One major application is in image-guided radiation therapy to allow precise cancer treatment. Studies have found that the free-breathing 3D-CBCT imaging may underestimate the motion range of tumors, which can potentially affect the tumor targeting accuracy [72]. The capture of on-board 4D tumor motion by 4D-CBCT will enable better alignment between 4D-CTs and 4D-CBCTs to achieve more accurate and robust patient setup and tumor targeting [73]. The reconstruction accuracy offered by SMEIR-type algorithms fulfills the clinical needs of accurate 4D tumor localization. The ability of SMEIR to reconstruct high-quality images from few projections also renders 4D-CBCT more time-efficient and safe. In comparison to the original SMEIR algorithm, our newly-developed SMEIR-Bio algorithm substantially improved the accuracy of intra-lung DVFs solved using limited-view projections. The derivation of a physically-plausible DVF, from displacement-based boundary conditions, allows us to capture the motion of small, intricate structures within the lung. The biomechanical modeling step requires the input of a lung contour, which was automatically-segmented using the “snake” method in this study. An investigation comparing different lung segmentation methods towards the SMEIR-Bio accuracy was not included in this article. However, since the lung is of high contrast at its boundaries, auto-segmentation is relatively easy and straightforward, and variations among different methods will be small. Furthermore, since DVFs are mostly piecewise constant and smooth, small variations between the lung boundaries segmented by different methods will bear minimal impacts on the extracted boundary condition and the corresponding biomechanical modeling results. In comparison to SMEIR-Bio, which involved the complex biomechanical modeling process, SMEIR-Unet provided similar 4D-CBCT reconstruction and DVF accuracy while with reduced computational workload and improved efficiency.

Currently, CBCT imaging is mostly applied at the beginning of each radiation therapy fraction, prior to radiation starts. Acquiring more intra-treatment 4D-CBCT images, either between consecutive radiation beams or within each beam [74], will enable continuous updates of tumor motion information to allow treatment adjustments on-the-fly. The ability to closely monitor the motion of moving targets can also promote safety margin reduction to spare more normal structures from being damaged by radiation beams [75]. Our recent developments towards SMEIR acceleration make it feasible to acquire and reconstruct the 4D-CBCT images for treatment guidance in a clinically-acceptable time frame [40], to allow multiple acquisitions and reconstructions during the treatment for continuous target monitoring. The evaluation of the SMEIR algorithm finds it could reconstruct high-quality 4D-CBCT images using as few as 20 projections per phase [38, 39]. Further imaging time reduction by acquiring fewer projections will help approach the goal of real-time volumetric, 4D imaging. Introducing prior information into the reconstruction, like the previously-acquired CT/CBCT images, or a prior motion model, may help to further lower sampling requirements for even faster imaging [25, 28]. On the other hand, high frame-rate non-ionizing imaging signals, such as the surface-guided optical imaging, have also found their potential in helping to achieve real-time volumetric imaging [76]. We envision the next evolution of real-time 4D-CBCT imaging will maneuver the potential of combining the prior information, on-board x-ray imaging, and on-board non-ionizing imaging sources. Through feeding them into a system comprised of advanced reconstruction techniques like SMEIR, biomechanical modeling and AI, a high-quality real-time volumetric image can potentially be reconstructed to maximize the accuracy of radiotherapy (Fig. 8).

Fig. 8
figure 8

The envisioned future developments towards real-time 4D-CBCT imaging, by combining real-time on-board information provided by non-ionizing imaging signals including the optical surface imaging, and prior information (prior CT/CBCT, prior motion model), and real-time on-board x-ray imaging. Through combining all these information sources, a reconstruction system comprising SMEIR, biomechanical modeling, and AI could potentially achieve real-time imaging to guide the most accurate treatment delivery, and allow on-the-fly plan adjustment to fit the patients’ daily variations. 4D-CBCT: 4-Dimensional cone-beam computed tomography; SMEIR: Simultaneous motion estimation and image reconstruction; AI: Artificial intelligence

In addition to providing anatomical and geometrical information towards more precise radiotherapy targeting, another potential benefit of 4D-CBCT is to calculate delivered radiation doses, and accumulate the delivered doses throughout the treatment course to dynamically assess the need of adaptive radiation therapy, and to provide dosimetric data for dose outcome analysis to fine-tune the treatment prescription [30, 31, 77]. With highly-accurate inter-phase DVFs, the 4D-CBCT solved by SMEIR-type algorithms allows direct and accurate 4D dose accumulation to determine the dose delivered to the gross tumor volumes as well as the normal tissues to evaluate the true dose coverage and the normal tissue toxicity (Fig. 9). By linking 4D-CBCTs acquired at different sessions through inter-session DVFs, we can also accumulate the overall doses delivered during the treatment course for comprehensive treatment evaluation. Currently, one major limitation of applying the CBCTs for direct dose calculation is the inaccurate HU values of the CBCT images, mostly due to the amplified scatter from the cone-beam geometry, as well as the limited field-of-view [78, 79]. Some studies proposed to circumvent the issue through direct CT-CBCT registration to calculate treatment doses on the deformed CT image instead [80, 81]. However, such methods are susceptible to the registration errors between the planning CT and on-board CBCT images. It would be ideal to directly improve the CBCT HU accuracy to allow direct dose calculation on CBCT. Multiple methods have been developed to reduce and correct the scatter signal, including experimental measurements, hardware-based corrections and Monte-Carlo simulations [79, 82, 83]. Introducing such methods into the clinic will allow substantial improvements of the CBCT HU accuracy. On the other hand, the issue of limited field-of-view can be potentially addressed through incorporating prior information into the CBCT reconstruction, which also enjoys the benefit of introducing more accurate HUs [30]. Recent developments of AI-based techniques were found effective to convert the CBCT images directly to improve their quality to match that of planning CT images for accurate dose calculation [84]. Such techniques could be grafted with SMEIR-type algorithms to yield the final 4D-CBCT directly applicable for accurate dose calculation. Based on the DVF-accumulated doses on each treatment fraction and throughout all fractions, and based on the DVF-propagated tumor and normal tissue contours [32, 85], decisions can be rendered in regards to the necessity of adaptive radiation therapy. Based on high-quality CBCTs, radiotherapy plans can be adapted and optimized to meet the patient’s clinical needs to maximize the radiotherapy benefits.

Fig. 9
figure 9

Proposed SMEIR-driven 4D dose accumulation and evaluation workflow for radiation oncology. After applying the rigid registration to align the 4D-CBCT to the 4D-CT coordinates, the 4D-CBCT uses the DVFs solved by SMEIR to deform 4D doses to a single reference phase, in a similar way as the dose accumulation process of 4D-CT. The 4D-CT uses a direct deformable registration algorithm to derive inter-phase DVFs, since the high-quality CT images allow direct inter-phase DVF solution. 4D-CBCT: 4-Dimensional cone-beam computed tomography; SMEIR: Simultaneous motion estimation and image reconstruction; DVF: Deformation-vector-field


The fast development of medical imaging has allowed more precise radiation therapy treatments and is continuously pushing the boundary of technology towards safer beam delivery and more effective normal tissue sparing. 4D-CBCT plays a key role in managing sites involving moving targets such as lungs and livers. Through combining motion estimation and motion-compensated reconstruction, the SMEIR algorithm allows low-dose and faster 4D-CBCT imaging to better meet the clinical demands. The introduction of biomechanical modeling and U-net based DVF fine-tuning further develops the original SMEIR algorithm to better reconstruct small, fine details in addition to bulky structures. Through introducing more information, including the prior information and the high frame-rate, non-ionizing imaging technique, we could potentially accelerate the 4D-CBCT acquisition to allow near real-time imaging to achieve the most precise tumor targeting with a minimal safety margin. In addition to image-guidance, the 4D-CBCT and the inter-phase DVFs solved by SMEIR-type algorithms also allow dose calculation and accumulation to guide intra-course adaptive radiation therapy to tailor the plan to better deliver patient-specific cancer care, and provides abundant information to assess the dose-treatment outcomes, on both tumor control and normal tissue toxicity.

Availability of data and materials

Please contact the corresponding author for data requests.



4-dimensional cone-beam computed tomography


Artificial intelligence


Algebraic reconstruction technique with total variation-regularization








Motion-compensated CBCT


Non-uniform rational B-spline cardiac and torso






Simultaneous algebraic reconstruction technique


Simultaneous motion estimation and image reconstruction


Biomechanical modeling-guided SMEIR


SMEIR with U-net based DVF fine-tuning


Universal quality index


  1. Zelefsky MJ, Kollmeier M, Cox B, Fidaleo A, Sperling D, Pei X et al (2012) Improved clinical outcomes with high-dose image guided radiotherapy compared with non-IGRT for the treatment of clinically localized prostate cancer. Int J Radiat Oncol Biol Phys 84(1):125–129.

    Article  Google Scholar 

  2. Jaffray DA, Siewerdsen JH (2000) Cone-beam computed tomography with a flat-panel imager: initial performance characterization. Med Phys 27(6):1311–1323.

    Article  Google Scholar 

  3. Jaffray DA, Siewerdsen JH, Wong JW, Martinez AA (2002) Flat-panel cone-beam computed tomography for image-guided radiation therapy. Int J Radiat Oncol Biol Phys 53(5):1337–1349.

    Article  Google Scholar 

  4. Lu J, Guerrero TM, Munro P, Jeung A, Chi PCM, Balter P et al (2007) Four-dimensional cone beam CT with adaptive gantry rotation and adaptive data sampling. Med Phys 34(9):3520–3529.

    Article  Google Scholar 

  5. Sweeney RA, Seubert B, Stark S, Homann V, Müller G, Flentje M et al (2002) Accuracy and inter-observer variability of 3D versus 4D cone-beam CT based image-guidance in SBRT for lung tumors. Radiat Oncol 7:81.

    Article  Google Scholar 

  6. Kincaid RE Jr, Yorke ED, Goodman KA, Rimner A, Wu AJ, Mageras GS (2013) Investigation of gated cone-beam CT to reduce respiratory motion blurring. Med Phys 40(4):041717.

    Article  Google Scholar 

  7. Sonke JJ, Zijp L, Remeijer P, van Herk M (2005) Respiratory correlated cone beam CT. Med Phys 32(4):1176–1186.

    Article  Google Scholar 

  8. Dietrich L, Jetter S, Tücking T, Nill S, Oelfke U (2006) Linac-integrated 4D cone beam CT: first experimental results. Phys Med Biol 51(11):2939–2952.

    Article  Google Scholar 

  9. Nakagawa K, Haga A, Kida S, Masutani Y, Yamashita H, Takahashi W et al (2013) 4D registration and 4D verification of lung tumor position for stereotactic volumetric modulated arc therapy using respiratory-correlated cone-beam CT. J Radiat Res 54(1):152–156.

    Article  Google Scholar 

  10. Vergalasova I, Cai J, Yin FF (2012) A novel technique for markerless, self-sorted 4D-CBCT: feasibility study. Med Phys 39(3):1442–1451.

    Article  Google Scholar 

  11. Sawant A, Smith RL, Venkat RB, Santanam L, Cho B, Poulsen P et al (2009) Toward submillimeter accuracy in the management of intrafraction motion: the integration of real-time internal position monitoring and multileaf collimator target tracking. Int J Radiat Oncol Biol Phys 74(2):575–582.

    Article  Google Scholar 

  12. Santhanam AP, Willoughby TR, Kaya I, Shah AP, Meeks SL, Rolland JP et al (2008) A display framework for visualizing real-time 3D lung tumor radiotherapy. J Display Technol 4(4):473–482.

    Article  Google Scholar 

  13. Feldkamp LA, Davis LC, Kress JW (1984) Practical cone-beam algorithm. J Opt Soc Am A 1(6):612–619.

    Article  Google Scholar 

  14. Chang J, Sillanpaa J, Ling CC, Seppi E, Yorke E, Mageras G et al (2006) Integrating respiratory gating into a megavoltage cone-beam CT system. Med Phys 33(7):2354–2361.

    Article  Google Scholar 

  15. Kim DW, Chung WK, Yoon M (2013) Imaging doses and secondary cancer risk from kilovoltage cone-beam CT in radiation therapy. Health Phys 104(5):499–503.

    Article  Google Scholar 

  16. Purdie TG, Bissonnette JP, Franks K, Bezjak A, Payne D, Sie F et al (2007) Cone-beam computed tomography for on-line image guidance of lung stereotactic radiotherapy: localization, verification, and intrafraction tumor position. Int J Radiat Oncol Biol Phys 68(1):243–252.

    Article  Google Scholar 

  17. Sidky EY, Pan XC (2008) Image reconstruction in circular cone-beam computed tomography by constrained, total-variation minimization. Phys Med Biol 53(17):4777–4807.

    Article  Google Scholar 

  18. Jia X, Lou YF, Li RJ, Song WY, Jiang SB (2010) GPU-based fast cone beam CT reconstruction from undersampled and noisy projection data via total variation. Med Phys 37(4):1757–1760.

    Article  Google Scholar 

  19. Jia X, Tian Z, Lou YF, Sonke JJ, Jiang SB (2012) Four-dimensional cone beam CT reconstruction and enhancement using a temporal nonlocal means method. Med Phys 39(9):5592–6602.

    Article  Google Scholar 

  20. Zhang H, Zeng D, Zhang H, Wang J, Liang ZR, Ma JH (2017) Applications of nonlocal means algorithm in low-dose X-ray CT image processing and reconstruction: a review. Med Phys 44(3):1168–1185.

    Article  Google Scholar 

  21. Jia X, Dong B, Lou YF, Jiang SB (2011) GPU-based iterative cone-beam CT reconstruction using tight frame regularization. Phys Med Biol 56(13):3787–3807.

    Article  Google Scholar 

  22. Wang Z, Bovik AC (2002) A universal image quality index. IEEE Signal Process Lett 9(3):81–84.

    Article  Google Scholar 

  23. Ren L, Chetty IJ, Zhang J, Jin JY, Wu QJ, Yan H et al (2012) Development and clinical evaluation of a three-dimensional cone-beam computed tomography estimation method using a deformation field map. Int J Radiat Oncol Biol Phys 82(5):1584–1593.

    Article  Google Scholar 

  24. Wang J, Gu XJ (2013) High-quality four-dimensional cone-beam CT by deforming prior images. Phys Med Biol 58(2):231–246.

    Article  Google Scholar 

  25. Zhang Y, Yin FF, Segars WP, Ren L (2013) A technique for estimating 4D-CBCT using prior knowledge and limited-angle projections. Med Phys 40(12):121701.

    Article  Google Scholar 

  26. Harris W, Zhang Y, Yin FF, Ren L (2017) Estimating 4D-CBCT from prior information and extremely limited angle projections using structural PCA and weighted free-form deformation for lung radiotherapy. Med Phys 44(3):1089–1104.

    Article  Google Scholar 

  27. Zhang Y, Tehrani JN, Wang J (2016) A biomechanical modeling guided CBCT estimation technique. IEEE Trans Med Imaging 36(2):641–652.

    Article  Google Scholar 

  28. Zhang Y, Yin FF, Pan T, Vergalasova I, Ren L (2015) Preliminary clinical evaluation of a 4D-CBCT estimation technique using prior information and limited-angle projections. Radiother Oncol 115(1):22–29.

    Article  Google Scholar 

  29. Wu QJ, Li TR, Wu QW, Yin FF (2011) Adaptive radiation therapy: technical components and clinical applications. Cancer J 17(3):182–189.

    Article  Google Scholar 

  30. Zhang Y, Yin FF, Ren L (2015) Dosimetric verification of lung cancer treatment using the CBCTs estimated from limited-angle on-board projections. Med Phys 42(8):4783–4795.

    Article  Google Scholar 

  31. Qin A, Gersten D, Liang J, Liu Q, Grill I, Guerrero T et al (2018) A clinical 3D/4D CBCT-based treatment dose monitoring system. J Appl Clin Med Phys 19(6):166–176.

    Article  Google Scholar 

  32. Zhang Y, Folkert MR, Li B, Huang XK, Meyer JJ, Chiu T et al (2019) 4D liver tumor localization using cone-beam projections and a biomechanical model. Radiother Oncol 133:183–192.

    Article  Google Scholar 

  33. Zhang Y, Ma JH, Iyengar P, Zhong YC, Wang J (2017) A new CT reconstruction technique using adaptive deformation recovery and intensity correction (ADRIC). Med Phys 44(6):2223–2241.

    Article  Google Scholar 

  34. Li T, Schreibmann E, Yang Y, Xing L (2006) Motion correction for improved target localization with on-board cone-beam computed tomography. Phys Med Biol 51(2):253–267.

    Article  Google Scholar 

  35. Li TF, Koong A, Xing L (2007) Enhanced 4D cone-beam CT with inter-phase motion model. Med Phys 34(9):3688–3695.

    Article  Google Scholar 

  36. Rit S, Wolthaus JWH, van Herk M, Sonke JJ (2009) On-the-fly motion-compensated cone-beam CT using an a priori model of the respiratory motion. Med Phys 36(6):2283–2296.

    Article  Google Scholar 

  37. Brehm M, Paysan P, Oelhafen M, Kunz P, Kachelrieß M (2012) Self-adapting cyclic registration for motion-compensated cone-beam CT in image-guided radiation therapy. Med Phys 39(12):7603–7618.

    Article  Google Scholar 

  38. Wang J, Gu XJ (2013) Simultaneous motion estimation and image reconstruction (SMEIR) for 4D cone-beam CT. Med Phys 40(10):101912.

    Article  Google Scholar 

  39. Dang J, Gu XJ, Pan TS, Wang J (2015) A pilot evaluation of a 4-dimensional cone-beam computed tomographic scheme based on simultaneous motion estimation and image reconstruction. Int J Radiat Oncol Biol Phys 91(2):410–418.

    Article  Google Scholar 

  40. Zhong ZC, Gu XJ, Mao WH, Wang J (2016) 4D cone-beam CT reconstruction using multi-organ meshes for sliding motion modeling. Phys Med Biol 61(3):996–1020.

    Article  Google Scholar 

  41. Huang XK, Zhang Y, Wang J (2018) A biomechanical modeling-guided simultaneous motion estimation and image reconstruction technique (SMEIR-bio) for 4D-CBCT reconstruction. Phys Med Biol 63(4):045002.

    Article  Google Scholar 

  42. Andersen AH, Kak AC (1984) Simultaneous algebraic reconstruction technique (SART): a superior implementation of the ART algorithm. Ultrason Imaging 6(1):81–94.

    Article  Google Scholar 

  43. Veress AI, Segars WP, Weiss JA, Tsui BMW, Gullberg GT (2006) Normal and pathological NCAT image and phantom data based on physiologically realistic left ventricle finite-element models. IEEE Trans Med Imaging 25(12):1604–1616.

    Article  Google Scholar 

  44. Kazemzadeh N, Modiri A, Samanta S, Yan YL, Bland R, Rozario T et al (2018) Virtual bronchoscopy-guided treatment planning to map and mitigate radiation-induced airway injury in lung SAbR. Int J Radiat Oncol Biol Phys 102(1):210–218.

    Article  Google Scholar 

  45. Fave X, Mackin D, Yang JZ, Zhang J, Fried D, Balter P et al (2015) Can radiomics features be reproducibly measured from CBCT images for patients with non-small cell lung cancer? Med Phys 42(12):6784–6797.

    Article  Google Scholar 

  46. Zhou ZG, Folkert M, Iyengar P, Westover K, Zhang YY, Choy H et al (2017) Multi-objective radiomics model for predicting distant failure in lung SBRT. Phys Med Biol 62(11):4460–4478.

    Article  Google Scholar 

  47. Brock KK, Sharpe MB, Dawson LA, Kim SM, Jaffray DA (2005) Accuracy of finite element model-based multi-organ deformable image registration. Med Phys 32(6):1647–1659.

    Article  Google Scholar 

  48. Al-Mayah A, Moseley J, Hunter S, Velec M, Chau L, Breen S et al (2010) Biomechanical-based image registration for head and neck radiation treatment. Phys Med Biol 55(21):6491–6500.

    Article  Google Scholar 

  49. Brock KK, Dawson LA (2010) Adaptive management of liver cancer radiotherapy. Semin Radiat Oncol 20(2):107–115.

    Article  Google Scholar 

  50. Al-Mayah A, Moseley J, Velec M, Brock K (2011) Toward efficient biomechanical-based deformable image registration of lungs for image-guided radiotherapy. Phys Med Biol 56(15):4701–4713.

    Article  Google Scholar 

  51. Li M, Castillo E, Zheng XL, Luo HY, Castillo R, Wu Y et al (2013) Modeling lung deformation: a combined deformable image registration method with spatially varying Young's modulus estimates. Med Phys 40(8):081902.

    Article  Google Scholar 

  52. Fang QQ, Boas DA (2009) Tetrahedral mesh generation from volumetric binary and grayscale images. In: Abstracts of 2009 IEEE international symposium on biomedical imaging, IEEE, Boston, 28 June-1 July 2009

  53. Si H (2015) TetGen, a Delaunay-based quality tetrahedral mesh generator. ACM Trans Math Softw (TOMS) 41(2):Article No.11. DOI:

    Article  MathSciNet  Google Scholar 

  54. Yushkevich PA, Gao Y, Gerig G (2016) ITK-SNAP: an interactive tool for semi-automatic segmentation of multi-modality biomedical images. In: Abstracts of the 2016 38th annual international conference of the IEEE engineering in medicine and biology society, IEEE, Orlando, 16-20 August 2016.

  55. Tehrani JN, Yang Y, Werner R, Lu W, Low D, Guo XH et al (2015) Sensitivity of tumor motion simulation accuracy to lung biomechanical modeling approaches and parameters. Phys Med Biol 60(22):8833–8849.

    Article  Google Scholar 

  56. Werner R, Ehrhardt J, Schmidt R, Handels H (2009) Patient-specific finite element modeling of respiratory lung motion using 4D CT image data. Med Phys 36(5):1500–1511.

    Article  Google Scholar 

  57. Naini AS, Patel RV, Samani A (2011) Measurement of lung hyperelastic properties using inverse finite element approach. IEEE Trans Biomed Eng 58(10):2852–2859.

    Article  Google Scholar 

  58. Shirzadi Z, Sadeghi-Naini A, Samani A (2013) Toward in vivo lung's tissue incompressibility characterization for tumor motion modeling in radiation therapy. Med Phys 40(5):051902.

    Article  Google Scholar 

  59. Al-Mayah A, Moseley J, Brock KK (2008) Contact surface and material nonlinearity modeling of human lungs. Phys Med Biol 53(1):305–317.

    Article  Google Scholar 

  60. Maas SA, Ellis BJ, Ateshian GA, Weiss JA (2012) FEBio: finite elements for biomechanics. J Biomech Eng 134(1):011005.

    Article  Google Scholar 

  61. Thompson RF, Valdes G, Fuller CD, Carpenter CM, Morin O, Aneja S et al (2018) Artificial intelligence in radiation oncology: a specialty-wide disruptive transformation? Radiother Oncol 129(3):421–426.

    Article  Google Scholar 

  62. Yang X, Kwitt R, Niethammer M (2016) Fast predictive image registration. In: Carneiro G, Mateus D, Peter L, Bradley A, Tavares JMRS, Belagiannis V, et al (eds) Deep learning and data labeling for medical applications. First international workshop, LABELS 2016, and second international workshop, October 2016. Lecture notes in computer science (lecture notes in artificial intelligence), vol 1008. Springer, Heidelberg, pp 48–57.

    Google Scholar 

  63. Sokooti H, de Vos B, Berendsen F, Lelieveldt BPF, Išgum I, Staring M (2017) Nonrigid image registration using multi-scale 3D convolutional neural networks. In: Descoteaux M, Maier-Hein L, Franz A, Jannin P, Collins DL, Duchesne S (eds) Medical image computing and computer-assisted intervention. 20th international conference, September, 2017. Lecture notes in computer science (Lecture notes in artificial intelligence), vol 10433. Springer, Heidelberg, pp 232–239.

    Chapter  Google Scholar 

  64. Rohé MM, Datar M, Heimann T, Sermesant M, Pennec X (2017) SVF-net: learning deformable image registration using shape matching. In: Descoteaux M, Maier-Hein L, Franz A, Jannin P, Collins DL, Duchesne S (eds) Medical image computing and computer-assisted intervention. 20th international conference, September, 2017. Lecture notes in computer science (Lecture notes in artificial intelligence), vol 10433. Springer, Heidelberg, pp 266–274.

    Chapter  Google Scholar 

  65. de Vos BD, Berendsen FF, Viergever MA, Staring M, Išgum I (2017) End-to-end unsupervised deformable image registration with a convolutional neural network. In: Cardoso MJ, Arbel T, Carneiro G, Syeda-Mahmood T, Tavares JMRS, Moradi M, et al (eds) Deep learning in medical image analysis and multimodal learning for clinical decision support. Third international workshop, DLMIA 2017, and 7th international workshop, September, 2017. Lecture notes in computer science (Lecture notes in artificial intelligence), vol 10553. Springer, Heidelberg, pp 204–212.

    Chapter  Google Scholar 

  66. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells WM, Frangi AF (eds) Medical image computing and computer-assisted intervention. 18th international conference, October, 2015. Lecture notes in computer science (lecture notes in artificial intelligence), vol 9351. Springer, Heidelberg, pp 234–241.

    Google Scholar 

  67. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization arXiv: 1412.6980

  68. Siddon RL (1985) Fast calculation of the exact radiological path for a three-dimensional CT array. Med Phys 12(2):252–255.

    Article  Google Scholar 

  69. Wang J, Mao WH, Solberg T (2010) Scatter correction for cone-beam computed tomography using moving blocker strips: a preliminary study. Med Phys 37(11):5792–5800.

    Article  Google Scholar 

  70. Gu XJ, Pan H, Liang Y, Castillo R, Yang DS, Choi D et al (2010) Implementation and evaluation of various demons deformable image registration algorithms on a GPU. Phys Med Biol 55(1):207–219.

    Article  Google Scholar 

  71. Al-Mayah A, Moseley J, Velec M, Hunter S, Brock K (2010) Deformable image registration of heterogeneous human lung incorporating the bronchial tree. Med Phys 37(9):4560–4571.

    Article  Google Scholar 

  72. Vergalasova I, Maurer J, Yin FF (2011) Potential underestimation of the internal target volume (ITV) from free-breathing CBCT. Med Phys 38(8):4689–4699.

    Article  Google Scholar 

  73. Sonke JJ, Rossi M, Wolthaus J, van Herk M, Damen E, Belderbos J (2009) Frameless stereotactic body radiotherapy for lung cancer using four-dimensional cone beam CT guidance. Int J Radiat Oncol Biol Phys 74(2):567–574.

    Article  Google Scholar 

  74. Ren L, Zhang Y, Yin FF (2014) A limited-angle intrafraction verification (LIVE) system for radiation therapy. Med Phys 41(2):020701.

    Article  Google Scholar 

  75. Grills IS, Hugo G, Kestin LL, Galerani AP, Chao KK, Wloch J et al (2008) Image-guided radiotherapy via daily online cone-beam CT substantially reduces margin requirements for stereotactic lung radiotherapy. Int J Radiat Oncol Biol Phys 70(4):1045–1056.

    Article  Google Scholar 

  76. Nasehi Tehrani J, McEwan A, Wang J (2016) Lung surface deformation prediction from spirometry measurement and chest wall surface motion. Med Phys 43(10):5493–5502.

    Article  Google Scholar 

  77. Thor M, Andersen ES, Petersen JBB, Sørensen TS, Noe ØK, Tanderup K et al (2014) Evaluation of an application for intensity-based deformable image registration and dose accumulation in radiotherapy. Acta Oncol 53(10):1329–1336.

    Article  Google Scholar 

  78. Ren L, Chen YX, Zhang Y, Giles W, Jin JY, Yin FF (2015) Scatter reduction and correction for dual-source cone-beam ct using prepatient grids. Technol Cancer Res Treat 15(3):416–427.

    Article  Google Scholar 

  79. Chen X, Ouyang L, Yan H, Jia X, Li B, Lyu QW et al (2017) Optimization of the geometry and speed of a moving blocker system for cone-beam computed tomography scatter correction. Med Phys 44(9):e215–e229.

    Article  Google Scholar 

  80. Yang Y, Schreibmann E, Li TF, Wang C, Xing L (2007) Evaluation of on-board kV cone beam CT (CBCT)-based dose calculation. Phys Med Biol 52(3):685–705.

    Article  Google Scholar 

  81. Giacometti V, King RB, Agnew CE, Irvine DM, Jain S, Hounsell AR et al (2019) An evaluation of techniques for dose calculation on cone beam computed tomography. Br J Radiol 92(1096):20180383.

    Article  Google Scholar 

  82. Zhu L, Xie YQ, Wang J, Xing L (2009) Scatter correction for cone-beam CT in radiation therapy. Med Phys 36(6):2258–2268.

    Article  Google Scholar 

  83. Xu Y, Bai T, Yan H, Ouyang L, Pompos A, Wang J et al (2015) A practical cone-beam CT scatter correction method with optimized Monte Carlo simulations for image-guided radiation therapy. Phys Med Biol 60(9):3567–3587.

    Article  Google Scholar 

  84. Liang X, Chen LY, Nguyen D, Zhou ZG, Gu XJ, Yang M et al (2019) Generating synthesized computed tomography (CT) from cone-beam computed tomography (CBCT) using CycleGAN for adaptive radiation therapy. Phys Med Biol 64(12):125002.

    Article  Google Scholar 

  85. Thor M, Petersen JBB, Bentzen L, Høyer M, Muren LP (2011) Deformable image registration for contour propagation from CT to cone-beam CT scans in radiotherapy of prostate cancer. Acta Oncol 50(6):918–925.

    Article  Google Scholar 

Download references


The authors would like to thank Dr. Paul Segars from the Duke University Medical Center for sharing the NCAT phantom for this study.


This work was supported in part by grants from the US National Institutes of Health, Nos. R01 EB020366 and R01 EB027898; the Cancer Prevention and Research Institute of Texas, Nos. RP130109 and RP160661, and from the University of Texas Southwestern Medical Center (Radiation Oncology Seed Grant).

Author information

Authors and Affiliations



YZ, XKH and JW participated in the methodology and experiment design, conducted the experiments and drafted the manuscript; all authors read and approved the final manuscript.

Corresponding author

Correspondence to You Zhang.

Ethics declarations

Ethics approval and consent to participate

This is a retrospective study under a departmental umbrella protocol approved by the internal review board (IRB).

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Y., Huang, X. & Wang, J. Advanced 4-dimensional cone-beam computed tomography reconstruction by combining motion estimation, motion-compensated reconstruction, biomechanical modeling and deep learning. Vis. Comput. Ind. Biomed. Art 2, 23 (2019).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: