Skip to main content

Preliminary landscape analysis of deep tomographic imaging patents

Abstract

Over recent years, the importance of the patent literature has become increasingly more recognized in the academic setting. In the context of artificial intelligence, deep learning, and data sciences, patents are relevant to not only industry but also academe and other communities. In this article, we focus on deep tomographic imaging and perform a preliminary landscape analysis of the related patent literature. Our search tool is PatSeer. Our patent bibliometric data is summarized in various figures and tables. In particular, we qualitatively analyze key deep tomographic patent literature.

Introduction

With the advances of artificial intelligence (AI), deep learning (DL) has emerged as a mainstream approach with successful applications in many areas. Since 2016, deep reconstruction or deep imaging methods have been actively developed, especially in the field of medical imaging [1,2,3]. Promising results on medical imaging are widely reported on diverse topics ranging from data acquisition and processing [4], image reconstruction and enhancement [5, 6] to radiomics and health analytics [7], and more. Clearly, AI/DL is paving an exciting way to improve or innovate medical imaging devices, and diagnostic and therapeutic procedures. With rigorous and systematic assessment and regulation, AI imaging software and devices may assist or compete effectively with radiologists, eventually transforming the current model of medical and healthcare practice in various aspects.

According to a 2020 press release from the Yole Group (https://www.yolegroup.com/press-release/medical-imaging-artificial-intelligence-changes-the-rules/), “AI has the potential to change all of our diagnostics and treatment procedures to enable more personalized and effective medicine.” “At Yole, we estimate the total market in 2025 for software generated revenues through the sale of AI tools will reach US$2.9 billion with a 36% CAGR (Compound Annual Growth Rate) between 2019 and 2025 (Fig. 1). These revenues can be shared between the main applications including improved image capture, noise reduction, image reconstruction, screening, diagnostic and treatment planning.”

Fig. 1
figure 1

AI revenue of the medical imaging companies from 2015 to 2025 (adapted from www.yolegroup.com)

Given the huge commercial potential of AI-based imaging technologies, intellectual property plays an increasingly important role in the imaging industry, research and user communities. In this context, for researchers and developers, patent landscape analysis and literature review are indispensable. To understand the landscape of AI-based deep tomographic imaging technologies that promise to be clinically relevant, we are motivated to survey relevant patent literature over the past decade or so. As used herein, “patent literature” includes issued patents and published patent applications (i.e., pre-grant patent application publications). As further used herein, “patent documents” (“PGPubs”) are also used to mean patent literature.

The rest of this paper is organized as follows. In the second section, we present our methodology. Our main patent search tool is PatSeer [8]. Our search strategy and analysis methods are described in this section. In the third section, we present our PatSeer search results in figures and tables. While the third section is largely data-driven, in the fourth section we analyze key deep tomographic patent documents aided by citation analysis. In the last section, we discuss relevant issues and conclude the paper.

Methods

Over recent years, the importance of patent literature has become more recognized in the academic setting, as evidenced by the establishment of the National Academy of Inventors in 2010. In the context of AI, DL, and data science, patents are relevant to not only industry but also academe and other communities. To perform our landscape analysis of the patent literature on deep tomographic imaging, we used the patent search and analysis tool PatSeer. We note that other patent search and analysis tools are also available including, but not limited to, AcclaimIP (available from Anaqua), LexisNexis TotalPatent One®, etc.

Specifically, the tool we used is PatSeer ProX, developed by Gridlogics. PatSeer ProX includes big-data analytic methods and performs relatively fast through, in our opinion, a very user-friendly interface. We find that the search rules used by PatSeer ProX are like those used by Scopus. According to the PatSeer website, the PatSeer ProX patent database covers more than 136-million patent publications, and 96-million full-text records. Furthermore, PatSeer maintains a scalable big-data platform with AI-based semi-automated algorithms to process and analyze raw data from over 300 sources.

As an illustration, with the “deep learning” as the search phrase in the title, abstract and claim fields from January 1, 2010 to the end of 2021 (TAC:("deep learning") AND PBD:[2010–01-01 TO 2021–12-31]), Figs. 2 and 3 were generated in seconds.

Fig. 2
figure 2

Patenting trend in the field of DL. The chart shows filing and grant data, with the applications being plotted by filing date and the grants by publication date

Fig. 3
figure 3

Distribution of inventors by country of residence. The left chart shows the number of records by the country of residence of inventors. The right chart shows the number of records by the country where the application was filed. Note that a record is included in each unique country (the same record may be repeatedly counted)

Results

Visualization of Patents on Deep Tomographic Imaging

In this section, we focus on our patent search results on AI-based medical imaging techniques. Initially, we performed keyword-based searches in the title, abstract and claim fields but we obtained quite many irrelevant results. For example, the inclusion of claims included French PGPub FR2733596B1 [9] which only mentions ‘Ultrasound’ in the claims. As another example, International PGPub WO0124700A1 [10] is for fingerprint detection. As a result, our searches were limited to the title and abstract fields only. After further deliberation, we defined the following search expression: TA:(("machine learning" or "deep learning" or "deep nets" or "neural network" or "deep network" or "deep neural network" or "artificial intelligence") and ((raw data process* or "k-space data" or "tomographic data" or "sinogram" or image reconstruct* or post-process*) or ("image quality" or "artifact reduction" or “low dose scan” or “fast scan” or “under sample” or “noise reduction”)) and (medical imag* or tomograph* or "CT" or "computed tomography" or "PET" or "positron emission tomography" or "SPECT" or "single photon emission computed tomography" or "nuclear imaging" or "MRI" or "magnetic resonance" or "Ultrasonography" or "ultrasound" or "optical coherence tomography" or "OCT") and not ("display apparatus" or assess*)), where TA means in the title and abstract fields. Note that the filter function, not ("display apparatus" or assess*), was empirically added to exclude irrelevant hits; for example, DE69031523D1 [11] matches the other parts of the overall expression but only describes a method to optimize a display window, and US8086007B2 [12] describes an image quality assessment method. In total, this search yielded 757 records.

Figure 4 summarizes the numbers of patents per application year grouped by the original assignee. Figure 5 lists the top 10 most cited patents in this domain. A few examples are described here. PGPub US20180018757A1 describes a method to improve the quality of the projection data using machine learning (ML) models. International PGPub WO2017223560A1 [13] describes ML-based tomographic/tomosynthetic techniques that use a neural network that includes more than three layers and can be applied to either raw data or initial image domains. PGPub US2020034998A1 [14] is an example of DL for MRI imaging, where a neural network model is used with data consistency.

Fig. 4
figure 4

Innovation timeline by original assignee vs application date, where innovation intensity is shown by the number of assigned patent applications per assignee per year

Fig. 5
figure 5

Top 10 most cited issued patents and PGPubs in the domain of interest

To assess the quality of an issued patent or published patent application quantitatively, PatSeer provides a 360° Quality metric, which is a weighted average of 4 contributing scores. The contributing scores include Citation Quality (CQ), Market Quality (MQ), Legal Quality (LQ) and Document Quality (DQ) [8]. CQ considers the number, recency and type of forward citations of a patent document. MQ is based on the patent family’s global market coverage. LQ reflects how aggressive the company protects the patent family. DQ measures the intrinsic quality of a patent based on its structural elements. Then, the portfolio value index is defined as the sum of the 360 Quality scores of all the patent families present in a portfolio, which, according to Gridlogics, is roughly proportional to the overall realizable value of the portfolio.

Figure 6 shows the most valuable portfolios, according to the PatSeer Quality metric. Among them, Generic Electric Co (GE), Canon Inc and Siemens AG hold the most valuable portfolios as industry leaders, while Rensselaer Polytechnic Institute (RPI), Zhengzhou University and Stanford University are the academic leaders.

Fig. 6
figure 6

Most valuable portfolios (benchmarked by PatSeer’s 3600 score) showing top players with the highest portfolio value index

Analysis of Patents on Deep Tomographic Imaging

We work in the medical imaging field with diverse interests but with an emphasis on computed tomography (CT). Given the broad view presented in the preceding section, in this section we analyze further deep CT reconstruction patent literature and other related technologies. This analysis provides a unique perspective that will help us plan future research and development activities. In the following sub-sections, we analyze patent literature based on specific applications and select representative patent documents for analysis. Tables 1, 2 and 3 list related patent documents and further include brief comments.

Table 1 Representative patent literature for image reconstruction using AI technology
Table 2 Representative patent literature for motion compensation, metal artifacts, and material decomposition
Table 3 Representative patents for radiomics

Tomographic image reconstruction

DL has significant implications for tomographic image reconstruction, as first described in our perspective paper on deep imaging [1]. Our perspective paper presents three specific examples with simulation results; i.e., CT image super-resolution, singogram quality improvement, and CT image denoising. More generally, when a dataset is truncated, for example, in cases of limited-angle, few-view and interior or local reconstruction, or an image is distorted or comprised by artifacts, including metal artifact, scatter artifact, motion artifact and beam-hardening effect, a synergistic combination of conventional tomographic methods and DL-based imaging may enhance image quality and diagnostic performance.

An international patent application (PGPub WO2017223560A1) [13] was filed with a priority date of June 24, 2016, describing tomographic image reconstruction systems and methods based on DL techniques. This PGPub discloses a general framework for image reconstruction from raw data directly or from an intermediate reconstructed image with the DL/ML approach, suggesting a potential for deep tomographic reconstruction to surpass classic reconstruction algorithms.

Another published international patent application (PGPub WO2019074879A1) [15] is directed to image reconstruction with DL/ML. The approach involves a ML model training process and use of the trained model for image generation. The ML model training may use multiple images generated from a single set of tomographic projections or images. The trained model may be used to generate a final image from the projection data, which is a less computationally intensive algorithm. The issued patent US10475214B2 [16] uses ML to solve tomographic reconstruction problems, which are usually in large scale and are space variant while correcting various artifacts. International PGPub WO2018126396A1 [17] presents DL-based raw data correction and estimation for tomographic reconstruction.

International PGPub WO2019060843A1 [18] discloses an image reconstruction method using a ML regularizer. Specifically, an iterative reconstruction technique can incorporate a ML model as a regularization filter for the image reconstruction. International PGPub WO2018236748A1 [19] describes an image reconstruction scheme using DL/ML for tomographic imaging. The method first produces a plurality of intermediate images using iterative reconstruction method and furthermore transforms selected intermediate images using a deep-learning based quasi-projection operator to a regularized intermediate image.

PGPub US2018/0018757A1 [20] discloses a technique for improving projection data via DL/ML. The method transforms projection data from low-quality projection to higher quality, and performs image reconstruction from the improved projection images. A DL model is trained from matched pairs of lower-quality (lower-dose) and corresponding higher-quality (higher-dose) projection data and used to transform lower-quality (lower-dose) projection data to higher-quality (higher-dose) ones after training is finished. PGPub US2020311878A1 [21] describes a feature-aware DL method for image reconstruction. A neural network model is trained to perform feature-aware reconstruction using a target image for spatially-dependent denoising and artifact suppression.

The last technology we comment on is virtual monochromatic CT image reconstruction. International PGPub WO2019067524A1 [22] discloses processing current-integrating data and images via ML to produce virtual monochromatic images. The neural network is configured to learn a nonlinear function to map a CT image reconstructed from projection data collected in a current-integrating X-ray detector to an image reconstructed from a virtual monochromatic projection dataset at a pre-specified kVp energy. The technique realizes monochromatic CT imaging and overcomes the beam hardening problem. PGPub US2020196973A1 [23] discloses an apparatus and method for a dual-energy CT system which uses sparse kVp-switching to collect data and a DL method to reconstruct the image. The deep network is fed with images that include artifacts and is trained to output images with little or no artifacts. Another deep network model can be trained to perform material decomposition from the artifact-free images.

Artifact reduction

Metal artifact reduction is one of the remaining problems in the CT field. European PGPub EP3743889A1 [24] discloses using DL to reduce metal artifacts. The neural network model is trained to generate a metal artifact image and generate a new image by subtracting the estimated metal artifacts from the input image. A relatively high number of patent documents target the same problem. Figure 7 lists some of them, ordered according to the number of forward citations.

Fig. 7
figure 7

Most often cited patent documents related to CT metal artifact reduction, ranked by the number of forward citations

The German PGPub DE102017219307B4 and Chinese PGPub CN109727203B [25, 26] describe a system and method for compensating motion artifacts via ML. The technology relates to an automatic compensation method for motion artifacts in a medical image. The technology further relates to a method of automatically identifying motion artifacts, and includes a compensation unit, a learning device, and a controlling device for medical imaging. A number of patent documents address this problem, as shown in Fig. 8, ranked according to the number of forward citations.

Fig. 8
figure 8

Most cited patent documents on CT motion artifact reduction, ranked by the number of forward citations

The Japanese PGPub (JP2020534929A) [27] presents a DL-based scattering correction for X-ray imaging. A neural network is trained with a Mone Carlo simulated imaging data. This simulation includes at least the X-ray photon scattering mechanism that contaminates CT data with scattering noise. The neural network model learns to remove the scattering noise in the CT data. After CT data are corrected by removing scatters in the projection space, the image reconstruction is performed to generate a scattering-corrected CT image. Scatter correction is also important for positron emission tomography (PET) imaging. A number of patent documents exist related to DL-based scatter correction for either CT or PET, as shown in Fig. 9.

Fig. 9
figure 9

Most cited patent documents on CT/PET scattering correction, ranked by the number of forward citations

Image analysis, radiomics and rawdiomics

International PGPub WO2018232388A1 [28] uses neural networks to integrate tomographic image reconstruction and radiomic analysis. CT screening, diagnosis or image analysis tasks are often performed with separate neural networks and algorithms. Integrating these elements into an end-to-end workflow may streamline the whole process and optimize the task-specific performance. As we know, while deep reconstruction is for image formation from raw data, image analysis or radiomics is for image analysis. Thus, the claimed integration of image reconstruction and image analysis is referred to as ‘rawdiomics’, where ‘rawd’ means raw data, and ‘i’ indicates images or informatics. International PGPub WO2018/220089A1 [29] applies DL/ML to raw medical imaging data analysis for clinical decision support. The techniques are intended for medical diagnosis from raw imaging data generated by a medical imaging machine with or without conventional image reconstruction.

Discussion and conclusions

First of all, we underline that although PatSeer ProX has provided useful information, we are still on the learning curve. We are not familiar with all of the functions and terminology of the PatSeer ProX tool. As a result, we believe that hidden information in the PatSeer dataset can be further mined. While we believe that we have obtained valuable data in this study, omissions and biases are unavoidable, due to the imperfect coverage of the database, dynamic nature of the field, and our limited capabilities. We apologize if we have made any misinterpretations.

Despite any problems possibly existing in this preliminary patent landscape analysis, we have learned significantly, facilitated by PatSeer ProX. Clearly, research and development in deep tomographic imaging has a strong momentum, engages both imaging companies and academic groups, and promises lasting impact on the further research and development as well as market and healthcare. Since the field of deep tomographic imaging is relatively young, more results and data are yet to be collected and analyzed to reveal the patent dynamics in terms of licensing, revenue, and translated outcomes. Also, it may be very informative to use PatSeer ProX, or similar tools, to identify and track emerging areas of AI-based imaging activities.

In conclusion, we have performed a preliminary landscape analysis on patent literature dedicated to deep tomographic imaging. Using the PatSeer ProX tool, we have systematically collected and analyzed relevant bibliometric data, and commented on representative deep tomographic imaging patent documents. Finally, we have discussed several issues and future work on these interesting topics.

Availability of data and materials

Not applicable

Abbreviations

AI:

Artificial intelligence

ML:

Machine learning

DL:

Deep learning

CQ:

Citation Quality

MQ:

Market Quality

LQ:

Legal Quality

DQ:

Document Quality

RPI:

Rensselaer Polytechnic Institute

GE:

Generic Electric Co

CT:

Computed tomography

2D:

Two-dimensional

3D:

Three-dimensional

FBP:

Filtered backprojection

SPECT:

Single photon emission computed tomograph

PET:

Positron emission tomography

References

  1. Wang G (2016) A perspective on deep imaging. IEEE Access 4:8914-8924. https://doi.org/10.1109/ACCESS.2016.2624938

    Article  Google Scholar 

  2. Wang G, Ye JC, Mueller K, Fessler JA (2018) Image reconstruction is a new frontier of machine learning. IEEE Trans Med Imag 37(6):1289-1296. https://doi.org/10.1109/TMI.2018.2833635

    Article  Google Scholar 

  3. Wang G, Jacob M, Mou XQ, Shi YY, Eldar YC (2021) Deep tomographic image reconstruction: yesterday, today, and tomorrow-editorial for the 2nd special issue "machine learning for image reconstruction". IEEE Trans Med Imaging 40(11):2956-2964. https://doi.org/10.1109/TMI.2021.3115547

    Article  Google Scholar 

  4. Lee H, Lee J, Kim H, Cho B, Cho S (2019) Deep-neural-network-based sinogram synthesis for sparse-view CT image reconstruction. IEEE Trans Radiat Plasma Med Sci 3(2):109-119. https://doi.org/10.1109/TRPMS.2018.2867611

    Article  Google Scholar 

  5. Guha I, Nadeem SA, You CY, Zhang XL, Levy SM, Wang G et al (2020) Deep learning based high-resolution reconstruction of trabecular bone microstructures from low-resolution CT scans using GAN-CIRCLE. Paper presented at the SPIE 11317, Medical Imaging 2020: Biomedical Applications in Molecular, Structural, and Functional Imaging, SPIE, Houston, 28 February 2020. https://doi.org/10.1117/12.2549318

  6. You CY, Yang LF, Zhang Y, Wang G (2019) Low-dose CT via deep CNN with skip connection and network-in-network. Paper presented at SPIE 11113, Developments in X-Ray Tomography XII, SPIE, San Diego, 10 September 2019. https://doi.org/10.1117/12.2534960

  7. Avanzo M, Wei LS, Stancanello J, Vallières M, Rao A, Morin O et al (2020) Machine and deep learning methods for radiomics. Med Phys 47(5):e185-e202. https://doi.org/10.1002/mp.13678

    Article  Google Scholar 

  8. Patent scoring in PatSeer. PatSeer. (2020, September 2). https://patseer.com/patent-scoring/. Accessed 4 Jan 2023

  9. Chelveder JC (1997) Method and device for identifying particles. FR Patent 2733596B1, 18 Jul 1997

  10. O'Gorman L, Schuckers S, Derakhshani R, Hornak L, Xia XM, Amour DM (2001) Spoof detection for biometric sensing systems. WO Patent 2001024700A1, 12 Apr 2001

  11. Ohhashi A (1997) Digital image display device. DE Patent 69031523D1, 06 Nov 1997

  12. Yan MXH, Chang TC, Lendl M, Boehm S, Fang T, Durlak P (2011) Method and system for human vision model guided medical image quality assessment. US Patent 8086007B2, 27 Dec 2011

  13. Wang G, Cong WX, Yang QS (2017) Tomographic image reconstruction via machine learning. WO Patent 2017223560A1, 28 Dec 2017

  14. Schlemper J, Salehi SSM, Kundu MSP, Wang ZY, Lazarus C, Dyvorne HA et al (2020) Deep learning techniques for magnetic resonance image reconstruction. US Patent 20200034998A1, 30 Jan 2020

  15. Thibault BJ, Srivastava S, Hsieh J, Bouman CA, Ye D, Sauer K (2019) Image generation using machine learning. WO Patent 2019074879A1, 18 Apr 2019

  16. Fu L, Ramani S, De Man BKB, Rui X, Ahn S (2019) Tomographic reconstruction based on deep learning. US Patent 10475214B2, 12 Nov 2019

  17. De Man KB, Erich B, Claus H, Hsieh J, Jin YN, Xing ZF (2018) Deep learning based estimation of data for use in tomographic reconstruction. WO Patent 2018126396A1, 12 Jul 2018

  18. Atria C, Ramesh N, Yatsenko D (2019) Image reconstruction using machine learning regularizers. WO Patent 2019060843A1, 28 Mar 2019

  19. Anastasio M, Matthews T, Kelly B (2018) Deep learning-assisted image reconstruction for tomographic imaging. WO Patent 2018236748A1, 27 Dec 2018

  20. Suzuki K (2018) Transforming projection data in tomography by means of machine learning. US Patent 20180018757A1, 18 Jan 2018

  21. Matsuura M, Zhou J, Yu Z (2020) Apparatus and method for image reconstruction using feature-aware deep learning. US Patent 20200311878A1, 1 Oct 2020

  22. Wang G, Cong WX (2019) Monochromatic CT image reconstruction from current-integrating data via machine learning. WO Patent 2019067524A1, 4 Apr 2019

  23. Zhou J, Liu Y, Yu Z (2020) Apparatus and method for dual-energy computed tomography (ct) image reconstruction using sparse kvp-switching and deep learning. US Patent 20200196973A1, 25 Jun 2020

  24. Xu SY, Dang H (2020) Using deep learning to reduce metal artifacts. EP Patent 3743889A1, 2 Dec 2020

  25. Allmendinger T, Schmidt B (2019) Method and system for compensating motion artifacts by machine learning. DE Patent 102017219307B4, 11 Jul 2019

  26. Allmendinger T, Schmidt B (2020) Method and system for compensating motion artifacts by means of machine learning. CN Patent 109727203B, 28 Jul 2020

  27. シユ スー, ペーター プリンセン, ジェンス ウィゲルト, ラヴィンドラ モハン マンジェシュワー (2021) Deep learning-based scattering correction. JP Patent 2020534929A, 3 Dec 2021

  28. Wang G, Kalra M, Hahn J, Kruger U, Cong W X, Shan H M (2018) Systems and methods for integrating tomographic image reconstruction and radiomics using neural networks. WO Patent 2018232388A1, 20 Dec 2018

  29. Swisher CM, Pien H (2018) Machine learning on raw medical imaging data for clinical decision support. WO Patent 2018220089A1, 6 Dec 2018

Download references

Acknowledgements

The authors are grateful for the standard free trial of the PatSeer ProX software provided by Gridlogics Technologies, Pune, India (www.patseer.com). In the trial period, Mr. Prashant Nair, Business Development Manager with PatSeer, has provided excellent technical support.

Funding

This work was partially supported by US National Institutes of Health, Nos. R01EB026646, R01CA233888, R01CA237267, R01HL151561, R21CA264772, and R01EB031102.

Author information

Authors and Affiliations

Authors

Contributions

QY drafted the manuscript. All the authors contributed in data analysis and paper revisions. The authors read and approved the final manuscript.

Corresponding author

Correspondence to Ge Wang.

Ethics declarations

Competing interests

Qingsong Yang, Wenxiang Cong, and Ge Wang are inventors of the deep tomographic imaging patents at Rensselaer Polytechnic Institute, Troy, New York, USA.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, Q., Lizotte, D.L., Cong, W. et al. Preliminary landscape analysis of deep tomographic imaging patents. Vis. Comput. Ind. Biomed. Art 6, 3 (2023). https://doi.org/10.1186/s42492-023-00130-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s42492-023-00130-x

Keywords