 Original Article
 Open access
 Published:
Features of hardware implementation of quasicontinuous observation devices with discrete receivers
Visual Computing for Industry, Biomedicine, and Art volume 5, Article number: 6 (2022)
Abstract
This article proposes an approach to the formalization of tasks and conditions for the hardware implementation of quasicontinuous observation devices with discrete receivers in remote sensing systems. Observation devices with a matrix are used in medicine, ecology, aerospace photography, and geodesy, among other fields. In the discrete receivers, the sampling of an image in the matrix receiver into pixels leads to a decrease in the spatial information of the object. In a greater extent, these disadvantages can be avoided by using photosensitive matrix with a regularly changing (controlled) density of elementary receiversmatrix (RCDOERmatrix). Currently, there is no substantiation of the tasks and conditions for the hardware implementation of RCDOERmatrix. The algorithmic formation of a quasicontinuous image of observation devices with the RCDOERmatrix is proposed. The algorithm used a formal pixelbypixel description of the signals in the image. This algorithm formalizes the requirements for creating a photosensitive RCDOERmatrix of a certain size, as well as for changing the mechanism for forming and saving a frame with observation results. The application of the developed method will allow multiplying the pixel size of the image relative to the pixel size of the RCDOERmatrix. Developed algorithms for RCDOERmatrix are supplemented by formalizing the tasks that arise when creating prototypes. In addition, the conditions for hardware implementation are proposed, which ensure the completeness of registration of the observation picture, and allow avoiding excessive pixel measurements. Thus, the results of the research carried out approximate the practical application of RCDOERmatrix.
Introduction
Electronic images in observation devices are obtained using the photosensitive matrix as a receiver. Matrix has a pixel size that allows for our eye to receive an image as continuously. Only at high magnification, the image is structured into individual pixels. Each of the pixels is in the state \({\tilde {x}_j}\), which depends on the brightness u_{j} of corresponding area in observation picture.
With such approach, the appearance of the U object can be considered as a set of N conjugate sections \({u_j}\) in the space Ω [1]:
According to Eq. (1), the image X of the object’s appearance is a set of signals \({\tilde {x}_j}\) [2, 3] functionally related to \({u_j}\) from photosensitive matrix:
Discretizing the image in the receiver into individual pixels leads to the loss of a portion of the spatial information. Within each pixel, there is no information about x in \({\tilde {x}_j}\). If the space \({\Delta} {\Omega}_j\) corresponds to an individual pixel, the description of image X in Eq. (2) can be refined as follows:
Some details of the image of an object fall on the insulating surface Δ in the interim of pixels in matrix. Such details are not involved in photoelectric conversion of the value \({\tilde {x}_j}\).
The photosensitive matrix is designed to minimize losses. The insulating surfaces Δ must be maximally thin, and the pixels maximally small. It can be written as follows:
Spatial discretization of the domain Ω in the observation devices depends on the design of the photosensitive receiver. Its standard design contains a system of elementary photosensitive receivers. The rectangular layout system assumes the presence of m rows with n elementary receivers.
The pixel size of the receiver is equal to the total number of elementary receivers. A signal is generated from each elementary receiver during an exposure. Such a signal sets the state of a specific pixel in the image. With such logic of operation, both the image and the receiver pixel sizes are the same. Then
According to Eq. (5), with a decrease of \({\Delta} {{\Omega}_j}\) the number of receivers increases. In this case, it is necessary to leave the insulating surfaces Δ maximally thin. Equation (4) can be refined as follows:
Thus, a successful species observation in advance presupposes the presence of an unstructured receiver. The full implementation of such requirements is technologically and physically difficult. The search of opportunity to fulfill Eq. (6) involves following hardware and algorithmic solutions: (1) constructive improvements [4, 5] in using multicamera observation devices [6] (which, however, complicate the design of observation devices); (2) a decrease in the size of an elementary receiver to the diffraction limit, so more receivers can be placed on a unit of surface matrix [7, 8] (the implementation of which is technologically complicated and limited by theoretical limits of applicability); (3) consideration of the design of the real tasks of a receiver for observing objects [9] (it reduces the possibility of observing all objects in the image); (4) taking into account the spectral properties of objects of observation and using additional signs of their recognition [10, 11] (during the hyperspectral observation of an object, a large amount of information is accumulated, and its interpretation requires a high and specific operator qualifications); (5) use of regularly changing (controlled) density of elementary receiversmatrices (RCDOERmatrices) in the structure of the species tool (RCDOERmatrices is only declared, the tasks and conditions of effective work are not formulated); and (6) application of special procedures for processing observation results [12,13,14,15,16] (the possibilities of algorithmic processing are a posterior, and they are always limited to the results of apparatus observation).
As a result, the problem of using RCDOERmatrices in systems of observation has not been completely solved at present.
The aim of this study is to formalize the tasks and conditions for the hardware implementation of quasicontinuous observation devices with discrete receivers in remote sensing systems. This goal is achieved because of the algorithmic formation of the RCDOERmatrix. The algorithm used is a formal pixelbypixel description of the signals in the image and the elements of the matrix calculus.
Methods
The use of the RCDOERmatrix provides elementbyelement scanning of the optical image. The photosensitive RCDOERmatrix is characterized by reduced density of distribution of receivers. The density is specified by the multiplicity of their arrangement along rows k_{1} and columns k_{2}. The fold in both directions is estimated by the ratio of the distance between two adjacent receivers to their linear size.
During the scanning, RCDOERmatrix is shifted within the optical image of the object. The nature of the displacement provides a view of all the details of the image. In the initial position and after each displacement by the specified number of pixels, the X_{r} snapshot is taken. The number of K snapshots provides the scanning of all elements of the optical image. Combining all K snapshots forms an X frame with the object image.
The formalization of the frame of the electronic image X will be as follows [14]:
In Eqs. (7) and (8), the maximum possible reduction in the size of individual pixel is taken into account. Its dimensions are limited only by the physical feasibility. The frame is composed of several snapshots. It allows increasing the pixel size. This is achieved through an intramatrix scanning of the optical image.
Let us formalize Eq. (7) for reading and memorization in the observation devices. In Eqs. (3) and (8), an image in a frame is considered as collection of individual pixels. This collection is determined by the sign of the summation. The similarity of a digital image to an optical image is achieved because of the systemic arrangement of the elementary receivers of the photosensitive matrix. These are arranged in rows and columns. Thus, the order of reading and storage of digital snapshots of one frame in a matrix form is determined.
In the RCDOERmatrix, x_{r,j} is a response to the input signal of pixel from jth column and ith row [4]. A matrix X_{r} of signals from the rth snapshot (r = 1, 2, …, K) consists of the signals x_{r,j} of all elementary receivers of the RCDOERmatrix. Equation (7) can be written in a matrix form as follows:
Its dimensions of the matrix X_{r} are (m × n). The matrix of X is obtained by summing the snapshots K. The elements of X are signals x_{r,i,j}:
The dimension of the matrix is then defined as follows:
An example of a frame formation is shown in Fig. 1 [14]. It corresponds to the operation of a RCDOERmatrix with multiplicities of placement k_{1} = 2 and k_{2} = 1. Pixels d are physically present on the matrix. The rest of pixels are virtual.
The frame consists of six snapshots. At the initial moment of time, the first snapshot is exposed. Its image is constructed using signals from elementary receivers d. This is snapshot 1 X_{1}. Next images are exposed after moving the matrix along the trajectory of positions 2, 3, 4, 5, and 6 at a distance d. The following pictures are exposed in each of the positions: X_{2}, …, X_{6}.
Each snapshot X_{r} includes pixels (m × n). Each pixel is a carrier of a signal about the brightness of a point in the environment associated with it. The total number of pixels in K frames m_{Σ}×n_{Σ} is equal to the number of discretization elements of the environment. The image has a pixel size Kfold larger than the pixel size of the photosensitive matrix.
The hardware implementation of this method involves solving two problems. A photosensitive RCDOERmatrix of size (m × n) needs to be created with reasonable multiplicity of placement for pixels horizontally and vertically.
The pixel density is set by the distance between adjacent horizontally and vertically pixels. This is a multiple of the size of the pixel size d. In such a case, RCDOERmatrix is equivalent to a matrix of size \(m_{\Sigma} \times n_{\Sigma}\). This is the virtual pixel size.
In a conventional matrix, the pixel size increases as the surface area increases or the pixel area decreases. In a RCDOERmatrix, an increase in the pixel size is achieved by a decrease in the pixel distribution density. In this case, the multiplicity of the pixel placement increases. The virtual pixel size of the RCDOERmatrix \(m_\Sigma \times n_\Sigma\) is calculated as follows:
A regularly changing density of the RCDOERmatrix of the elementary receivers of the same pixel size, but with different multiplicities of placement, will have different virtual pixel sizes. The pixel size of the image is determined by the virtual size of the RCDOERmatrix, which will increase the resolution and quality of the image.
Results and Discussion
In Table 1; Figs. 2 and 3, the results of the calculations and experiments for the virtual sizes of the RCDOERmatrix with different physical pixel sizes are shown. In the calculations, the following equation was used:
The calculation results are provided for case k_{1} = k_{2} = k. The magnitude of the multiplicity k is considered within the range of 1–8. The pixel dimensions of the RCDOERmatrix are as follows: 0.31, 2.0, and 4.0 MP.
The virtual pixel size increased 81fold when the multiplicity of the pixel placement from 1 to 8 is changed. For a matrix with a physical size of 0.31 MP, the virtual size increases to 25.11 MP; for a 2 MP matrix, up to 162 MP; for a 4 MP matrix, up to 324 MP. Examples of images with the virtual pixel size of the equivalent matrix are shown in Fig. 3. Information about the parameters of the equipment that was used in the experiment is presented in Table 2. For modeling, a part of the image had a pixel size of 16 × 16.
In addition, a change in the mechanism is needed for forming and saving a frame with the observation results. Snapshot contains signals from pixels. For the rth snapshot, the x_{r,i,j} signal comes from pixel located in the ith row and jth column of the RCDOERmatrix. The set of all values of the snapshot signals comprises a rectangular matrix of m × n in size. None of the snapshots contained duplicate pixels.
In this case, Eqs. (9) and (10) are considered as matrices. The elements of the matrix description of the frame are signals from the pixels of all images in the snapshots. The matrix description of the frame image contains matrix elements of the following types [14]:
For Fig. 1, the added snapshots are taken at the displacement along the r_{1} row and r_{2} column. In this case,
Then, Eq. (10) can be rewritten in a matrix form:
The matrices X and X_{r} must have the same size horizontally and vertically. Only such a matrix can be added.
From Eq. (13), it follows that the values of \(m_\Sigma\) and \(n_\Sigma\) are determined by Eq. (12). The size of the matrix signals of each snapshot should be \(m_\Sigma \times n_\Sigma\). To achieve this, zeros were added to the columns and rows. The rows and columns that were not used in this snapshot were given a value of zero. If \({k_1} \ne 0\), zero rows should appear, if \({k_2} \ne 0\) zero columns should appear. If \({k_1} \ne 0\) and \({k_2} \ne 0\), then zero rows and columns will appear.
The general rule for the sequence of recording signals from the frame pixels will be as follows [14]:
where j_{r} is the ith pixel number in the rth frame snapshot.
It is necessary to observe the sequence of pixels, which are recorded by both the number of elementary receivers and the number of snapshots. It is convenient to form a frame from snapshots in a matrix form. The structure of the frame memorization register is determined by dimension of the image memorization register, as well as the trajectory of the image movement during intramatrix scanning. The simplest version of the trajectory is linear. In this case, the measured value x_{r,i,j} is written into each ijcell of the register sequentially after each snapshot.
For observation devices with a conventional photosensitive matrix, the concepts of a frame and a snapshot coincide: \(X={M_s}\left\{ {{x_i}} \right\}={M_f}\left\{ {{x_i}} \right\}\). The sets of M_{s} and M_{f} correspond to the number of bins N of the image. The number of bins N of image X is equal to the number of elementary receivers (m × n) of the matrix. The same number is for the bins per frame, i.e., \(X=\left\{ {{x_i}} \right\}\):
When constructing an image using the RCDOERmatrix, a frame with M_{f} sampling elements is formed from K snapshots. Each snapshot had N bins. The design of the RCDOERmatrix and scanning system ensures that certain conditions are met.
At the physical level, Eq. (18) provides a digital representation of the environment. Two basic requirements are formalized. Equations (14), (15), (17), and (18) set the relationship between the number of snapshots in a frame. This is a condition for the completeness of documentation to all discrete elements of space Ω.
The fulfillment of the lower condition presupposes the absence of common signals from sampling elements of space Ω in the snapshots. This is the condition for the absence of intersections between images.
The size of the frame matrix horizontally (1 + k_{1})fold and vertically (1 + k_{2})fold is larger than the size of the photograph frame matrix. The matrix summation is applied only with matrices of the same size. Therefore, for \({k_1},{k_2} \ne 0\), zero rows and columns must be entered into the snapshot signal of the matrix. If \({k_1} \ne 0\), zero rows should appear, if \({k_2} \ne 0\), zero columns should appear. In terms of hardware, the matrix analogy simplifies the register distribution of signals from pixels and the reproduction of a frame image.
Hardware requirements need to be defined. Let us describe the theoretical justification for the time that is need for creating snapshots and frames. The exposure time of one snapshot is t_{e}. Moving the matrix by one pixel (2, …, 8 μm) is t_{s}. This did not exceed the exposure time. During this time, the pn transitions of photosensitive receivers relax (10^{−6} s). The exposure time t_{f} of the frame is as follows:
The standard average shutter speed at the lowest sensitivity level in the camera is 0.01 s. Then, up to 20 frames were formed in 1 s. With an average photosensitivity, this is already several tens of frames, according to Eq. (19). This frame rate will allow video recording.
Displacement of the image during formation of each frame is not a problem. In each camera there is a mechanism for compensating for imageshift. It provides image displacement in much the same way as RCDOERmatrix.
Physical limitations on the possibility of observation with RCDOERmatrix are: (1) the minimum possible or reasonable pixel density; and (2) precision regarding the realization of the image scanning trajectory.
We can use RCDOERmatrices in all technical devices designed to observe objects by its digital image. The advantages of using RCDOERmatrices as a receiver of radiation are as follows: (1) obtaining of a digital image with pixel size that is multiples of the pixel size of the matrix (all pixels of a digital image are result of physical measurement of the brightness of the conjugate area, and it is not the result of calculation); (2) simplification of the technology of manufacturing photosensitive matrices (this is important if you need to use multimegapixel matrices); and (3) decrease of diffraction influence at elementary receiver of the matrix on measured values of brightness in neighboring pixels (this property is interesting when miniaturizing the size of photosensitive matrices, as well as when constructing images in the high frequency range).
For using RCDOERmatrices in real equipment it is necessary to solve the following issues: (1) to determine the rational level of density of elementary receivers in the photosensitive matrix [it corresponds of multiplicity of the arrangement of elementary receivers along the horizontal and vertical (Eqs. 11, 12, and 13), the limiting factor is the degree of dynamism of the object that is supposed to be observed with the equipment, as consequence, this is a limitation on the number of shots in a single frame (Eq. 10)]; (2) to set the trajectory of movement of optical image along the RCDOERmatrix [the reproduction of observation depends on accuracy of the trajectory (Eq. 18)]; (3) to place accurately the elementary receivers in RCDOERmatrix [it will provide adequate representation of the observed picture in its digital image and prevent intersections between images of individual snapshots (Eq. 18)]; and (4) to adapt the recording of signals from pixels of the frame to the conditions of using the RCDOERmatrix (Eq. 16).
Conclusions
Observation devices with a matrix are used in various fields, including medicine. Sampling of the image into pixels will decrease spatial information of the object. This disadvantage can be avoided by using RCDOERmatrix. For example, with it we can get 324 megapixel images by using a 4 megapixel matrix. We do not need to increase the pixel size of the photosensitive matrix. It is achieved by inside matrix scanning of the optical image. The algorithms that were developed for the functioning of RCDOERmatrix have a declarative nature. The research results, formalize tasks that arise for creating prototypes of RCDOERmatrix. The conditions for hardware implementation are proposed.
The structure of the frame memorizing register is proposed. Also, it is determined by the dimension of the image memorizing register, as well as the trajectory of the image movement during intramatrix scanning. The disadvantages of using RCDOERmatrices in observation devices are assessed. Thus, the results of the research carried out approximate the practical application of RCDOERmatrices for observation devices.
Availability of data and materials
Not applicable.
Abbreviations
 RCDOER:

Regularly changing (controlled) density of elementary receivers
References
Krasilnikov NN (2011) Digital Processing of 2D and 3D Images. BHV, St. Petersburg
Krasilnikov NN, Krasilnikova OI (2002) Study of the efficiency of the human visual system in recognizing static images. J Opt Technol 69(6):397–403. https://doi.org/10.1364/JOT.69.000397
Mitchell EN (1984) Photographic science. John Wiley and Sons, New York
Popov M (2018) On the technology of creation of new technologies in the area of remote sensing of the Earth. Ukrainian J Remote Sens 17:4–9. https://doi.org/10.36023/ujrs.2018.17.127
Popov MA, Stankevich SA, Shklyar SV (2016) Algorithm for enhancing the distributional efficiency of images displaced by subpixels. Mathematical Machines and Systems 56:29–36
LSST (2021) The largest digital camera in the world has a resolution of 3.2 gigapixels. https://tehnobzor.ru/stati/samajabolshajakamera. Accessed 27 Jan 2021
Burgett G (2021) OWC unleashes 2nd gen ThunderBlade SSD drives with a top speed of 5000 MBps. https://www.dpreview.com/news/2707357090. Accessed 18 Apr 2021
Rehm L (2021) Samsung launches 1/3.4” 20 MP sensor for use in smartphone front cameras and tele modules. https://www.dpreview.com/news/5407916317. Accessed 24 Jan 2021
Korobchynskyi M, Slonov M, Rudenko M, Maryliv O, Pylypchuk V, Moldovan V (2020) Improvement of the algorithm of determination parameter of photography in the conditions of light sensitivity. In: Abstracts of the 2020 IEEE 3rd international conference on data stream mining and processing, IEEE, Lviv, 2125 August 2020. https://doi.org/10.1109/DSMP47368.2020.9204161
Ferraris V, Dobigeon N, Wei Q, Chabert M (2018) Detecting changes between optical images of different spatial and spectral resolutions: A fusionbased approach. IEEE Trans Geosci Remote Sens 56(3):1566–1578. https://doi.org/10.1109/TGRS.2017.2765348
Popov MO, Stankevich SA, Kozlova AO (2007) Method for enhancing spectral variability of lagospectral aerospace images.Patent 81195 UA, 12 Oct 2007.
Stankevich S, Maslenko O, Andronov V (2020) Neural network technology adaptation to the smallsize objects identification in satellite images of insufficient resolution within the graphic reference images database. Ukrainian J Remote Sens 27:13–17. https://doi.org/10.36023/ujrs.2020.27.175
Kwan C (2018) Image resolution enhancement for remote sensing applications. In: Abstracts of the 2nd international conference on vision, image and signal processing, Association for Computing Machinery, Las Vegas, 2729 August 2018. https://doi.org/10.1145/3271553.3271590
Slonov M, Maryliv O (2021) A method of quasicontinuous image formation in observation devices with discrete receivers. Ukrainian J Remote Sens 8(2):4–11. https://doi.org/10.36023/ujrs.2021.8.2.192
Bodonyi A, Kunkli R (2020) Efficient object location determination and error analysis based on barycentric coordinates. Vis Comput Ind Biomed Art 3:18. https://doi.org/10.1186/s4249202000052y
Guo ML, Zhao M, Cheong AMY, Dai HJ, Lam AKC, Zhou YJ (2019) Automatic quantification of superficial foveal avascular zone in optical coherence tomography angiography implemented with deep learning. Vis Comput Ind Biomed Art 2:21. https://doi.org/10.1186/s4249201900318
Acknowledgements
Not applicable.
Funding
Not applicable.
Author information
Authors and Affiliations
Contributions
OM and MS wrote the manuscript; OM did editing. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Maryliv, O., Slonov, M. Features of hardware implementation of quasicontinuous observation devices with discrete receivers. Vis. Comput. Ind. Biomed. Art 5, 6 (2022). https://doi.org/10.1186/s42492022001027
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s42492022001027