 Original Article
 Open Access
 Published:
Vector textures derived from higher order derivative domains for classification of colorectal polyps
Visual Computing for Industry, Biomedicine, and Art volume 5, Article number: 16 (2022)
Abstract
Textures have become widely adopted as an essential tool for lesion detection and classification through analysis of the lesion heterogeneities. In this study, higher order derivative images are being employed to combat the challenge of the poor contrast across similar tissue types among certain imaging modalities. To make good use of the derivative information, a novel concept of vector texture is firstly introduced to construct and extract several types of polyp descriptors. Two widely used differential operators, i.e., the gradient operator and Hessian operator, are utilized to generate the first and second order derivative images. These derivative volumetric images are used to produce two anglebased and two vectorbased (including both angle and magnitude) textures. Next, a vectorbased cooccurrence matrix is proposed to extract texture features which are fed to a random forest classifier to perform polyp classifications. To evaluate the performance of our method, experiments are implemented over a private colorectal polyp dataset obtained from computed tomographic colonography. We compare our method with four existing stateoftheart methods and find that our method can outperform those competing methods over 4%13% evaluated by the area under the receiver operating characteristics curves.
Introduction
Imaging tissue textures have become a widely researched topic in the field of medical diagnosis within recent years. As machine learning methods have grown through more powerful computers and computational algorithms, the research on tissue textures, particularly the lesion textures, has grown rapidly in the field because of the everexpanding number of medical applications. One of the preliminary applications for texture analysis is lesion classification based on the heterogenous characteristics of the image contrast within and surrounding the lesion [1, 2]. Many studies have demonstrated that textures should be an important expression for the heterogeneities of medical images which describe some distinct morphological and phenotypic profiles and has become a critical measure in benign and malignant differentiability [3,4,5,6,7,8]. Using some texture measures computed from medical images, the tissue textures have demonstrated a powerful ability for computeraided detection and diagnosis across a spectrum of diseases [9,10,11]. Among the most popular texture measures, some common examples include the graylevel cooccurrence matrix (GLCM) measure [12], graylevel runlength features [13], and the first order statistics features [14].
Research on tissue textures has shown the importance of the image intensity gradient within medical images [14,15,16,17]. By using the derivative operator on these medical images, the gradient information can be effectively encoded within the texture. The most notable examples of using the gradient information for textures are the histogram of oriented gradient (HOG) features [18] and the cooccurrence of local anisotropic gradient orientations (CoLIAGe) features [19]. The HOG features aim to bin the orientations of the gradients and use each histogram bin as an input feature for classification. The CoLIAGe features aim to compute the local entropy using the cooccurrence matrix (CM) of patches of voxels and the gradient angular information. The computed local entropy values are binned in a histogram, and then the histogram bins are used as input features for classification. These two methods have demonstrated a way of using the directional information in the gradient domain and coding the information into texture measures.
An alternative strategy was explored to investigate what additional information the higher order derivative images can provide beyond the original intensity image [20]. The idea is to magnify the original image contrast distribution at different orders for different texture patterns, aiming to encode as much information as possible about the lesion heterogeneity into quantitative texture measures. By using the first and secondorder derivatives to obtain the corresponding gradient and curvature images, exploratory studies [20, 21] provided an alternative way of encoding the higher order derivative image information into texture measures like GLCM.
While the above feature extraction methods have shown their potential in applications, they have their own limitations. For example, HOG method lacks any operation to extract information from the neighboring voxels that a CMbased method can provide. While the CoLIAGe method uses the CM, it limits itself only to the entropy values and considers only the local patches. The previous methods [20, 21] limit their use of GLCM to only the magnitude of the higher order images.
To address the limitations of the above methods, we further explore the use of higher order derivative information for polyp description and classification via computed tomographic colonography (CTC). Colorectal cancer (CRC) is one of the leading causes of cancer related deaths worldwide and accurate diagnosis remains a challenging task for radiologists [22]. While standard endoscopic colonoscopy is still the most popular screening method for CRC, CTC has grown to become a viable screening option to detect and diagnose both the precursor polyps and cancerous lesions. CTC is a noninvasive procedure which often makes it more palatable for patients, but unlike endoscopic colonoscopy, it cannot resect any concerning polyps. Accurate diagnosis through imaging textures of these polyp masses can better assist physicians to determine a plan of treatment while reducing costs for biopsy and pathology procedures [23], though computeraided diagnosis of CTC polyps have been studied [21, 24, 25]. We focus on a dataset of polyp masses, or those polyps which have grown to be larger than 30 mm. These polyps require surgical intervention to remove, and the choice for how aggressively surgeons may cut into the colon for removal is determined by the malignancy of the lesion.
In this study, we look forward to expand upon the vector model introduced in ref. [26] to more deeply evaluate the image textures generated from the first and secondorder derivative information. Two differential operators, i.e., the gradient and Hessian operators, are employed to extract six local geometric measures, three of which are from the gradient operator and the other three are from the Hessian operator. The three local geometric measures of gradient operator are utilized to construct an anglebased and a vector (including both angle and magnitude)based vector texture images (VTIs). Similarly, three local geometric measures of Hessian operator are utilized to construct an anglebased and a vectorbased VTIs. From each VTI, a vectorbased cooccurrence matrix (VCM) is proposed to compute a twodimensional (2D) texture pattern along an angle through the VTI space, similar to the application of GLCM to the intensity images [12]. A series of 2D texture patterns is then obtained along different angles and called VCMs hereafter. From the computed series of 2D texture patterns or VCMs, several texture features can be derived and then fed to a classifier to perform the polyp classification.
The remainder of this study is organized as follows. Methods/experimental section describes the methods used to generate the new vector texture features (VTFs) and Results and discussion section presents the results obtained from all classification experiments using these new VTFs. Discussions and conclusions are drawn in last section.
Methods/experimental
To overcome the challenges of limited soft tissue contrast from CT images, we first utilize the derivative operator to magnify the contrast. Then four VTIs are generated, two of which are from the gradient operator and the other two are from the Hessian operator. By applying the vectorbased CM or VCM operator to each VTI, a series of VTFs are obtained and fed to a classifier. Figure 1 shows the flowchart of this work to outline the methods used to generate the derivative images, convert them into the associated vectors, and then input into a CM to form a set of vectortextures.
Vector texture definition in gradient domain
Suppose a scalar function or intensity image \(I=I(x,y,z)\) in threedimensional (3D) space. Its gradient would be represented by the following vector function:
where \(\partial {I}_{x}, \partial {I}_{y},\mathrm{ and }\partial {I}_{z}\) are three partial differential parts. Since the intensity image is discrete and not continuous, we employ the wellestablished Sobel Operator kernels [27] to acquire the partial derivatives of the image. It is difficult to describe the object using the coordinates directly since the coordinate is a relative metric which could be changeable under different geometric transforms [28, 29]. Fortunately, a vector could be expressed by the geometric metrics of magnitude and direction. In 3D space, the direction of the vector has two components, the azimuth angle and polar angle, as shown in Fig. 2. To calculate the magnitude and direction of a gradient vector, a common method used is to translate the vector from Euclidean coordinates to spherical coordinates as follows:
where \(\left\Delta I\right\) is the magnitude, \({\theta }_{g}\) is the azimuth angle and \({\phi }_{g}\) is the polar angle. Their definitions are given below:
Unlike the CoLIAGe approach [19], where the azimuth and polar angles are used individually to extract texture features, we treat these two angles together as a description of gradient angle vector (GAV) of unit magnitude as:
where GAV only contains partial information of gradient assuming unit magnitude. In addition, we further employ the magnitude and direction to compose a total gradient vector (TGV), which preserves all gradient information:
Our next step is to quantify the Formulas (6) and (7) to generate two VTIs. By their definitions, the range of the azimuth \({\theta }_{g}\) is \(\left[0,\right.\left. 2\pi \right)\), and the range of the polar angle \({\phi }_{g}\) is in \(\left[0,\right.\left. \pi \right)\). Their digitalization could be realized by:
where \({Q}_{g}^{a}\) and \({Q}_{g}^{p}\) denote their gray levels, respectively, and \(\lfloor{X}\rfloor\) represents the infimum of X.
The range of the gradient magnitude tends to be very wide, however most of that information is clustered within a narrow region. This can be seen in the distribution of magnitudes for our polyp dataset in Fig. 3b. Moreover, the CM is very sparse when computed from the unbalanced value distribution. A nonuniform sampling based on histogram equalization [30] can be a solution, which would generate very flat histogram as shown in Fig. 3c. This type of sampling methods treats every image component or region equally and ignores the different contributions from image component in different polyp subtypes. Hence, before quantification on the gradient image, we use a onetoone mapping to change the pixel/voxel magnitude distribution by:
where \(\left\Delta I\right\) is the gradient magnitude of the original gradient image and t is an integer. Then a uniform gray scaling method is applied on \({\Delta I}_{g}\) by:
where \({\Delta I}_{g}\) was defined before as the remapped gradient magnitude, \({\Delta I}_{g}^{max}\) and \({\Delta I}_{g}^{min}\) are the maximum and minimum of \({\Delta I}_{g}\), and \({Q}_{g}^{m}\) is its maximum gray level number. Its histogram is shown in Fig. 3d.
After the quantification via Formulas (8, 9, 10 and11), we obtain two VTIs, corresponding to the definitions of GAV and TGV in the gradient image domain, as follows
Vector texture definition in Hessian domain
In mathematics, Hessian operator of a scalar function or intensity image \(I=I(x,y,z)\) in 3D space could be defined by a 2^{nd} order derivative matrix as the follows:
where I_{xx}, I_{xy}, I_{xz}, I_{yy}, I_{yz}, and I_{zz} are the 2^{nd} order derivatives of \(I(x,y,z)\). To compute these partial derivatives of the polyp images, we use the wellestablished Deriche filters [31] with parameter \(\alpha =1\).
Due to the number of unique variables in the Hessian matrix we use matrix decomposition to reduce the dimensionality and extract the three eigenvalues \({\lambda }_{1}\ge {\lambda }_{2}\ge {\lambda }_{3}\). Since the Hessian matrix is a real symmetric matrix, this guarantees its eigenvalues and eigenvectors are all real values. We compose a 3D vector field, similar to the vector image construction in the gradient domain, as follows:
Like the gradient, this vector could be expressed by spherical coordinates as below:
where \(\uplambda\) is the magnitude, \({\theta }_{h}\) and \({\phi }_{h}\) are two angles representing the direction as shown in Fig. 4. They are defined as follows:
Again, the Hessian angle vector (HAV) is defined as:
The magnitude, azimuth angle and polar angle compose the total Hessian vector (THV) given by:
Similar to the presentation in the gradient domain, we use the same method to perform the digitalization of \({\theta }_{h}\) and \({\phi }_{h}\) in the Hessian domain by:
where \({Q}_{h}^{a}\) and \({Q}_{h}^{p}\) denote their gray levels or quantification numbers, respectively, and ⌊X⌋ was defined before as the infimum of X.
The gray scaling of Hessian magnitude is implemented by:
where \({\Delta I}_{h}=\sqrt[t]{\uplambda }\) is the remapped Hessian magnitude similar to Formula (10), \({\Delta I}_{h}^{max}\) and \({\Delta I}_{h}^{min}\) are the maximum and minimum of \({\Delta I}_{h}\), and \({Q}_{h}^{m}\) is the maximum gray level number.
Thus, another two VTIs, corresponding to the definitions of HAV and THV in the Hessian image domain, are obtained as follows:
Vectorbased CM or VCM
Given above vector discretization, we present a new method based on the traditional GLCM called the vectorbased CM, or VCM, and is expressed below as:
where \({T}_{i}\) represents the VTIs in 3D space, \(i\in \{1, 2, 3, 4\}\), (R, C, S) is the volume size, \({V}_{1}\) and \({V}_{2}\) are a vector pair in \({T}_{i}\), and \(d\) is the displacement between two voxels along the direction\(\left(\alpha ,\beta \right)\), and \(d*(\alpha ,\beta )=(d*cos\alpha cos\beta , d*sin\alpha cos\beta , d*sin\beta )\).
Given each of the four VTIs from Formulas (12), (13), (25), and (26), the Formula (27) generates a VCM along a direction in a similar way as the Haralick method [12] does. The Haralick method generates a GLCM along a direction from the original intensity image and our proposed method generates a VCM along a direction from the higher order VTIs. Because of this similarity, the Haralick method will be implemented as reference for comparison purpose.
A voxel within a 3D volume consists of 26 nearest neighbor voxels from which the VCM directions can be derived from. However, half of these directions produce redundant information as a transpose of their opposite direction. Therefore, only 13 directions are used for the VCM calculation and are denoted as: (0, 0, 1), (0, 1, 0), (1, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 0), (1, 1, 0), (0, 1, 1), (1, 0, 1), (1, 1, 1), (1, 1, 1), (1, 1, 1) and (1, 1, 1). Thus, using the VCM definition of Formula (27), we will compute 13 VCMs along 13 directions through a polyp volume [20]. By the use of the 28 texture measures from each direction [21], we will have total of 364 (= 13 × 28) texture measures per polyp. Since these 364 texture measures are derived from a 3D VTI of a polyp, they will be called VTFs hereafter. Classifying these VTFs and evaluating the classification outcomes are presented in the following section. The summary algorithm of generating these VTFs from for either the gradient vector or Hessian vector can be found below in Algorithm 1.
Classification of the VTFs
Once the VTFs are computed for each polyp’s TVI, we use the R package ‘randomForest’ to perform classification using random forest (RF) [32, 33], as it has been shown to be effective in previous polyp classification experiments [34, 35]. Due to the limited data size of only 63 polyps, the RF method (and all machine learning algorithms) is susceptible to bias from the input data, or overfitting, depending on how it is divided into training and testing datasets. To reduce this overfitting effect, we utilize two different crossvalidation methods to measure the robustness of the model. The first is a common twofold crossvalidation method where we randomly divide the total dataset into equally sized testing and training sets, with the criteria that the ratio of benign and malignant lesions are maintained in each set. For a small dataset such as the one we are using, this method uses the least amount of training data and can therefore still lead to overfitting of the classification model. To alleviate this issue, we generate the random training and testing sets 100 times, and all results reported in Sect. 3 are the listed averages across all 100 randomly generated sets. For the second crossvalidation scheme, we utilize a leaveoneout style approach where all except one polyp is used in the testing set, and the “left out” polyp is tested. This is repeated until all polyps have been left out of the training set and the final classification results summarize all iterations of the leaveoneout models. The results of the leaveoneout cross validation method are detailed in the Appendix.
To perform classification, we use the function ‘randomForest’ with the settings “ntreetry = 5000” and “doBest = True.” To perform the forward stepwise feature selection [36], we first do a preliminary classification using all available features and obtain the importance value for each feature by its GINI index [32]. The features are then sorted based on a decreasing GINI index value. The RF classification is then run with the 3 highest importance features, and repeated with iteratively increasing number of features added in importance order [21]. This procedure is repeated for each of the 100 randomized sets of testing and training data. The results are evaluated for each group, and the reported values show to average maximum area under the curve (AUC) of the receiver operating characteristic (ROC) curve. The results for the leaveoneout crossvalidation method use this same feature selection procedure, only the distribution of training and testing sets are different.
Lastly, we note our choice of using RF classifier compared to other classifiers such as support vector machine (SVM) [37] or Knearest neighbors [38]. Through internal experiments, we have generally found that we achieve better classification through RF over the other two methods. RFs have shown good classification in a variety of applications [39], and has been found that they tend to outperform SVM in instances of low resolution, such as for the dataset of polyp images used in this study [40]. Further, RF is probabilistic in its design, and is not under the same linear constraints required by SVM for its hyperplane segmentation. This work demonstrates the feasibility and efficacy of these proposed vector textures through RF classification, and comparisons across multiple classification modalities are subjects of future works.
Results and discussion
In this section, we first describe the dataset and evaluate two parameters for gradient and Hessian magnitude to obtain some preliminary results for further experiments. Then polyp classification is performed using the VTFs extracted from the four VTIs. At the end, we compare our classification results with four existing methods.
Polyp dataset
The data set used for these experiments consisted of 59 patients with a total number of 63 polyps found through CTC screening. These polyps were all at least 30 mm or larger that were scheduled for surgical removal later. When they were removed, the pathology reports were obtained to verify whether the polyps were indeed cancerous as an adenocarcinoma or were benign/premalignant as an adenomatous polyp. The breakdown of the dataset can be seen in Table 1. For classification, the dataset was divided into binary categories of malignant (adenocarcinoma) and premalignant (all others).
The regions of interest (ROIs), used to compute the VTFs described below, were manually selected around the polyp region of the CTC image. For each polyp, a 3D volume was extracted, which was confirmed by radiologists to ensure accuracy for the manual procedure. We note that a thresholding preprocessing step was used to discard all voxels below 450 HU within these ROIs as being predominately air from the lumen of the colon. The information encoded in these voxels from partial volume effects is mostly air and therefore contributes more noise to the polyp features for classification. Since the first and second order derivatives apply kernels to the polyp images, the border voxels still use the intensity values provided outside of the ROI from the original CTC image. Examples of a polyp ROI slice are shown in Fig. 5 with each of the different image metrics used, i.e., the images in the original intensity domain, the gradient domain, and the Hessian (eigenvalue) domain. From the images, it is clear that each of the azimuth, polar, and magnitude images from the two derivative orders provide different types of information or different texture patterns. We hypothesize that given the same polyp descriptor or VCM in this study, the extracted VTFs from the gradient domain would be different from the Hessian domain, reflecting the polyp information at different orders.
While this dataset is small, we note the medical application to these polyp masses can still be quite significant. These polyp masses are large enough that removal will always be necessary, but the type of treatment may change based on the pathology. For example, accurate classification may help guide physicians to prevent unnecessary biopsy procedures which can help reduce medical costs and potential risks to the patient.
Experimental results
Two parameters of magnitude images
Gradient magnitude and Hessian eigenvalue magnitude are two kinds of geometric measurements which represent the vector length. Since gradient and Hessian images both contain image edge information, they could enhance the contrast of the image to some extent, most notably at the boundaries. Therefore, their magnitude might vary in an interval with a wide range. However, most of the information is clustered in a very narrow interval as shown in Fig. 3. So, when we employ the tth root mapping of Formula (10) to alleviate the issue, it is necessary to make sure how to set the value of t to obtain a top performance. To this goal, t is changed from 1 to 8 by the interval of 1 to calculate the polyp descriptors according to Formulas (5) and (12) while the gray level is kept 32. Table 2 lists the AUC scores of magnitudes of gradient and three Hessian eigenvalues. By consideration of their averages and their stabilities, we find t = 2 and t = 4 should be reasonable choices for gradient and Hessian magnitude images in the following experiments.
The second important parameter of magnitudes is its gray levels which determine the magnitude images and the VCM size. A larger gray level will demonstrate more details of metric images while smaller gray levels will over smooth the VCMs which will affect the discrimination of the texture features. Therefore, it is necessary to test the gray level’s influence in the polyp classification as shown in Table 3.
Outcomes of anglebased VCM
Both GAV and HAV are anglebased vector images which could be fed to VCM using Formula (27) to calculate the eHMs after digitalization. GAV represents the gradient angle vector image while HAV reflects the novel angle vector image derived from the Hessian eigenvalue vector. Both contain the two components of the azimuth and polar angles. In the angular vector digitalization, there are four parameters corresponding to the four angles, via \({Q}_{g}^{a}\),\({Q}_{g}^{p}\),\({Q}_{h}^{a}\) and \({Q}_{h}^{p}\) of Formulas (8), (9), (22) and (23). To avoid the matrix sparsity, we calculate the VCMs while \({Q}_{g}^{a}\in \left\{7, 8, 9, 10, 11\right\}\), \({Q}_{g}^{p}\in \left\{4, 5, 6\right\}\), and \({Q}_{h}^{a} and {Q}_{h}^{p}\in \left\{5, 6, 7, 8\right\}\). After classification via the RF classifier, their AUC scores are listed in Tables 4 and 5.
Table 4 illustrates that the GAV is very robust when the gray level varies from 28 to 66. The mean of AUC is about 0.94 while its standard deviation is almost less than 0.03. Comparatively, AUC scores of HAV in Table 5 show that its fluctuation of their averages is a little wider than GAV, which varies from 0.897 to 0.949. Meanwhile, most of the standard deviations are in the range of [0.03, 0.04]. From the best results of GAV and HAV, both could reach a similar classification level as high as 0.949.
Outcomes of vector (angle and magnitude)based VCM
Dissimilar to GAV and HAV which only contain angles, TGV and THV consist of all information including both gradient and Hessian eigenvalues, i.e., the magnitudes and the azimuth and polar angles. Both vector images have six parameters in their quantization, i.e., \({Q}_{g}^{m}\), \({Q}_{g}^{a}\),\({Q}_{g}^{p}\), \({Q}_{h}^{m}\),\({Q}_{h}^{a}\) and \({Q}_{h}^{p}\). Like the anglebased VCM, we set some reasonable quantification levels for these parameters to generate nonsparse VCMs based on the best results of GAV and HAV. We hereby test TGV by changing \({Q}_{g}^{m}\) in the range {1, 2, 3, 4} while \({Q}_{g}^{a}\) keeps 10 and \({Q}_{g}^{p}\) varies in the range {4, 5} as shown in Table 6. Dissimilarly, we test \({Q}_{h}^{m}\) in {4, 5},\({Q}_{h}^{a}\) in {3, 4} and \({Q}_{h}^{p}\) in {2, 3, 4, 5, 6, 7, 8} as shown in Table 7 where tth power is equal to 4. To measure the classification performances, their AUC scores are obtained to show their effectiveness.
Table 6 demonstrates that TGV improves the average AUC scores and decreases their standard deviation compared with GAV’s classification results. That means all geometric components of gradient could provide some contribution to enhance the discrimination in polyp classification. Additionally, we find the similar trend for THV’s results in Table 7 which exceeds more than 1% by their best AUC scores.
Comparison to other methods
For reference on how well our proposed method performs, we compare our results to some similar texture extraction methods and a stateoftheart deep learning method as follows.

Extended Haralick features (eHF) – this method extracts a set of 60 texture features from the GLCMs of the intensity image [21].

HoG3D – this method counts the occurrences of gradient orientation in some cropped portions of the intensity image and generates some histograms which are joined to form gradient features [18].

CoLIAGe – this model employs gradient angles to extract the entropy of every local patch to form global texture features by two joint histograms [19].

VGG16 – it is a widely cited deep learning method. Total of 20 salient slices were extracted from each polyp volume to feed to the VGG16 pipeline for polyp classification [41].
Table 8 shows the results of comparison to the similar texture extraction methods listed above, where the same RF classifier was used, and the deep learning method. Our proposed method performed the best. We can see quite clearly that when comparing to the extended Haralick feature method, the T_{4} derived texture features outperformed the Haralick texture features with an AUC of 0.962 against 0.876. Comparing against other gradient angular features, our proposed angular texture features improved the performance substantially over the 3D HOG features (AUC = 0.804) and the CoLIAGe angular features (AUC = 0.923). The gain is also substantially higher over the VGG16 outcome (AUC = 0.833). We note that deep learning methodologies such as VGG16 have a much higher data requirement than conventional machine learning approaches. While we include the comparison to the VGG16 model for comparison, we expect that the results of VGG16 and other deep learning models such as those presented in refs. [42, 43] would be more comparable on a much larger dataset. Therefore, the eHF, HOG3D, and CoLIAGe models provide a more representative comparison and evaluation of the proposed vectortextures since they may be utilized under the same circumstances of fewer data entries.
Using a Wilcoxin ranked ttest, we obtained the quantitative measures for significant difference between the results of our proposed method and those of the reference methods as listed in Table 9. In all except one instance, we find that our results performed significantly better (p < 0.05) than the comparison methods.
Conclusions
To enhance the image contrast of the original CTC polyp images, this study utilizes gradient operator and Hessian operator to generate the corresponding gradient image and Hessian image. The gradient image is represented by a vector field. The Hessian image is represented, according to the definition, by a matrix field. To avoid the difficulty of manipulating matrices, we take the three eigenvalues of the Hessian matrix at each image voxel as a vector and reduce the matrix field as a vector field similar to gradient field, thus all operations in the gradient domain are adapted to the Hessian domain. In addition, a novel concept of VTIs is proposed by the use of the vector geometric measures through the two vector fields in the corresponding gradient and Hessian domains, i.e., the GAV images (T_{1}), the TGV images (T_{2}), the HAV images (T_{3}) and the THV image (T_{4}). Moreover, another novel concept of vectorbased CM or VCM is introduced to extract 2D texture patterns from these 2D/3D VTIs. These 2D texture patterns or VCMs can be viewed as the projection of the 2D/3D VTIs at different angles. From the projected VCMs, texture measures can be extracted as VTFs and classified by an existing classifier, such as RF as an example. Experimental outcomes demonstrated that the proposed VTF extraction method can outperform the stateofthe art feature extraction methods for polyp classification.
The novel textures introduced in this work are based on the CM. We had chosen to focus on the CM as previous studies demonstrated GLCM textures had typically outperformed those of related matrixbased textures such as those from graylevel runlength matrices. While we only focused on the cooccurrencebased matrices here, we believe these other matrices can be another avenue to explore these vectortextures and will be considered in our future works. We also note that these vectortextures place an emphasis on characterizing more microscopic properties of the lesions since they are based on the higher order derivative information. For classification with handcrafted features, it is important to introduce as much relevant information to the classifier while minimizing redundant information. Another topic for our future works will be looking into the integration of these vectortextures with other textures that emphasize more macroscopic lesion properties, such as the gray level size matrix. Appropriate feature selection methods will also be examined on how best to integrate these novel textures into the already existing library of handcrafted features.
Availability of data and materials
The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.
Abbreviations
 HOG:

Histogram of oriented gradient
 CoLIAGe:

Cooccurrence of local anisotropic gradient orientations
 CM:

Cooccurrence matrix
 GLCM:

Graylevel cooccurrence matrix
 CRC:

Colorectal cancer
 CTC:

Computed tomographic colonography
 VTI:

Vector texture image
 VCM:

Vectorbased cooccurrence matrix
 GAV:

Gradient angle vector
 TGV:

Total gradient vector
 HAV:

Hessian angle vector
 THV:

Total Hessian vector
 VTF:

Vector texture feature
 eHF:

Extended Haralick features
 2D:

Twodimensional
 3D:

Threedimensional
 AUC:

Area under the curve
 ROI:

Region of interest
 ROC:

Receiver operating characteristic
 SVM:

Support vector machine
References
Gatenby RA, Grove O, Gillies RJ (2013) Quantitative imaging in cancer evolution and ecology. Radiology 269(1):8–15. https://doi.org/10.1148/radiol.13122697
Bashir U, Siddique MM, Mclean E, Goh V, Cook GJ (2016) Imaging heterogeneity in lung cancer: techniques, applications, and challenges. AJR Am J Roentgenol 207(3):534–543. https://doi.org/10.2214/AJR.15.15864
Freeman HJ (2018) Heterogeneity of colorectal adenomas, the serrated adenoma, and implications for screening and surveillance. World J Gastroenterol 14(22):3461–3463. https://doi.org/10.3748/wjg.14.3461
Motai Y, Siddique NA, Yoshida H (2017) Heterogeneous data analysis: online learning for medicalimagebased diagnosis. Patt Recogn 63:612–624. https://doi.org/10.1016/j.patcog.2016.09.035
Summers RM (2017) Texture analysis in radiology: does the emperor have no clothes? Abdom Radiol 42(2):342–345. https://doi.org/10.1007/s0026101609501
Cao WG, Liang ZR, Pomeroy MJ, Pickhardt PJ, Barish MA, Abbasi AF et al (2018) A new look at graylevel cooccurrence for multiscale texture descriptor with applications to characterize colorectal polyps via computed tomographic colonography. Paper presented at the 2018 IEEE nuclear science symposium and medical imaging conference proceedings, IEEE, Sydney, 10–17 November 2018. https://doi.org/10.1109/NSSMIC.2018.8824639
Choi TW, Kim JH, Park SJ, Ahn SJ, Joo I, Han JK (2018) Risk stratification of gallbladder polyps larger than 10 mm using highresolution ultrasonography and texture analysis. Eur Radiol 28(1):196–205. https://doi.org/10.1007/s0033001749541
Ren YC, Ma JC, Xiong JF, Chen Y, Lu L, Zhao J (2019) Improved false positive reduction by novel morphological features for computeraided polyp detection in CT colonography. IEEE J Biomed Health Inform 23(1):324–333. https://doi.org/10.1109/JBHI.2018.2808199
Castellano G, Bonilha L, Li LM, Cendes F (2004) Texture analysis of medical images. Clin Radiol 59(12):1061–1069. https://doi.org/10.1016/j.crad.2004.07.008
Rathore S, Hussain M, Ali A, Khan A (2013) A recent survey on colon cancer detection techniques. IEEE/ACM Trans Comput Biol Bioinform 10(3):545–563. https://doi.org/10.1109/TCBB.2013.84
Li H, Zhu YT, Burnside ES, Huang E, Drukker K, Hoadley KA et al (2016) Quantitative MRI radiomics in the prediction of molecular classifications of breast cancer subtypes in the TCGA/TCIA data set. NPJ Breast Cancer 2:16012. https://doi.org/10.1038/npjbcancer.2016.12
Haralick RM, Shanmugam K, Dinstein IH (1973) Textural features for image classification. IEEE Trans Syst Man Cybern 3(6):610–621. https://doi.org/10.1109/TSMC.1973.4309314
Tang XO (1998) Texture information in runlength matrices. IEEE Trans Image Process 7(11):1602–1609. https://doi.org/10.1109/83.725367
Ji TL, Sundareshan MK, Roehrig H (1994) Adaptive image contrast enhancement based on human visual properties. IEEE Trans Med Imaging 13(4):573–586. https://doi.org/10.1109/42.363111
Lu J, Healy DM, Weaver JB (1994) Contrast enhancement of medical images using multiscale edge representation. Paper presented at the SPIE’s international symposium on optical engineering and photonics in aerospace sensing, SPIE, Orlando, 15 March 1994. https://doi.org/10.1117/12.170070
Kim YT (1997) Contrast enhancement using brightness preserving bihistogram equalization. IEEE Trans Consumer Electr 43(1):1–8. https://doi.org/10.1109/30.580378
Aggarwal N, Agrawal RK (2012) First and second order statistics features for classification of magnetic resonance brain images. J Signal Inform Proc 3(2):19553. https://doi.org/10.4236/jsip.2012.32019
Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. Paper presented at the 2005 IEEE computer society international conference on computer vision and pattern recognition (CVPR’05), IEEE, San Diego, 20–25 June 2005. https://doi.org/10.1109/CVPR.2005.177
Prasanna P, Tiwari P, Madabhushi A (2016) Cooccurrence of local anisotropic gradient orientations (CoLIAGe): a new radiomics descriptor. Sci Rep 6:37241. https://doi.org/10.1038/srep37241
Song BW, Zhang GP, Lu HB, Wang HF, Zhu W, Pickhardt PJ et al (2014) Volumetric texture features from higherorder images for diagnosis of colon lesions via CT colonography. Int J Comput Assist Radiol Surg 9(6):1021–1031. https://doi.org/10.1007/s1154801409912
Hu YF, Liang ZR, Song BW, Han H, Pickhardt PJ, Zhu W et al (2016) Texture feature extraction and analysis for polyp differentiation via computed tomography colonography. IEEE Trans Med Imaging 35(6):1522–1531. https://doi.org/10.1109/TMI.2016.2518958
American Cancer Society (2021) Cancer facts and figures 2021. American Cancer Society, Atlanta
Pyenson B, Pickhardt PJ, Sawhney TG, Berrios M (2015) Medicare cost of colorectal cancer screening: CT colonography vs. optical colonoscopy. Abdom Imaging. 40(8):2966–2976. https://doi.org/10.1007/s0026101505381
Grosu S, Wesp P, Graser A, Maurus S, Schulz C, Knösel T et al (2021) Machine learningbased differentiation of benign and premalignant colorectal polyps detected with CT colonography in an asymptomatic screening population: a proofofconcept study. Radiology 299(2):326–335. https://doi.org/10.1148/radiol.2021202363
Tan JX, Gao YF, Liang ZR, Cao WG, Pomeroy MJ, Huo YM et al (2019) 3DGLCM CNN: A 3dimensional graylevel Cooccurrence matrixbased CNN model for polyp classification via CT colonography. IEEE Trans Med Imaging 39(6):2013–2024. https://doi.org/10.1109/TMI.2019.2963177
Cao WG, Liang ZR, Gao YF, Pomeroy MJ, Han FF, Abbasi A et al (2021) A dynamic lesion model for differentiation of malignant and benign pathologies. Sci Rep 11(1):3485. https://doi.org/10.1038/s41598021830952
Sobel I, Feldman G (1973) A 3×3 Isotropic gradient operator for image processing. Stanford Artificial Intelligence Project (SAIL). https://www.researchgate.net/publication/239398674_An_Isotropic_3x3_Image_Gradient_Operator
Xu D, Li H (2008) Geometric moment invariants. Patt Recogn 41(1):240–249. https://doi.org/10.1016/j.patcog.2007.05.001
Cao WG, Hu P, Liu YJ, Gong M, Li H (2013) Gaussiancurvaturederived invariants for isometry. Sci China Inf Sci 57(9):1–12. https://doi.org/10.1007/s114320114453y
Pomeroy M, Lu HB, Pickhardt PJ, Liang ZR (2018) Histogrambased adaptive gray level scaling for texture feature classification of colorectal polyps. Paper presented at the medical imaging 2018: computeraided diagnosis, SPIE, Houston, 10–15 February 2018. https://doi.org/10.1117/12.2293884
Monga O, Benayoun S (1995) Using partial derivatives of 3D images to extract typical surface features. Comput Vis Image Underst 61(2):171–189. https://doi.org/10.1006/cviu.1995.1014
Breiman L (2001) Random forests. Mach Learn 45(1):5–32. https://doi.org/10.1023/A:1010933404324
Pal M (2005) Random forest classifier for remote sensing classification. Int J Remote Sens 26(1):217–222. https://doi.org/10.1080/01431160412331269698
Ma M, Wang HF, Song BW, Hu YF, Gu XF, Liang ZR (2014) Random forest based computeraided detection of polyps in CT colonography. Paper presented at the 2014 IEEE nuclear science symposium and medical imaging conference, IEEE, Seattle, 8–15 November 2014. https://doi.org/10.1109/NSSMIC.2014.7430920
Cao WG, Pomeroy MJ, Pickhardt PJ, Barish MA, Stanley III S, Liang ZR (2019) A local geometrical metricbased model for polyp classification. Paper presented at the SPIE medical imaging, SPIE, San Diego, 16–21 February 2019. https://doi.org/10.1117/12.2513056
Hastie T, Tibshirani R, Tibshirani RJ (2017) Extended comparisons of best subset selection, forward stepwise selection, and the lasso arXiv preprint arXiv: 1707.08692
Suthaharan S (2016) Support vector machine. In: Machine learning models and algorithms for big data classification, vol 36. Springer. Boston. 207–235. https://doi.org/10.1007/9781489976413_9
Saadatfar H, Khosravi S, Joloudari JH, Mosavi A, Shamshirband S (2020) A new Knearest neighbors classifier for big data based on efficient data pruning. Mathematics 8(2):286. https://doi.org/10.3390/math8020286
Joloudari JH, Joloudari EH, Saadatfar H, GhasemiGol M, Razavi SM, Mosavi A et al (2020) Coronary artery disease diagnosis; ranking the significant features using a random trees model. Int J Environ Res Public Health 17(3):731. https://doi.org/10.3390/ijerph17030731
Sheykhmousa M, Mahdianpari M, Ghanbari H, Mohammadimanesh F, Ghamisi P, Homayouni S (2020) Support vector machine versus random forest for remote sensing image classification: a metaanalysis and systematic review. IEEE J Sel Top Appl Earth Observat Remote Sensing 13:6308–6325. https://doi.org/10.1109/JSTARS.2020.3026724
Simonyan K, Zisserman A (2014) Very deep convolutional networks for largescale image recognition arXiv preprint arXiv: 1409.1556
Shamshirband S, Fathi M, Dehzangi A, Chronopoulos AT, AlinejadRokny H (2021) A review on deep learning approaches in healthcare systems: taxonomies, challenges, and open issues. J Biomed Inform 113:103627. https://doi.org/10.1016/j.jbi.2020.103627
Sharifrazi D, Alizadehsani R, Joloudari JH, Band SS, Hussain S, Sani ZA et al (2020) CNNKCL: automatic myocarditis diagnosis using convolutional neural network combined with kmeans clustering. Math Biosci Eng 19(3):2381–2402. https://doi.org/10.20944/preprints202007.0650.v1
Acknowledgements
The authors acknowledge and thank the NIH and National Natural Science Foundation of China for supporting this work.
Funding
This work was partially supported by the NIH/NCI, Nos. CA206171 and CA220004; Dr. Lu was supported by the National Natural Science Foundation of China, No. 81871424.
Author information
Authors and Affiliations
Contributions
WC and MJP computed the vector textures, ran the classification experiments, and wrote the manuscript; PJP generated the polyp image dataset used; AFA and HL oversaw the segmentation of the polyp images; ZL and WC inspired and developed the scientific and experimental idea. The authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
Some experimental results using leaveoneout scheme
The results of using a leaveoneout cross validation method are presented below. For each model training instance in a leaveoneout method, only a single polyp is removed for the testing set at a time. The process is repeated until each polyp has been removed once, and the output classification probability from each polyp can be used to evaluate the overall performance. Since the leaveoneout method uses all but one polyp each instance, there is much more training data available to generate a model. This is especially important when the dataset used is already small with only 63 polyps. As can be seen in the tables below, the repeated experiments with a leaveoneout classifier generally perform better than the twofold cross validation equivalent since there is more data in the training set each time.
The experiments for the results of Tables 4, 5, 6 and 7 of the twofold cross validation method were repeated using the leaveoneout validation method and the outcomes are shown in Tables 10, 11, 12 and 13 respectively. We find for the GAV (or T_{1}) model and the HAV (or T_{3}) model, we can achieve AUC values of up to 0.960 and 0.975 respectively by changing the scaling parameters. We similarly find the TGV (or T_{2}) and the THV (or T_{4}) models can achieve AUC values up to 0.970 and 0.982 respectively. When comparing these values to those in Tables 4, 5, 6 and 7, we do find that the leaveoneout cross validation is able to obtain better overall classification performance by virtue of having more data available at each training instance.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visithttp://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Cao, W., Pomeroy, M.J., Liang, Z. et al. Vector textures derived from higher order derivative domains for classification of colorectal polyps. Vis. Comput. Ind. Biomed. Art 5, 16 (2022). https://doi.org/10.1186/s42492022001081
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s42492022001081
Keywords
 Machine learning
 Gradient
 Hessian matrix
 Haralick feature
 Random forest
 Image texture