Linear Regression Supporting Vector Machine and Hybrid LOG Filter-Based Image Restoration

Abstract The image restoration (IR) technique is a part of image processing to improve the quality of an image that is affected by noise and blur. Thus, IR is required to attain a better quality of image. In this paper, IR is performed using linear regression-based support vector machine (LR-SVM). This LR-SVM has two steps: training and testing. The training and testing stages have a distinct windowing process for extracting blocks from the images. The LR-SVM is trained through a block-by-block training sequence. The extracted block-by-block values of images are used to enhance the classification process of IR. In training, the imperfections on the image are easily identified by setting the target vectors as the original images. Then, the noisy image is given at LR-SVM testing, based on the original image restored from the dictionary. Finally, the image block from the testing stage is enhanced using the hybrid Laplacian of Gaussian (HLOG) filter. The denoising of the HLOG filter provides enhanced results by using block-by-block values. This proposed approach is named as LR-SVM-HLOG. A dataset used in this LR-SVM-HLOG method is the Berkeley Segmentation Database. The performance of LR-SVM-HLOG was analyzed as peak signal-to-noise ratio (PSNR) and structural similarity index. The PSNR values of the house and pepper image (color image) are 40.82 and 36.56 dB, respectively, which are higher compared to the inter- and intra-block sparse estimation method and block matching and three-dimensional filtering for color images at 20% noise.


Introduction
Recorded images generally degrade due to environmental effects and imperfections in the imaging system [22]. Image restoration (IR) plays a major role in digital image processing, and it aims to reconstruct the highfrequency details or eliminate noises from the image [2]. This IR process takes place in deblurring, denoising, and medical applications [3,8,16,24]. There are numerous methods available for image denoising, which are divided into four different categories: spatial filtering methods, transform domain filtering methods, partial differential equation-based methods, and variational methods [33]. Passive millimeter wave images (PMMWs) become affected by the high reflectivity of metal objects, and these PMMWs are used in aviation, security, and environmental monitoring [20]. IR over a hyperspectral image (HSI) requires rich spectral information, and this HSI is a three-dimensional data cube [12,41]. The super-resolution reconstruction uses in IR-based image processing technique. A regularization method is used to perform the IR, and this regularization defines the quality of the image [29,49]. Some of the filtering techniques include Wiener filtering and wave atom transform used in degraded images and those affected by noise, blur, etc. [4,23].
The conventional methods used in IR are support vector machine (SVM)-based blur identification, residual-based deep convolutional neural network (CNN) for image dehazing, and multi-resolution CNN. SVM is used for classifying non-local feature vectors, and mapping of SVM is made between the vector and blur parameters [26]. Based on the transmission map of residual CNN, the haze from the image is eliminated and this network is trained with the help of the NYU2 depth dataset [18]. A multi-resolution CNN gives the reconstructed image by removing more patterns related to a specific frequency band [32]. In multi-scale vectorial total variation, the spatial-dependent regularization parameters are utilized to reconstruct the images [10]. The combination of discriminative learning techniques and advanced proximal optimization algorithms restores the image. Along with that, the denoising and deconvolution of the corrupted image are performed in the training part of discriminative learning [40]. In addition, the combination of non-local regularization technique with structured sparse representation is used for restoring the images, and it depends on the sparse nature of the transform coefficients of image patches [31]. In this paper, IR over corrupted images is done by linear regression-based (LR)-SVM. Here, the hybrid Laplacian of Gaussian (LOG) filtering technique (combination of LOG and Gaussian) is used for enhancing the image blocks extracted from the testing stage.
The major contributions of this research work are as follows: -A total of three kinds of noise, including Gaussian noise, speckle noise, and salt and pepper noise, are considered in this research work. -The images are restored through separating the image imperfections by separating the noise samples from the image blocks while performing LR-SVM. The proposed LR-SVM-HLOG method-based IR is applicable for both color and grayscale images. -The performance of this research work is improved by placing the hybrid LOG filter at the testing stage. Both the restoration and denoising of the LR-SVM-HLOG method are enhanced by providing the block-by-block information of the entire image.
This research work is composed as follows: Section 2 presents a brief review of some papers based on IR. Section 3 describes the details about the problem formulation and gives the solution with the help of LR-SVM-HLOG. Section 4 briefly describes how the image is restored from the noisy images. Section 5 describes the experimental results of an LR-SVM-HLOG method and conventional methods. The conclusion of this research work is given in Section 6.

Survey of Related Works
Several existing techniques related to IR from noise, blur, etc., are surveyed in this section. A brief evaluation of some contributions to the existing literature is presented in the Table 1. It also explains about the limitation of existing methodologies and the purpose of the work. Typically, the images are often corrupted by several noises, haze, fog, and blur due to an error generated in noisy sensors or communication channels. In recent years, researchers have developed several methods, such as the cascaded model of Gaussian conditional random fields [28], splitting method for structured total least squares [13], Bayesian model [19,30], the combination of iterative VanCittert algorithm with noise reduction modeling [21], K-means singular value decomposition [39], joint log likelihood function [45], and filters [42], for restoring corrupted images. In addition, various kinds of neural networks and classifiers have been utilized in the IR process. The main processes present in all neural networks are noise identification (training stage), denoising (testing stage) the image, and storing the respective image [5, 7, 11, 17, 25, 27, 34-38, 44, 46, 47]. However, some of the techniques are not fit for those three kinds of noises. In this research, LR-SVM is used for restoring the noisy images.

Problem Definition and Solution
The present issues of IR are analyzed in this section, which also states how the LR-SVM-HLOG method gives a solution to this problem. The problems of IR are as follows: -Difficulty in the prediction of noise; In this RBFNN, the weight is not connected between the input and output layers.
Haibin Duan and Xiaohua Wang [11] 2016 Echo state network (ESN) and orthogonal pigeon-inspired optimization (OPIO) ESN is a recurrent neural network that has been characterized by recurrent loops in synaptic connection pathways. This ESN is used for estimating the original images.
The ESN parameters is optimized by OPIO because the typical ESN has a more complicated fitting system due to the large amount of dynamic reservoir internal processing units.

Yunjin Chen and
Thomas Pock [5] 2016 Trainable non-linear reaction diffusion (TNRD) The training of TNRD is similar to the other neural networks (NNs).
Here, only the Gaussian noise is considered in the image denoising process.
Manjun Qin et al. [27] 2018 Deep convolutional neural network (CNN) The different dehazing abilities were performed by training the deep CNN based on different levels of haze samples. Then, the dehazing result is produced by fusing the outputs of CNN individuals.
The Landsat 8 Operational Land Imager (OLI) dataset was used in this research, and the method was only used for the purpose of dehazing.
Ilke Turkmen [34] 2011 Neuro-fuzzy detector-based median filter (NFDMF) The hybrid tuning algorithm was used for tuning the fuzzy parameters. This leads to decreased or minimized errors.
Only three different kinds of images are used in the training process.
Hang Zhao et al. [47] 2016 Neural networks Denoising of the images is performed by investigating the impact of different loss function layers.
The convergence properties of the NN decide the IR performances.
Jian Zhang et al. [44] 2014 Group-based sparse representation (GSR) An effective self-adaptive dictionary learning method is developed to make the GSR tractable and robust.
Here, only the noisy image blocks are given to the training process.
-Works only in appropriate noises; and -Selection of appropriate training sequence.

Diflculty in the Prediction of Noise
The prediction of impulse noises from the data acquisition process is difficult, and there is also no perfect noise reduction method [1,43]. Solution: In this LR-SVM-HLOG method, the noisy image blocks are identified by computing the average intensity values of the neighborhood pixel for the respective pixel. By using this process, the noises from the images can be predicted easily and the respective image blocks can also be restored in the dictionary.

Works Only in Appropriate Noises
The restoration of images is performed only on appropriate noises, for example trainable non-linear reaction diffusion (TNRD) [5] and group-based sparse representation (GSR) [44]. Here, the TNRD training experiments are performed based on noise levels of 15%, 25%, and 50%, and the GSR receive the noisy images at different noise levels such as 10%, 20%, 30%, etc. However, this TNRD considers only the Gaussian noise and the GSR method considers only the impulse and Gaussian noise.

Solution:
In the LR-SVM-HLOG method, three kinds of noises (Gaussian noise, speckle noise, and salt and pepper noise) are considered. The corrupted images are denoised by identifying noise imperfections based on the separable plane of LR-SVM. The denoising process is improved by converting the entire input image into a block-by-block sequence.

Selection of Appropriate Training Sequence
For an effective restoration of the images, the training sequences that are given to the system need to be in proper manner. However, in some of the methods, the training sequences are not in a proper manner. For example, IR of motion blurred images is obtained by using edge information, related coefficients, and next-layer information in kernel function [48].
Solution: This training problem is overcome by providing the appropriate training sequences. In this SVM, the values of the noisy image array are given as input and the target value for this SVM is set as values of the original image array. By giving the denoised image in the testing section, the imperfections in the image are analyzed and tested with trained input values to restore the respective image. Here, the input image is converted into a block-by-block sequence by using the windowing process, and these block values are tested with the entire database for effectively improving the restoration process.

Proposed Methodology for Effective IR
As the demand for high-quality and high-resolution images are growing, computational efficiency has become a main issue in IR. Several techniques have been developed for restoring the images from a corrupted image; however, some of the techniques are not fit for all types of noises. Hence, this research determines the technique for restoring different kinds of noises such as Gaussian noise, speckle noise, and salt and pepper noise. This LR-SVM-HLOG method has two stages: LR-SVM training and LR-SVM testing, which are clearly explained in the following sections.

LR-SVM Training
The training of LR-SVM has three main processes: image acquisition, block extraction, and classifications using LR-SVM. Figure 1 shows the block diagram for LR-SVM training.

Image Acquisition and Block Extraction
Initially, the input image is taken from the Berkeley Segmentation Database (BSD) and then the image is transformed into a grayscale image. After adding the noise in the original images, both the noise and the original images are given to block extraction. The block extraction process is performed by a distinct windowing process. The block extracted from the windowing process has a size of 3 × 3. Then, these blocks are arranged into an array for giving the input to the LR-SVM.

Training Using LR-SVM
In this LR-SVM, the noisy image array values are given as input and the target value for this LR-SVM is set as the original image array value. The LR-SVM is trained by the noisy image array (i.e. blocks) of three different noises, such as Gaussian noise, speckle noise, and salt and pepper noise, with standard deviations of 10%-90%. Based on the input and target values, this LR-SVM is trained and the training process of the LR-SVM is explained as follows: SVMs are the set of related supervised learning methods for classification and regression. The LR-SVMs are actually introduced for the classification problems and are later also used to resolve the regression issues. Based on the subset of the training data, the model is generated. The cost function for constructing the model does not consider the training points that lie beyond the region. Similarly, the model generated by the LR-SVM is based on a subset of training data, because the cost function for building the model avoids the training data that are close (i.e. within the threshold ϵ) to the model prediction. Table 2 presents the specifications of LR-SVM.
Here, the set of training samples (image blocks) that is given to the LR-SVM is denoted as , −1, n, and d are the training sample dimensionality; and k and φ are the positive definite kernel and its feature mapping, respectively.
The dual SVM problem is solved by Eq. (1): where the Lagrange multiplier is denoted as ⃗ α = (α1, ...α n) and the weight The samples that are given in the LR-SVM only belong to its dimensionality, and the j th dimension in the The gradient G measures the changes in f (︀ ⃗ α )︀ with respect to α i , which is given in Eq. (3):

a. Linear Regression for Gradient Estimation
Both the gradient computation during training and the computation of prediction function during testing hinge on the term ⃗ w T φ (︀ ⃗ x )︀ . With properly chosen explanatory variables, linear regression can achieve this goal. The computational bottleneck (g(⃗ q)) of LR-SVM is given in Eq. (4): where α i , y i , and x i are constants in this function. A linear regression model will estimate the output of this function, g(.), which is given in Eq. (5): where the function of computational bottleneck is g(⃗ q), the explanatory variable function is e (︀ ⃗ q )︀ , the linear regression coefficient is ⃗ β, and the error tem is ∈.

b. Updating the Linear Regression Coefficients
There is still a need for a way to update the classifier w to complete the LR-SVM-HLOG linear regressionbased approach. The weight (2) and gradient (3) are updated by fixing i among 1 to n. When α i is changed by ∆α i = α i − ⃗ α i , using the definition of t, it is easy to show that t is changed by Eq. (6): where s(⃗ x i ) is the similarity function. After performing the training process, the trained values are stored in the dictionary.

Loss Function
The loss function used in the LR-SVM corrects the class of each image pixel, which has a score greater than the incorrect classes based on the fixed margin value (∆). The results of the LR-SVM loss function should be less. The loss function of the LR-SVM is expressed in Eq. (7): where L i is the loss function, y i is the label that specifies the index of the correct class, and s j is the j th element score in Eq. (8): where x i is the pixel of the image and the weight is W.

LR-SVM Testing
The corrupted images are given as an input to LR-SVM testing. LR-SVM testing runs only one time when the given input image is in grayscale. If the input image is a color image (RGB image), the LR-SVM processes the image three times (i.e. for different color features such as RGB) because the testing stage requires to test three colors (R, G, and B). Figure 2 presents an example of an input image of LR-SVM testing. Similar to LR-SVM training, the block extraction is performed in this training. The block values from the windowing process are used for enhancing the restoration process. Before giving the image blocks to the input of LR-SVM testing, the image blocks are replicated as three arrays because the LR-SVM is trained as three arrays. The noise present   in the image pixels is found based on the intensity values of each pixel. The noise calculation of the respective pixel is performed by considering the intensity values of neighborhood pixels. Initially, the maximum and minimum boundary value of intensity is fixed to identify the noisy pixel. The pixel's intensity value either goes higher than the maximum boundary value or lower than the minimum boundary value. Then, the respective pixel is referred to as noisy pixel. Finally, the noisy pixels are compared with the trained dictionary value to restore the respective pixel from the stored dictionary. The trained dictionary value comprises image blocks of three different noises with various standard deviations. Moreover, the image obtained from the LR-SVM testing is enhanced by the hybrid LOG filter. Figure 3 represents the block diagram of the LR-SVM testing.

Hybrid LOG Filtering
The hybrid LOG filter receives the input from the testing stage of LR-SVM, and it is a combination of LOG and Gaussian filter. The given input is in the form of a block-by-block sequence of image values, and this input format is used for enhancing the denoising process. Generally, the Laplacian filters are derivative filters and generally used in finding the areas of edges of images. As this derivative filter is very sensitive to noise, the research work used Gaussian filters for smoothening the image. In LOG filter, the Gaussian filtering is performed before the Laplacian filtering. After these two processes, again the Gaussian filtering is applied to smoothen the images. The output from the LOG filter is given as an input to the Gaussian filter. The filtering of LOG is shown in Eq. (9), and the Gaussian filtering of LOG is shown in Eq. (10).
The LOG scale space representation is where the Laplacian operator is ∇ 2 , standard deviation is σ, and x and y are coordinates of the images. The standard deviation plays a major role in the filtering behavior of the HLOG filter. The regions of rapid intensity change are highlighted by Laplacian. The LOG filter is an orientation-independent operator and the scale is provided by σ. The corners, curves, and locations have varied intensity function, which is in non-linear manner breakdown by using the LOG filter. A set of images with various levels of smoothness is achieved by applying Gaussian filter of different scales of standard deviation to an image. In Gaussian filter, zero crossings of the second-order derivatives are found to detect the edges from the images. The purpose of the hybrid LOG filter in the LR-SVM-HLOG method is to remove the noise from the restored images because the images from the database may be corrupted by noises from the environment. Thus, here, the hybrid LOG filter is used to enhance the contrast and brightness of the restored image. The hybrid LOG filter provides higher significance to the pixels near the edge. Effective smoothening and noise removal is obtained by using the combination of LOG and Gaussian filtering in the LR-SVM-HLOG method.

Experimental Results and Discussion
This section gives a detailed description of the experimental results of the LR-SVM-HLOG method. Correspondingly, the experimental setup, type of data set, and performance measure are explained in this section. The performance of the LR-SVM-HLOG method is analyzed by restoring the images from the noisy images.

Experimental Set-up
The LR-SVM-HLOG method is analyzed with the help of MATLAB simulator software version 2018b. The entire work is done by using an I3 system with 3 GB RAM. Moreover, the better performance of the LR-SVM-HLOG method is shown by comparing the LR-SVM-HLOG method with conventional techniques. The performance of the LR-SVM-HLOG method is analyzed in terms of peak signal to-noise ratio (PSNR) and structural similarity index (SSIM).

Dataset Description
The evaluation of the LR-SVM-HLOG method is carried out on the 150 images of the image denoising benchmark dataset (BSD). The experimental image used in this LR-SVM-HLOG method is shown in Figure 4. The LR-SVM-HLOG method evaluated the performance of the IR by removing the Gaussian noise, speckle noise, and salt and pepper noise. The zero-mean Gaussian noise, speckle noise, and salt and pepper noise with standard deviations of 10%, 20%, 30%, 40%, and 50% noise are added to each image to evaluate the performance of the LR-SVM-HLOG method.

Performance Measure
Typically, the success of the IR process is evaluated by comparing the PSNR of the denoised and noisy images with regard to reference. However, it is well known that PSNR does not correlate with the visual quality of the results. Their SSIM considers three reasonably independent image components: luminance, contrast, and structure. SSIM gives a better prediction of subjective image quality than PSNR and other existing quality measures. The SSIM takes values in [0, 1], where 1 indicates that the reference and target images are identical.
a. Mean Square Error (MSE) MSE is defined as the squared intensity of the original image pixels (i.e. given at the testing side) to the output image pixels, and the mathematical expression for MSE is given in Eq. (11) and that for RMSE is given in Eq. (12). The MSE is identified for the P × Q-sized image: where I is the noise image and J is the noise-free image.
b. PSNR PSNR is used for computing the difference between two images: an input image and the restored image from the LR-SVM-HLOG method. The PSNR helps estimate the quality of the reconstructed image with respect to the original image. The PSNR is calculated using MSE by Eq. (11). The PSNR (dB) of an image is expressed in Eq. (13): where Max I is the maximum possible pixel value of the image.

Performance Analysis of Noisy IR
Here, the performance of the LR-SVM-HLOG method is evaluated for three different noises: Gaussian noise, speckle noise, and salt and pepper noise. The performance of the LR-SVM-HLOG method is evaluated for various images such as baboon, Barbara, parrot, cameraman, house, peppers, starfish, monarch, Lena, boat, etc. These images are taken from the BSD. The performance analysis of the PSNR and SSIM of the restored images is given in Table 3, and one example is given in Figure 5 for different levels of noise. From the analysis, it is concluded that the LR-SVM-HLOG method gives an effective performance for the three different noises. Figure 5A describes the input image that is given in the testing stage, and Figure 5B    Figure 6 describes the input image that is given in the testing stage with the presence of 50% Gaussian noise. The noise present in the input images of Figure 6 is identified by the intensity values of neighborhood pixels. Then, the noisy image blocks are restored from the dictionary values of LR-SVM. The restored images are shown in Figure 7. This LR-SVM-HLOG method is also analyzed in the medical images. Some of the examples for IR through the medical image are given in Figures 8 and 9. Figures 8A and 9A show the noisy image that is given at the testing stage of LR-SVM-HLOG with Gaussian noise (σ = 50%). The restored image from the dictionary is shown in Figures 8B and 9B. The performance of the LR-SVM-HLOG method is analyzed with some conventional techniques such as GSR [44], non-locally centralized sparse representation (NCSR) [9], inter-and intra-block sparse estimation (IIBSE) [14], block matching and three-dimensional filtering (BM3D) [6], and weighted nuclear norm minimization (WNNM). NCSR is a sparse representative model, and it reduces the noise of an image by introducing a centralized sparse constraint and iterative shrinkage function for solving the minimization problem [9]. IIBSE is also used for norm minimization problems. Here, the intra-block sparsity priors are inter-block sparsity errors [14]. BM3D-based image denoising is introduced to denoise the images. The BM3D for color images is named as C-BM3D [6]. WNNM is introduced to denoise the images by using the image non-local self-similarity [15]. Table 4 and Figure 10 show the comparative analysis for grayscale images of the LR-SVM-HLOG method with some conventional techniques such as NCSR, IIBSE, GSR, and BM3D. Figure 10 illustrates the comparison of LR-SVM-HLOG with the conventional method GSR [44]. From the analysis, it is concluded that the LR-SVM-HLOG method gives effective IR performances. The performance of GSR is high at 30% and 40%        noise, because it works only in medium-level noise. However, the LR-SVM-HLOG method works both at low and medium noise levels. The comparative analysis for color images is given in Table 5 and Figure 11. Figure 11 shows the PSNR comparison of the color images at 20% noise level. From the comparison, it is concluded that the IR over the color images provides an effective performance in terms of PSNR. The main reason is that the LR-SVM-HLOG method tests each image based on the block-by-block sequence. By testing the images in a block-by-block sequence, the entire information from the image is obtained, which is used for an effective IR.

Conclusion
IR technology is one of the major technical areas in digital image processing. In this paper, the LR-SVM-HLOG method is introduced for restoring the quality of corrupted images. This LR-SVM-HLOG method mainly comprises two steps: LR-SVM training and LR-SVM testing. Here, the training of LR-SVM is based on noisy image blocks with the original image blocks. The trained values are stored in the dictionary. The noisy image is given to the testing stage for comparing the input image with stored dictionary values. It provides the respective