Blind Restoration Algorithm Using Residual Measures for Motion-Blurred Noisy Images

Abstract Image de-blurring is an inverse problem whose intent is to recover an image from the image affected badly with different environmental conditions. Usually, blurring can happen in various ways; however, de-blurring from a motion problem with or without noise can pose an important problem that is difficult to solve with less computation task. The quality of the restored image in iterative methods of blind motion de-blurring depends on the regularization parameter and the iteration number, which can be automatically or manually stopped. Blind de-blurring and restoration employing image de-blurring and whiteness measures are proposed in this paper to automatically decide the number of iterations. The technique has three modules, namely image de-blurring module, whiteness measures module, and image estimation module. New whiteness measures of hole entropy and mean-square contingency coefficient have been proposed in the whiteness measures module. Initially, the blurred image is de-blurred by the employment of edge responses and image priors using point-spread function. Later, whiteness measures are computed for the de-blurred image and, finally, the best image is selected. The results are obtained for all eight whiteness measures by employing evaluation metrics of increase in signal-to-noise ratio (ISNR), mean-square error, and structural similarity index. The results are obtained from standard images, and performance analysis is made by varying parameters. The obtained results for synthetically blurred images are good even under a noisy condition with ΔISNR average values of 0.3066 dB. The proposed whiteness measures seek a powerful solution to iterative de-blurring algorithms in deciding automatic stopping criteria.


Introduction
Image blurring is one of the prime causes of poor image quality in digital imaging. Two main causes of blurry images are out-of-focus shots and camera shake. The image blurring process is commonly modeled as the convolution of a clear image with a shift-invariant kernel plus noise [13,19]. The image recovery problem is to estimate an image from the blurred image by use of various methods [7]. Essentially, it tries to perform an operation on the image that is the inverse of the imperfections in the image formation system [10]. Unfortunately, the restoration of blurred and noisy images is an ill-posed inverse problem [22]. Linear inverse problems arise in a wide range of applications such as astrophysics, signal and image processing, statistical inference, and optics [3].
Generally, image de-blurring (ID) is the operation of taking a blurred image and estimating the clean original image. There are two tightly coupled sub-problems: estimating the blur kernel and estimating the clear image using the estimated blur kernel. Existing ID methods can thus be classified into two categories: blind ID (BID) that jointly solves the above two sub-problems and non-blind ID (NBID) that only solves the second sub-problem [5,8,13]. Research on ID can be divided into NBID, in which the blur filter is assumed known, and BID, in which both the image and the blur filter are (totally or partially) unknown. The most popular NBID methods are the Wiener filter [25] and Lucy-Richardson [15,18] methods. Despite its narrower applicability, NBID is already a challenging problem to which a large amount of research has been devoted, mainly due to the ill-conditioned nature of the blur operator: the observed image does not uniquely stably determine the underlying original image. The problem becomes worse if there is even a slight mismatch between the assumed blur and the true one. Most of the NBID methods overcome this difficulty through the use of an image regularizer, or prior, the weight of which has to be tuned or adapted [4]. BID seeks a solution in terms of a correct pair of an unknown point-spread function (PSF) and original image from multiple combinations of these two unknowns. Some models and algorithms have been proposed that estimate blur kernel in the parametric form [16]. This simplified approach cannot define ideally true motion blurring, and an iterative regularization-based approach is used to estimate complex blurs successfully.
The BID methods require proper selection of iteration number and tuning of the regularizing parameter. In already existing methods, the iterations have to be manually stopped when a good image estimate with high increase in signal-to-noise ratio (ISNR) value is obtained. The method is iterative and starts by estimating the main features of the image, using a large regularization weight, and gradually learns the image and filter details, by slowly decreasing the regularization parameter. From an optimization point of view, this can be seen as a continuation method designed to obtain a good local minimum of the underlying non-convex objective function. The drawback of the method is that it requires manual stopping, which corresponds to choosing the final value of the regularization parameter. In fact, adjusting the regularization parameter and/ or finding robust stopping criteria for iterative (blind or not) ID algorithms is a long-standing, but still open, research area [1]. More work has been devoted to choosing the accurate regularization parameter. The discrepancy principle (DP) [21] selects the regularization parameter such that residual image (the difference between the observed image and the blurred estimate) variance matches to the variance of the noise. The extended versions of DP are based on residual moments [9]. Stein's unbiased risk estimate-based [11,12] approaches are not useful in BID, as it requires full knowledge of the degradation model, but is useful in NBID. It provides an estimate of the mean-square error (MSE), by assuming knowledge of the noise distribution and requiring an accurate estimate of its variance [26].
In this paper, blind de-blurring and restoration employing ID and whiteness measures are proposed. The proposed technique consists of modules of ID, whiteness measures, and image estimation. In the de-blurring module, the blurred image is de-blurred by the employment of edge responses and image priors using PSF. In the whiteness measures module, eight whiteness measures, including covariance, weighted co-variance, entropy, block covariance, block weighted co-variance, block entropy, mean-square contingency coefficient, and holoentropy, are computed for the de-blurred image. Finally, the best image is selected based on whiteness measures, minimum MSE (MMSE), and ISNR values to have the estimated image in the image estimation module.
The rest of the paper is organized as follows: Section 2 gives the literature review, and Section 3 gives the contribution of the paper. Section 4 describes the proposed technique, and Section 5 gives the results and discussion. The conclusion is summed up in Section 6.

Review of Related Works
Many researchers have developed several approaches for ID and restoration. Among them, a handful of significant researches are presented in this section.
Ji and Wang [13] presented a convex minimization model that explicitly takes account of errors in the blur kernel. The resulting minimization problem was efficiently solved by the so-called accelerated proximal gradient method. In addition, a new boundary extension scheme was incorporated in the proposed model to further improve the results. Tai et al. [19] investigated the role that non-linear camera response functions (CRFs) have on ID. They presented a comprehensive study analyzing the effects of CRF on motion de-blurring. In particular, they showed how non-linear CRFs can cause a spatially variant blur to behave as a spatially varying blur. They proved that such non-linearity can cause large errors around edges when directly applying de-convolution to a motion-blurred image without CRF correction. These errors were inevitable even with PSF and with state-of-the-art regularization-based de-convolution algorithms. In addition, they showed how CRFs can adversely affect PSF estimation algorithms in the case of blind de-convolution.
Wang et al. [24] pointed out the weaknesses of the deterministic filter and unified the limitation latent in two kinds of Bayesian estimators. They further explained why the conjunctive de-blurring algorithm (CODA) was able to handle rather large blurs beyond Bayesian estimation. Finally, they proposed a method to overcome several unreported limitations of the CODA. Carlavan and Blanc-Féraud [6] presented two recent methods to estimate this regularizing parameter, and they first proposed an improvement of these estimators that takes advantage of confocal images. Following these estimators, they secondly propose to express the problem of the de-convolution of Poisson noisy images as the minimization of a new constrained problem. The proposed that constrained formulation was well suited to this application domain, as it was directly expressed using the antilog likelihood of the Poisson distribution, and therefore does not require any approximation. They showed how to solve the unconstrained and constrained problems using the recent alternating-direction technique, and they presented results on synthetic and real data using well-known priors, such as total variation and wavelet transforms. Among these wavelet transforms, they especially focused on the dual-tree complex wavelet transform and on the dictionary composed of curvelet and an undecimated wavelet transform.
Ponti et al. [17] presented a restoration approach through band extrapolation and de-convolution that deals with the noise. An extrapolation algorithm using constraints on both spatial and frequency domains with a smoothing operator were combined with the Richardson-Lucy iterative algorithm. The results of the method for simulated data were compared with those obtained by the original Richardson-Lucy algorithm and also regularized by total variation. The extrapolation of frequencies was also analyzed both in synthetic and in real images. The method improved the results with higher signal-to-noise ratio (SNR) and quality index values, performing band extrapolation, and achieving a better visualization of the three-dimensional (3D) structures. Takeda and Milanfar [20] developed de-blurring with a 3D space-time-invariant PSF, instead of removing the motion blur as a spatial blur. Instead of de-blurring video frames individually, a fully 3D deblurring method was proposed to reduce motion blur from a single motion-blurred video to produce a highresolution video in both space and time. Unlike other existing approaches, the proposed de-blurring kernel was free from knowledge of the local motions. Most important, due to its inherent locally adaptive nature, 3D de-blurring was capable of automatically de-blurring the portions of the sequence that were motion blurred, without segmentation and without adversely affecting the rest of the spatiotemporal domain, where such blur was not present. Their method was a two-step approach: first, they upscale the input video in space and time without explicit estimates of local motions and then perform 3D de-blurring to obtain the restored sequence.

Contribution of the Paper
Blind restoration employing ID and whiteness measures are proposed in this paper. The contribution of the work lies in addition of two whiteness measures, i.e. holoentropy and mean-square contingency coefficient. This measure exhibits a clear lowest point and helps select the iteration number. The existing whiteness measures include covariance, weighted co-variance, entropy, block covariance, block weighted co-variance, and block entropy [2]. A major contribution has been made also in the image estimation module where the best de-blurred image is selected based on the metrics of ISNR calculated for all eight whiteness measures. The proposed method consists in selecting the regularization value or the final iteration of the algorithm even under noisy conditions.

Proposed Blind Restoration Technique
Blind restoration using ID technique and whiteness measures is proposed in this paper. The block diagram of the proposed technique is given in Figure 1.

Image De-blurring Module
The blurred image is de-blurred in this module with the use of image priors, image edge response, and power spectral density [1].
Initially, the blurred image Z is obtained with the use of motion filter. The input image g is blurred with an unknown PSF h. The blurring process [1] can be given as Here, "⋅" is the convolution operator and η is the additive white Gaussian noise involved. The notion behind the de-blurring process is the fact that most images tend to have sparse leading edges, and edges of  the blurred images are less sparse as an area of the blurred image edges are larger. The prior and the cost function are modified accordingly such that lower-frequency components are given more weights. The initial iterations correspond to the estimation of lower frequency, and later iterations give the estimation of higher frequency. The modification also aims to reduce the noise encountered. The minimization cost function [1] is given as Here, h is the PSF kernel to be found, β is the scaling factor, and r[f(g)] is the regularization function, where f(g) is the edge response. The cost function is minimized with the use of a conjugate gradient method. The scaling parameter is at first a large value and then decreased over iterations.
The edge responses [1] of the blurred image are defined by the function f(g) given by Normally, φ values of 0, 45, 90, and 135 are taken into consideration. Image priors consider that the edges are spare and edge intensities are independent of each another. The sparse prior [1] with density given the edge intensity for a pixel j represented by f j (g) is defined by k adjusts for the scale of edge intensities, q controls the prior's sparsity, and ∈ is a small parameter. Taking the noise into consideration as Gaussian likelihood [1] is given by The log-likelihood maximization is similar to having the cost function minimized.
Maximizing this likelihood is equivalent to minimizing the cost function [1] Here, the λ = kσ 2 regularization parameter functions over the edge response and regularize was chosen, which favors the sharp edges, or the priors are selected that reach a sparser edge response.

Whiteness Measures Module
In this module, eight whiteness measures are found out from the de-blurred images. The existing whiteness measures include covariance, weighted co-variance, entropy, block covariance, block weighted co-variance, and block entropy [2]. Apart from these, mean-square contingency coefficient and holoentropy whiteness measures are proposed in this paper. The selection of the stopping criteria is based on the measures of the fitness of the image estimate and the blur estimate. This is analyzed based on the residual image given by The nature of the residual is then compared with the additive noise n. The motivation behind employing the whiteness measures to find the adequacy of the image and the blur estimates is the fact that noise taken into consideration is spectrally white. Initially, the image is set to zero mean and unit variance. Let this residual image represented as d: Here, d′ is the sample mean and var(d) is the sample variance of d. The auto-covariance [2] of the normalized residual image (having two-dimensional lag (x, y) can be estimated by where summation is carried out over the de-blurred image and C represents an irrelevant constant. The δ function can be considered as the auto-covariance of a spectrally white image. It can be mathematically represented as Hence, the distance between the auto-covariance function and the δ function forms a whiteness measure and is termed covariance whiteness measure. Considering a (2M + 1) × (2M + 1) window, the covariance whiteness measure can be defined as As the auto-covariance for large lags is smaller than for small lags, weight is incorporated to the normal covariance whiteness measure to have a weighted version [1] given by where ∆ is the weight matrix. The next whiteness measure is the entropy, for which let the power spectral density being represented by dd ψ and the transformation can be expressed as Here, φ is the discrete Fourier transform. The white signal has flat power spectral density due to the fact that its auto-covariance is a δ function. Entropy measure [2] is employed to evaluate the flatness of the power spectral density. Hence, the entropy whiteness measure can be given by The above whiteness measures are found out, assuming that the residual image is stationary. However, it may not be stationary and, in this case, the measures are modified by employing the image block concept. The block-based auto-covariance is given by Here, b represents the index of the image block and P b is a pixel set in the block. Here, the difference is that the residual image is normalized to zero mean and unit variance in a block-by-block fashion rather than the whole image. The whiteness measures of covariance, weighted covariance, and entropy are computed block-wise to have block covariance whiteness measure, block weighted covariance whiteness measure, and block entropy whiteness measure [2]. Let these be represented by ( ), ( ) and ( ).
Normalized power spectral density ψ′(m, n) is calculated in a block-by-block fashion and used as a vector a. Holoentropy considered gives equal weight to all attributes; however, in real conditions, this may not give the best results. Hence, weighted holoentropy by weighting attributes can give increased effectiveness. Here, weighting is carried out such that the more important of those attributes have small entropy values. Weighting is carried out using the reverse sigmoid function of the entropy given by The weighted holoentropy W o (a i ) is the summation of weighted entropy in each attribute of the random vector a.
The best image selection is the one with a minimum value of weighted holoentropy. The final whiteness measure taken is the mean-square contingency coefficient. The mean-square contingency coefficient is a measure of correlation and defined as  (20) where normalized power spectral density ij ψ′ is calculated in a block-by-block fashion, and d 1 and d 2 are the sizes of the block. j ψ′ and i ψ′ are sums of normalized power spectral density ij ψ′ along the row and column (marginal totals) and calculated as Having computed the contingency coefficient in a block-by-block fashion and then averaged to select the best image estimate, best image selection is done when the mean-square contingency coefficient is minimum.

Image Estimation Module
In this module, image estimation is proposed by choosing the best de-blurred image with the employment of whiteness measures and evaluation metrics. The evaluation metrics employed are SNR and MMSE. After the de-blurring process, a set of de-blurred images is obtained based on the number of iterations (denoted by N) in the de-blurring process, and these are represented as D = {D 1 , D 2 , …, D N }. From this set, the best image is selected based on criteria by calculating the whiteness measure for every image. The eight whiteness measures are covariance, weighted co-variance, entropy, block covariance, block weighted co-variance, block entropy, mean-square contingency coefficient, and holoentropy.
From the set of images, eight images are selected initially based on the whiteness measure. The images are selected based on the maximum values for the first six measures and minimum for the remaining two measures. Let the selected images be represented as Hence, the de-blurred image is obtained using whiteness measures and evaluation matrices.

Results and Discussion
The results obtained for the proposed technique is given in this section. Section 5.1 gives the dataset description, experimental setup, and evaluation metrics employed. Section 5.2 gives the simulation results, and Section 5.3 gives the performance analysis.

Dataset Description, Experimental Setup, and Evaluation Metrics
The proposed technique is implemented in MATLAB in a system having 8 GB RAM and 2.6 GHz Intel i-7 processor. The evaluation metrics used are ISNR, MSE, and structural similarity index (SSIM) [23,14]. The dataset consists of various standard images such as Lena, Barbara, Goldhill, Baboon, Boats, Cameraman, and other images obtained from public databases. Figure 2 gives the sample images.

Simulation Results
The simulation results obtained for the proposed technique are given in this section. Table 1 shows the simulation results for three images with degradation parameters length-10, theta-190, and noise level-40 dB. Here, the images obtained for whiteness measures such as covariance (represented as M1), weighted co-variance (represented as M2), entropy (represented as M3), block covariance (represented as M4), block weighted covariance (represented as M5), block entropy (represented as M6), mean-square contingency coefficient (represented as P1), and holoentropy (represented as P2) are given along with the final restored image. Figure 3 shows graphs of various whiteness measures vs. iteration. Graphs are drawn for cameraman image with degradation parameters length-10, theta-110, and noise level-30 dB. The maximum ISNR obtained is for iteration number 22, which is shown in the graph of ISNR vs. iteration, which matches with the selection of iteration number based on whiteness measure by maximizing already existing whiteness measures such as covariance, weighted co-variance entropy, block covariance, block weighted co-variance, and block entropy, and by minimizing newly proposed mean-square contingency coefficient and holoentropy.

Performance Analysis
In this section, the proposed technique is evaluated using evaluation metrics employed such as ISNR, MSE, and SSIM with degradation parameters length-10, theta-190, and noise level-40 dB. The tables below give the evaluation metric value obtained for different images using various whiteness measures.   Inferences from Tables 2-5: -The tables give the performance evaluation of the proposed technique. Table 2 gives the ISNR values obtained and ΔISNR (maximum ISNR along iterations − maximum ISNR obtained out of eight whiteness measures). It shows that all the ISNR values are optimal but the highest ISNR is achieved in Barbara image, i.e. 1.06. However, the minimum ISNR is reached in Baboon, i.e. 0.0; the ISNR value changes based on the performance of particular images. Table 3 gives the MSE values, and          Tables 5-7 give the performance analysis by varying the length, theta, and noise level. We take into consideration four cases as given in the table. Here, ISNR, MSE, and SSIM are also calculated for all eight whiteness measures for boat image and the ΔISNR average value was 0.21685 dB. Figure 4 shows the comparative analysis of six images based on ISNR values with respect to existing techniques. For the comparison of filters, the proposed model motion filter reaches the maximum value when compared to regularization filter, Wiener filter, and Kalman filter.

Conclusion
In this paper, blind de-blurring and restoration employing ID and whiteness measures of the residual image are proposed. The technique has three modules, namely ID module, whiteness measures module, and image estimation module. New whiteness measures of holoentropy and mean-square contingency coefficient have been proposed in the whiteness measures module. The choice of the stopping condition depends on the measures of the fitness of image measure and the blur estimate. Good estimates of both the image and the blurring operator are obtained considering all eight whiteness measures, and the best one is selected based on ISNR. The results are obtained for all eight whiteness measures by employing evaluation metrics of ISNR, MSE, and SSIM. The results are obtained for standard images degraded with uniform motion blurs, and performance analysis is made by varying parameters. The best image is selected on the basis of the whiteness measures and proves to be effective for various images with ISNR losses with respect to best ISNR of only 0.3066 dB on average. The quality of the restored image is good even when the blurred image has noise, and promises the iteration number selection that is the best compromise between image detail and artifacts. The disadvantage of the strategy is that it requires manual halting, which compares to picking the final estimation of the regularization parameter. In fact, modifying the regularization parameter as well as finding regularization parameter for iterative (daze or not) ID algorithms is a long-standing, yet open, research area. In further research, we intend to expand our method to deal with other ID applications, for example de-blurring video sequences or out-of-center deblurring. Our strategies can likewise be connected to a hybrid image system or joined with coded exposure photography.