Due to the intensive software development over the recent years, computer calculations have been becoming more and more advanced. Innovative applications used in Information Technology support actions from the scope of both: science and industry. Significant group of solutions used in most research fields is the set of the numerical modeling and visualization applications [1, 2, 3]. Their primary function is to visually represent the processes taking place within different physical phenomena. Visualization techniques can influence the comprehension of obtained view. Use of stereoscopic methods dramatically increases the spectrum of visualization capabilities. It allows for a more accurate space representation and perception. Three-dimensional simulation and visualization techniques are also widely used in the film industry. There are more and more movies, that are realised in innovative techniques and are becoming the integral part of our lives, not only as pastime but also as a way of communication.
Considering only the number of 3D screens, there are almost 90 thousands three-dimensional cinemas worldwide where millions of projections are shown every day. Such a big infrastructure created many opportunities for the industry, but on the other hand innovative technologies made cinematography completely dependent on the computer applications aiding the process of making a movie. In the case of the three-dimensional techniques one of the most crucial issue is the quality of the synthesized stereoscopic view . The forementioned problem is analogous in both, visualization applications as well as in the case of movie. In this context, it is significant to detect all view defects at the stage of material production in order to provide correct final product. Omission or negligence of step may very likely result in release of a flawed material, which causes visual discomfort or other complaints. Detecting the defects enables making corrections and providing correct interpretation of the analyzed data. The process of reference 3D image quality analysis consists of the comparison of stereo views in order to identify artifacts in the stereo view.
2 Vertical disparity
The correctness of applied stereoscopic scene is the main factor conditioning the high quality of obtained view. The elementary implementation of this solution is based on the usage of two parallel cameras placed on the specialized trivet called RIG (Figure 1). It provides a portable and easy-to-set tool allowing to get three-dimensional visual data . Achieving an appropriate final view by using the cameras is determined by correct positioning and calibration of the whole equipment. RIG constitutes a tool for control of the camera and provides a precise calibration device. It is worth to mention, that despite the above described solution, the proper setting of cameras is still a non-trivial issue. Moreover, the initial calibration of the RIG does not guarantee the correctness during the whole shooting. It is particularly noticeable in the case of difficult weather conditions during the shooting.
Anomalies in the calibration not only lower the quality of the final product by causing a variety of image dysfunctions , but even worse, may be the source of the user’s health problems. There are many different kinds of scene implementation anomalies with various reasons and effects. The major artifacts observed in three-dimensional materials are anomalies of stereo views rotation, zoom and vertical disparity.
Vertical disparity is one of the most common, and exceptionally inexpedient defects from the scope of scene implementation. The phenomenon occurs in a three-dimensional images. The vertical displacement of cameras or convergence point decalibration is the source of the problem. The incorrect localization and calibration of the cameras is the reason why the objects from the left and from the right view of the stereo pair are shifted with one another (Figure 2). The incorrect mutual translation between objects of the stereo pair is the direct root of the image defects .
The desirable value of the above mentioned displacement is zero. It means that the image is correct with regard to the disparity issue. If the value does not equals zero (h), the image contains the disparity error. User may properly interpret the three-dimensional visualizations only if the objects from their stereo pair have no vertical displacement. Occurrence of this artifact precludes proper image receiving. The defect has a negative impact on the product quality, but also may results in serious health consequences in the case of longer exposure of the viewer to the discussed anomaly. The only way to avoid such problems is to conduct precise tests of 3D visualizations in order to detect visual artifacts, especially the vertical disparity. Defects connected with the occurrence of the disparity have to be entirely eliminated before releasing the product.
3 Vertical disparity detection method
The quintessential task of the vertical disparity detection algorithm is calculating the translation vector between stereo views. The first step to realize the method is to determine the proper correspondence between objects of the stereo pair. This process is run by mutual matching of characteristic areas of stereo views. The determined matches are the base to calculate vertical displacement between the objects within the three-dimensional image .
Nowadays on the market, there is a wide variety of methods which enables finding a correlation of the stereo pair. Nevertheless, it is still a serious problem to apply them in HD image. The substantial impediment is the operation time. Precise and detailed processing of the high resolution 3D images requires too much time. Such a time-consuming method can not be applied in real time applications because they require immediate calculations and results. It implies the necessity to optimize the calculation time without lowering the high quality. Fulfilling both conditions is a major issue in the subject area. The calculation cost may be lowered by using the method of partial image analysis. The algorithm, instead of using whole images, compares only the most representative subset of image’s pixels.
The binary patterns matching method is the solution enabling to lower the calculation costs by working on the image’s pixels subset. With this the method the matches between two images can be found. It is used with regard stereo views in particular. Knowing the correlation between pixels in stereo pair it is possible to mark out the local vertical disparity values. The displacement is calculated in the Y coordinates of corresponding images. The calculated result equals the value of the local vertical disparity. The method of binary patterns matching processes only the chosen, representative pixels. As a result the amount of processed data is much smaller than in the case of considering on all pixels of the image and therefore it is possible to save much calculation time. Unfortunately this method entails also the risk of incorrect matching. The inaccuracy in generated correlations is associated with processing the single pixels, which, unlike when using the whole images, may cause incorrect points matching within the stereo pair. The pixels of different objects may be very similar to each other in their numerical space, even though they represent something completely various. If they are recognized as the corresponding ones, the inappropriate matching occurs (Figure 3). The error makes it impossible to run correctly further stages of the calculations, blocking the process of detecting the value of vertical disparity.
The larger the image, the bigger the risk of finding an incorrect correlation. Also having the repetitive patterns in the image enlarge significantly the probability of incorrect match. To avoid the negative consequences of the error, filtering mechanism has to be applied. The solution enlarges the correctness of the binary pattern matching process, but it does not eliminate all problems. If the patterns in a stereo pair are repeated too many times, the filtering mechanism may not be enough, what finally equals receiving factually incorrect results .
4 Improved algorithm of disparity detection
To dispose entirely of the above mentioned error, it is necessary to redefine the view processing area. Dividing the image into smaller subimages enables mitigating the risk of repeating similar pixels in the processed area. With this method it is possible to receive only correct matches . The sliding window mechanism is a tool enabling division of stereo pair into subimages. The obtained, splitted segments correspond to each other in the framework of the primary stereo view. The process of division utilizes a rectangular, equal-sized window, which sets the boundaries of the processed area, so that the pixels from the outside of the border are not taken into account. The window moves across the surface of the stereo pair marking out consecutive subimages. Moreover, it guarantees that every region is taken under consideration only once, without needlessly repeated calculations in the already processed fragment. It allows to save time and make the method a robust and optimalized tool.
The subimages from the corresponding sliding windows from the left and from the right view are compared utilizing the binary patterns matching algorithm to find the matches belonging to the processed area. From all matches marked out in the framework of a single subimage pair, only the best matches are chosen. Not only one match (as in the previous version of the algorithm), but several best matches are used. This means selecting a set of matches that fulfill the condition of minimum vectorial distance between the feature vectors representing the pixels of the right and left views, respectively. This modification makes it possible for more explicit image areas to have significant roles in estimating the correlation of stereo images The chosen matches are used in further steps of disparity estimation process. The solution improves the correctness of final results (Figure 4). The way of estimating the value of local vertical disparity does not differ from calculating the disparity in the case of the basic algorithm - it means marking out the displacement value in the Y coordinates of matched pixels. After all pairs of subimages are processed according to the described algorithm (Algorithm 1), the stereo pair analysis may be considered as a completed one. The mean of obtained in this way values of local disparities equals the global stereo view disparity value. Provided that the sliding window mechanism has correctly estimated parameters, the described algorithm enables elimination the problem occurring in the binary patterns matching process.
Pseudocode of the sliding window disparity detection algorithm
VL LeftStereoscopicView, VR RightStereoscopicView
SL LeftSubimage, SR RightSubimage
KL LeftKeypoints, KR RightKeypoints
DL LeftDescriptors, DR RightDescriptors
MLR LeftToRightMatches, MRL RightToLeftMatches
for w in 1:min(VL.width,VR.width) step SWW do
for h in 1:min(VL.height,VR.height) step SWH do
MF:=MLR ⋂ MRL
for each b ∈ MB do
DV.add(b.leftPoint.y - b.rightPoint.y)
Conducting tests is a crucial step in the process of specifying the parameters and detecting faults in the developed method. There are two possible ways of assessing the improvement of the algorithm: comparing basic and modified implementations, or considering only the metadata of a testing set. To obtain as solid and complete results as possible, the Middlebury Stereo Dataset  was used in the testing process of the primary implementation of the disparity detection algorithm. Middlebury Stereo Dataset is a set of three-dimensional images and metadata i.e. the stereo pairs parameters including the vertical disparity.
The tests have been carried out for a couple of cases differing between each other in sliding window parameters. The below presented table (Table 1) shows the most significant results.
Considering the data contained in the chart, appears the conclusion, that the propounded improvement of the binary patterns matching mechanism makes the process of calculating the vertical disparity much more efficient and precise. The changes are evidently visible especially in the comparison of columns E, F, G. The values from the G column are approximate to the values from the E column, unlike the values from column F, where the corresponding numbers differ from each other much more. The more similar the values are to each other, the more accurate and correct the results are received in the process of estimating the vertical disparity. It means that the obtained values of displacement are not the effect of algorithm error, but they are nearly equal to the ground truth values. The main condition for the satisfying work of the mechanism is the proper calibration of sliding window parameters (shown in the column B). To avoid the problem of differences in the views from the corresponding stereo subimages, the width of a sliding window has to be estimated at 20% of a stereo image (it is conditioned by the fact that the same objects have to occur in both subimages, what arises from the basic rules of stereoscopy). The height of a sliding window was tested in the cases ranging from 1 to 100% of a stereo pair height. The tests indicated that the more reduced height of the sliding window, the more accurate and satisfying results, but on the other hand the height should not go down under the 5% of a height of the stereopair. It is caused by the fact that if the value of vertical disparity is bigger than the height of the sliding window, some objects contained in one analyzed subimage, may not occur in the corresponding one. Such situation does not allow to carry out correct calculations. To avoid it, it is crucial to provide a sliding window, which is higher than the value of vertical disparity in the analyzed material. In the three-dimensional scenes the vertical disparity, which values are above 10%, is visible to the naked eye and do not require any additional ways of evaluation. Taking under consideration all the above mentioned conditions, it is best to use a sliding window, which dimensions are ranging between 20%x10% and 20%x5% of an analyzed image.
Encapsulating the tests process, one more improvement should be highlighted. The modified algorithm requires significantly smaller amount of matches to calculate the vertical disparity value than the basic one, which is visible in the comparison of columns C and D. Due to it, it is possible to reduce the amount of data to analyze and therefore to save calculation time and optimize the process. To sum up, taking all the above mentioned conclusions under consideration, it is clear that the developed algorithm is a much better tool than the basic version of it.
It is worth to mention, that the proposed method is tested during three-dimensional movie shooting (Figure 5) to evaluate stereoscopic image quality and properly calibrate the camera set. This makes it possible to evaluate the developed algorithm in real conditions and continuously work on its improvement.
The presented method of improving the algorithm involves both: the issue of correctness and the question of processing time and optimizing it. Due to the usage of the sliding window mechanism it is possible to minimize the problem of finding incorrect correlations in the binary patterns matching process. At the same time the method of qualifying only the best matches to the further calculations enables to accelerate the operations. As a result, the robust and precise vertical disparity detection algorithm is obtained. It is successfully utilised in the film industry in the Cinema Vision Project so that the produced movies are released in better quality . What is important, providing certainty to producers, is that their product will not cause any health problems to the users.
The developed algorithm may be used in the case of any three-dimensional material, not only in commercial movies, but also in visualizations. This means that the developed mechanism can be used at any stage of 3D material production beginning with the verification of the three-dimensional scene construction, and ending with the correctness analysis of the post-production process. It makes the algorithm a very universal and powerful tool to detect the vertical disparity and estimate the quality of stereoscopic images.
Wiak S., Smolka K., Dems M., Komeza K., Numerical (solid) modeling of 3-D intelligent comb drive accelerometer structure - Mechanical problems, Przeglad Elektrotechniczny, 2006, 82, 68-71 Google Scholar
Wiak S., Napieralski P., Visualization Method Of Magnetic Fields With Dynamic Particle Systems, Przeglad Elektrotechniczny 2012, 7b, 56-59 Google Scholar
Woods A., Docherty T., Koch R., Image Distortions in Stereoscopic Video Systems, Stereoscopic Displays and Applications, Proceedings of the SPIE, 1993, 36-49 Google Scholar
Cinema Vision Project: http://cinemavision.com/
Voronov A., Vatolin D., Sumin D., Napadovsky V., Borisov A., Methodology for stereoscopic motion-picture quality assessment, Stereoscopic Displays and Applications, 2013 Google Scholar
Lambooij M., IJsselsteijn W., Fortuin M., Heynderickx I., Visual discomfort and visual fatigue in stereoscopic displays : a review, Journal of Imaging Science and Technology, Society for Imaging Science and Technology, 2009 Web of ScienceGoogle Scholar
Kowalczyk M., Fornalczyk K., Napieralski P., Staniucha R., Disparity detection in three-dimensional scenes implementing virtual reality, Computer Game Innovations, Lodz University of Technology Monograph, 2016, 35-42 Google Scholar
Lazaros N., Sirakoulis G., Gasteratos A., Review of stereo vision algorithms: from software to hardware, International Journal of Optomechatronics, Taylor &Francis Group, 2008, 435-462 Google Scholar
Kowalczyk M., Napieralski P., Noise resistant method enabling detection of vertical disparity in three-dimensional visualizations, 18th International Symposium on Electromagnetic Fields in Mechatronics, Electrical and Electronic Engineering (ISEF) Book of Abstracts, 2017 Google Scholar
Kowalczyk M., Napieralski P., An Effective Client-Side Object Detection Method on the Android Platform, Journal of Applied Computer Science, Technical University Press Łódź, 2, 2015, 29-38 Google Scholar
Rosten E., Drummond T., Machine learning for high-speed corner detection, Computer Vision-ECCV 2006, Springer, 2006, 430-443 Google Scholar
Calonder M., Lepetit V., Strecha C., Fua P., BRIEF: Binary Robust Independent Elementary Features, Computer Vision–ECCV 2010, Springer, 2010, 778-792 Google Scholar
Middlebury dataset: http://vision.middlebury.edu/stereo/data/
Rogalska A., Napieralski P., A model of saliency-based visual attention for movie retrospection, 18th International Symposium on Electromagnetic Fields in Mechatronics, Electrical and Electronic Engineering (ISEF) Book of Abstracts, 2017 Google Scholar
About the article
Published Online: 2017-12-29
Citation Information: Open Physics, Volume 15, Issue 1, Pages 1028–1033, ISSN (Online) 2391-5471, DOI: https://doi.org/10.1515/phys-2017-0128.
© 2017 Piotr Napieralski and Michał Kowalczyk. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. BY-NC-ND 4.0