An innovative binocular vision-based method for displacement measurement in membrane structures

: This article presents a new binocular vision method for accurate deformation measurements of ﬂ ex-ible membrane structures. Using enhanced marker points on the membrane, the method identi ﬁ es areas for displacement measurements, ﬁ ltering out unwanted image features with scale-invariant feature transform and threshold correlation. It integrates Canny edge recognition and quadratic weighted averaging for precise positioning of measurement points. By comparing reference images and utilizing the principle of minimum distance between matching points, the method achieves fast matching and determines the three-dimensional coordinates of marker points, enhancing measurement e ﬃ ciency and robustness. This approach has been empirically tested on membrane structures, providing new insights. The results highlight that our novel algorithm can achieve high-precision measurements down to millimeters, and its accuracy increases with the actual displacement of the membrane structure. Notably, this groundbreaking measurement method remains una ﬀ ected by the form of the membrane surface, addressing a long-standing challenge in the ﬁ eld.


Introduction
Membrane structures have the advantages of larger span, lighter weight, and adjustable stiffness compared with other typical flexible structural designs [1][2][3][4].Thus, it is widely applied to architectural, mechanical, aerospace, and other engineering fields such as gymnasiums and museums [5,6].With a membrane structure building in an area prone to extreme environments, long-term operation service might have a certain degree of deformation, resulting in structural defects and damages [7].Membrane structures can be degraded over time due to various factors such as extreme temperature, wind load, and other environmental effects.The membrane structure can be easily deformed due to its lightweight and flexible nature [8,9].During the construction and operation of membranestructured buildings, it is necessary to monitor and inspect the deformation of the membrane to ensure the stability and integrity of the structure under various conditions.
In recent years, visual measurement has emerged as an alternative method of non-contact measurement technologies for the deformation measurement of membrane structures, aiming to improve accuracy and reliability and overcome the limitations of traditional methods.Various techniques have been proposed including common methods such as strain gauge, displacement transducer, and more recent methods.Ryan and Shan [10] used a photogrammetry method to measure folds caused by thermal expansion by measuring the membrane surface on the optimally fitted plane.Wong et al. [6] used drone photogrammetry technology to evaluate the deformation of the air-supported structure of large floating covers.Vien et al. [11] examined a method of strain measurement of the floating cover air membrane of an aerospace structure by combining droneassisted measurement and the finite-element method to assess the integrity of the floating cover structure.Zhao et al. [12] used the three-dimensional photogrammetry system to monitor the creep of the air pillow membrane and proposed a stress and strain determination method based on non-contact photogrammetry.Xing et al. [13] used commercial image processing software to obtain the displacement of the identification point of the aerated membrane surface and simulated the overall deformation.Wu et al. [14] conducted response observations for membrane structure air bomb model tests based on binocular stereo vision technology and proposed more suitable observation schemes.Liu et al. [15] proposed a three-dimensional (3D) dynamic measurement technology of dot matrix projection of thin membrane structure combined with stereo vision using a laser scanner.Li et al. [16] used a commercial photogrammetry system to measure the overall displacement of the air pillow forming process.He then combined the approximate membrane surface displacement, numerically simulated the plane formation process of the air pillow at various temperatures, and validated the model's applicability.Cao et al. [17] proposed that the membrane was speckle-experimentally tested, and the 3D displacement field of the square membrane was measured to obtain deformation morphology using digital image correlation technology.Chucheepsakul et al. [18] performed numerical simulation to verify the analytical equilibrium equations of large deflection measurement of circular and elliptic membrane structures.
It is important to note that the proposed method in this article is a significant contribution to the field of structural monitoring systems.Binocular vision technology has been recently adopted for structural monitoring system; nevertheless, the algorithms used to process the image capture can be complex, and they need to be improved to make the system more accurate [19,20].By using binocular vision technology, the proposed method can provide a more accurate result for measuring the deformation of flexible membrane structures.The use of the scale-invariant feature transform (SIFT) feature and threshold correlation method to identify the effective region of the identification point is an innovative approach that improves the accuracy of the measurements.Additionally, the integration of Canny edge detection with a secondary weighted averaging algorithm to obtain the central coordinate of the identification point is a novel technique that can provide more accurate displacement change results.In this article, rigorous experimental research is conducted to verify the accuracy of the measurements, and the analysis of the membrane structure observations under different forms demonstrates the effectiveness of the proposed method.Overall, the proposed method can be a promising approach for monitoring the displacement of membrane structures and can potentially be used in various engineering applications.

Overall flow
In image recognition, the accuracy of results hinges significantly on the methodology used for extraction.This article introduces a novel approach to pinpointing the target point.Figure 1 illustrates the comprehensive flow of the image recognition process, wherein the innovative method proposed in this study is applied during the target point identification phase.In the process of measuring the deformation of binocular stereo vision, it is common to install artificial identification points on the key parts of the surface of the object to be measured and then establish the binocular visual measurement system to identify and process the identification points.

Identification of effective areas based on SIFT characteristics and threshold correlation methods
The membrane surface is marked with key points, and image data are acquired using a binocular vision measurement system.During image acquisition, the range of pictures taken by the camera is much larger than the marked area, making it difficult to avoid interference of invalid areas during the extraction of the image identification point.In this article, a method based on SIFT characteristics and threshold correlation is proposed to identify the effective area of the identification point.This method is divided into two steps: identifying the candidate area for the identification point and then determining the effective identification point area.The common identification points are generally the ones shown in Figure 2a.However, this article proposes a new improved marking point, as shown in Figure 2b.First, the acquired image area is filtered and then binarized.In the binary image analysis, the algorithm of a connected area marker is adopted by marking the target pixels in the binary image.Each individual connected area forms an identified area block, so as to further obtain the outline, centroid, and other parameters.The two-pass algorithm is used to analyze and extract the connected areas.For the identification of the effective area of the identification point, the SIFT local feature detection algorithm is proposed, and the Fast Library for Approximate Nearest Neighbors (FLANN) region feature matching algorithm is combined to complete the extraction and matching of the SIFT feature points on the template image and the candidate area image.The SIFT feature point extraction is divided into four steps: (i) constructing scale space, (ii) difference of Gaussian (DOG) space extreme value detection, (iii) key point direction allocation and description, and (iv) keypoint matching.The Gaussian kernel function is used to construct a scale space, as presented in Eq. (1).
where x, y are the coordinates of pixels.The image continuously transforms the scale by the Gaussian kernel function and finally obtains a multi-scale spatial sequence.The scale space of a two-dimensional image can be represented as follows: where , , is called Gaussian-scale space, and σ is the size determining how smooth the image is.
To detect extremum points and present in scale space, each pixel is compared to all adjacent points of its image threshold and scale threshold.According to the DOG definition formula, the local maximum value is searched for in different scale spaces, and the detection of the key points of the DOG space is completed by comparing them with the neighboring points.The DOG definition formula is as follows: where k is a parameter depending on the test situation.For the local extremum points of the detected DOG space, the DOG function is expanded using the Taylor series by fitting to the DOG function of the scale space, as shown in Eq. ( 4), where D is the differential operator.Determine the location and scale of key points and eliminate some unstable edge feature points to improve the accuracy of matching The key point description subset is established for the template image and the screened candidate area image, and the target is identified by comparing the key descriptors in the two sets, and the similarity of the key point descriptors is measured by Euclidean distance matching method, as shown in Eq. ( 5).

(
) ∥ ∥ ( ) To exclude feature points without matching relationships due to the confusion and complexity of the image background, the SIFT key points in the template image are extracted using the FLANN feature matching nearest neighbor algorithm and combined with the k-nearest neighbor method.The first two key points are found in the candidate area according to the minimal distance rule.If the ratio between the two distances is judged to be an error match point, the matching process is shown in Figure 3.By matching the template features and the characteristics of the current candidate area, the number of key points and matches is obtained.If the number of key points and matches surpass the set threshold, the area is considered a valid area.

Common methods of detection
Since the light intensity and the noise are sensitive to the change of the environment in practice, this article proposes an adaptive Canny edge detection algorithm and combines the OTSU (proposed by Ōtsu Nobuyuki) algorithm [21] to identify the target in the image.In the circular marking point often used in binocular stereo vision measurement experiments, when the position of the circular marking point is not perpendicular to the optical axis of the camera, whether the Steger method is used to extract the center of the ring or the OpenCV native blob method to obtain the center of the ring, there is a certain accuracy problem between the actual center of the measured feature point and the center of the feature point on the captured image.The optical axis of the camera and the position of the identification point are not always vertical during the actual shooting process and, in practice, the membrane structure usually has large displacement due to its flexibility.
The characteristics of the membrane structure should be fully combined when posting the marking point.It is necessary to choose an identification point that can be easily detected and identified and can accurately reflect the actual deformation displacement of the corresponding part.This article makes a new improvement on the basis of the circular identification point, that is, a pair of rightangle sectors are added to the center of the inner ring position.Figure 4 shows the result of image binarization using the OTSU method on the target point.
Two methods are considered to detect the target point: right-corner angle detection and ellipse edge detection.In this article, the right-corner edge is used to obtain the center coordinate at the corner point, noted as P rc .The edges of the outer ring that identify them are then inspected.Since circles tend to appear elliptical in camera imaging, the benefits translate into the detection of elliptical edges.There are two ellipses in the image.The center coordinate detected from the edges of the outer ellipse is noted as P oc , and the coordinates derived from the inner ellipse are noted as P ic .Right-angle edge detection and ellipse edge detection are shown in Figure 5.The center coordinates (P wcr ) of the obtained inner and outer ellipses are weighted and averaged, and the pixel coordinates after weighted fusion are obtained, and the formula is as follows:

Proposed detection method
The right-angle method and the ellipse edge method are typically effective at obtaining accurate results when the angle or ellipse is situated on a plane.However, significant deformations occur on the surface of a membrane structure during the load applied, causing the initial right angle and circle to undergo large deformations and not lay on the plane.Therefore, there can be significant errors in the detection of these methods.Interestingly, the right-angle method usually yields a far point, and the ellipse-edge method yields a near point.Therefore, the weighted fused ellipse center coordinates are fused with the center coordinates based on the edge detection of corner points by a second weighted average, and the result is regarded as the final pixel coordinates that identify the center of the point, with the following formula: where P wcr is the center coordinate of the identity point of a weighted average fusion, P rc is the center coordinate of the fit of the right-angle method, and the center coordinate of the identification point, P wc the secondary weighted average fusion is the final pixel coordinate of the identification point.Hence, considering the complex environment of binocular vision technology and the object to be measured, the center coordinates can be weighted and averaged by the results of three methods combining Eqs. ( 6) and (7).The final formula is shown in Eq. ( 8).
where α 1 , α 1 , α 1 are the coefficients and . The coefficients are determined by the shape of the target figure.In this article, the weighted coefficients are chosen as the three Gauss points 0.5, 0.25, and 0.25, respectively.
In the article, a method for image matching is presented that integrates a grayscale-based normalized cross-correlation (NCC) algorithm with Euclidean distance matching.This technique calculates the cross-correlation values between a template image and a sequence image to determine their degree of matching, focusing on the grayscale values.Additionally, it employs Euclidean distance matching, where the nearest Euclidean distance between identification points in the template and those in a sequence image sub-graph is compared against the sub-progressive Euclidean distance of all corresponding points.This comparison helps in identifying correct matching pairs.The method is applied to both left and right image sequences, ensuring accurate matching of identification points over time.The principles and process of this matching technique are visually detailed in Figure 6.

Specimens
In this study, the experiments have been completed in collaboration with Shanghai WeiBo Membrane Company of Yancheng City, Jiangsu Province.Using the self-made membrane tension equipment, the four levels of tension membrane were loaded.The membrane selected for the test is made of polyvinylidene difluoride (PVDF).The material properties are shown in Table 1.
The specimens used in the tests were divided into four sizes.Specimens 1 and 2 correspond to the two design  schemes under the uniaxial test of planar tensile conditions, specimen 3 is a design under the biaxial test of planar tensile conditions, and specimen 4 is a saddleshaped tensile membrane.The cutting process and production of the specimens were carried out by the Weibo Company, as shown in Figure 7.
The design scheme is shown in Figure 8, where there are five object points in specimens 1, 2, and 3 and six object points in specimen 4. The layout of the object points is shown in Figure 8, and practical object points can be seen in the specimens, as shown in Figure 9.
The physical drawings of the four specimens are shown in Figure 6.To ensure the precision of the marker points, we adopted a distinct marking method for specimen 2 compared to specimens 1, 3, and 4. Specifically, for specimen 2, we marked an annulus, whereas for specimens 1, 3, and 4, we marked the points as a ring edge plus an annulus.

Test apparatus and instruments
The test apparatus was independently developed and designed by our research team.The support structure is a welded steel bracket support system with a total size of 3,080 mm × 3,080 mm and an overall height of 1,500 mm, and the brackets were made of square steel pipes measuring 40 mm × 40 mm.The design drawing and actual structure are shown in Figure 10.Displacement measurement in membrane structures  7 The test was conducted to consider different pretension levels and various load cases.The pretension was applied at the edge of the membrane.To apply point load to the surface of the membrane, bolts were fastened onto the membrane surface.The bolt weights were 0.2, 0.5, and 1 kg, respectively, to represent three different point load cases.The pretension device consisted of a self-locking winch and a wire rope fixed around the support.For flat tension membranes, the membrane boundary was connected to the membrane using strip steel plywood and wire rope.For surface tension membranes, wire ropes and manual winches were connected using rigid splint connections at the corners.During the test, the rotating manual winch tightened the wire rope, thereby providing a certain tension to the membrane, as shown in Figure 11.
The test utilizes the DYLY-103 S tensile force sensor to measure the pretension value.The sensor has a measuring range of 1.0-1.5 tons with an accuracy of 0.02%.To acquire displacement data of each measurement point on the membrane surface, this article employs the HBVCAM binocular camera model as the image acquisition device of the binocular stereo measurement system.The specific parameters of the camera are listed in Table 2.  To evaluate the precision of the method, multiple HG-C1200 Panasonic laser displacement sensors were deployed.These sensors feature a measurement range of ±80 mm and offer an accuracy of 0.1% along with a resolution of 0.01%.

Experimental process
There are 14 working conditions in total, considering different pretension loads and the rise-to-span ratio of the membrane surface.A summary of the test conditions is shown in Table 3.
For each working condition, the deformation measurement of every point on the membrane surface took into account four factors: tension force, the arrangement of data acquisition devices, the application of load, and the data acquisition process itself.In the setup phase, the laser displacement sensors were securely mounted beneath the membrane surface, aligned with each measurement point.The binocular camera was positioned at a diagonal angle above the membrane, supported by a tripod to maintain absolute stability.It was crucial to ensure that the camera remained completely stationary throughout the process.The camera's light was oriented at an approximate 45°a ngle to the ground, and it was positioned roughly 1.0 m from the membrane surface, as depicted in Figure 12.
Prior to load application, the internal and external parameters of the binocular camera were calibrated [22].Figure 13 illustrates the checkerboard calibration plate employed in this process.At each stage of loading and unloading, the membrane structure was captured using the binocular camera.It is important to clarify that the 16 images captured by the cameras were taken at random   angles within a range of 0-60 degrees.This approach was deliberately chosen to test the robustness of the calibration method rather than adhering to a set of fixed angles.
4 Results and discussions

Measurement accuracy test
In order to verify the accuracy of different extraction methods, three extraction methods were used: the right angle fitting method named M1, weighted fusion based on the inner and outer circles named M2, and based on the weighted method of M1 and M2, named M3.The specimen with a pretension load of 320 kN was used for validation.The error between the calculated displacement and the actual displacement was then calculated using the Euclidean distance method and shown in Table 4, where the "Dis" means the displacement detected by laser measurement and "Cal" represents the calculated value.It should be noted that for the M1, M2, and M3 methods, the coefficients of (α 1 , α 2 , α 3 ) used for Eq. ( 3) are (1, 0, 0), (0, 0.5, 0.5), and (0.5, 0.25, 0.25), respectively.The errors in measuring displacement using lasers for methods M1, M2, and M3 are presented in Figure 14.This article suggests a method that combines the corner point, inner circle, and outer circle in a weighted manner to obtain the center coordinate of the identification point.As illustrated in Figure 14, this approach has a relative error of less than 5%, which is considerably lower than the results obtained from M1 and M2.Although the identification point extraction based on corner point fitting and elliptical fitting can demonstrate changes in the target measurement point's displacement, it does not offer the same level of accuracy as the proposed approach in this article.

Displacement results
After completing the calibration of the binocular camera, the images collected by the camera at each stage were combined, and image processing based on the OpenCV  4.4.0 environment on the Visual Studio 2017 platform was used to obtain the 3D displacement information of each measurement point on the membrane surface using binocular stereo vision technology.The displacement data of each measuring point on the membrane surface were collected using a laser displacement sensor.The displacement results of each measuring point under different working conditions and three loading stages were compared.In this section, only the displacement results of measurement points A and C under pretension of the four membrane types are reported, as shown in Figure 15.It is important to highlight that only the out-of-plane deformation of the membrane, subjected to three loading stages, was considered, while the in-plane deformation was disregarded.
The relative error of the displacement measured at 480 N with the span ratio of 1/8 is shown in Figure 16.It should be noted that the axis label of "test position" in Figure 16 corresponds to the data from Figure 15 presented in order.It can be seen that the relative error of all cases is less than 10%.
Due to the variety of membrane structures, in order to explore the applicability of the binocular visual measurement method to different membrane surface forms, the displacement results of each measuring point under the full load state of the four pretension membrane forms are compared, as shown in Figure 17.There are five test positions named as "A," "B," "C," "D," "E," respectively, for each case.Thus, each figure displays five pairs of data.Figure 18 shows the displacement data obtained from specimen 4 with different span ratios.It is evident from Figures 17-18 that the displacement error is very small when compared to the measurements obtained through a laser displacement sensor.Figure 19 illustrates the relative error of displacement for specimens 3 and 4. The results are the relative error derived from Figures 17 and 18 for each case.Notably, the majority of the measured values have a relative error of less than 10%.However, there is one exceptionpoint E in the 480 N casewith a relative error of 16%, which is highlighted in Figure 19.The observed error at this point is attributable to significant local deformation in the membrane.The deformation caused by this localized effect appears to have a disproportionate impact on the measurement accuracy, resulting in the comparatively larger relative error observed at point E.
From the measurement results, it can be seen that the binocular visual measurement method proposed herein is consistent with the displacement acquisition result of the identification point and the measured result of the laser displacement sensor.There is no significant change in the error between the two, which can indicate that the accuracy of the proposed measurement method will not be affected by the change in the form of the tension-drawn membrane.

Conclusions
This article proposes an alternative novel deformation measurement method for flexible membrane structures and improves the accuracy of the detection method.This binocular vision technology identifies the effective area of the objective points based on the SIFT feature and threshold correlation algorithm by applying the surface paste to the measured membrane structure.The center of the objective point is extracted by combining Canny edge recognition and the quadratic weighted average algorithm.Taking into account the minimum distance between two corresponding points of mutual matching, the center distance of the identification point is obtained sequentially from the benchmark imagery.The corresponding matching points of each objective point in the left image are assessed in turn in the right image, completing the 3D matching of the identification point.Furthermore, the grayscale-based image matching normalization product algorithm (NCC algorithm) and the distance matching method are employed to complete the timing matching of the image identification points at different times, obtain the 3D coordinates of the identification points, and find the corresponding displacements, thereby enabling accurate measurement of multi-point displacements on the surface of the membrane structure.
The application of this binocular visual measurement technology to deformation measurements of membrane structures demonstrates that the improved algorithm can meet the need for high precision and is better than other traditional methods such as strain gauge, displacement, and transducers.In addition, Eq. ( 3) is validated for image detection, with recommended coefficients of 0.5, 0.25, 0.25.A comparative study on displacement errors across four different membrane surface forms indicates that the pretension membrane's form does not influence the precision of the measurement system.Further analyses reveal that the measurement error remains below 10%.It should be noted that these parameters can be extended to other applications.With these advancements in the algorithm, it is evident that the introduced method holds significant potential for accurately measuring extensive deformations in structural membranes.Furthermore, binocular vision systems are more cost-effective than laser scanners.This makes them accessible for a wider range of applications.

Figure 1 :
Figure 1: Calculation flow of image recognition.

Figure 2 :
Figure 2: Marking points: (a) common marking points and (b) proposed marking point.

Figure 3 :
Figure 3: FLANN feature matching: (a) error match point and (b) match the points correctly.

Figure 4 :
Figure 4: OSTU binarization results: (a) target point on specimen and (b) gray map of the target point.

Figure 5 :
Figure 5: Marking point edge recognition: (a) right angle corner detection and (b) ellipse edge detection.

Figure 6 :
Figure 6: Principle of marker point matching.

Figure 10 :
Figure 10: Test support device: (a) design drawings and (b) equipment for support.

Figure 11 :
Figure 11: Test equipment: (a) the planar membrane test and (b) the surface membrane test.

Figure 12 :
Figure 12: Binocular camera acquisition process: (a) binocular camera placement and (b) image collection.

Figure 15 :
Figure 15: Results of displacement of specimens 1-4 at points A and C measured at 480 N and the rise-to-span ratio 1/8: (a) measurement point A of specimen 1, (b) measurement point C of specimen 1, (c) measurement point A specimen 2, (d) measurement point C specimen 2, (e) measurement point A of specimen 3, (f) measurement point C of specimen 3, (g) measurement point A of specimen 4, and (h) measurement point C of specimen 4.

Figure 16 :Figure 17 :
Figure 16: Relative error of the displacement at the case of 480 N and span ratio 1/8.

Table 2 :
Parameters of HBVCAM binocular camera

Table 3 :
Summary of test conditions

Table 4 :
Feature point location results and relative errors (mm) Figure 14: Errors in laser measurement of the specimens.