Estimating 3D displacement vectors from line-of-sight observations with application to MIMO-SAR

: Displacements in typical monitoring applications occur in 3D but having sensors capable of measuring such 3D deformations with areal coverage is rare. One way could be to combine three or more line-of-sight measurements carried out from different locations at the same time and derive 3D displacement vectors. Automotive Multiple-Input-Multiple-Output Synthetic Aperture Radar (MIMO-SAR) systems are of interest for such monitoring applications as they can acquire line-of-sight displacement measurements with areal coverage and are associated with low cost and high ﬂexibility. In this paper, we present a set of algorithms deriving 3D displacement vectors from line-of-sight displacement measurements while applying spatial and temporal least squares adjustments. We evaluated the algorithms on simulated data and tested them on experimentally acquired MIMO-SAR acquisitions. The results showed that especially spatial parametric and non-parametric least squares adjustments worked very well for typical displacements occurring in geomonitoring and structural monitoring (e.g. tilting, bending, oscillating, etc.). The simulations were con-ﬁrmed by an experiment, where a corner cube was moved step-wise. The results show that acquisitions of off-the-shelf automotive-grade MIMO-SAR systems can be combined to derive 3D displacement vectors with high accuracy.


Overview
Monitoring of natural and artificial structures is carried out by public authorities and private entities to mitigate the risk of failure or fatal events. It is often crucial to know not only the amplitude and speed but also the direction of movement to assess the deformation mechanisms [1]. Various sensors with different strengths and weaknesses are being applied in deformation monitoring [2] but they are typically expensive (e.g. total stations [3]) or their configuration cannot be adapted flexibly by the user (e.g. satellite-based systems [4]). Radar sensors in combination with radar interferometry are of interest for deformation monitoring because of their ability to detect surface deformations with sub-millimetre precision and a temporal resolution on the level of milliseconds [5,6]. However, radar systems are mainly sensitive in the line-of-sight direction when using the phase information [7] or orthogonal on a planar area when exploiting the amplitudes [8]. Therefore the actual movement as a 3D displacement vector cannot be derived with a single instrument.
Automotive Multiple-Input-Multiple-Output Synthetic Aperture Radar (MIMO-SAR) systems are particularly interesting for deformation monitoring applications because of their low cost and the potential to use multiple systems concurrently [9]. Those systems are designed to distinguish objects ahead of a vehicle. Usually, they have a wide fieldof-view (almost 180 • ) with fine range and azimuth resolutions but generally coarse elevation resolution. A single instrument can derive a 2D map of LOS displacements by using interferometry. Combining acquisitions of three or more instruments allows the derivation of 3D displacement vectors.
In this work, we investigate how to determine 3D displacement vectors out of LOS displacement vectors. The algorithms can of be applied to line-of-sight measurements from various measuring instruments. Due to the previously mentioned advantages, we are focusing on MIMO-SAR systems, and we tested and evaluated the algorithms on simulated and real MIMO-SAR acquisitions. To this end, we deploy three MIMO-SAR systems for an experimental setup with a point scatterer in the form of a corner cube mounted on a motorised translation stage as well as a concrete sample. We derive the 3D displacement vectors and compare them to ground truth data.
After surveying the state-of-the-art, we describe in Section 2 the processing workflow and the adjustment algorithm to derive 3D displacement vectors from LOS displacement vectors. In Sections 3 and 4 we test, validate, and discuss the algorithm on simulated and real radar acquisitions and we conclude in Section 5.

State-of-the-art
Deriving real 3D displacement measurements based on radar interferometry requires utilising at least three radar systems simultaneously from different locations. This can be done by using three or more active (i.e. sending and receiving) radar systems [10,11] or by using bi-static configurations with one active and multiple passive (i.e. receiving) radar systems [12]. It is also possible to employ a single [13] or a combination of ground-based and satellite-based sensors [14] for mapping slowly moving objects at different locations and points in time.
Other approaches derive pseudo-3D displacements by using one or two radar sensors while taking geophysical or mathematical assumptions into account. For example: -A bridge is expected to deform mainly in the direction of gravity. A single sensor can be used to derive vertical displacements based on the interferometric phase, and the line-of-sight with its relation to the direction of gravity [9]. -An open pit mine exhibiting sliding or toppling motion.
The interferometric phase measurements of two sensors can be combined to derive pseudo-3D displacements by utilising a weighting function for the height values [15]. -A object moving between two acquisitions does change not only the interferometric phase but also the amplitude. This spectral diversity between two co-registered acquisitions can be exploited and allows deriving the orthogonal displacements (i.e. up-down and left-right) [8,16]. However, this comes with a severe drawback in terms of accuracy as the accuracy depends now on the angular and range resolution and not anymore on the wavelength of the emitted electromagnetic wave.
All those approaches did not model any uncertainties and weighting of the input data to improve the reliability of the derived 3D displacement vectors. Furthermore and to the authors' knowledge, only single-time acquisitions have been used to derive 3D displacements. In this paper, we present an approach where time series were used to quantify the uncertainties of line-of-sight displacement measurement for each observed object or bin. We implemented those variances in the stochastic model, and afterwards, we carried out parametric and non-parametric least squares (LSQ) adjustments based on temporal or spatial correlation.

Methods
We give an overview of the output of a radar acquisition with some relevant properties and definitions in Section 2.
1. An overview of the processing workflow is then given in Section 2.2. Afterwards, we describe the preprocessing of the data in Section 2.3, followed by the description of the transformation of line-of-sight displacement vectors to the final 3D displacement vectors in Section 2.4.

Data characteristics
The basic output of a radar acquisition is a single look complex image (SLC). The dimensions of this complex-valued matrix correspond to the number of resolution cells in range and azimuth (Figure 1(b)). The complex number of a cell B k,l represents the amplitude A S and phase S of the radar signal scattered back by all the scatterers S within the bin (Figure 1(a)). The bin size depends on the resolution Δr and Δ in range and azimuth direction (Figure 1(c)), respectively. Following and extending [17,18], we define the polar coordinate system of the MIMO-SAR system as follows: -X R denotes the cross-range direction in parallel to the linear antenna array; -Y R denotes the along-range direction in parallel to the antenna boresight; -withZ R being orthogonal to X R and Y R ; -the range R is the distance between the origin of all antenna pairs and the scatterer S; -the unit vector LOS encodes the direction from the origin to S; -the azimuth is the deflection of LOS from Y R in the X R Y R -plane; and the elevation is the deflection of LOS from the X R Y R -plane.
It shall be noted that the azimuth in the context of linear SAR is not equal to the geometric azimuth but is defined as with N VA being the number of unique and evenly-distributed synthesised antenna positions along the cross-range direction (Figure 1(c)). The output provided by the manufacturer for the MIMO-SAR system is then in Cartesian coordinates as A simple radar system with one fixed receiving antenna (RXA) can only derive the range component R from the received signal scattered back from a scatterer S. Using multiple antennas, as is the case for a Figure 1: Radar acquisition of a scatterer S: (a) measurement principle of an FMCW radar sensor, (b) SLC matrix with the axes oriented in range and azimuth; (c) topview and (d) cross-section of the polar coordinate system of a MIMO-SAR system and the relationship to the radar Cartesian coordinate system. MIMO-SAR system, it is also possible to compute the angle of arrival. If the antennas are arranged linearly, the angle of arrival in the azimuth direction can be detected but not the angle of arrival in the elevation direction (Figure 1(c) and (d)). Getting the relative location in 3D coordinates of the scatterer S observed by a linear MIMO-SAR can therefore be established by either (a) having a 3D model acquired by other means and projecting the bins to the surface [19] or (b) moving the radar sensor along the axis Z R [20]. Determining a 3D displacement vector would require having at least three known LOS displacement vectors in one common frame. As each radar system uses its own, local polar coordinate system a common global coordinate system has to be defined and implemented ( Figure 2).

Workflow and data preprocessing
We propose a four-step processing workflow to acquire 3D displacement vectors from SLCs. An overview can be seen in Figure 3. The steps are described in detail in their respective sections. As input data, a time series of SLC images and a point cloud are required with information regarding their relative location and orientation towards each other ( Figure 2) [18]. A preprocessing has to be carried out to transform each dataset to one common coordinate and temporal (i.e. temporal shift of timestamps) system and create the interferograms containing the displacements d LOS in line-of-sight. The interferograms are then matched with the point cloud, and the line-of-sight displacement vectors are calculated. In a final step least squares adjustments are applied to determine the 3D displacement vectors d 3D representing the actual displacements of the corresponding scatterers.
As a first step, the SLC images of each radar instrument have to be converted to interferograms by applying on all bins of the image. Here, t contains the phase information at time t, is the wavelength of the emitted electromagnetic wave (Figure 1(a)) and k is an index variable (Figure 1(b)). The resulting estimate d k LOS for the line-of-sight displacement only coincides with the actual line-ofsight displacement if the absolute value is smaller than ∕4. For larger movements, phase unwrapping has to be performed (e.g. [21]).
In a second step, we analysed the time series of interferograms by calculating for each bin B k,l three different values: (a) The coherence COH after [22] as a measure of noise in the spatial neighbourhood, (b) the amplitude stability index ASI after [23,24] as a measure of noise in the temporal neighbourhood, with ASI being related to the wellknown amplitude dispersion index (ADI) [25] by ASI = 1 − ADI, and (c) the maximum displacement MRD as a measure to detect outliers. We defined thresholds for those measures after empirically analysing the images with the aim of keeping persistent scatterers. Bins with a coherence COH < 0.8, an amplitude stability index ASI < 0.2, and a maximum displacement MRD > 50 mm are omitted. Optionally, we applied further filtering based on time constraints (i.e. within a given time frame) and area of interest (i.e. within a given range and azimuth).

Coregistration and geometrical projections
The point cloud and interferograms need to be in one common coordinate system to perform a successful matching. In parallel to the previous steps, the Cartesian coordinates C G PC of the point cloud will first be transformed to the Cartesian coordinate system of the radar sensor thereby yielding C R PC and afterwards to the polar coordinate system of each radar sensor. For this purpose the centre coordinates

RI
] ⊺ and orientation ( rotation around X G , rotation around Y G , and rotation around Z G ) of the radar sensor has to be known. In our case study, this information has been extracted from a point cloud acquired by a laser scanner [18]. The transformation can be applied by using homogeneous coordinates on where M , M , M , and T are the (homogeneous versions of the) Tait-Bryan rotation matrices and the translation vector, respectively. For potential future work, we already implemented three rotations even though only two would be sufficient for linear array geocoding. The result is a mapping from the matrix C G PC onto the matrix C R PC containing the Cartesian coordinates of the point cloud in the system of the radar instrument.
These Cartesian coordinates can then be converted to polar coordinates The interferograms and point cloud are now in the same coordinate system, and the actual matching can be done. The number of points in the point cloud can be extensive and consists of many points not being in the field of view (FOV) of the radar instrument. Therefore, a first rough point cloud filtering is applied where points are kept if they fulfil the conditions The lower and upper bound (range), as well as the left and right bound (azimuth) of each bin B k,l , are defined by the range and azimuth resolution. Therefore, each remaining point of the point cloud can be compared to those bounds. If a point S is within the limits of a bin B k,l then the line-of-sight displacement d LOS is assigned to this point. The line-of-sight defined by the unit vector can now be calculated by applying subtraction followed by a scaling with i and j being the indices of the matrix with the constraints i = j and ∀i ∈ {1, … , n}, to receive the unit vectors Finally, for each radar instrument, a list of matched points exists. Those lists will be merged by only keeping the points matched in all radar instruments.

Estimation of 3D displacements
The following section will explain the mathematical background of the algorithms. We start with the observation equation for 3d displacement vectors to line-of-sight observations in Section 2.4.1 followed by a detailed description of the parametric and non-parametric least squares adjustments in Section 2.4.2 and 2.4.3, respectively. The first one requires pre-knowledge of the function describing the data, while the latter one requires a function describing the structure of the correlations of the data.

Observation equations:
Assuming a scatterer S moved between observations carried out at times t 1 and t 2 (see Figure 4) then the displacement can be described as a vector where d X , d Y , and d Z are the coordinate components of the displacements in X, Y, and Z, respectively. A radar instrument k can derive through interferometry, the displacement in line-of-sight d LOS,k . This is the projection of d 3D onto the (unit) vector e LOS,k which describes the location of the scatterer S with respect to the radar instrument k. The relation can be expressed by Reformulated and expressed in matrix notation this can be written as (14) and requires at least three independent observations d LOS to solve for d 3D . Using more than the minimum required observations will lead to an over-determined equation system. A least squares (LSQ) adjustment [26,27] can be beneficial. The stochastic model for this adjustment problem can be formulated as l + = Ax (15) with l ∈ ℝ n being the vector of observations, x ∈ ℝ 3 the vector of unknown parameters, A is the design matrix and are the deviations assumed to be an i.i.d Gaussian random vector. The goal would then be to find x such that min which would correspond to x being the maximum likelihood estimator for the parameters [28, p. 150]. The solution for Eq. (15) in combination with a weighting matrix P to incorporate knowledge of the variance of observations would then be [28, p. 139] Applying this equation would provide the best solution for each scatterer S not accounting for temporal or spatial neighbourhood relations.

Parametric LSQ adjustment:
Assuming n instr instruments acquired n time observations of a scatterer S over a period of time. Applying Eq. (17) with Eq. (14) would give independent displacement vectors d 3D with coordinate components in X, Y, and Z for each instrument and point in time. The same counts for all observed n point scatterers. Apparent, high-frequency movement attributable to noise (e.g. instrumental noise, atmospheric variation) would be visible in the resulting displacement vectors. The aim would be to define a function with n para parameters that can describe the observations l such that the resulting vectors are smooth in the temporal or spatial domain. The parametric function estimation requires as an input a function matching the expected behaviour of the data itself. We model x as a linear superposition of basis functions g m leading to the 3D displacement vectors x being expressible as where G is a design matrix, g m the mth basis function or column of G, the parameter vector, and x the 3D displacement vector. The 3D displacement vector has the shape of with n f referring to either n time (i.e. temporal LSQ adjustment) or n point (i.e. spatial LSQ adjustment). The parameter vector consists of n para parameters with where n para = in case of temporal and n para = in case of spatial adjustment with n deg being the degree of the parametric function. The design matrix describes the transformation and is built as with in case of a polynomial function of degree n deg for an input f k with , spatial LSQ (23) where, in the spatial context, we denote by the term (f k ) n the three . As this choice of basis functions does not contain multivariate polynomials, the model only permits non-linearities that are separate in the individual space directions. Depending on the expected behaviour of the displacement field, the inclusion of more cross-terms might be appropriate. The projection of the 3D displacement vectors onto the line-of-sight vectors to get the line-of-sight displacement vectors l for each observation and instrument is made in a second step. The linear equation Eq. (18) will be extended to with l containing all the observations and the design matrix Q having a block-diagonal shape with E consisting of the unit vectors of the scatterer S to each radar instrument I. E has already been introduced in Eq. (11) but for the readers' convenience we define the design more precisely as The aim would be to find 1 to n para such that min‖Qx − l‖ 2 .
The solution comes out of Eq. (17) with the weighting matrix P having a block-diagonal shape and p a submatrix of P consisting of the squared inverse of the standard deviations derived from the observations, with Weighting is necessary to balance the different observations according to their reliability. For MIMO-SAR systems, the observation's noise depends on various factors (e.g. strength of reflected signal) and can depend on the target's location and orientation with respect to the radar instrument. We determined the empirical standard deviation s for each radar instrument i and the respective scatterer S for a period of no displacement before the actual movement was initiated. The inverse of the weighting matrix P is denoted by . It is the covariance matrix quantifying uncertainties of the observations.

Non-parametric LSQ adjustment:
The non-parametric vector field estimation [29, p. 159ff.] does not require as an input a specific function matching the data itself but a function describing the structure of the correlations. We start with the assumption that a linear function maps the 3D displacement vectors in x to the line-of-sight displacement vectors l. The latter is equal to the one introduced in Eq. (25). The vector x is rearranged compared to Eq. (19) and has the form of The rearrangement has been done to simplify the design matrix F which has the shape of with E X , E Y , and E Z being the unit vectors as introduced in Eq. (27). Describing the correlation of the data is done with squared exponential functions. They are typically used to model smooth stochastic processes [30, p. 83-84] and are typically used for interpolation and estimation in geospatial context and machine learning. They are separated for each coordinate component and defined as with q ∈ {x, y, z}. s q describes the assumed variances and determines the size of plausible deformations, whereas m j quantifies the expected correlation length, thereby determining the smoothness of the estimator. In other words, s q is a scaling factor for the correlation. Large s q results in high correlations, while s q ≪ 1 results in low correlations.  The quantity f is defined as in Eq. (23). For each component, a matrix of size (n f × n f ) is created and filled. A block-diagonal covariance matrix K is built as The aim is to find the 3D displacement vectors such that the following condition is minimised Applying this equation would provide the best solution for a smooth temporal or spatial deformation.

Numerical model
We implemented the above parametric and non-parametric LSQ adjustment in Matlab and provided them, and other related scripts via GitHub [31]. We simulated radar acquisitions with various deformations to evaluate and compare the different approaches.
First, a planar 3D point cloud with a regular grid of 31 by 31 points (9 by 9 m) was initialised. The point cloud did not move over a period of 50 observations before initiating -tilt, -rotation, -areal bending (ceiling), -linear bending (bridge), -random correlated deformation, -step-wise deformation, or -oscillating deformation with damping with a maximum displacement of 5 mm, followed by a static period of another 50 observations, see   15 m orthogonal to the planar point cloud. We simulated five instruments because we wanted to test the algorithm on an over-determined system. We also randomly generated the radar locations once and then used the same locations in all numerical simulations to suppress the influence of the radar locations on the results and thus simplify the comparison of the results. A random noise of = 25 μm was added to the line-of-sight displacements to simulate real measurements comparable to the results reported in [9,17].

Quality of estimations
Examples of the outcomes for the approaches can be seen in Figure 7. It depicts line-of-sight displacements of one point as seen from one of the five simulated radar instruments. The first 50 observations are without any displacements followed by a predefined movement for 200 observations followed by another 50 observations without changing displacements. Visually looking at those samples it can be seen that the cubic function utilised in the temporal cubic parametric adjustment (TCP) can only describe continuous linear movements, like the ones shown for the rotating or tilting cases (Figure 7(a) and (b)), well. The cubic function for the spatial parametric adjustment (SCP) works quite well except for random correlated deformations where it deviates from the ground truth. The temporal (TNP) and spatial (SNP) non-parametric adjustments work visually well with TNP slightly over-smoothed at times of changes of relative displacements.
These visual impressions can be supported numerically by deducting the adjusted line-of-sight displacement from the ground truth and calculating the root mean square errors (RMSE) of these differences. This has been done for all simulated radar observations and is summarised in Table 1. The TCP solutions have an RMSE of ≥32 μm, which is larger than the simulated noise of 25 μm. It shows that TCP degraded the observations and cannot describe the deformations well for these examples. The TNP adjustment performed well except for the step-wise and oscillating movements, where the smoothing of the interpolated curve was too strong. The adjustment could have been improved by decreasing the smoothness of the estimator (see Eq. (35)) but this was not done to allow a comparison of the methods. The spatial adjustments performed very well, except for random-correlated deformations. Overall it can be summarised that spatial parametric adjustments would work well if a parametric function could describe the deformation. However, non-parametric adjustments are simpler to use as they don't require prior knowledge of the underlying deformation behaviour and can improve the observations in most cases.

Experimental device and setup
We conducted our experimental investigation using three Texas Instruments TIDEP-01012 MIMO-SAR systems. We refer for detailed specifications to previous publications [9,17]. The range resolution depends on the chosen sweep bandwidth of the emitted chirp. The range resolution was 4 cm using the settings reported in Table 2 for the experiment. Three instruments were set up in a large hall next to each other, each controlled by a different laptop. The radar instruments were mounted on tripods at different heights above ground and faced a corner cube at about 12.5 m at boresight (Figure 8(a)-(c)). Additionally, a planar concrete sample of 50 × 50 cm (Figure 8, S) was put up left of the corner cube (Figure 8, CC). The corner cube was mounted on a motorised translation stage (ThorLabs MTS50/M-Z8) and moved at steps of 0.5 mm. The sample was moved manually with a rotation of 40 mrad from its initial position, while the radar instruments acquired their measurements.
A 3d model of the experimental set-up is required to project the line-of-sight acquisitions. This model was  acquired using a laser scanner (Leica RTC360). A local network was established using a total station (Leica TS60) to combine the 3d model of the environment and the SLCs acquired by the three different radar instruments. The axis Y G was defined by the direction of a wall in the hall (see Figure 8(a)); the axis Z G was defined to be vertical; and the axis X G was defined to be orthogonal to Y G and Z G in a righthanded coordinate system (see Figure 2). A spherical prism was mounted with the corner cube onto the translation stage, and the movements were tracked with the total station to derive the ground truth 3d displacement vector. Five BOTA8 targets [32] were mounted on the wall and pillars, and their centre coordinates were measured reflectorless with the total station. The targets were used to register the 3d model acquired with the laser scanner. The registered laser scanning point cloud was used to derive the locations and orientations of the radar instruments, the corner cube, and the concrete sample. Figure 9 (left column) shows the amplitude images for each of the instruments for acquisitions from the locations as described in the previous section in Figure 8. Radar instruments A and B show similar amplitudes with strong reflectors corresponding to pillars, concrete sample, tripods, and corner cube. Floor and ceiling reflect weakly instead. Instrument C shows higher and noisier amplitude values overall due to being an older version of the TIDEP-01012 (Revision C vs. Revision E). In the second column the coherence images are given for the first and last SLC acquired within an acquisition period of about 2 min and a neighbourhood 3 × 3 bins. It can be seen that the corner cube (CC) shows a high coherence in all three images. The concrete sample S also has high coherence for instruments A and B but lower coherence for instrument C. Strong reflectors increase the effect of azimuthal sidelobes [33] and are visible as circular areas (e.g. Figure 9). This can make it difficult to separate objects due to overlapping signals. For this experiment, we took care that no overlapping of strong reflectors occurred and that the signals were always separated by at least 0.5 m in azimuth and range direction. An excerpt of the observed line-of-sight displacements for the step-wise moving corner cube (CC) can be seen in Figure 10. The three time series were temporally coregistered at time T. The stability of data storage depends on the specific instruments with one of the instruments having a data loss of approximately 7%. This difference causes a miss-alignment of the data the further away from the point of temporal co-registration the data are visualised. To validate the algorithm presented in this work, we filtered the data. We only kept acquisitions where no change of displacements have been observed for all three instruments (i.e. flat areas).

Estimation quality: corner cube
A corner cube was moved in a predefined direction, and the direction of movement is visualised as a bold black line in Figure 11. For the interpretation of the ground truth, it has to be considered that for the given acquisition geometry (i.e. location of the total station with respect to the corner cube) and instrument specification (i.e. accuracy of distance and angle measurement) a 3D accuracy of 0.6 mm can be expected.
The 3D displacement vectors derived from the LSQ adjustment can be seen in Figure 11(a) as red arrows. Even though the corner cube only corresponds to measurements in very few radar bins, our algorithm produces a displacement field assigning one separate deformation vector to every point of the laser scanning point cloud in the extent of the corner cube. It is obvious that the vectors are not parallel as they should be for a rigid body. The same is true for the temporal adjustments based on TCP (b) and TNP (c). The advantage of spatial adjustments is clearly visible with all vectors being parallel for SCP (d) and SNP (e), the latter being closer to the ground truth. Calculating the absolute values from the 3D displacement vectors as defined in Eq. (11) gives the actual displacement. In Figure 12 (left column), we show the displacements for the time series of the corner cube. The step-wise movements of 0.5 mm can clearly be seen for the TNP, SCP, and SNP adjusted vectors. The TCP function did not represent the data well enough, and the result is a smoothed time series as already shown in Table 1 and Figure 7. The programmed step-sizes of the motorised translation stages were taken as ground truth. These values were deducted from the absolute displacements, and the results can be seen in the right column of Figure 12 and SCP (h) adjustments seem to perform similarly well with deviations from the ground truth in the range of approx. −0.05 to +0.2 mm. The visually best performing algorithm is the SNP (j) adjustment with a single peak deviation of about +0.07 mm and otherwise being between 0 and +0.02 mm.
These statements are underlined by Table 3 showing the RMSE for each series. Most of the values coincide well with the simulated ones shown in Table 1. The 27 μm for the real case scenario with SCP is larger compared to the 2 μm for the simulated scenario. Looking at the respective Figure 12(h) indicates that the larger error occurs when the corner cube has been moved, with a larger impact at the beginning and end of the time series. Nevertheless, it is still a factor of 2 better than the LSQ. Finally, the SNP (j) adjustment resulted in an RMSE of 6 μm indicating that SNP works very well for step-wise movements of a corner cube.

Estimation quality: planar concrete sample
The planar concrete sample S stood on the ground and was manually rotated for approximately 40 mrad or 10 mm at the edges in the clockwise direction. The line-of-sight displacements for one of the radar instruments can be seen in Figure 13. The first 20 s of the time series in (a) do not show any deformations for any of the observed points on the sample. Then the rotation is performed within about 5 s and afterwards, the points are again in a stable state with displacements within −10 and some +10 mm. That a rotation occurred can be seen when looking at Figure 13(b) where the LOS displacements from (a) have been projected to the radar instruments' line-of-sight. We processed the data the same way as we did for the point scatterer, but we could not get areal 3D displacements.  All three radar instruments have been calibrated using the procedure outlined in the Texas Instruments' mmWave Studio software [34]. While comparing the amplitude images with geodetic measurements, we observed a range and azimuth shift in the order of magnitude of 12-20 cm and  0 to 1 • respectively. The shifts are not constant for all instruments and relative locations. For a point scatterer like a corner cube, it was possible to spatially co-register the acquisitions with the amplitude image such that the correct radar bin was projected to the correct laser scanning point. However, applying the same corrections as derived from the corner cube signal to the concrete sample failed. Experiments carried out outside of the framework of this paper indicated non-constant range and azimuth shifts within the field of view of the radar sensor. Unfortunately, the concrete sample has a weaker more dispersed amplitude image and it was impossible to estimate the corrections and to map all the observed displacements such that they coincide for all the instruments at all the points. Solving this for areal targets requires a more sophisticated calibration procedure. This goes beyond the scope of this paper and has to be addressed in future work.

Conclusions
Deriving 3D displacements from line-of-sight displacements is beneficial for the understanding of the actually occurring deformations in many monitoring applications. In this paper, we describe a set of algorithms to derive 3D displacements from line-of-sight displacement measurements acquired by simulated and real MIMO-SAR sensors. The displacement measurements are least squares adjusted with spatial or temporal conditions based on parametric and non-parametric functions. The resulting 3D displacement vectors are improved compared to a simple adjustment without applied conditions. The numerical simulation involved typical displacement patterns as they could occur in geomonitoring or structural monitoring applications (e.g. tilting, bending, oscillating, etc.). We showed that the simulated line-of-sight displacements measurements with a simulated noise of 25 μm could be improved when using spatial cubic parametric, temporal or spatial non-parametric functions in most cases and get standard deviations of 2-9 μm. The temporal adjustment did not work well for step-wise or oscillating movements with RMSE of 24 and 43 μm, respectively.
The spatial adjustments worked well for all cases except for random correlated displacement patterns where the results were degraded with an RMSE of 114 and 454 μm, respectively.
The experiment with a corner cube moving step-wise showed that the algorithm works with real data and that the results coincide with the simulated data. The radar instrument could also measure the displacements occurring on non-point scatterers like a planar concrete sample of 50 by 50 cm but the instruments need more sophisticated spatial calibrations for mapping multiple instruments on a common point cloud and combine the measurements. We also observed a temporal miss-alignment due to data acquisition gaps varying for each instrument. Deriving a calibration procedure and mitigating the impact of the temporal missalignment are out of the scope of this paper and have to be addressed in future work.
Overall it can be concluded that the proposed algorithms work well and improve the results in general compared to a simple least square adjustment. Especially the non-parametric adjustments are favourable compared to traditional, parametric adjustments since prior knowledge of the expected displacement is not necessary. We also provide MATLAB scripts of the algorithms via GitHub such that they are publicly accessible and can be used by other researchers and potential users.
Future work should investigate the combination of spatial and temporal adjustment and include further constraints (e.g. allowing only vertical/horizontal displacements). The algorithms should also be tested and evaluated on other types of line-of-sight sensors, displacement patterns, and areal deformation measurements to further prove the general applicability.