Building element recognition with MTL-AINet considering view perspectives

2.1 Projections from multiple-view perspectives

Generally, both the color and texture are used for superpixel segmentation. A building surface has rich color and texture semantics. Building texture refers to the similar structures presented by the color and material of the building surface, and the color and texture can characterize the 2D features of different objects on the building surface, such as walls, windows, and doors [31,32]. For example, in Figure 2(a), the color and texture features of the building facade, windows, and roofs differ significantly, and the building detail elements such as windows, facades, and roofs can be accurately identified. However, as shown in Figure 2(b), the color and texture of the building facade are not consistent with those of the facade, and it is difficult to accurately identify the building elements using only the color and texture. The dilapidated building facade in Figure 2(c) also shows an inconsistency between the 2D color and texture and the 3D feature of the facade. In conclusion, although 2D color and texture semantics can be important for building detail recognition, it is difficult to achieve an accurate recognition of complex building details using them.

Figure 2

Different building color and texture semantics: (a) distinct differences in color and texture features, (b) interference with color and texture features, and (c) color and texture of old building surfaces.

It is difficult to achieve an accurate recognition of complex building details using only 2D color and texture semantics. Adding 3D features can overcome the limitations of inaccurate 2D features and increase the recognition accuracy. Figure 3 shows the geometric schematics and projection interpretation of multiple features for a building. The color feature is displayed in Figure 3(a). As shown in Figure 3(b), n 1 and n 2 , respectively, denote the normal vectors for the roof and the facade of a building. The geometric relationship about the orientation and roll angle features is described in Figure 3(c). The parameter θ in Figure 3(c) represents the roll angle, i.e., the included angle between the normal vector of a plane (e.g., roofs and facades of a building in Figure 3) and the horizontal plane. n 1 ′ represents the projection of the normal vector n 1 in the horizontal plane, and the direction is represented by the intersection angle between n 1 ′ and the true north direction. In this study, the normal vectors of the point cloud model are calculated based on the planar-fitting method. The combined use of direction, roll angle, color, and texture features can enable an accurate, quick, and automatic characterization of the detailed elements of buildings. Figure 4(a–d) shows the schematic diagrams of projection for building facades under multiple-view perspectives. Each building facade is filling with a color.

Figure 3

Geometric interpretation of multiple features for building elements: (a) a building, (b) normal vectors of the roof and the facade of building, and (c) the orientation and roll angle of the building roof.

Figure 4

The schematic diagrams of projection of building facades under various view perspectives: (a) three view perspectives, (b) four view perspectives, (c) five view perspectives, and (d) six view perspectives.

The point cloud obtained from the sequence image was used to calculate the direction and roll angle semantics of the building; however, the buildings in different scenes had different facade orientations and structural postures. To establish the correlation mapping between 2D and 3D features while considering the overall shape of the building, this study combines the facade orientation of an actual building with the design of multi-angle projection, divides it into different perspectives, and selects multiple perspectives of the building for projection to establish the mapping relationship between the direction and roll angle of different building perspectives and the color and texture semantics (Figure 4(d)), which provides the basic data for multi-task learning.

2.2 Multi-task learning and segmentation based on MTL-AINet

AINet adopts an encoder–decoder architecture to acquire deeper features and output feature maps with superpixel embeddings as the input image sensory field increases in the encoding stage, which is fed into the decoding stage to generate the pixel–superpixel association maps. Meanwhile, the superpixel and pixel embeddings generated in the encoding and decoding stages, respectively, realize the direct interaction between the pixel and its field through the AI module, i.e., the superpixel is embedded around pixel p so that the network can capture the association between p and its neighboring grids and realize the pixel–superpixel association, which is more in line with the goal of superpixel segmentation. A matrix Q of h × w × 9 is finally obtained as the relationship between p and the surrounding nine superpixels [33], i.e., the probability that p belongs to the surrounding nine superpixels. The central information of the superpixel c s = ( u s , l s ) is calculated using the association matrix, and the image is reconstructed based on the superpixel features using the association between the pixel and the superpixel. The specific calculation is as follows:

where f(p) is the feature of pixel p, q s ( p ) is the association of pixel p with superpixel s in Q , N p is the number of superpixels around the pixel, u s represents the feature information, and l s represents the position information.

To make the superpixels fit more closely to the edges of the object, a boundary perception loss was constructed, which was applied to a pixel-by-pixel embedding to improve the boundary accuracy. It includes two main components: the first makes pixels of the same category closer to the mean, i.e., closer to each other, and the second increases the variability between pixels belonging to different categories. The overall concept of the cross-entropy (CE) function was used. Finally, the loss function was constructed based on the CE of the semantic labels and position vectors, L2 reconstruction loss, and boundary-perceiving loss using the following equations:

where l s ( p ) is the truth label, l s ′ ( p ) is the semantic label of the estimated association matrix Q reconstruction, B is the sampling block, l B is the boundary perception loss, and α and β are the two weighting factors.

Because the AINet superpixel segmentation network is only applicable to 2D color texture features of images, it is difficult to handle 3D feature data, such as roll angle and direction. Therefore, a multi-task learning segmentation model, MTL-AINet, was designed based on texture, color, direction, and roll angle semantics. AINet was used as the basis function, and the mapping association map obtained from the projection of texture and color, direction, and roll angle features was input into the model for multi-task association learning. Therefore, multi-task learning considering the texture, color, direction, and roll angle features could be performed. The specific steps are as follows: for building facade information segmentation in different scenes, the dense point cloud model of the building is first reconstructed using the image sequence of the building according to the MVS algorithm [34], calculating the direction and roll angle semantics, dividing it into different viewpoints based on the overall structure and form to obtain the multi-view, and then projecting it onto the 2D image to obtain the pixel-level association of direction and roll angle maps and color and texture maps to be used as the input of the model. The fused pixel–superpixel association matrix (shown in Figure 5) is obtained via multi-task-integrated learning through the AINet network.

Figure 5

Structure of the MTL-AINet network.

In the proposed algorithm, two sets of pixel embeddings corresponding to the semantic features of color and direction and roll angle, respectively, denoted as E rgb ∈ R rgb H × W × D and E pose ∈ R pose H × W × D , are obtained through the deep neural network. The embeddings of pixel p, involving the color and direction and roll angle semantic features, can be denoted as e p rgb ∈ R rgb D and e p pose ∈ R pose D , respectively. Let S be the sampling interval. The input image is compressed through multiple convolution and maximum pooling operations to generate two grid cell feature maps with multidimensional semantics, i.e., M rgb ∈ R rgb h × w × D ′ and M pose ∈ R pose h × w × D ′ , where h = H / S , and w = W / S .

The feature maps M rgb ∈ R rgb h × w × D ′ and M pose ∈ R pose h × w × D ′ are transformed into new feature maps M ˆ rgb ∈ R rgb H × W × D and M ˆ pose ∈ R pose H × W × D , respectively, through a 3 × 3 convolution. Thus, the embedding of the nine grid cells around pixel p is defined using equation (5), which directly associates a pixel with a semantic block:

where SP = SP rgb SP pose , and m ˆ | ∙ | = m ˆ | ∙ | rgb m ˆ | ∙ | pose .

The association graph can be predicted through a 3 × 3 convolution and equation (6):

The proposed method uses the same loss function as the AINet superpixel segmentation method, including the three losses: CE loss, pixel reconstruction loss, and boundary-perceiving loss (as in equation (3)).

A new set of embedded pixels E ′ = E rgb ′ E pose ′ can be computed using equations (5) and (6), which directly reflect the pixel–superpixel semantic association regarding the color and roll angle. In the proposed method, AINet is used as the base learner, and the multi-feature semantic association projection maps, including the color and texture feature map and the direction and the roll angle map, are used as multiple inputs for MTL-AINet. The soft association maps Q rgb and Q pose are obtained through MTL-AINet, which are, respectively, used to describe the probabilities each pixel belonging to its adjacent superpixels. Furthermore, Q rgb and Q pose are integrated to calculate the fusion association map Q Fusion by equation (7), which is the output considering multi-feature semantics of the MTL-AINet. Subsequently, a set of semantic blocks is extracted according to the soft association map Q Fusion :

where δ 1 and δ 2 denote the weight factors of association maps Q rgb and Q pose , respectively, satisfying δ 1 + δ 2 = 1 .

Figure 6 shows the detailed computation process of the soft association graph Q Fusion . In Figure 6, ρ 1 ∼ ρ 9 represents a probability attribution distribution on color and texture of pixel p ( i , j ) , and similarly, μ 1 ∼ μ 9 represents a probability attribution distribution of pixel p on the direction and roll angle. This pixel attribution reflects the similarity between pixel p ( i , j ) and its nine adjacent grid cells in 2D and 3D features, respectively; τ 1 – τ 9 represents the fusion probability distribution of pixel p ( i , j ) on multi-feature semantics. The maximum of the nine multi-feature probabilities for each pixel yields a mapping of the label, which corresponds to the result of the semantic block segmentation. In summary, by calculating the feature maps of color and texture, and direction and roll angle, and using the AINET basis function to calculate their soft association mappings Q rgb and Q pose , respectively, the optimal association matrix Q Fusion is obtained using a multi-task learning strategy, and the value with the highest probability is considered the final homogeneous semantic block segmentation result.

Figure 6

Detailed computation procedure of the soft association graph Q Fusion for pixel p ( i , j ) .

2.3 Building information extraction based on semantic clustering

On the basis of obtaining the initial segmentation results of each building facade, the lowest level of homogeneous regions is obtained. For recognition tasks with different requirements, a region adjacency graph based on 2D color and texture features and 3D direction and roll angle features is constructed. Region merging is considered as an approximation problem of the image, and the final clustering results are obtained using a stepwise iterative optimization method based on the nearest neighbor graph for the nodes with the smallest edge weights for region merging [35].

The texture characteristics of the image are measured using the joint probability distribution histogram of LBP (local binary pattern) and LC (local contrast) [36] by combining the structure and intensity of image texture. The G-statistic method is used to perform the analysis. Let x and y, respectively, represent a random variable set, and let g denote the probability density function, then the G-statistic formula is as follows [37]:

where G ( x , y ) represents the value of the pixel at position ( x , y ) on the image. g i represents the number of pixels (pixel frequency) with gray level i in the image, and t represents the number of gray levels in the image. The results are highly capable of texture description and differentiation.

Buildings are geographic entities with certain shape information, and in the pixel merging process, it is often difficult to appropriately distinguish the contour edges of buildings using only texture information [38]; therefore, the geometric orientation constraint is adopted to obtain the object with a good edge fit. During merging, neighboring regions with longer common edges are prioritized to obtain more compact objects. The influence of the common edge length is introduced based on equation (9), and the geometric directional heterogeneity property is defined as H, where l xy i is the common edge length of neighboring regions x and y, and i is the influence factor of the common edge. When i = 0, l xy i = 1, i.e., the common edge has no influence on the regional heterogeneity, whereas when i ≠ 0, the longer the common edge, the smaller the heterogeneity [39].

In the region adjacency graph- and the nearest neighbor graph-based area merging, the problem of minimizing the adjacent-area approximation error is gradually transformed into that of calculating the area with the smallest merging cost. The merging cost is the weight of the edge of an adjacent area with a common edge and is defined as:

where Q ( x , y ) denotes the combined generational values of regions x and y; S x and S y denote the areas of x and y, respectively; and H ( x , y ) denotes the heterogeneous properties of the two regions.

Using the strategy of building element extraction based on the aforementioned clustering strategy, we combined feature maps with hierarchical clustering to extract building elements in different scenes.

3.1 Segmentation extraction for building facades with similar texture

The Florentine Cathedral multi-view sequence image dataset (Figure 7(a)) was used in the experiment. A total of 105 images were selected with a resolution of 1,296 × 1,936 and a focal length of 29 mm. The sparse point cloud and camera parameters were first obtained using the structure from motion (SfM) technique, and the dense point cloud was then obtained using the MVS method. In total, 39.01 million points were obtained in the experiment, and the dense reconstruction results are shown in Figure 7(b).

Figure 7

Building image models with similar texture features: (a) single image, (b) dense point cloud.

Based on the geometric orientation of the building, the color and texture, and direction and roll angle projections were selected from the left, front, and right views of the building, and directional feature maps were obtained. Figure 8(a–c) shows the color and texture projections of the left, front, and right views of the building, respectively. Figure 8(d–f) shows the direction and roll angle projections of the left, front, and right views of the building, respectively. The figures show that the color and texture and the direction and roll angle projections express different details of the building facades.

Figure 8

Different view images of buildings and the corresponding directional semantic features: (a and d) left-view, (b and e) front-view, (c and f) right-view.

Figures 9–11, respectively, show the segmentation results of the four segmentation methods under different viewing angles. From regions marked with the two red rectangles, it can be seen that the proposed MTL-AINet method performs better on those regions with similar color and texture but different façade orientations than the other three traditional methods; different facades were not be segmented successfully with the three traditional methods, while the proposed method can effectively distinguish different facades with similar textures, which facilitates the clustering task.

Figure 9

Left-view semantic segmentation result maps: (a) SLIC, (b) LSC, (c) AINet, and (d) MTL-AINet.

Figure 10

Front-view semantic segmentation result maps: (a) SLIC, (b) LSC, (c) AINet, and (d) MTL-AINet.

Figure 11

Right-view semantic segmentation result maps: (a) SLIC, (b) LSC, (c) AINet, and (d) MTL-AINet.

The clustering results of each facade shown in Figures 12–14 indicate that the AINet method can easily identify the roofs as the same facade, which cannot be separated by the AINet method due to the overly similar color texture. The proposed MTL-AINet method can effectively utilize the directional features of each facade, distinguish different facades with similar textures (such as the roofs and edges of each facade), and can accurately extract the detailed structure of each building. Thus, the details of the facade structure can be extracted accurately. Therefore, the MTL-AINet method is robust for buildings with similar color and texture features but large facade differences.

Figure 12

Left-view clustering result maps: (a) AINet and (b) MTL-AINet.

Figure 13

Front-view clustering result maps: (a) AINet and (b) MTLAINet.

Figure 14

Right-view clustering result maps: (a) AINet and (b) MTL-AINet.

The experimental results show that for buildings with different facades but very similar texture features, the segmentation method relying only on single-texture semantics is prone to misclassification, and it is difficult to extract sufficient detail information. The segmentation method based on color and texture and direction and roll angle semantics considerably overcomes this limitation and can efficiently extract fine details of each building facade.

3.2 Segmentation extraction for complex buildings with occlusions and shadows

The multi-view sequence image dataset of Örebro Castle (Figure 15a) was used in this experiment. A total of 136 images were selected with an image resolution of 1,936 × 1,296 and a focal length of 29 mm. The sparse point cloud and camera parameters were first obtained through SfM, and the dense point cloud was then obtained through the MVS method. A total of 77.72 million points were obtained in the experiment, and the dense point cloud reconstruction result is shown in Figure 15(b).

Figure 15

Complex building image model with occlusion and shadow: (a) single image and (b) dense point cloud reconstruction result.

The building was divided into four views (front, back, right, and left) according to its architectural characteristics, and a directional semantic feature map was constructed, as shown in Figure 16. In this experiment, the four views of the building were selected for the color and texture and direction and roll angle projections, and directional feature maps were obtained. Figure 16(a–d) shows the color and texture projections of the front, right, back, and left views of the building, respectively. Figure 16(e–h) shows the direction and roll angle projections of the front, right, back, and left views of the building, respectively. From the figures, it is evident that the texture and color and the direction and roll angle projection maps can express different details of the building facades. In particular, the direction and roll angle projection maps can significantly reduce the influence of shadows.

Figure 16

Different view images of buildings and the corresponding directional semantic features: (a) the front view image, (b) the right view image, (c) the back view image, (d) the left view image, (e) the front view semantic features, (f) the right view semantic features, (g) the back view semantic features, and (h) the left view semantic features.

Because all the facades of the building are affected by shadows and occlusions to different degrees, the three traditional superpixel segmentation methods based on texture semantics can only segment the areas with obvious color–texture features and does not sufficiently distinguish between the shadowed parts and occlusions, such as the boundaries of the column-shaped buildings in Figures 17(a–c) and 18(a–c) and the roof and bottom fences in Figures 19(a–c) and 20(a–c). The proposed MTL-AINet method compensates for these shortcomings by weakening the influence of shadows and occlusions and improving the segmentation accuracy; for example, the top protrusion in Figure 17(d) is not mixed with the roof, the shadows and edges in Figures 18(d) and 20(d) are not divided separately, and the bottom fence in Figure 19(d) is not divided with the background but into a homogeneous surface.

Figure 17

Front-view semantic segmentation result maps: (a) SLIC, (b) LSC, (c) AINet, and (d) MTL-AINet.

Figure 18

Right-view semantic segmentation result maps: (a) SLIC, (b) LSC, (c) AINet, and (d) MTL-AINet.

Figure 19

Back-view semantic segmentation result maps: (a) SLIC, (b) LSC, (c) AINet, (d) MTL-AINet.

Figure 20

Left-view semantic segmentation result maps: (a) SLIC, (b) LSC, (c) AINet, (d) MTL-AINet.

From Figures 21(a) and 22(a), it is evident that the clustering result maps based on the superpixel segmentation results generated by AINet erroneously divides the shadows into a separate facade. Moreover, because the shaded parts present similar texture features with other facades, they can be easily mixed into the same facade, as shown in Figures 23(a) and 24(a), which considerably impacts the extraction of building elements. In contrast, the proposed MTL-AINet method can effectively avoid effects of occlusions and shadows, and improve the accuracy of facade structure extraction, as shown in Figures 21(b), 22(b), 23(b) and 24(b), where the large areas of shadows and shaded areas are efficiently distinguished and extracted. This significantly weakens the influence of shadows and facilitates subsequent tasks.

Figure 21

Right-view clustering result maps: (a) AINet and (b) MTL-AINet.

Figure 22

Left-view clustering result maps: (a) AINet, (b) MTL-AINet.

Figure 23

Front-view clustering result maps: (a) AINet and (b) MTL-AINet.

Figure 24

Back-view clustering result maps: (a) AINet, (b) MTL-AINet.

The experimental results show that because of shadows and occlusions, building facades cannot be segmented accurately based only on texture. However, a segmentation method based on color and texture, and direction and roll angle semantics can efficiently extract detailed elements of building facades.

3.3 Segmentation extraction for old buildings with inconsistent textures

The Martenstroget multi-view sequence image dataset (Figure 25(a)) was used in this experiment. Twelve images with an image resolution of 2,592 × 3,872 and a focal length of 30 mm were selected. The sparse point cloud and camera parameters were first obtained through SfM, and the dense point cloud was then obtained through the MVS method. A total of 23.2 million points were obtained in the experiment, and the dense point cloud reconstruction results are shown in Figure 25(b).

Figure 25

Image sequence and point cloud model of an old building: (a) single image, (b) dense point cloud reconstruction result.

The building was divided into three views according to its architectural characteristics and a directional semantic feature map was constructed, as shown in Figure 26. In this experiment, three views of the building – front, right, and left – were selected for the color and texture and direction and roll angle projections, and directional feature maps were obtained. Figure 26(a–c) shows the color and texture projections of the front, right, and left views of the building, respectively. Figure 26(d–f) shows the direction and roll angle projections of the front, right, and left views of the building, respectively. From the figures, it is evident that the texture and color and direction and roll angle projection maps can reveal protrusions and depressions.

Figure 26

Different view images of buildings and the corresponding directional semantic features: (a) the left view image, (b) the front view image, (c) the right view image, (d) the left view semantic features, (e) the front view semantic features, and (f) the right view semantic features.

The color texture of the old building surface interferes with the recognition of building detail elements, and it is obvious from Figures 27(a–c) and 28(a–c) that the three traditional superpixel segmentation methods based on texture semantics mistakenly segment walls that have similar color textures but do not belong to the same facade into the same blocks. From Figures 28(a–c) and 29(a–c), it can be seen that the three traditional methods will mistakenly divide the regions belonging to the same facade but with dissimilar color textures into different categories. As seen from Figures 27(d), 28(d) and 29(d), the proposed MTL-AINet method effectively overcomes this limitation by dividing the regions that belong to the same facade but are not similar in color and texture into the same similar semantic facets, and distinguishing the regions that do not belong to the same facade but are more similar in color and texture, which provides a better basic data for the subsequent task of hierarchical clustering.

Figure 27

Left-view semantic segmentation result maps: (a) SLIC, (b) LSC, (c) AINet, and (d) MTL-AINet.

Figure 28

Right-view semantic segmentation result maps: (a) SLIC, (b) LSC, (c) AINet, and (d) MTL-AINet.

Figure 29

Front-view semantic segmentation result maps: (a) SLIC, (b) LSC, (c) AINet, and (d) MTL-AINet.

From the clustering results shown in Figures 30 and 31(a), it can be concluded that the traditional clustering method can only extract areas with more evident color textures and performs more false extractions. In contrast, the proposed method incorporates direction and roll angle features and can achieve a finer extraction of detailed information on old buildings, such as the protrusions and depressions, as shown in Figures 30 and 31(b).

Figure 30

Left-view clustering result maps: (a) AINet and (b) MTL-AINet.

Figure 31

Right-view clustering result maps: (a) AINet and (b) MTLAINet.

Figure 28 shows the segmentation and clustering results of the front view of the building, wherein it is evident that the surface textures of the old buildings have large differences. Figure 28(a–c) shows the texture-based segmentation results of the three traditional methods, and it can be seen that segment regions belonging to the same facade but with variations in brightness and colors were wrongly segmented into separate facets. Better segmentation results are obtained by the proposed MTL-AINet method because it considers the direction and roll angle semantics, as shown in Figure 28(d). From Figure 32(a) and (b), it is evident that the clustering results based on AINet superpixel segmentation still cannot distinguish the edge details of old buildings very well. However, the proposed MTL-AINet performs better because it considered both direction and roll angle constraints.

Figure 32

Front-view clustering result maps: (a) AINet and (b) MTL-AINet.

Thus, the proposed MTL-AINet method obtains a higher accuracy and more accurately extracts details in depressions and protrusions. The experimental results show that for building facades with inconsistent color and texture dilapidation and similar textures, but with geometric depressions on the facade, the MTL-AINet superpixel segmentation method based on color and texture and direction and roll angle semantics has better extraction accuracy.

3.4 Evaluation and discussion

To verify the reliability of the proposed method, the results of building facade segmentation obtained in the experiment were evaluated using the super-pixel evaluation metrics REC, UE, CO, ASA, and ICV, and compared with three traditional methods based on SLIC, LSC, and AINet. The quantitative evaluation of building facade segmentation results across different scenes is shown in Tables 1–3.

As indicated by the results presented in Tables 1–3, in the detail segmentation experiments of each building facade in different scenes and clustering for different recognition extraction tasks, the indicators based on the proposed MTL-AINet method have been greatly improved in comparison with the other three traditional methods. From the quantitative comparative evaluation results, it can be concluded that AINet and MTL-AINet have greatly improved the CO and under-segmentation of different building facades compared with traditional SLIC and LSC. Comparing with AINet, the average BR and UE rates of 11 building facades segmentation results from MTL-AINet were, respectively, improved by approximately 26 and 3%. Building facade segmentation results from MTL-AINet achieve the highest average REC rate and ASA. For MTL-AINet, as it considered multiple semantic features for superpixel segmentation, the ICV has a relatively moderate increase in comparison of AINet, but obviously better than SLIC and LSC methods. In conclusion, compared with the three traditional methods, the proposed MTL-AINet method has achieved better results, which reflects the superiority.

According to these experimental results, compared with the single semantic features that cannot adequately consider the detailed features of different facades of buildings, there are many problems of over- or under-segmentation. The MTL-AINet method proposed in this study integrated color and texture and direction and roll angle semantics of buildings, which can achieve an effective and accurate segmentation of target objects. In particular, for building objects with similar colors and textures but different directions and angles, building objects with severe shadow and occlusion effects, and old and new aging building objects with different colors and textures, the proposed method can accurately extract building detail elements with higher robustness.