Cross-modal biometric fusion intelligent traffic recognition system combined with real-time data operation

3.1 Cross-modal re-identification method for hyperspherical manifold embedding

For the cross-modal vehicle re-identification problem, learning the cross-modal vehicle image feature representation with the embedded identity information is the primary problem. In the traditional vehicle re-identification problem, many methods have used embedded representations based on identity information as deep features of vehicle images. In cross-modal vehicle re-identification methods, the embedded representation based on identity information is often used as an important constraint to extract cross-modal vehicle image features [9]. For example, in the deep zero-padding method, the objective function of the deep neural network is the embedded representation based on the identity information. The overall framework of the embedded representation based on identity information is shown in Figure 1.

Figure 1

Embedding representation based on identity information.

A set of samples is often used as a batch to optimize the model during network training, resulting in the following formula:

When the fully connected layer without a bias term is used as the classification layer to calculate this probability value, the formula is converted as follows:

The different modulus values of the vehicle and the angle between the two vectors are shown in Figure 2. From this, it is more intuitive to know which category the vector is classified into, depending on the modulo value of the weight vector of this category and the size of the included angle.

Figure 2

Spatial relationship between two cross-entropy losses. (a) Angle 1 between the vehicle modulus and the vector, (b) Angle 2 between the vehicle modulo value and the vector.

According to the aforementioned representation, the vector of the classification layer can be modified to obtain the following formula:

After another correction, the weight vector is re-calculated into the following formula:

Then finally, the following formula is obtained:

It can be observed from these formulas that, for the feature of any vehicle, its classification result is only related to the size of the included angle of the weight of a certain category of the feature. Its spatial relationship is shown in Figure 3.

Figure 3

Schematic diagram of hypersphere manifold embedding.

If it is assumed that a batch of input different modalities contains all samples M, the formula can be obtained:

Thus, the divergence formula is obtained as follows:

From the hyperspherical flow loss and divergence formulas, the calculation formula can be obtained as follows:

3.2 Cross-entropy loss function

Cross-entropy loss function is the most commonly used loss function in image classification problems, and it is also a loss function currently used in cross-modal traffic re-identification. When using the cross-drop loss function to solve the image classification problem, generally after the image feature extraction, a fully connected layer will be connected according to the number of categories M of the image, and the input of the fully connected layer is the dimension of the given image feature vector. The output is the number of categories M of the image. As shown in formula (10), the obtained result represents the probability that the image belongs to this category [10].

The cross entropy loss function actually calculates the deviation between the classification probability output by the network and the expected classification probability. The following formula is obtained:

After the constraints, the following formula is obtained:

It can be seen that the ordering of the loss function, so that the formula is:

Then, by measuring the material, the constraint formula between the modes can be obtained as follows:

Combined with the idea of loss function, the bidirectional loss between modes can be formulated as follows:

However, for the entire network, its objective function is expressed as follows:

If the two functions are directly calculated by the cosine of the included angle, for two features, its distance formula is given as follows:

Finally, the formula can be obtained:

3.3 Neural network model to build cross-modal traffic re-identification model

Traffic re-identification, as the problem of finding and matching images with the same ID as the query image from vehicle images, is often abstracted as an image classification problem. The classification problem generally refers to the application of vehicle ID attributes for classification training [11]. In the image classification problem, a very important point is how to convert the two-dimensional information of the image into one-dimensional feature vectors that are relatively independent of each other. Before the convolutional neural network convolutional neural networks (CNN) was widely used, we usually used algorithms such as feature point matching, translation, rotation, histogram of oriented gridients, and local binary patterns (LBP) to find the similarity of two images [12]. Traditional manual feature extraction algorithms face the problems of complex manual design, poor robustness, and slow image matching. In this article, CNN-based deep neural network is used for feature extraction. Currently, the more common image feature extraction networks mainly include AlexNet, VGG, GoogLeNet, and ResNet. The corresponding network structure parameters are presented in Table 1.

From the basic principles of deep neural networks, generally, the deeper the network, the better the ability to extract image features. The number of convolutional layers in ResNet is significantly more than the other three feature extraction networks. Therefore, this article gives priority to using the ResNet network for feature extraction. As shown in Figure 4, after a large number of experiments, the accuracy of the test set and training set decreases simultaneously after the number of network layers is deepened. So the problem is not due to overfitting.

Figure 4

Training and test errors for 20-layer and 56-layer networks.

There are already batch normalization included. Methods such as group normalization alleviate the problems of gradient disappearance and gradient nonconvergence to a certain extent, but the convergence effect of the deep network is still relatively poor [13]. Some scholars have proposed that for deep neural networks, the worst result of each layer of the network is that it does not work, which is equivalent to passing the input directly to the output. In this way, the deep neural network degenerates into a shallow neural network, and there will be no network degradation and gradient nonconvergence problems. For this, identity maps can be proposed to construct residual blocks, as shown in Figure 5.

Figure 5

Schematic diagram of the ResNet residual block structure.

The standard residual block structure of ResNet34 and ResNet50 is shown in Figure 6. The improvement of the latter relative to the former is mainly through the use of 1 * 1 convolution kernels for dimension increase and dimension reduction, which reduces the amount of parameters of the convolution layer.

Figure 6

Basic residual block and improved residual block structure diagram.

4.1 Traffic identification combined with real-time data operation

4.2 Cross-modal traffic re-identification image classification experiment

From the aforementioned algorithms, in Table 2, we present the classification results of different algorithms. The final classification result of the JointBoostI2C algorithm is better than that of the logic data center (LDC) and JointBoost algorithms [15]. It can be seen that while designing an appropriate weak classifier, the combination of weak classifiers should also be considered, which is beneficial to improve the classification effect.

We compare the average classification performance of images with national broadband neural networks (NBNN) and LI2C using the method proposed in this section in 15 scenes and Caltech101 image database. In both databases, the average classification accuracy was compared for each of the 15 categories of images in 15 scenes and the 6 categories of images in Caltech101, as shown in Figure 7.

Figure 7

Comparison of the classification results of three methods for each type of image in the Scene-15 and Caltech101 image libraries: (a) Scene-15 and (b) Calthech-101.

As can be seen from Figure 7, for which images are easily distinguishable, the method used in this article is similar to the other two methods such as Suburb and Office, of Scene15, and Carside and Face of Caltech101. For those images that are difficult to distinguish, the classification performance of the method proposed in this section is higher than the other two methods such as Scene15’s living room, store, Caltech101 bufferfly, and bear [16].

Table 3 presents the average classification performance of the three methods. Overall, the average performance of the image classification used in this article has a large improvement over the other two methods. In this article, the training and image classification of the JointBoost I2C classifier takes 381, 4 and 74, and 1.5 s for 15 types of images in Scene 15 and 6 types of images in Caltech 101, respectively.

The recognition results of the method in this chapter on the Caltech 101 dataset are presented in Table 4. The position (i, j) in the table represents the number of images that the ith image is classified as the j-type image, and the value of the cell on the diagonal in the table represents the number of correctly classified images. The larger the value, the better the classification effect [17]. It can be seen from the table that the method in this article can effectively re-identify multiple types of vehicles, and its average recognition rate is more than 80%.

A cross-modal vehicle feature registration method based on double-aligned feature embedding, which uses a fusion loss function as the constraint of the network. Therefore, this article first conducts two ablation experiments on the RegDB database for the three loss functions. It proves the validity of the three functions and the necessity of combining them [18]. Figure 8 presents the quantitative experimental results. The results of the ablation experiments show that the intraclass distribution loss function can improve the Rank1 recognition rate and mAP value by 6–7 percentage points based on the baseline method. This shows that it improves the modal invariance feature by reducing the distribution difference between different modal images of the same vehicle. It can effectively deal with the imbalance of feature information caused by modal changes.

Figure 8

Ablation experiment results. (a) First ablation experiment (b) second ablation experiment.

When using infrared images for retrieval on the RegDB dataset, the experimental results are shown in Figure 9. Compared to the task of retrieving infrared images using visible light images, the results of all models degrade when retrieving infrared images. Especially the ZeroPadding method has the most obvious drop. This is because the method uses only one deep network and cross-drop loss to simultaneously extract features from different modalities. Neither the spatial misalignment of cross-modal vehicle images nor the modal misalignment are effectively addressed. Therefore, the feature robustness is poor, and the effect drops significantly in the infrared-visible light retrieval task [19].

Figure 9

Experimental results under different retrieval directions. (a) The first mode and (b) the second mode.

4.3 Acquisition frame rate test of cross-modal traffic recognition acquisition module

The test of the acquisition frame rate of this module considers the test parameters from two aspects: (1) the choice of light and (2) the choice of time unit.

1. Light selection: In the same day, selecting the noon time period with better light and the dusk time period with poor light for testing.

2. Time unit selection: To avoid errors as much as possible, the tests were carried out at 10, 30, 60, 120, and 600 s. The test results are presented in Table 5.

The results presented in Table 5 show that after the data collection at different time units to obtain the average, there is a gap of about 2 frames between the average frame rates of better light and poor light. This shows that light has a certain influence on the acquisition frame rate of image recognition. Good light will be more conducive to the operation of the system [20]. It also shows that the reason for the intensity of light is also an important factor affecting the identification of the traffic system. It is important to treat the light factor as a distraction in the research process.

Based on the self-built car logo image library, this article chooses a feature extraction method with rotation invariance and scale invariance for car logo recognition. It is also known as improved scale-invariant feature transform (SIFT) feature extraction [21]. Because the vehicle recognition is affected by the natural environment, rain, and snow weather when capturing images, it is necessary to remove the interference items from the images. At the same time, the image capture is carried out on the wireless client. It requires the image to have a certain degree of adaptability to its own rotation and scaling, and the improved SIFT feature satisfies these performance requirements [22]. The experimental results are shown in Figure 10. To make the experiment more scientific and accurate, it is divided into two kinds of experiments: simple car logo and complex car logo.

Figure 10

Different feature vector extraction for simple and complex car logos. (a) Extraction of different feature vectors for simple car logos and (b) extraction of different feature vectors for complex vehicle logos.

Through the aforementioned experimental results, it can be found that when the feature extraction is performed on several simple car logo images, the correct recognition rate of the car logo by various feature extraction methods can basically reach more than 85%. In the feature extraction of Mazda, Honda, Lexus, Citroen, Chevrolet, and Mercedes-Benz with relatively simple structures, the advantages of SIFT are not obvious. But the overall performance is still better than several other feature vector extraction methods [23]. But in the identification of complex structure vehicle logo, the efficiency is obviously better than other several feature extraction methods. During the experiment, relatively complex car logos such as Rolls Royce, Maserati, Koenigsegg, Saab, Aston Martin, and Jaguar were selected for the comparison of experimental results. It can be seen that although the final recognition rate of various types of feature extraction methods has reached more than 80%, the recognition accuracy of complex vehicle logos is lower than that of simple vehicle logos to a certain extent. But the drop rate of the SIFT feature extraction method is the smallest. Therefore, the recognition effect of SIFT is also the best [24].