Show Summary Details
More options …

# Open Physics

### formerly Central European Journal of Physics

Editor-in-Chief: Seidel, Sally

Managing Editor: Lesna-Szreter, Paulina

IMPACT FACTOR 2018: 1.005

CiteScore 2018: 1.01

SCImago Journal Rank (SJR) 2018: 0.237
Source Normalized Impact per Paper (SNIP) 2018: 0.541

ICV 2017: 162.45

Open Access
Online
ISSN
2391-5471
See all formats and pricing
More options …
Volume 15, Issue 1

# A novel fast target tracking method for UAV aerial image

Liu Jianfang
/ Zheng Hao
/ Gao Jingli
• Pingdingshan University, Pingdingshan, Henan 467000, China
• College of electrical engineering, Zhejiang university, Hangzhou, Zhejiang 310027, China
• Other articles by this author:
• De Gruyter OnlineGoogle Scholar
Published Online: 2017-06-16 | DOI: https://doi.org/10.1515/phys-2017-0046

## Abstract

Unmanned aerial vehicles (UAV) are able to achieve autonomous flight without drivers, and UAV has been a key tool to extract space data. Therefore, how to detect the trajectories of targets from UAV aerial image sequences is of great importance. Because local features are suitable to detect target tracking, we exploit scaleinvariant feature transform (SIFT) features to describe the interesting keypoints of targets. The main innovation of this paper is to utilize Multiple hypothesis tracking (MHT) algorithm to track an object (target) in a series of image sequences. Particularly, we develop a MHT framework based on a multidimensional assignment formulation and a sliding time window policy. To obtain target tracking from UAV aerial image sequences, three steps should be done, that is, 1) Breaking each track set into tracklet at a specific time, 2) Estimating the association cost of each track set, 3) Merging trajectory fragments to a longer one iteratively. Finally, we collect several UAV aerial image sequences with different target density to construct a dataset, and experimental results demonstrate the effectiveness of the proposed algorithm.

PACS: 07.05.Pj; 89.20.Ff

## 1 Introduction

Unmanned aerial vehicles (UAV), that are also called drones, can implement autonomous flight without drivers. Moreover, the flight status and routes of UAV are controlled by a wireless remote controller or computer programs [1]. UAV utilizes the air power to support its flight in the atmosphere, and it has many advantages, such as wide viewing angle, huge potential, and so on [2]. UAV, especially the small unmanned helicopter with ability of autonomous flying, is suitable to be used in the field of military, civil and scientific research [3]. The basic structure of UAV is made up of 6 modules: 1) the UAV body, 2) navigation system, 3) flight control system, 4) wireless information communication system, 5) ground station system and 6) the mission-application system [4].

As an important part of the aircraft, unmanned aerial vehicle (UAV) has been a crucial way to extract space data. In the application domain of military and civilian, UAV is regarded as an effective mode to solve many complex problems [5]. Particularly, in recent years, UAV has been successfully used for military purposes, with the outstanding performance in the Gulf War and the war in Afghanistan. Afterwards, UAV has been exploited in applications of science and business in the past decade [6]. Currently, various types of unmanned aerial vehicles are able to be utilized in microwave links, military surveillance, traffic management, scientific research, atmospheric monitoring, geographic mapping, forest fire prevention, emergency rescue and so on.

In this paper, we concentrate on the problem of fast target tracking for UAV aerial image sequences, and the aim of the image sequence processing is to process images extracted from the UAV vision system when flying across particular locations. As is well known that fast target tracking for UAV aerial image sequences is very complex, and the process needs to use low cost and high performance techniques [7]. Particularly, UAV system receives video frames obtained by a nadir camera, and then input texture features to the image sequence target tracking system. Furthermore, in real flights, information of rotating and scaling of the frames should be corrected to enhance the performance of the UAV onboard instruments navigation [8].

The main innovation of this paper lies in the following aspects:

1. We utilize multiple hypothesis tracking (MHT) algorithm to track an object (target) in a series of image sequences.

2. SIFT features are used to describe visual contents of UAV aerial images.

3. The MHT framework is designed, based on a multidimensional assignment formulation and a sliding time window scheme.

The paper is organized as follows. In the next section, we introduce the related works about visual object tracking. Section 3 illustrates Basic theory of the fast target tracking using UAV aerial image sequences. Section 4 proposes a novel fast target tracking algorithm based on multiple hypothesis tracking for UAV aerial image sequences. Afterwards, the experiments are conducted in section 5 to make performance evaluation, and this paper is concluded in section 6.

## 2 Related works

In recent years, visual object tracking has become an important research field in computer vision and image processing community. Meanwhile, many researchers have studied on target tracking methods which are suitable to be used in many different applications, and related works about target tracking are listed as follows.

Gaxiola et al. proposed a reliable target tracking algorithm based on dynamically adaptive correlation filtering, and the proposed algorithm is able to track targets with high accuracy without the offline training process. Particularly, this work chooses the target in advance, and then optimizes a composite correlation filter optimized for distortion tolerant pattern recognition [9].

Wu et al. proposed a multi-sensor multi-target tracking method, which regards the data association and registration separately. Main idea of this paper lies in that the probability hypothesis density filter is utilized to prevent the complex. In particular, this paper presents an augmented state Gaussian mixture filter with registration errors for multi-target tracking via Doppler radars [10].

Zhang et al. exploited the probability hypothesis of density filter to implement a suboptimal Bayesian multi-target filter, and the authors provided a new multiple target tracking algorithm. The main innovations of this paper lie in two aspects: 1) a novel weight redistribution method of targets is developed to revise the weights of closely spaced targets, and 2) a false alarm detection approach is given to reduce the disturbance of clutter [11].

He et al. proposed an infrared target tracking approach using a robust low-rank sparse representation which aims to search for the maximum-likelihood estimation solution of the residuals in the target tracking system. Finally, experimental results demonstrate the effectiveness of the proposed method [12].

Gostar et al. proposed a novel approach to implement a multi-Bernoulli-based multi-target tracking system. In particular, the proposed algorithm is able to solve the general multi-target tracking problem without any prior knowledge, and then utilizes a novel task-driven objective function [13].

Li et al. proposed a multi Bernoulli smoother to solve the multi-target tracking problem, and the proposed method is made up of forward filtering and backward smoothing. Particularly, the forward filtering is developed by a cardinality-balanced multi-target multi-Bernoulli filter, and the smoothed multi-target probability density is estimated through a multi-Bernoulli density [14].

Yi et al. proposed a novel structure of the hierarchical data association tracking method based on a branch partition. In particular, a dynamic viewpoint model and an iterative computation algorithm are proposed to track several targets. Extensive data experiments on several public benchmarks prove the proposed algorithm is effective [15].

Apart from the above studies, other methods and different environments of target tracking are listed as follows. Conte et al. studied on how to track underwater targets [16], Asmaa et al. introduced target tracking in wireless sensor networks [17]. Moreover, there are other studies about target tracking, such as underwater passive target tracking [18], target tracking with range-only information [19], target tracking using interactive multiple model [20], target tracking utilizing underwater wireless sensor networks [21], multi-target tracking with multi-bernoulli filter [22].

## 3.1 Key points description

As local features are effective to detect target tracking, we utilize SIFT features to describe the interesting keypoints of targets. SIFT algorithm is able to extract features which are invariant to scaling, orientation, affine transforms and illumination varying. For each keypoint, users compute a local feature descriptor using the local image gradient, which is transformed to estimate orientation invariance. In particular, SIFT feature vector is defined as follows. $V(Fi)=v1(Fi)⋯vni(Fi),loc1(Fi)⋯locni(Fi)$(1) where v1(Fi) refers to the feature vector of the keypoint Fi, and function loc1(Fi) is used to define the location of an image. Furthermore, to compute the local descriptor, the derivatives Ix1 and Ix2 of a particular region I(x) are calculated by the following two equations. $Ix1(x1,x2)=I(x1,x2+1)−I(x1,x2−1)$(2) $Ix2(x1,x2)=I(x1+1,x2)−I(x1−1,x2)$(3)

Thus, the magnitude and orientation of an image are computed for a given image region: $M(x1,x2)=Ix1(x1,x2)2,Ix2(x1,x2)2$(4) $θ(x1,x2)=tan−1Ix2(x1,x2)Ix1(x1,x2)$(5)

## 3.2 Multiple hypothesis tracking

Multiple hypothesis tracking (MHT) [2325] is designed to iteratively construct the association Θ* (k) which maximizes P {Θk+d | Zk+d}. For the kth frame, the former processing steps give the track set (denoted as Θ*k-1), and then the target tracks which belong to the d + 1th frame using the measurements from the track set (denoted as Zk:k+d). Structure of the multiple hypothesis tracking is given in Figure 1.

Figure 1

Structure of the multiple hypothesis tracking

Afterwards, given a track ${\theta }_{j}^{\ast k-1}\in {\mathit{\Theta }}^{\ast k-1}$, potential track set ${\mathit{\Gamma }}_{tj}^{k+1}$ is constructed, and ${\mathit{\Gamma }}_{ci}^{k+d}$ refers to the potential track set to construct the cluster ci. In the process of association choosing for the cluster ci, a subset of potential tracks ${\mathit{\Theta }}_{ci}^{\ast k+d}$ should be constructed, and the highest possibility of ${\mathit{\Theta }}_{ci}^{\ast k+d}\subset {\mathit{\Gamma }}_{ci}^{k+d}$ is equal to $L\left({\mathit{\Theta }}_{ci}^{k+d}\right)$. Next, the optimal association is obtained by integrating the association of each cluster: $Θ∗k+d=⋃iΘci∗k+d$(6)

The track set is made up of n elements Θl = {θj}, j ∈ {1, 2, ···, n}, and the likelihood of $ΨΘk+d=∏t∈1,2,⋯,k+dpZ0t⋅∏jpθjk+d,ztjk+d=Φ^Θk+d⋅ΦΘk+d$(7) where $p\left({\theta }_{j}^{k+d},{z}_{{t}_{j}}^{k+d}\right)$ denotes the joint probability of the associations which are chosen for track tj, and Φ (Θk+d), Φ̃ (Θk+d) means the product of track probability and measurements probability, respectively. Based on the above definition, a node is valid only when the following equation is satisfied: $LΘ≤ΦΘ⋃θj≤ΦΘ$(8)

## 4 The proposed algorithm for fast target tracking

In this section, we will discuss how to solve the task of target tracking by the multiple hypothesis tracking technology, which is used as a tracking framework in this paper. Particularly, we implement the MHT framework via a multidimensional assignment formulation and the sliding time window policy. The target tracking problem is illustrated in Figure 2, which aims to track an object (target) in a series of image sequences (frames).

Figure 2

Illustration of the target tracking problem

Afterwards, the flowchart of the target tracking system is given in Figure 3.

Figure 3

Flowchart of the target tracking system

From Figure 3, we can see that the key problem of the target tracking process is how to utilize the MHT algorithm to find target tracking with SIFT descriptors. MHT aims to maximize the aposteriori probability of track associations. Assume that the current time clock is T, and then the set of measurements of time t is represented as Z(t), i ∈ {1, 2, · · · T}, and the number of measurements at time t is Mk = |Z(t)|. Thus, the data association process is described as searching a dividing scheme of the set Z to a track set Tr, which maximizes the probability p (Tr |Z). $Tr~=arg⁡maxTrpTrZ=arg⁡maxTrnpZTrnTrn⋅pTrn$(9) where the symbol ZTrn refers to the image sequences which are allocated to the track set Trn.

Furthermore, we suppose that Z is the image sequences which have been allocated to the track set Tr in the time range [t0, t0 + T], and the track set Tr is maintained until time t0 + T −1. Afterwards, the MHT algorithm with sliding time window is illustrated as follows:

• Step 1: Deleting the image sequences from Z which have been allocated to the track set Tr

• Step 2: Constructing a multiple hypotheses tree with gating, and then allocating image sequences which are saved at T to the track set Tr

• Step 3: Clustering hypotheses into disjoint trees.

• Step 4: Utilizing the Greedy Randomized Adaptive Local Search Procedure algorithm, and then obtain the track set $\stackrel{~}{Tr}$

• Step 5: Track classification

1. If a track in the set $\stackrel{~}{Tr}$ is extended from tracks of Tr, classify it to the “continuing track class”.

2. If a track in the set $\stackrel{~}{Tr}$ is not overlapped with the track which belongs to a track in Tr, classify it to the “new track class”.

3. The rest tracks are classified to the “ending class”, which ends at time t0 + T −1.

Afterwards, given the track sets obtained from UAV aerial image sequences, to tackle the target tracking problem the following three steps should be done: 1) Breaking each track set into tracklet at a specific time, 2) Calculating the association cost of each track set, 3) Iteratively merging trajectory fragments to a longer one until no fragments can be merged any more.

## 5 Experiment

In this section, we test the proposed target tracking algorithm under different experimental settings. In particular, we collect UAV aerial image sequences with different target density to construct four datasets (denoted as D1, D2, D3, and D4). The target density is defined as the number of targets in a frame, and target densities of D1, D2, D3, and D4 are set to 1,2,4, and 6, respectively. In order to estimate the accuracy of a track Ti, the similarity between each object position xi on track Ti and the position xj on the actual trajectory Aj is computed by the Euclidean distance. Furthermore, we define the track distance as follows. $DTi,Aj=1ΔTi,Aj⋅∑t∈ΔTi,Ajxti−xtj$(10) With the above equation one is able to calculate the sum of all instances in the time index set Δ (Ti, Aj), and function D(·) is used to be an error measure for the difference between the observed data and the actual data. For an actual trajectory Aj, we define f (Aj) to minimize the total track distances between observed tracks and actual ones: $fAj^=arg⁡minfAj∑Ti∈fAjDTi,Ajs.t.(1)ITi,Tj=∅(2)Ti,Tj∈fAj$(11) where I (Ti, Tj) refers to the time index set of Ti and Tj.

Next, to test the quality of target tracking, we utilize the following metrics to make performance evaluation:

• Metric 1: Track completeness (TC) [26] $TC=∑j∑Ti∈fAj^ITi,Aj∑jAj$(12)

• Metric 2: Track accuracy (TA) [27] $TA=∑j∑Ti∈fAj^DTi,Aj∑jAj$(13)

• Metric 3: Track fragmentation (TF) $TF=∑jfAj^AjfAj^≠∅$(14)

• Metric 4: Ratio of the phantom track (PTR) $PTR=Ti∀Aj,Ti∉fAj^AjfAj^≠∅$(15)

Then, we test the performance of our proposed algorithm by the above four metrics. Particularly, for each of the scenarios, the dataset is made up of 100-frame UAV aerial image sequences. Moreover, to be fair, we utilize the same parameters for each experiment setting. Performance of the proposed algorithm under different level of target density is provided in Figure 4 to Figure 7.

Figure 4

Track completeness for different level of target density

Figure 5

Track accuracy for different level of target density

Figure 6

Track fragmentation for different level of target density

Figure 7

Ratio of the phantom track for different level of target density

It can be observed from Figure 4 to Figure 7 that the proposed algorithm allows achieving high quality of target tracking from UAV aerial image sequences.

On the other hand, in order to lower the phantom track ratio, we utilize a longer size of sliding time window. We want to estimate a suitable size of sliding time window to remove short track fragments, and then detect phantoms. In the following section, we test the performance of the proposed algorithm with different sliding window sizes (shown in Figure 811).

Figure 8

Track completeness for different sliding window size

Figure 9

Track accuracy for different sliding window size

Figure 10

Track fragmentation for different sliding window size

Figure 11

Ratio of the phantom track for different sliding window size

Integrating experimental results from Figure 811, we can see that the given four evaluation metrics almost decrease with the sliding window size increasing. The reason is that short true tracks are removed incorrectly when sliding window size increases. Therefore, considering the influence of trade-off, the size of sliding window for all the above evaluation metrics are selected from experimentation.

From all above experimental results, we can see that the proposed UAV aerial image sequences target tracking algorithm performs well for all four metrics. The reasons lie in that 1) MHT is able to maximize the aposteriori probability of track associations, and 2) In our proposed algorithm, trajectory fragments are iteratively merged to a longer one until no fragments can be merged any more.

## 6 Conclusion

In this paper, we focus on the problem of detecting the trajectories of targets from UAV aerial image sequences. We exploit multiple hypothesis tracking algorithm to track an object from image sequences, and then we develop a MHT framework based on a multidimensional assignment formulation and a sliding time window policy. In order to achieve the aim of this paper, three steps are included in the proposed algorithm: 1) Breaking track set, 2) Estimating the association cost, and 3) Merging trajectory fragments. In the end, experimental results prove that the proposed algorithm can rapidly track targets from UAV aerial image sequences.

In the future, we will try to utilize other performance evaluation metrics to test the performance of our proposed method, and we will also make a comparison of other target tracking methods.

## References

• [1]

Nishar A., Richards S., Breen D., Robertson J., Breen B., Thermal Infrared Imaging of Geothermal Environments and by an Unmanned Aerial Vehicle (UAV): A Case Study of the Wairakei - Tauhara Geothermal Field, Taupo, New Zealand, Renew Energ, 2016, 86, 1256-1264. Google Scholar

• [2]

Xu J., Cai C.X., Cai G.W., Zou Y., Robust H-Infinity Control for Miniature Unmanned Aerial Vehicles at Hover by the Finite Frequency Strategy, IET Control Theory A, 2016, 10, 190-200.

• [3]

Kim H., Hyun O.T., Kwon S., Simple Catalyst Bed Sizing of A NaBH4 Hydrogen Generator with Fast Startup for Small Unmanned Aerial Vehicles, Int J Hydrogen Energ, 2016, 41, 1018-1026.

• [4]

Goncalves J., Henriques R., Alves P., Rita S.S., Monteiro A.T., Lomba A., Marcos B., Honrado J., Evaluating an Unmanned Aerial Vehicle-Based Approach for Assessing Habitat Extent and Condition in Fine-Scale Early Successional Mountain Mosaics, Appl Veg Sci, 2016, 19, 132-146.

• [5]

Dong X.W., Zhou Y., Ren Z., Zhong Y.S., Time-Varying Formation Control for Unmanned Aerial Vehicles with Switching Interaction Topologies, Control Eng Pract, 2016, 46, 26-36.

• [6]

Martin P.G., Payton O.D., Fardoulis J.S., Richards D.A., Yamashiki Y., Scott T.B., Low Altitude Unmanned Aerial Vehicle for Characterising Remediation Effectiveness Following the FDNPP Accident, J Environ Radioactiv, 2016, 151, 58-63.

• [7]

Wang D.J., Chen L., Wu J., Novel In-flight Coarse Alignment of Low-cost Strapdown Inertial Navigation System for Unmanned Aerial Vehicle Applications, T Jpn Soc Aeronaut S, 2016, 59,10-17.

• [8]

Moreno A.I., Jarzabek A., Gonzalez M.A., Perales J.M., Optimizing Multidisciplinary Scaled Tests in Terrestrial Atmosphere for Extraterrestrial Unmanned Aerial Vehicle Missions, P I Mech Eng G-j Aer, 2016, 230, 77-89.

• [9]

Gaxiola L.N., Diaz-Ramirez V.H., Tapia J.J., Garcia-Martinez Pascuala, Target Tracking with Dynamically Adaptive Correlation, Opt Commun, 2016, 365, 140-149.

• [10]

Wu W.H., Jiang J., Liu W.J., Feng X., Gao L., Qin X., Augmented State GM-PHD Filter with Registration Errors for Multi-Target Tracking by Doppler Radars, Signal Process, 2016, 120, 117-128.

• [11]

Zhang H.Q., Ge H.W., Yang J.L., Yuan Y.H., A GM-PHD Algorithm for Multiple Target Tracking Based on False Alarm Detection with Irregular Window, Signal Process, 2016, 120, 537-552.

• [12]

He Y.J., Li M., Zhang J.L., Yao J.P., Infrared Target Tracking Based on Robust Low-Rank Sparse Learning, IEEE Geosci Remote S, 2016, 13, 232-236.

• [13]

Gostar A.K., Hoseinnezhad R., Alireza B.H., Multi-Bernoulli Sensor-Selection for Multi-Target Tracking with Unknown Clutter and Detection Profiles, Signal Process, 2016, 119, 28-42.

• [14]

Li D., Hou C.P., Yi D.Y., Multi-Bernoulli Smoother for Multi-target Tracking, Aerosp Sci Technol, 2016, 48, 234-245.

• [15]

Yi Y., Mo Z.W., Tan J.W., A Novel Hierarchical Data Association with Dynamic Viewpoint Model for Multiple Targets Tracking, J Vis Commun Image R, 2016, 34, 37-49.

• [16]

Conte G., Capua G.P., Scaradozzi D., Designing the NGC System of a Small ASV for Tracking Underwater Targets, Robot Auton Syst, 2016, 76, 46-57.

• [17]

Ez-Zaidi A., Rakrak S., A Comparative Study of Target Tracking Approaches in Wireless Sensor Networks, J Sensor, 2016, Article NO 3270659. Google Scholar

• [18]

Kumar D.R., Rao S.K., Raju K.P., Integrated Unscented Kalman Filter for Underwater Passive Target Tracking with Towed Array Measurements, Optik, 2016, 127, 2840-2847.

• [19]

Chaudhary G., Sinha A., Tripathy T., Borkar A., Conditions for Target Tracking with Range-Only Information, Robot Auton Syst, 2016, 75, 176-186.

• [20]

Vasuhi S., Vaidehi V., Target tracking using Interactive Multiple Model for Wireless Sensor Network, Inform Fusion, 2016, 27, 41-53.

• [21]

Zhang Q., Zhang C.J., Liu M.Q., Zhang S.L., Local Node Selection for Target Tracking based on Underwater Wireless Sensor Networks, Int J Syst Sci, 2015, 46, 2918-2927.

• [22]

Yuan C.S., Wang J., Lei P., Bi Y.X., Sun Z.S., Multi-Target Tracking Based on Multi-Bernoulli Filter with Amplitude for Unknown Clutter Rate, Sensors, 2015, 15, 30385-30402.

• [23]

Zhang H.Q., Ge H.W., Yang J.L., Adaptive Gaussian mixture probability hypothesis density for tracking multiple targets, OPTIK, 2016, 127, 3918-3924.

• [24]

Yang W., Fu Y.W., Li X., Joint detection, tracking and classification of multiple maneuvering targets based on the linear Gaussian jump Markov probability hypothesis density filter, Opt Eng, 2013, 52, AR 083106.

• [25]

Lim Y.C., Kim J.H., Lee C.H., Lee M., Stereo-Based Tracking-by-Multiple Hypotheses Framework for Multiple Vehicle Detection and Tracking, Int J Adv Robot Syst, 2013, 10, AR 293.

• [26]

María R., María L.G., Multiplier method and exact solutions for a density dependent reaction-diffusion equation, Appl Math Nonlin Sci, 2016, 1, 311-320. Google Scholar

• [27]

Francisco B., On problems of Topological Dynamics in non-autonomous discrete systems, Appl Math Nonlin Sci, 2016, 1, 391-404. Google Scholar

## About the article

Accepted: 2016-11-14

Published Online: 2017-06-16

Citation Information: Open Physics, Volume 15, Issue 1, Pages 420–426, ISSN (Online) 2391-5471,

Export Citation