Optimization of target acquisition and sorting for object - ﬁ nding multi - manipulator based on open MV vision

: To optimize the mechanical arm target capture and classi ﬁ cation of the open multiple - view ( MV ) visua - lization program, the open MV visualization program - ming and deep learning detection method combined with the di ﬀ erent capture strategies of robotic arm, a method to extend the research is proposed. For the pro - posed sorting robot ’ s multi - cargo grasping, the analysis required to detect a wide variety of goods in a storage environment that lacks color or structural features uni - formly. On the basis of SSD target detection method regression, the object ’ s 3D position information is recon - structed by default preselected cell selection. 3D coordinate accuracy of binocular navigation system was veri ﬁ ed as 8% when the target cargo location distance is more than 5 cm, and binoculars matching success rate is 89.7%. The success rate of Sorting and hoarding is increased from 6% to 85% by adding a change to the scoring points of the target products of unevenquality, with this we have achieved e ﬃ cient and accu - rate import.


Introduction
With the continuous breakthrough and innovation in multi-rotor unmanned aerial vehicles (UAV) technology, with its advantages of flexible movement, simple operation, stability, and reliability, it has played an increasingly important role in agricultural plant protection, military security, meteorological monitoring, disaster relief, and other fields. In recent years, UAVs have been used in a wide range of civil applications, including urban planning, environmental monitoring, agriculture, governance, utilities, and commercial uses. UAVs are less expensive to acquire and operate than manned aircraft, which are utilized in traditional photography [1]. They also create photographs with comparable or superior spatial resolution. The disadvantage is that a single flight's region coverage is limited. UAVs can (and often must) fly at extremely low altitudes (less than 200 m, mainly below clouds) and as frequently as resources (e.g., batteries) allow. UAVs can thus play an essential role in monitoring smallholder agricultural systems, where persistent cloud cover makes picture acquisition difficult for optical remote sensing systems [2,3].
UAV is mainly used by pilots to monitor the specified locations in the form of aerial photography, and it cannot actively interact with the surrounding environment, which has certain application limitations [4]. UAVrobotic arm structure, also known as flying robotic arm, has greatly expanded the application field of UAV. In addition, it can replace manual operations to reduce risks, improve working environment, and improve work efficiency [5]. The main research content is the autonomous navigation function of multi-rotor flight arm system based on open multiple-view (MV) visual dispatching system, and the target identification and grasping research based on navigation. The motion planning problem of humanoid robot arm grasping space object is a typical motion planning problem with multiple constraints in high-dimensional C-space, which is a challenging frontier research field. For the complex system of tightly coupled humanoid robot, the complex problem of motion planning in highdimensional space involved by humanoid robot can be decomposed into a series of sub-problems in low-dimensional space, the constraint coordination scheme is utilized and the subproblem is solved incrementally [6]. Automatic sorting robots will be responsible for handling, sorting, packaging, and other tasks, among which, the sorting operation is to intelligently pick, sort, and mix a variety of goods according to the order requirements. Diverse environments on the production floor, shortage of experienced workers, functional correctness, and reliability are just a few of the issues that are preventing smart industrial robots from being used more widely. The automation and deployment of industrial robot systems replace repetitive, meaningless jobs that offer a significant danger of injury, as well as low-skill work that formerly needed human intellect. To be precise, the robot must be capable of making order decisions, handling movable objects, and interacting with information. At this time, robots require a high level of comprehensive perception, decision-making, and execution technologies, which necessitates the use of a variety of robot control technologies such as visual recognition, autonomous judgment, and motion planning [7].
Zhang et al. constructed a target grabbing system for robotic arm based on Kinect, and used the depth difference method to extract the contour of desktop object to complete the target detection. RRT algorithm is used to plan the trajectory of the manipulator to achieve the target grasping, but this method cannot be effectively applied to the non-uniform objects. How to carry out trajectory planning under spatial constraints is also the key to ensure the work efficiency of the sorting robot [8]. Candelieri et al. proposed a trajectory planning method of robot joint space under multiple constraints based on trapezoidal velocity curve, it can give full play to the driving performance of the robot joint while making the movement as smooth as possible [9]. Yin et al. adopted the energy storage systems method to reduce the selection error of feature points, and optimized the binocular feature matching algorithm points with the improved RANSAC algorithm, the effectiveness of the method is proved by the indoor and outdoor robot self-positioning comparison experiments [10]. UAVs can be controlled remotely by a pilot at a ground control station or autonomously by following a pre-programmed flight plan. The history of these systems may be traced back to hobby and professional radio-controlled aircraft, as well as military applications, such as flying over hostile or otherwise dangerous area. UAVs are planes that do not have a pilot on board. UAVs are gaining popularity among scientists as innovative instruments for remote sensing [11]. Due to this unique properties of data that it can acquire, the UAV is able to fills the unfilled niche when compared to more traditional aircraft or satellite-based platforms. Because of its low operating altitude, it can generate data with ultra-high spatial resolution across relatively tiny geographical areas [12]. For topographic modeling and feature recognition, Zhao et al. [13] and Lin et al. [14] employed UAV-based LiDAR. Stefanik et al. [15] employed UAVs for stereo-image 3D landscape modeling. According to Rudol and Doherty [16] and Hinkley and Zajkowski [17], UAVs are assisting in the fast replacement of traditional aerial photography in the field of remote sensing by producing high spatial resolution aerial photographs of the earth's surface. UAV images may be used for a variety of purposes, including large-scale mapping, urban modeling, and flora structure mapping. UAVs can conduct effective surveys in disaster-prone or physically unreachable places, as well as can quickly assess the damage caused by landslides, floods, and earthquakes to enable relief efforts.
On the basis of the above planning technology, this study proposes research on open MV visual scheduling, hardware system, software system, and overall structure layout and function of the sorting robot by studying the operating requirements of the unmanned storage application scenario. The sorting motion path of the six degree of freedom (DOF) manipulator in the storage scenario is developed based on the operating requirements of goods sorting, the features of the grab surface of warehousing items are examined, and two end grab strategies, bent suction and side suction, are proposed.

Grasping and sorting by mechanical arm
Based on the D-H convention, the reference coordinate system, linkage coordinate system, and workspace of the right arm of the humanoid robot with seven DOF are set to q 1 = [q 1 , …, q 7 ] T , sin q i (s qi ) and cos q i (c qi ) are denoted as s i and c i , respectively, then the homogeneous matrix i−1 T(q i ) between coordinate system i and i − 1 can be expressed as Eq.
The forward kinematics equation of the humanoid robot's right arm can be described by the homogeneous coordinate transformation matrix of the position and pose of the end effector Σ 7 of the right arm relative to the reference coordinate system Σ 0 Eq. (2): The right arm of the robot described in Eq.
(2) has redundancy, and can be described by rotation around the center of the elbow through the wrist and the Σ 0 origin axis. Once the position of the elbow is determined, an analytical form of the inverse kinematics of the arm can be derived. According to the elbow circle of the right arm, the position 0 T4 (Eq. (3)) of the elbow joint of the right arm can be deduced and calculated.
The position of the end effector of the right arm relative to the right elbow joint is (4):

(4)
Thus, the 7-DOF inverse kinematics problem of the right arm is decomposed into Eqs. (3) and (4), which represent two sub-inverse kinematics problems with smaller dimensions. By multiplying both sides of Eq.
The equation formed by matrices (3,4) and other elements on both sides of Eq. (5) can be obtained: Multiply both sides of Eq.
By comparing the matrix elements on the left and right sides of Eq. (8), q 3 and q 4 can be obtained: , , According to Eq. (4): The equation formed by the corresponding elements of the matrices on both sides of Eq. (11) can be solved as follows: In the space and material of a collaborative robot arm, follow the search scenario of this theme. The positioning capability of the 3D coordinate of the depth camera and the binocular camera is more suitable for the requirements of the scene; however, during the automatic classification process, the autonomy of the camera is significantly limited by space, for example, when using eye-to-eye method to penetrate the target of the lower shelf, the mechanical arm can only try to avoid noise with the shelf captured by the party. Greatly limiting the working space of the manipulator, therefore, visual positioning system uses eye-to-hand method. The binocular structure was combined with the grabbing strategy on the fixed side of the sorting robot platform to achieve the collision free multi-target positioning and grabbing experiment [18].
In open MV machine vision applications, the relationship between a three-dimensional space point and its presence in a two-dimensional image is closely related to the camera imaging model, Figure 1. In the relationship between pixels coordinate system and the physical coordinate system of the imaging plane, the coordinates in the pixel coordinate system (O uv ) are (u, v), the origin O 0 of (O uv ) is generally selected at the upper left corner of the image; however, (u, v) does not represent the physical location of the pixel, instead, the physical position of the pixel is described in the imaging plane physical coordinate system (O xy ), the origin of (O xy ) is O 1 , the intersection point of the camera optical axis and the image plane, its X-axis and Y-axis are parallel to the U-axis and V-axis of the pixel coordinate system, respectively [19]. Set the coordinate of O 1 as (u 0 , v 0 ) in the pixel coordinate system, assuming that the actual physical dimensions of pixels on the imaging plane in the X and Y directions are dx and dy, then the corresponding relationship between pixel coordinates and imaging physical coordinates is: As shown in Figure 2, the coordinate p(x c , y c , z c ) of the object in the camera is described by the camera coordinate system O c , the origin of P is generally the optical center of the camera [20], and the z axis of the coordinate system is the optical axis of the camera. When the binocular camera observed a coordinate point p(x c , y c , z c ) of barium O 1 , coordinate point p(x c , y c , z c ) is represented differently in the two camera coordinates as shown in Figure 2. Figure 2 is the mapping between camera coordinate system O c and imaging plane physical coordinate system (O xy ), the relationship between the actual position of object O c and the question of (O c ) is: O w represents the world coordinate system, and the homogeneous transformation matrix can be used between it and O c . T c w represents the transformation of coordinates, assuming that the object's world coordinate is p w , the homogeneous coordinate is [X W , Y W , Z W , 1] T , the camera coordinate system is [X C , Y C , Z C ] T , then the relationship between the two coordinate systems can be expressed as follows: where R represents the rotation transformation matrix, t represents the translation transformation vector, and then, according to Eqs. (13)- (16), the relationship between pixel coordinates (u, v) and p w can be obtained as follows:  Formula (17) represents the position mapping relationship between the points in O c and the world coordinate system O w .
Open MV (machine vision) system is the process of object spatial position perception, due to the influence of illumination change, external noise, perspective distortion, and other factors, the processing of two-dimensional images inevitably involves the problem of visual maladaptation, that is, the characteristics of the corresponding points projected from the spatial positioning points of the target goods to the physical imaging plane of the binocular camera are different [21,22]. In the storage environment, the binocular visual recognition system of the sorting robot is faced with the unknown category and location of goods. For a feature point or piece of image located in an image, there may be multiple similar candidate matches in another image, so other information or constraints are required for further judgment. In order to determine a unique match, the matching accuracy of the object capture information between images will determine the final object spatial positioning accuracy [23,24]. The spatial representation of grasping target is reconstructed based on the parallax principle, which provides a reliable basis for the space motion planning of manipulator arm and the precise grasping operation of actuator. According to the information points of the target image, the matching strategy of polar line geometry is adopted to complete the three-dimensional matching of the capture information, which effectively guarantees the uniqueness of the capture point of the object, so as to realize the accuracy of the spatial positioning of the object.

Experimental analysis
Unmanned storage scenario has narrow aisles, many shelves, and dense goods. In the process of dynamic operation, get open MV 3D location information with the visual scheduling system. According to the position information of the goods, the manipulator can obtain the end attitude through the inverse solution of the analytical method, then plan specific subtasks or sequences of actions. Given the starting points, intermediate points, end points, and some necessary constraints, plan and describe the trajectory of the manipulator. The flexibility of the end actuator is used to adjust the grasping strategy to avoid collision between the manipulator and surrounding obstacles, stable and fast cargo grabbing can be realized. The control system diagram of the sorting robot is shown in Figure 3.
Select products with typical shapes in the unmanned storage scene, as shown in Figure 3, these goods are in lying or are in vertical state. This scheme simulates the human perspective and grasping mode, select the surface closest to the outside of the shelf when the goods are placed naturally at the grasping surface. The combination of steering gear and flexible sucker cleverly adds a 180°to the end of the manipulator arm. DOF and the suction angle can be set freely. For different shapes of goods, the manipulator end actuator adopts two grasping strategies, the first grasping strategy is called the forward suction position, and the second grasping strategy is called the side suction position. The increase in the end actuator extends the motion range of the manipulator arm, and the end of the manipulator arm can keep the end grasping state further deep into the shelf, the operation comfort of the manipulator and the flexibility of grasping goods are improved. Under the condition that the end posture of the manipulator remains the same, the first approach extends the extension length of the manipulator by 16 cm, the second side suction attitude extends the extension length of the manipulator by 19 cm. The increase in the end actuator extends the motion range of the manipulator arm, improves the operation comfort of the manipulator arm and the flexibility of grasping goods. The end actuator's flexibility is employed to alter the grasping strategies to minimize collisions between both the manipulator and surrounding obstructions, allowing for steady and quick cargo grabbing. The manipulator arm's end is skillfully turned 180°thanks to the combination of steering gear and flexible sucker. The suction angle may be  flexibly set due to the degree of freedom. The first approach extends under the constraint that the manipulator's ultimate position stays the same.

Experimental results
Aiming at two kinds of obstacles that interfere with the sorting operation of the sorting robot, the goal of trajectory planning of the manipulator is to realize each subtask in detail under these constraints, because the sorting robot works in the unmanned storage scenario, and the target goods are randomly distributed. Through the analysis of order requirements, it is determined that the operation requirements of the robot arm are mainly composed of sub-tasks: 1) Initial pose in non-working state INIT: After all sorting tasks are completed, the robot arm needs to return to its non-working initial posture, and it must ensure that all parts of the manipulator are not in the field of view of the visual system, so as not to affect the visual system identification and positioning. Before the robot arm is ready for the sorting operation, it needs to return from the current arbitrary posture to the non-working initial state.
2) The initial posture of the working state MOVE. After the binocular recognition system detects the ordered goods, the mechanical arm is in the initial position of working state, and it should be able to reach the upper target point or the lower target point of the shelf without collision. 3) Motion to target point REACH: The manipulator moves to the sorting target point and controls the end effector to carry out the cargo gripping operation.
Place the goods in LOOSE posture: After the manipulator completes the target grasping, the position of the cargo placement point should be determined by the cargo basket and binocular recognition structure device. After the goods are placed, the robot arm returns to the initial working state from the current position to prepare for the next target sorting. The cycle continues until all sorting operations are completed.
In the sorting process of the manipulator arm, the end actuator moves in a straight line in space. In order to prevent the end actuator of the manipulator arm from entering the area near the placing of goods, interfering with other goods or shelves due to wrist rotation, it is, therefore, necessary to move the end-actuator from its initial posture to a position parallel to the surface of the cargo in advance, a certain distance away from the grasping surface and 400 mm away from the grasping point of the current sporting goods, which is called the approach attitude. Then, the end actuator moves in a straight line from the approach attitude to the cargo grasping point. When the cargo is successfully drained, it returns to the approach point, and then rotates the waist joint. In order to prevent the collision between the manipulator body, the binocular recognition device and the basket device, the manipulator moves vertically above the basket to place the goods. Put (i = 0, 1…6) to the subscript of P are the path nodes passed by the end-effector. The sorting process is described as serial movements as shown in Table 1 by referring to the poses of these nodes.
It takes 17 s for the goods to be sorted. Considering the inertia of the manipulator, stay for 1 s and return to the initial attitude of the working state to prepare for the next cargo grab. The total sorting time of 100 target goods was 40 min, and the average sorting time of single target goods was 22 s, and the success rate of multi-target sorting was 89%. The experiment shows that: The path planning of the manipulator arm based on the joint blank point effectively reduces the interference between the manipulator arm and the shelf, binocular bracket, and basket in the sorting process. The operation efficiency and stability of the open MV visual dispatching and sorting system are ensured.

Conclusion
A robotic system that can identify and locate multiple targets in an unmanned storage environment and has the ability to execute, is developed, according to the characteristics of various targets and environments in the aerial storage scene. Multi-target identification and positioning classification robot was designed and verified by simulation and practice. Simulation and construction of a gallery-type unmanned storage scenario, analysis, and design of a sorting robot control module were performed to solve problems in the storage environment, such as narrow channels between shelves, the small distance between the goods layers of the shelves, the variety of planning system is proven to be 8%. When the target merchandise order distance is greater than 5 cm, the open MV match success rate is 89.7%. By increasing the offset of the uneven target goods, the sorting and picking success rate is increased from 6 to 85%, so that the end of the mechanical arm can reach the position of the cargo easily, conveniently and quickly.
Funding information: The authors state no funding involved.
Author contributions: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.