Infants and young children, between months and 1 year old, will gradually develop hand-eye coordination by playing with some simple toys. In the field of robotics, we need to give it the ability to coordinate with the hand, this process is generally called "hand-eye calibration." Only with the ability of hand-eye coordination can a robot complete complex work and truly call it "smart." When an infant sees an object, first obtains information through the eye, transmits it to the brain for understanding, and finally completes the grasping and placing of the toy by hand. Hand, eye and brain synergy is a special skill of humans and a few animals. By playing some simple educational toys, you can gradually exercise to enhance your child's hand-eye coordination ability, and at the same time, can be applied to the brain to promote children's intellectual development. In the field of intelligent robots, robots need to complete efficient and complex work, otherwise they cannot be called "smart". However, it is not easy for a robot to acquire the same coordination capabilities as humans. How can we achieve hand-eye coordination like humans? Go back to this game. When a child is playing with a toy, there is a coordinate system in the brain, which can be regarded as a basic coordinate system; a flexible hand forms a trajectory in motion, which is called the coordinate system of the hand; the point where the toy is to be placed has a target coordinate system. . The three coordinate systems work together to make the game play smoothly. If you add coordinates to this game scene, it will look like this, as shown in Figure 2. In the field of intelligent robots , four-axis, six-axis, parallel and other mechanical arms can be considered as the robot's arm (although the degree of freedom is high and the speed is fast, it is far less flexible than the hand). The eyes of the robot are generally referred to as cameras (including 2D and 3D). The mainstream of intelligent robots today is the use of 3D vision cameras. The brain of a robot generally refers to a computer or embedded chip that performs core control and calculation. How to establish a relationship between two coordinate systems? This is the core issue to be solved by hand-eye calibration. The hand-eye calibration and core is to solve a matrix of AX=XB. The algorithm for solving this matrix is ​​the hand-eye calibration algorithm. Xiaolan (Hangzhou Blue Core Technology Co., Ltd.) relies on the Eigen library to implement the classic Tsai method and open source. How to perform hand-eye calibration Note: The following involves formula derivation, and you don't want to push the math formula to skip. Understanding the core of hand-eye calibration is how to substitute the robot's hand-eye relationship into the formula of AX=XB. First of all, we must understand what A, B, and X represent respectively. X, indicating the unknown amount, that is, the transfer matrix of the hand-eye calibration. A and B? First look at the schematic diagram of the separation of the robot's hand. In Figure 4, the robot base is generally considered to be the origin of the world coordinate system. The position recognized by the camera (eye) can be converted to the robot coordinate system (also the world coordinate system) by the transfer matrix. In the process of hand-eye calibration, it is necessary to continuously move the arm end to capture the pose of a set of end effectors; at the same time, the camera captures the pose of the mark (eg, checkerboard) hung on the end effector to form another pose. Group data. Early warning: Here are some real mathematical derivations. among them According to the pose is the pose in space, we can get (the poses of both sides are marked) Therefore, the equation has two known quantities, two unknowns, and one set of data is unsolvable! Simultaneous equation At this point, recalling the law of matrix multiplication, we were pleasantly surprised to find that the form of the equation is AX=XB! The mathematical derivation is over. Children, no, engineers, can use this data to bring into the standard algorithm calculation! to sum up This article belongs to pure original article, please indicate Hangzhou Blue Core Technology Co., Ltd. Power Steering Motor,Power Brake Motor,Electric Power Steering Motor,Electric Power Brake Motor Fuxin Dare Automotive Parts Co., Ltd. (DARE AUTO) , https://www.darepump.com
Merchants have also seen this area, designing a lot of educational toys and games, as shown in Figure 1. The child needs to grab the toy and put it in the car. The child's spatial thinking ability and coordination ability have been exercised in the game. 
figure 1 
figure 2
In general, the "hand-eye coordination" of a robot requires a "hand-eye calibration" process. Hand-eye calibration is a classic problem in the robot field. The core is to calculate the transfer matrix of the camera coordinate system in the robot coordinate system. The transfer matrix is ​​a 4 x 4 matrix, generally expressed in homogeneous coordinates. The two most important components are the rotation matrix R and the translation vector t, which represent the rotation component and the translation component, respectively. After the transfer matrix is ​​obtained, the coordinates under the camera can be converted to the robot arm coordinate system to complete the work. As shown in Figure 3. 
image 3 
Figure 4
Suppose the collected data has N pairs. According to the geometric relationship in the figure, there are two unknowns, which are the poses marked under the end effector. 
And the coordinates of the camera in the world coordinate system 
. Is the target of our hand-eye calibration, 
How much we don't care. 
Observe the above formula, 
Unknown, is the target matrix, invariant; 
Known, is the data that can be read by the camera, variables; 
Known, it is the end reading of the arm, variable; 
Unknown, we don't like it very much, not variable.
If a set of data doesn't work, then there are more groups!
Suppose we have two sets of data, which are the jth group of the i-th combination, and the two sets of coordinates are represented in parentheses in the formula.
Looking back at the middle school algebra, we can eliminate the ones we don't like by solving the similar equations. 
.
There are the following two sets of equations, 
And finally we get, 
Where A and B are respectively known, 
what? What is A and B? Watch for yourself!
In general, the more data collected, the more accurate the calibration results. When collecting, in order to prevent falling into a local trap, it is necessary to collect the robot posture as much as possible. At the same time, it is necessary to ensure that the mark is fixed at the end of the arm, and the relative posture of the camera and the arm must be kept fixed.
Some people say, no, the camera I used is not like this. I put the camera on the arm and move it with it. Can I use this method?
The answer is of course yes.
This involves two modes of hand-eye collaborative robots, eye-in-hand and eye-to-hand. What we have been studying all the time is the eye-to-hand mode. For the eye-in-hand mode, a similar method can be used to eliminate the intermediate variables we don't like, and finally it comes down to the form of AX=XB, which can be solved.
Both modes are available in the open source code.
1, the child's hand-eye coordination ability will be exercised in the game, as long as the toy allows her / him to play, the gene will gradually teach her / his flexible hand-eye coordination ability.
2, the coordination of hand-eye coordination of intelligent robots requires the careful training of engineers! In general, the more data, the smaller the error. The hand-eye coordination ability is the premise for the robot to complete the sorting, grabbing, and delivery work. 
Figure 5 Hand-eye coordination picking