Autonomous and teleoperated sensor pointing on a mobile robot
A computer-implemented method executed by data processing hardware of a robot causes the data processing hardware to perform operations. The operations include receiving a sensor pointing command that commands the robot to use a sensor to capture sensor data of a location in an environment of the robot. The sensor is disposed on the robot. The operations include determining, based on an orientation of the sensor relative to the location, a direction for pointing the sensor toward the location, and an alignment pose of the robot to cause the sensor to point in the direction toward the location. The operations include commanding the robot to move from a current pose to the alignment pose. After the robot moves to the alignment pose and the sensor is pointing in the direction toward the location, the operations include commanding the sensor to capture the sensor data of the location in the environment.
1 . A computer-implemented method that when executed by data processing hardware of a legged robot causes the data processing hardware to perform operations comprising:
receiving, from a user computing device, a sensor pointing command that commands the legged robot to navigate to a waypoint and use a first sensor of the legged robot to capture sensor data of a target location in an environment of the legged robot by pointing the first sensor in a target direction toward the target location at the waypoint;
commanding a first movement of one or more legs of the legged robot to move the legged robot to the waypoint based on receiving the sensor pointing command, wherein the first movement of the one or more legs transitions the legged robot from a first pose of the legged robot to a second pose of the legged robot using a set of joints of the legged robot;
determining a first orientation of the first sensor relative to the target location based on the second pose of the legged robot;
obtaining, from a second sensor that is different than the first sensor, first image data;
determining, based on the sensor pointing command, the first image data obtained from the second sensor, and the first orientation of the first sensor relative to the target location, a third pose of the legged robot that points the first sensor in the target direction toward the target location, wherein each of the first pose, the second pose, and the third pose correspond to a respective set of positions of the set of joints;
commanding a second movement of the one or more legs based on receiving the sensor pointing command, wherein the second movement of the one or more legs transitions the legged robot from the second pose to the third pose using the set of joints; and
commanding the first sensor to capture the sensor data of the target location based on receiving the sensor pointing command.
2 . The method of claim 1 , wherein the sensor pointing command comprises one or more sensor data capture parameters, the one or more sensor data capture parameters comprising an angle of the first sensor, a height of the first sensor, a proximity of the first sensor relative to the target location, or a direction of the first sensor relative to the target location.
3 . The method of claim 1 , wherein the first sensor comprises a camera, wherein the sensor data of the target location comprises second image data of the target location, wherein the sensor pointing command comprises an identifier of an object, and wherein the operations further comprise:
detecting, using the second image data of the target location, the object within the environment based on the sensor pointing command.
4 . The method of claim 1 , wherein the first sensor comprises a pan-tilt-zoom (PTZ) sensor.
5 . The method of claim 4 , wherein the operations further comprise:
determining, based on the sensor pointing command and the first orientation of the first sensor relative to the target location, a second orientation of the first sensor relative to the target location for aligning a field of view of the PTZ sensor with the target direction; and
commanding a third movement of the PTZ sensor based on determining the second orientation of the first sensor relative to the target location, wherein a combination of the third movement of the PTZ sensor and the second movement of the one or more legs causes the first sensor to point in the target direction toward the target location.
6 . The method of claim 5 , wherein the operations further comprise:
receiving, from the PTZ sensor, feedback data indicating an error associated with the field of view of the PTZ sensor, wherein determining the third pose is further based on the feedback data.
7 . The method of claim 6 , wherein the error indicates that a difference between the field of view of the PTZ sensor and a field of view associated with pointing in the target direction toward the target location is greater than or equal to a threshold difference.
8 . The method of claim 1 , wherein receiving the sensor pointing command comprises receiving a user input indicating a selection of a ray or a point relative to a coordinate frame of the legged robot.
9 . The method of claim 1 , wherein the sensor pointing command comprises a model of an object, and wherein the operations further comprise:
obtaining second image data;
identifying, using the model of the object, the object within the second image data; and
determining the target direction based on identifying the object.
10 . The method of claim 1 , wherein the sensor pointing command comprises an output of an object detector, and wherein the operations further comprise:
obtaining second image data;
identifying, using the output of the object detector, an object within the second image data; and
determining the target direction based on the object and a field of view of the first sensor.
11 . The method of claim 1 , wherein the first sensor is fixed to the legged robot.
12 . The method of claim 1 , wherein the operations further comprise:
determining inverse kinematics associated with the legged robot, wherein determining the third pose is further based on determining the inverse kinematics associated with the legged robot.
13 . The method of claim 1 , wherein the operations further comprise:
coordinating a movement of the first sensor and the second movement of the one or more legs to align the first sensor with the target direction.
14 . The method of claim 1 , wherein the first orientation of the first sensor relative to the target location is updated based on the second movement of the one or more legs, and wherein commanding the first sensor to capture the sensor data is further based on the first orientation of the first sensor relative to the target location.
15 . A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed by the data processing hardware cause the data processing hardware to:
receive, from a user computing device, a sensor pointing command that commands a legged robot to navigate to a waypoint and use a first sensor disposed on the legged robot to capture sensor data of a target location in an environment of the legged robot by pointing the first sensor in a target direction toward the target location at the waypoint;
command a first movement of one or more legs of the legged robot to move the legged robot to the waypoint based on receiving the sensor pointing command, wherein the first movement of the one or more legs transitions the legged robot from a first pose of the legged robot to a second pose of the legged robot using a set of joints of the legged robot;
determine a first orientation of the first sensor relative to the target location based on the second pose of the legged robot;
obtain, from a second sensor that is different than the first sensor, first image data;
determine, based on the sensor pointing command, the first image data obtained from the second sensor, and the first orientation of the first sensor relative to the target location, a third pose of the legged robot that points the first sensor in the target direction toward the target location, wherein each of the first pose, the second pose, and the third pose correspond to a respective set of positions of the set of joints;
command a second movement of the one or more legs based on receiving the sensor pointing command, wherein the second movement of the one or more legs transitions the legged robot from the second pose to the third pose using the set of joints; and
command the first sensor to capture the sensor data of the target location based on receiving the sensor pointing command.
16 . The system of claim 15 , wherein the sensor pointing command comprises one or more sensor data capture parameters, the one or more sensor data capture parameters comprising an angle of the first sensor, a height of the first sensor, a proximity of the first sensor relative to the target location, or a direction of the first sensor relative to the target location.
17 . The system of claim 15 , wherein the first sensor comprises a camera, wherein the sensor data of the target location comprises second image data of the target location, wherein the sensor pointing command comprises an identifier of an object, and wherein execution of the instructions by the data processing hardware further causes the data processing hardware to:
detect, using the second image data of the target location, the object within the environment based on the sensor pointing command.
18 . The system of claim 15 , wherein the first sensor comprises a pan-tilt-zoom (PTZ) sensor.
19 . The system of claim 18 , wherein execution of the instructions by the data processing hardware further causes the data processing hardware to:
determine, based on the sensor pointing command and the first orientation of the first sensor relative to the target location, a second orientation of the first sensor relative to the target location for aligning a field of view of the PTZ sensor with the target direction; and
command a third movement of the PTZ sensor based on determining the second orientation of the first sensor relative to the target location, wherein a combination of the third movement of the PTZ sensor and the second movement of the one or more legs causes the first sensor to point in the target direction toward the target location.
20 . The system of claim 19 , wherein the execution of the instructions by the data processing hardware further causes the data processing hardware to:
receive, from the PTZ sensor, feedback data indicating an error associated with the field of view of the PTZ sensor, wherein determining the third pose is further based on the feedback data.
21 . The system of claim 20 , wherein the error indicates that a difference between the field of view of the PTZ sensor and a field of view associated with pointing in the target direction toward the target location is greater than or equal to a threshold difference.
22 . The system of claim 15 , wherein to receive the sensor pointing command, execution of the instructions by the data processing hardware further causes the data processing hardware to:
receive a user input indicating a selection of a ray or a point relative to a known coordinate frame of the legged robot.
23 . The system of claim 15 , wherein the sensor pointing command comprises a model of an object, and wherein execution of the instructions by the data processing hardware further causes the data processing hardware to:
obtain second image data;
identify, using the model of the object, the object within the second image data; and
determine the target direction based on identifying the object.
24 . The system of claim 15 , wherein the sensor pointing command comprises an output of an object detector, and wherein execution of the instructions by the data processing hardware further causes the data processing hardware to:
obtain second image data;
identify, using the output of the object detector, an object within the second image data; and
determine the target direction based on the object and a field of view of the first sensor.
25 . The system of claim 15 , wherein the first sensor is fixed to the legged robot.
26 . The system of claim 15 , wherein execution of the instructions by the data processing hardware further causes the data processing hardware to:
determine inverse kinematics associated with the legged robot, wherein determining the third pose is further based on determining the inverse kinematics associated with the legged robot.
27 . The system of claim 15 , wherein the first orientation of the first sensor relative to the target location is updated based on the second movement of the one or more legs, and wherein commanding the first sensor to capture the sensor data is further based on the first orientation of the first sensor relative to the target location.