Collision detection, estimation, and avoidance
An example method involves obtaining a log of sensor data indicative of an environment during a prior time period, with the log of sensor data including a sequence of image frames, and determining that the log of sensor data relates to a collision involving a physical object in the environment at a particular time within the prior time period. The method also involves, responsive to determining that the log of sensor data relates to the collision, generating a training data set for the collision from the log of sensor data. The training data set for the collision may include multiple image frames of the sequence of image frames that are prior to an image frame in the sequence of image frames that corresponds to the particular time.
1. A collision detection system stored on a non-transitory computer-readable storage medium, wherein the collision detection system is manufactured by a process comprising:
obtaining a log of sensor data indicative of an environment during a prior time period, wherein the log of sensor data comprises a sequence of image frames;
determining that the log of sensor data relates to a collision involving a physical object in the environment at a particular time within the prior time period;
responsive to determining that the log of sensor data relates to the collision, generating a training data set for the collision from the log of sensor data, wherein the training data set for the collision comprises multiple image frames of the sequence of image frames that are prior to an image frame in the sequence of image frames that corresponds to the particular time; and
training the collision detection system with the training data set to predict future collisions.
2. The collision detection system of claim 1 , wherein determining that the log of sensor data relates to the collision comprises determining that the collision is a physical collision between the physical object and a robotic device.
3. The collision detection system of claim 2 , the process further comprising obtaining accelerometer data for at least part of the prior time period, wherein determining that the log of sensor data relates to the collision comprises determining that the accelerometer data is indicative of a collision between the physical object and the robotic device.
4. The collision detection system of claim 2 , the process further comprising obtaining contact sensor data for at least part of the prior time period, and wherein determining that the log of sensor data relates to the collision comprises determining that the contact sensor data is indicative of a collision between the physical object and the robotic device.
5. The collision detection system of claim 2 , wherein the process is accomplished by a computing device of the robotic device.
6. The collision detection system of claim 2 , wherein the log of sensor data further comprises a log of velocity data indicative of movement of the robotic device in the environment during the prior time period, and wherein generating the training data set for the collision from the log of sensor data comprises generating the training data set to further comprise at least a portion of the log of velocity data.
7. The collision detection system of claim 2 , wherein the log of sensor data further comprises payload data indicative of a payload of the robotic device, and wherein determining that the log of sensor data relates to the collision comprises determining that the payload data is indicative of a collision between the physical object and the robotic device.
8. The collision detection system of claim 1 , wherein determining that the log of sensor data relates to the collision comprises determining that the collision is a virtual collision between the physical object and a virtual boundary surrounding a robotic device or a virtual collision between a virtual boundary surrounding the physical object and the robotic device.
9. The collision detection system of claim 8 , the process further comprising obtaining proximity sensor data for at least part of the prior time period, wherein determining that the log of sensor data relates to the collision comprises determining that the proximity sensor data is indicative of a distance between the physical object and the robotic device transitioning from being more than a threshold distance to being less than the threshold distance.
10. The collision detection system of claim 1 , wherein determining that the log of sensor data relates to the collision comprises determining that the collision is a virtual collision between the physical object and a software robot in a form of an augmented reality graphic.
11. The collision detection system of claim 10 , wherein at least two images of the sequence of image frames comprise depth information indicative of a distance to the physical object, and wherein determining that the log of sensor data relates to the collision comprises determining, based on the depth information, that the physical object moved from a first position to a second position and that the second position overlaps with a position of the augmented reality graphic.
12. The collision detection system of claim 1 , wherein the multiple image frames comprise depth data for one or more pixels, and wherein the depth data is indicative of movement of the physical object prior to the collision.
13. The collision detection system of claim 1 , wherein the environment comprises a room, wherein the log of sensor data further comprises room information indicative of a type of room of the room, and wherein generating the training data set for the collision from the log of sensor data comprises generating the training data set to further comprise the room information.
14. The collision detection system of claim 1 , the process further comprising transmitting, to a machine learning server device, at least one file representing the training data set for the collision, wherein the machine learning server device is configured to train a machine learning model of the collision detection system using the training data set.
15. The collision detection system of claim 1 , configured to:
receive additional sensor data indicative of the environment; and
determine that the additional sensor data is indicative of a potential collision at a future time in the environment.
16. A computer-implemented method comprising:
receiving sensor data indicative of an environment at a current time; and
determining through inputting the sensor data into a trained model whether the sensor data is indicative of a potential collision in the environment at a future time, the trained model trained according to a process comprising:
obtaining a log of sensor data indicative of another environment during a prior time period, wherein the log of sensor data comprises a sequence of image frames,
determining that the log of sensor data relates to a collision involving a physical object in the environment at a particular time within the prior time period,
responsive to determining that the log of sensor data relates to the collision, generating a training data set for the collision from the log of sensor data, wherein the training data set for the collision comprises multiple image frames of the sequence of image frames that are prior to an image frame in the sequence of image frames that corresponds to the particular time, and
training the trained model with the training data set to detect collisions.
17. The computer-implemented method of claim 16 , wherein determining that the log of sensor data relates to the collision comprises determining that the collision is a physical collision between the physical object and a robotic device.
18. The computer-implemented method of claim 17 , the process further comprising obtaining accelerometer data for at least part of the prior time period, wherein determining that the log of sensor data relates to the collision comprises determining that the accelerometer data is indicative of a collision between the physical object and the robotic device.
19. The computer-implemented method of claim 17 , the process further comprising obtaining contact sensor data for at least part of the prior time period, and wherein determining that the log of sensor data relates to the collision comprises determining that the contact sensor data is indicative of a collision between the physical object and the robotic device.
20. The computer-implemented method of claim 17 , wherein the process is accomplished by a computing device of the robotic device.
21. The computer-implemented method of claim 17 , wherein the log of sensor data further comprises a log of velocity data indicative of movement of the robotic device in the other environment during the prior time period, and wherein generating the training data set for the collision from the log of sensor data comprises generating the training data set to further comprise at least a portion of the log of velocity data.
22. The computer-implemented method of claim 17 , wherein the log of sensor data further comprises payload data indicative of a payload of the robotic device, and wherein determining that the log of sensor data relates to the collision comprises determining that the payload data is indicative of a collision between the physical object and the robotic device.
23. The computer-implemented method of claim 16 , wherein determining that the log of sensor data relates to the collision comprises determining that the collision is a virtual collision between the physical object and a virtual boundary surrounding a robotic device or a virtual collision between a virtual boundary surrounding the physical object and the robotic device.
24. The computer-implemented method of claim 23 , the process further comprising obtaining proximity sensor data for at least part of the prior time period, wherein determining that the log of sensor data relates to the collision comprises determining that the proximity sensor data is indicative of a distance between the physical object and the robotic device transitioning from being more than a threshold distance to being less than the threshold distance.
25. The computer-implemented method of claim 16 , wherein determining that the log of sensor data relates to the collision comprises determining that the collision is a virtual collision between the physical object and a software robot in a form of an augmented reality graphic.
26. The computer-implemented method of claim 25 , wherein at least two images of the sequence of image frames comprise depth information indicative of a distance to the physical object, and wherein determining that the log of sensor data relates to the collision comprises determining, based on the depth information, that the physical object moved from a first position to a second position and that the second position overlaps with a position of the augmented reality graphic.
27. The computer-implemented method of claim 16 , wherein the multiple image frames comprise depth data for one or more pixels, and wherein the depth data is indicative of movement of the physical object prior to the collision.
28. The computer-implemented method of claim 16 , wherein the other environment comprises a room, wherein the log of sensor data further comprises room information indicative of a type of room of the room, and wherein generating the training data set for the collision from the log of sensor data comprises generating the training data set to further comprise the room information.
29. The computer-implemented method of claim 16 , wherein the trained model is trained as a machine learning model.
30. A non-transitory computer-readable storage medium with encoded instructions that, when executed by a processor, cause the process to accomplish steps of:
receiving sensor data indicative of an environment at a current time; and
determining through inputting the sensor data into a trained model whether the sensor data is indicative of a potential collision in the environment at a future time, the trained model trained according to a process comprising:
obtaining a log of sensor data indicative of another environment during a prior time period, wherein the log of sensor data comprises a sequence of image frames,
determining that the log of sensor data relates to a collision involving a physical object in the environment at a particular time within the prior time period,
responsive to determining that the log of sensor data relates to the collision, generating a training data set for the collision from the log of sensor data, wherein the training data set for the collision comprises multiple image frames of the sequence of image frames that are prior to an image frame in the sequence of image frames that corresponds to the particular time, and
training the trained model with the training data set to detect collisions.