Context aware measurement
Embodiments include systems and methods for remotely measuring distances in an environment captured by a device. A device captures a video stream of a device along with AR data that may include camera pose information and/or depth information, and transmits the video stream and AR data to a remote device. The environment may be analyzed to identify objects such as lines, edges, curves, shapes, anchors/corners, products, e.g., appliances, and other things of interest. Some or all of objects may be identified to the remote device to facilitate selecting an object or region of interest. Selected points for an object may be more precisely located by snapping them to corresponding anchor points for an object. Using anchor points facilitates more precise identification and/or measurement of an aspect of an object, such as one of its dimensions, the volume of a space, or performing other actions such as replacement of an object.
1. A method for measuring a region of interest in a video stream of an environment, the environment having an associated model including coordinates for locations within the model of the environment, comprising:
receiving, from a device, the video stream of the environment;
identifying, using object recognition within the video stream, an object;
associating one or more edges and anchor points with the identified object;
receiving, from the device, a user selection of a region of interest in the environment, the region of interest including the identified object;
receiving, from the device, a user selection of the identified object;
snapping a starting point to a first anchor point proximate to the starting point, the first anchor point one of the anchor points of the identified object;
monitoring a selection proximate to the region of interest;
identifying a stopping point for the selection along the region of interest;
determining whether the stopping point is on the identified object, and if not, presenting an interface with options for confirming the stopping point;
snapping the stopping point to a second anchor point proximate to the stopping point;
correlating the first and second anchor points with coordinates in the model; and
calculating a distance between the first and second anchor point based at least in part on coordinates for the first anchor point relative to coordinates for the second anchor point.
2. The method of claim 1 , the correlating comprising projecting the first and second anchor points onto the model to respectively determine 3D coordinates in the model for anchor points.
3. The method of claim 2 , further comprising deskewing the region of interest prior to the correlating.
4. The method of claim 1 , wherein modeling data is associated with the video stream and comprises augmented reality (AR) data correlated to the video stream.
5. The method of claim 1 , further comprising:
determining the selection along the region of interest is proximate to a third anchor point; and
revising the region of interest based at least in part on the first and third anchor points.
6. The method of claim 1 , further comprising
determining a measurement of an enclosure for the identified object.
7. The method of claim 6 , further comprising:
highlighting the identified object within the model; and
presenting a prompt to display on the device to confirm the identified object for the measurement of the enclosure.
8. A method for a receiving device in communication over a network with a sending device to measure an object in a video stream sent by the sending device, the video stream displaying an environment having an associated model including a coordinate system to place objects within the model, comprising:
receiving, by the receiving device, the video stream from the sending device;
identifying objects in the environment to the sending device;
associating one or more edges and anchor points with each of the identified objects;
monitoring a dragging across a selected one of the identified objects, including snapping a starting point for the dragging to a first anchor point proximate to the starting point, the first anchor point associated with the selected identified object;
determining whether a stopping point is on the selected identified object, and if not, presenting an interface with options for confirming the stopping point;
snapping the stopping point to a second anchor point proximate to the stopping point;
correlating the first and second anchor points with the coordinate system; and
calculating a measurement of the selected one of the identified objects based at least in part on the first and second anchor points.
9. The method of claim 8 , further comprising:
receiving modeling data corresponding to the video stream; and
computing the model based at least in part on the modeling data.
10. The method of claim 8 , further comprising an Artificial Intelligence analyzing a selected one or more of the video stream or the model to perform the identifying objects.
11. The method of claim 8 , further comprising sending a recommended next action from the receiving device to at least the sending device.
12. The method of claim 11 , further comprising:
determining a replacement is needed for the selected one of the identified objects; and
locating the replacement.
13. A non-transitory computer readable medium (CRM) comprising instructions for measuring a region of interest in a video stream of an environment, the environment having an associated model including coordinates for locations within the model of the environment, the instructions, when executed by an apparatus, cause the apparatus to perform:
receiving the video stream of the environment;
identifying, using object recognition within the video stream, an object;
associating one or more edges and anchor points with the identified object;
receiving a user selection of a region of interest in the environment, the region of interest including the identified object;
receiving a user selection of the identified object;
snapping a starting point to a first anchor point proximate to the starting point, the first anchor point one of the anchor points of the identified object;
monitoring a selection proximate to the region of interest;
identifying a stopping point for the selection along the region of interest;
determining whether the stopping point is on the identified object, and if not, presenting an interface with options for confirming the stopping point;
snapping the stopping point to a second anchor point proximate to the stopping point;
correlating the first and second anchor points with coordinates in the model; and
calculating a distance between the first and second anchor point based at least in part on coordinates for the first anchor point relative to coordinates for the second anchor point.
14. The CRM of claim 13 , wherein the instructions for correlating comprise further instructions that are to further cause the apparatus to perform: projecting the first and second anchor points onto the model to respectively determine 3D coordinates in the model for anchor points.
15. The CRM of claim 14 , wherein the instructions include instructions that are to further cause the apparatus to perform: deskewing the region of interest prior to the correlating.
16. The CRM of claim 13 , wherein the instructions include instructions that are to further cause the apparatus to perform:
determining the selection along the region of interest is proximate to a third anchor point; and
revising the region of interest based at least in part on the first and third anchor points.
17. The CRM of claim 13 , wherein the instructions include instructions that are to further cause the apparatus to perform
determining a measurement of an enclosure for the identified object.
18. The CRM of claim 17 , wherein the instructions include instructions that are to further cause the apparatus to perform:
highlighting the identified object within the model; and
presenting a prompt to display on the device to confirm the object for the measurement of the enclosure.
19. A non-transitory computer readable medium (CRM) comprising instructions for a receiving device in communication over a network with a sending device to measure an object in a video stream sent by the sending device, the video stream displaying an environment having an associated model including a coordinate system to place objects within the model, the instructions, when executed by an apparatus, cause the apparatus to perform:
receiving, by the receiving device, the video stream from the sending device;
providing a selected one or more of the video stream or the model to an artificial intelligence (AI);
receiving an analysis from the AI recognizing objects in the environment;
associating one or more edges and anchor points with each of the recognized objects;
sending an identification of the recognized objects to the sending device;
monitoring a dragging across a selected one of the recognized objects, including snapping a starting point for the dragging to a first anchor point proximate to the starting point, the first anchor point associated with the selected identified object;
determining whether a stopping point is on the selected identified object, and if not, presenting an interface with options for confirming the stopping point;
snapping the stopping point to a second anchor point proximate to the stopping point;
correlating the first and second anchor points with the coordinate system; and
calculating a measurement of the selected one of the identified objects based at least in part on the first and second anchor points.
20. The CRM of claim 19 , wherein the instructions include instructions that are to further cause the apparatus to perform:
receiving modeling data corresponding to the video stream; and
computing the model based at least in part on the modeling data.