Systems and methods for processing captured images
Systems, methods, and apparatuses described herein may provide image processing, including displaying, by a mobile device, an image of an object located perpendicular to a reference object, calculating, based on at least one depth measurement determined using a depth sensor in the mobile device, the predicted height of the mobile device when the image was captured, calculating scale data for the image based on the predicted height, determining a reference line identifying the location of the object and the reference object in the image, segmenting pixels in the object in the image from pixels in the image outside the object, measuring the object based on the pixels in the object and the scale data, and generating model data comprising the object, the scale data, and the measurements.
1 . A method for generating a model of an object, comprising:
capturing, by a mobile device, an image comprising an object and a reference object;
determining, by the mobile device, from a plurality of candidate lines and corresponding confidence metrics, a reference line identifying a location of the object and the reference object in the image;
segmenting, by the mobile device, pixels in the object in the image from pixels in the image outside the object, wherein segmenting pixels in the object in the image from pixels in the image outside the object includes applying a first machine learning classifier to generate a heat map marking a foreground including the object from a background in the image and generating a binary mask to remove the background from the image;
measuring, by the mobile device, the object based on the pixels in the object and based on determining scale data of the image, wherein the scale data is determined using a second machine learning classifier and metadata from the captured image and based on a predicted height of the mobile device calculated from at least one depth measurement; and
generating, by the mobile device, model data comprising the object and measurements of the object.
2 . The method of claim 1 , further comprising generating, by the mobile device,
a product size recommendation based on the model data.
3 . The method of claim 1 , further comprising:
determining, using a depth sensor in the mobile device, a depth measurement of the object; and
correcting, by the mobile device and based on the depth measurement, a perspective of the model data.
4 . The method of claim 1 , further comprising determining, by the mobile device, that the mobile device is aligned with the object wherein determining that the mobile device is aligned with the object includes leveling the mobile device by alignment indicators on the mobile device.
5 . The method of claim 1 , further comprising generating image data based on the model data, the image data including model data of the object and metadata associated with one or more settings specific to a user.
6 . The method of claim 1 , wherein measuring the object includes:
calculating, by the mobile device and based on at least one depth measurement determined using a depth sensor in the mobile device, the predicted height of the mobile device when the image was captured.
7 . The method of claim 6 , wherein calculating the predicted height of the mobile device when the image was captured includes:
projecting one or more laser beams toward the object using the depth sensor, wherein the depth sensor includes a light detection and ranging (LIDAR) sensor;
detecting, by the LIDAR sensor, a reflected laser beam being reflected back to the LIDAR sensor;
determining a time duration between projecting the one or more laser beams and detecting the reflected laser beam; and
calculating the predicted height based on the time duration.
8 . The method of claim 6 , wherein calculating the predicted height of the mobile device when the image was captured includes using the depth sensor to take a plurality of depth measurements, and averaging the plurality of depth measurements to calculate the predicted height.
9 . The method of claim 6 , wherein generating the model data includes generating one or more mesh models of the object based on projections onto the object from the depth sensor.
10 . The method of claim 1 , wherein determining the reference line identifying the location of the object and the reference object in the image includes projecting, by a depth sensor in the mobile device, one or more beams or dots onto the object and determining the reference line based on the one or more beams or dots.
11 . The method of claim 1 , wherein segmenting pixels in the object in the image from pixels in the image outside the object includes: validating the heat map to ensure that portions identified as the foreground are correct as a precursor to generating the binary mask, and providing an object tracking and segmentation application programming interface by the mobile device to refine designated edges of the object in the captured image in combination with the binary mask.
12 . An apparatus comprising:
at least one image capture device;
a processor; and
memory storing computer readable instructions that, when executed, cause the apparatus to:
capture an image comprising an object and a reference object;
determine, from a plurality of candidate lines and corresponding confidence metrics, a reference line identifying a location of the object and the reference object in the image;
segment pixels in the object in the image from pixels in the image outside the object, wherein segmenting pixels in the object in the image from pixels in the image outside the object includes applying a first machine learning classifier to generate a heat map marking a foreground including the object from a background in the image and removing the background from the image;
measure the object based on the pixels in the object and based on determining scale data of the image, wherein the scale data is determined using a second machine learning classifier and metadata from the captured image and based on a predicted height of the at least one image capture device calculated from at least one depth measurement; and
generate model data comprising the object and measurements of the object.
13 . The apparatus of claim 12 , further comprising at least one depth sensor that includes at least one of an infrared sensor, a light detection sensor, or a LIDAR sensor.
14 . The apparatus of claim 12 , further comprising a depth sensor, and wherein measuring the object includes:
calculating, based on at least one depth measurement determined using the depth sensor, a predicted height of the apparatus when the image was captured.
15 . The apparatus of claim 14 , wherein calculating the predicted height of the apparatus when the image was captured includes:
projecting one or more laser beams toward the object using the depth sensor, wherein the depth sensor includes a LIDAR sensor;
detecting, by the LIDAR sensor, a reflected laser beam being reflected back to the LIDAR sensor;
determining a time duration between projecting the one or more laser beams and detecting the reflected laser beam; and
calculating the predicted height based on the time duration.
16 . The apparatus of claim 14 , wherein generating the model data includes generating one or more mesh models of the object based on projections onto the object from the depth sensor.
17 . The apparatus of claim 16 , wherein generating the model data includes determining edge locations of the object based on the one or more mesh models.
18 . The apparatus of claim 12 , further comprising a depth sensor, and wherein generating the model data includes determining contours of the object based on a depth map generated using the depth sensor.
19 . The apparatus of claim 12 , further comprising a depth sensor, and wherein determining the reference line identifying the location of the object and the reference object in the image includes projecting, by the depth sensor, one or more beams or dots onto the object and determining the reference line based on the one or more beams or dots.
20 . A method for generating a model of an object, comprising:
capturing, by a mobile device, an image comprising an object and a reference object;
measuring, by the mobile device, motion of the mobile device as the mobile device is moved toward a reference plane;
detecting, by the mobile device, the object in the image;
determining, by the mobile device, from a plurality of candidate lines and corresponding confidence metrics, a reference line identifying a location of the object and the reference object in the image;
segmenting, by the mobile device, pixels in the object in the image from pixels in the image outside the object, wherein segmenting pixels in the object in the image from pixels in the image outside the object includes applying a first machine learning classifier to generate a heat map marking a foreground including the object from a background in the image and generating a binary mask to remove the background from the image;
measuring, by the mobile device, the object based on the pixels in the object and based on determining scale data of the image; and
generating, by the mobile device, model data comprising the object and measurements of the object and based on determining scale data of the image, wherein the scale data is determined using a second machine learning classifier and metadata from the captured image and based on a predicted height of the mobile device calculated from at least one depth measurement.