Systems and methods for generating a 360-degree viewing experience
A system for generating a 360-degree viewing experience may receive a plurality of images of an object from an image capture device, wherein each of the plurality of images corresponds to a different rotational orientation of the object relative to the image capture device. The system may detect, using a first machine learning model, the object in each of the plurality of images. The system may detect, using a second machine learning model, regions associated with identifiable object features in one or more images of the plurality of images. The system may assign feature metadata to the one or more images, the features metadata associated with one or more detected regions of the detected regions of the object in the one or more images. The system may publish, with an application programming interface, the plurality of images and the feature metadata for the 360-degree viewing experience.
1. A system for generating a 360-degree viewing experience, comprising:
one or more processors; and
a memory in communication with the one or more processors and storing instructions that, when executed by the one or more processors, are configured to cause the system to:
receive a plurality of images of an object from an image capture device, wherein each of the plurality of images corresponds to a different rotational orientation of the object relative to the image capture device;
detect, using a first machine learning model, the object in each of the plurality of images;
detect, using a second machine learning model, regions associated with identifiable object features in one or more images of the plurality of images;
assign feature metadata to the one or more images, the feature metadata associated with one or more detected regions of the detected regions of the object in the one or more images of the plurality of images; and
publish, with an application programming interface, the plurality of images and the feature metadata for the 360-degree viewing experience.
2. The system of claim 1 , wherein the first machine learning model comprises a first single shot detector (SSD), a first Faster Region-Based Convolutional Neural Network (Faster R-CNN), a Region-Based Fully Convolutional Net (R-FCN), a Mask R-CNN, SegNET, a pixel-wise contextual attention network (PiCANet), or a combination thereof, wherein the second machine learning model comprises a You Look Only Once image segmentation model (YOLO), YOLO version 2 (YOLOv2), YOLO9000, RetinaNET, a second Mask R-CNN, a second SSD, or a combination thereof, and wherein the object is a vehicle.
3. The system of claim 1 , wherein the feature metadata comprises at least one region label and coordinates associated with the one or more detected regions.
4. The system of claim 3 , wherein the instructions, when executed by the one or more processors, are further configured to cause the system to:
receive a feature list of the object; and
determine whether a first region label associated with the one or more detected regions matches a first feature on the feature list beyond a threshold level.
5. The system of claim 4 , wherein the instructions, when executed by the one or more processors, are further configured to cause the system to:
receive a list of high value features;
determine whether the first feature matches a first high value feature from the list of high value features beyond the threshold level;
responsive to determining that the first region label matches the first feature beyond the threshold level and responsive to determining that the first feature matches the first high value feature beyond the threshold level, generate a visual indicator to be displayed at a center of the detected region based on the coordinates; and
responsive to determining that the first region label does not match the first feature beyond the threshold level or that the first feature does not match the first high value feature beyond the threshold level, refrain from generating the visual indicator for display.
6. The system of claim 5 , wherein generating the visual indicator comprises (i) determining that a series of images of the plurality of images each comprise a common first region, (ii) performing a regression analysis on a center of the common first region in each of the series of images, (iii) updating the coordinates corresponding to the visual indicator for each of the series of images based on the regression analysis; and (iv) generating the visual indicator for display in each of the series of images at a location based on the updated coordinates.
7. The system of claim 6 , wherein the instructions, when executed by the one or more processors, are further configured to cause the system to:
load the plurality of images and the visual indicator; and
display the plurality of images sequentially to give the appearance of the object rotating about an axis along with the visual indicator.
8. The system of claim 7 , wherein the instructions, when executed by the one or more processors, are further configured to cause the system to:
crop each of the plurality of images based on detecting the object,
wherein the displaying of the plurality of images with the visual indicator and the generation of the visual indicator occurs substantially simultaneously.
9. The system of claim 1 , wherein the object is rotated into a first rotational orientation position, a second rotational orientation position, and a third rotational orientation position with respect to the image capture device, and wherein the image capture device captures a first image at the first rotational orientation position, a second image at the second rotational orientation position, and a third image at the third rotational orientation position.
10. A system for generating a 360-degree viewing experience, comprising:
one or more processors; and
a memory in communication with the one or more processors and storing instructions that, when executed by the one or more processors, are configured to cause the system to:
receive a video capturing approximately 360 degrees of an object;
split the video into a plurality of images;
detect, using a first machine learning model, the object in each of the plurality of images;
detect, using a second machine learning model, regions associated with identifiable object features in one or more images of the plurality of images;
assign feature metadata to the one or more images, the feature metadata associated with one or more detected regions of the object in the one or more images of the plurality of images; and
publish, with an application programming interface, the plurality of images and the feature metadata for the 360-degree viewing experience.
11. The system of claim 10 , wherein the first machine learning model comprises a first single shot detector (SSD), a first Faster Region-Based Convolutional Neural Network (Faster R-CNN), a Region-Based Fully Convolutional Net (R-FCN), a Mask R-CNN, SegNET, a pixel-wise contextual attention network (PiCANet), or a combination thereof, wherein the second machine learning model comprises a You Look Only Once image segmentation model (YOLO), YOLO version 2 (YOLOv2), YOLO9000, RetinaNET, a second Mask R-CNN, a second SSD, or a combination thereof, and wherein the object is a vehicle.
12. The system of claim 10 , wherein the feature metadata comprises at least one region label and coordinates associated with the one or more detected regions.
13. The system of claim 12 , wherein the instructions, when executed by the one or more processors, are further configured to cause the system to:
receive a feature list of the object;
receive a list of high value features;
determine whether a first region label assigned to the one or more detected regions matches a first feature on the feature list beyond a threshold level;
determine whether the first feature matches a first high value feature from the list of high value features beyond the threshold level;
responsive to determining that the first region label matches the first feature beyond the threshold level and responsive to determining that the first feature matches the first high value feature beyond the threshold level, generate a visual indicator for display at a center of the detected region based on the coordinates; and
responsive to determining that the first region label does not match the first feature beyond the threshold level or that the first feature does not match the first high value feature beyond the threshold level, refrain from generating the visual indicator for display.
14. The system of claim 13 , wherein generating the visual indicator comprises (i) determining that a series of images of the plurality of images each comprise a common first region, (ii) performing a regression analysis on a center of the common first region in each of the series of images, (iii) updating the coordinates corresponding to the visual indicator for each of the series of images based on the regression analysis; and (iv) generating the visual indicator for display in each of the series of images at a location based on the updated coordinates.
15. The system of claim 14 , wherein the instructions, when executed by the one or more processors, are further configured to cause the system to load the plurality of images and the visual indicator.
16. The system of claim 15 , wherein the instructions, when executed by the one or more processors, are further configured to cause the system to display the plurality of images sequentially to give the appearance of the object rotating about an axis along with the visual indicator.
17. A system for generating a 360-degree viewing experience of an interior of an object, comprising:
one or more processors; and
a memory in communication with the one or more processors and storing instructions that, when executed by the one or more processors, are configured to cause the system to:
receive a 360-degree image of the interior of the object;
detect, using a machine learning model, regions associated with identifiable object features in the 360-degree image;
assign feature metadata to the 360-degree image, the feature metadata associated with one or more detected regions of the interior of the object in the 360-degree image; and
publish, with an application programming interface, the 360-degree image and the feature metadata for the 360-degree viewing experience.
18. The system of claim 17 , wherein:
the 360-degree image is formed by stitching together two or more images captured by an image capture device of the interior of the object, and
the feature metadata comprises at least one region label and Cartesian coordinates associated with the one or more detected regions.
19. The system of claim 17 , wherein the instructions, when executed by the one or more processors, are further configured to cause the system to:
receive a feature list of the object;
receive a list of high value features;
determine whether a first detected region of the one or more detected regions matches a first feature on the feature list beyond a threshold level;
determine whether the first feature matches a first high value feature from the list of high value features beyond the threshold level;
responsive to determining that the first detected region matches the first feature beyond the threshold level and responsive to determining that the first feature matches the first high value feature beyond the threshold level, translate the Cartesian coordinates of a center of the detected region to polar coordinates and generate a visual indicator for display at the center of the detected region based on the polar coordinates; and
responsive to determining that the first detected region does not match the first feature beyond the threshold level or that the first feature does not match the first high value feature beyond the threshold level, refrain from generating the visual indicator for display.
20. The system of claim 19 , the instructions, when executed by the one or more processors, are further configured to cause the system to stretch the 360-degree image into a spherical view.