IP Library Granted Patent US 10,997,413
Granted Patent B2
US 10,997,413 · App. 16/174,772 · Granted May 4, 2021

Method and system for obtaining vehicle target views from a video stream

Inventors: Mark Endras (Toronto, CA); Li Zhang (Toronto, CA); Yunlei Zhang (Toronto, CA); Nataliya Portman (Toronto, CA); Akbar Nurlybayev (Toronto, CA)
Assignee: NthGen Software Inc.
G06K9/00664G06K9/00718G06T7/0002G06T2207/10016G06T2207/30248
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,997,413
App. No.
16/174,772
Granted
May 4, 2021
Kind
B2
Abstract

A system and method for obtaining target views of a vehicle is disclosed. A seller of a vehicle may seek to obtain one or more target views of the vehicle. To obtain the target views, the user may use a smartphone with an app that accesses the video stream while the user walks around or inside the vehicle. When the app identifies a frame in the video stream as one of the target views sought, the app tags the frame as an image of one of the target views. Further, the user may provide additional input, such as voice input (as part of the video stream) or manual taps on the touchscreen of the smartphone. The additional input may be used for damage assessment or sentiment analysis of the vehicle.

Claims (105)

1. A mobile device comprising:

a camera configured to generate a video stream;

a display;

a memory configured to store indicators for a plurality of predetermined views of a vehicle; and

a processor in communication with the camera, the display, and the memory, the processor configured to:

access the video stream;

determine, for a frame of the video stream, whether a specific predetermined view from the plurality of predetermined views, is present in the frame of the video stream;

responsive to determining that the specific predetermined view is present in the frame of the video stream:

identify the frame as an image of the specific predetermined view;

output an indication that the image of the specific predetermined view has been obtained;

responsive to identifying the frame as the image of the specific predetermined view, analyze sensor data indicative of a position of the mobile device relative to the vehicle in order to determine whether the mobile device has moved closer to the vehicle since the camera generated the frame; and

responsive to determining that the mobile device has moved closer to the vehicle, determine, based on the determined specific predetermined view, at least one aspect of damage for the specific predetermined view.

2. The mobile device of claim 1 , wherein the processor is configured to determine, based on the determined specific predetermined view, the at least one aspect of the damage for the specific predetermined view by:

accessing a damage model that correlates the determined specific predetermined view to potential types of damage; and

determining a type of damage using the accessed damage model and selected from the potential types of damage.

3. The mobile device of claim 1 , wherein the processor is further configured to receive an indication of damage from a user of the mobile device; and

wherein the processor is configured to determine the at least one aspect of the damage for the specific predetermined view based on both the specific predetermined view and the indication of damage from the user of the mobile device.

4. The mobile device of claim 3 , wherein the indication of damage from the user of the mobile device comprises an indication of a location of damage in the specific predetermined view; and

wherein the processor is configured to determine the at least one aspect of the damage for the specific predetermined view based on the specific predetermined view by determining a type of damage at the location of damage in the specific predetermined view.

5. The mobile device of claim 1 , wherein the specific predetermined view comprises a first specific predetermined view;

wherein the processor is further configured, after identifying the frame as the image of the first specific predetermined view but before identifying another frame as an image of a second specific predetermined view, to receive damage input from a user of the mobile device; and

wherein the processor is configured, responsive to receiving the damage input, to assess damage, based on the damage input, to the first specific predetermined view.

6. The mobile device of claim 5 , wherein the damage input comprises input indicative of location of damage and input indicative of type of damage; and

wherein the processor is configured to assess the location of damage in the specific predetermined view based on the input indicative of the location of damage and the type of damage in the specific predetermined view based on the input indicative of type of damage.

7. The mobile device of claim 1 , wherein the processor is configured to determine, based on the determined specific predetermined view, the at least one aspect of damage for the specific predetermined view by:

responsive to determining that the specific predetermined view is present in the frame of the video stream:

generating a list of potential options for damage for the specific predetermined view; and

receiving user input of at least one potential option in the list of potential options.

8. The mobile device of claim 7 , wherein the processor is configured to:

automatically analyze the frame of the video stream in which the specific predetermined view is present;

reduce, using the automatic analysis of the frame of the video stream, a potential list of options for damage in the frame of the video stream to generate a reduced list of one or more reduced options to present to a user;

present, on the display, the reduced list of one or more reduced options to the user; and

receive input from the user for at least one of the one or more reduced options.

9. The mobile device of claim 8 , wherein the processor is configured to reduce the potential list of options to generate a reduced list of at least two reduced options to present to the user; and

wherein the processor is configured to receive input as to one of the at least two reduced options presented to the user.

10. The mobile device of claim 8 , wherein the processor is configured to automatically analyze the frame of the video stream using machine learning.

11. The mobile device of claim 1 , further comprising a motion sensor configured to generate motion sensor data;

wherein the processor is further configured to enter damage input mode responsive to analyzing the motion sensor data generated by the motion sensor resident on the mobile device;

wherein processor is configured to determine the at least one aspect of damage responsive to entering damage input mode;

wherein the processor is further configured to identify a frame as an image of another predetermined view separate from the specified predetermined view; and

wherein, responsive to identifying the frame as the image of the another predetermined view separate from the specified predetermined view, the processor is further configured to exit the damage input mode.

12. The mobile device of claim 1 , wherein the processor is configured to determine, based on the determined specific predetermined view, the at least one aspect of damage for the specific predetermined view by:

generating on the display a first-level menu indicative of a list of vehicle parts for the specific predetermined view;

receiving a selection of one of the list of vehicle parts for the specific predetermined view;

generating on the display a second-level menu of a types of damage associated with the one of the list of vehicle parts selected; and

receiving a selection of one of the types of damage associated with the one of the list of vehicle parts selected.

13. The mobile device of claim 1 , further comprising a sensor configured to generate the sensor data indicative of the position of the mobile device relative to the vehicle.

14. The mobile device of claim 13 , wherein the sensor comprises an accelerometer configured to generate the sensor data indicative of the position of the mobile device relative to the vehicle;

wherein, responsive to identifying the frame as the image of the specific predetermined view, the processor is configured to analyze the sensor data generated by the accelerometer in order to determine whether the mobile device has moved closer to the vehicle;

wherein, responsive to determining that the mobile device has moved closer to the vehicle, the processor is further configured to enter damage input mode; and

wherein, responsive to entering the damage input mode, the processor is configured to determine the at least one aspect of damage.

15. The mobile device of claim 1 , wherein the processor is configured to analyze the sensor data by analyzing the video stream generated by the camera.

16. The mobile device of claim 1 , wherein the processor is further configured to enter damage input mode with regard to the specific predetermined view responsive to determining that the mobile device has moved closer to the vehicle since the camera generated the frame.

17. A mobile device comprising:

a camera configured to generate a video stream;

a display;

a memory configured to store indicators for a plurality of predetermined views of a vehicle; and

a processor in communication with the camera, the display, and the memory, the processor configured to:

access the video stream;

determine, for a frame of the video stream, whether a specific predetermined view from the plurality of predetermined views, is present in the frame of the video stream;

responsive to determining that the specific predetermined view is present in the frame of the video stream:

identify the frame as an image of the specific predetermined view;

output an indication that the image of the specific predetermined view has been obtained; and

determine, based on the determined specific predetermined view, at least one aspect of damage for the specific predetermined view by:

determining whether the specific predetermined view comprises an exterior predetermined view or an interior predetermined view;

responsive to determining that the specific predetermined view comprises an exterior predetermined view, analyzing the image of the specific predetermined view limited to damage for an exterior of the vehicle; and

responsive to determining that the specific predetermined view comprises an interior predetermined view, analyzing the image of the specific predetermined view limited to damage for an interior of the vehicle.

18. The mobile device of claim 17 , wherein the processor is configured to analyze the image of the specific predetermined view limited to damage for an exterior of the vehicle by:

accessing a plurality of potential types of damage to the exterior of the vehicle; and

analyzing the image in order to reduce the plurality of potential types of damage to the exterior of the vehicle.

19. The mobile device of claim 18 , wherein the processor is configured to analyze the image in order to reduce the plurality of potential types of damage to the exterior of the vehicle by analyzing the image to reduce a number of potential types of damage to the exterior of the vehicle to be less than the plurality and to be at least two potential types of damage; and

wherein the processor is further configured:

present to a user of the mobile device the at least two potential types of damage for the user to select from; and

receive a selection from the user from the at least two potential types of damage.

20. A mobile device comprising:

a camera configured to generate a video stream;

a display;

a memory configured to store indicators for a plurality of predetermined views of a vehicle; and

a processor in communication with the camera, the display, and the memory, the processor configured to:

access the video stream;

determine, for a frame of the video stream, whether a specific predetermined view from the plurality of predetermined views, is present in the frame of the video stream;

responsive to determining that the specific predetermined view is present in the frame of the video stream:

identify the frame as an image of the specific predetermined view;

output an indication that the image of the specific predetermined view has been obtained; and

determine, based on the determined specific predetermined view, at least one aspect of damage for the specific predetermined view by:

receiving, via the display, a first user input from a user indicative of a location of damage in the specific predetermined view;

identifying a portion of the frame that is at or around the location of damage manually input from the user;

receiving a second user input from the user indicative of a type of damage; and

automatically analyzing at least the portion of the frame in order to identify the type of damage in the specific predetermined view,

wherein automatically analyzing the at least the portion of the frame is configured to output at least two options of different types of damage and wherein the second user input selects the type of damage from the at least two options of different types of damage that is output; or

wherein automatically analyzing the at least the portion of the frame is configured to use both of the first user input and the second user input in identifying the type of damage in the specific predetermined view.

21. The mobile device of claim 20 , wherein the specific predetermined view is segmented into a plurality of different sections;

wherein the first user input comprises a user tap on the display;

wherein the processor is configured, based on the user tap, to identify one of the plurality of different sections of the specific predetermined view; and

wherein the processor is configured to analyze the one of the plurality of different sections of the specific predetermined view to determine the type of damage in the specific predetermined view.

22. The mobile device of claim 20 , wherein the first user input comprises a user tap on the display, the user tap indicative of spatial coordinates; and

wherein the processor is configured to analyze the frame using the spatial coordinates to determine the type of damage in the specific predetermined view.

23. The mobile device of claim 20 , wherein the first user input comprises a user tap on the display; and

wherein the processor is configured to analyze, using the location of damage in the specific predetermined view, the frame to determine a type of damage in the specific predetermined view by:

analyzing potential damage at the location associated with the user tap in order to generate, from a plurality of potential types of damage, a narrower set of potential types of damage in order to present to the user for selection.

24. The mobile device of claim 20 , wherein the second user input comprises an audio input.

25. The mobile device of claim 24 , wherein the audio input is indicative of a user's assessment of the damage; and

wherein the automatic analysis is configured to use both of the first user input and the audio input in order to confirm or dispute the user's assessment of the damage.

26. The mobile device of claim 20 , wherein the first user input comprises a tap on the display indicative of the location of the damage in the specific predetermined view; and

wherein the second user input comprises a user selection of the type of damage from the at least two options of different types of damage that is output on the display.

Assignments (3)
SECURITY INTEREST Recorded Apr 19, 2024
From: NTH GEN SOFTWARE INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 067169/0745 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 15, 2024
From: NTHGEN SOFTWARE INC.
To: NTH GEN SOFTWARE INC.
Reel/Frame 067108/0439 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 1, 2018
From: ENDRAS, MARK; ZHANG, LI; ZHANG, YUNLEI; PORTMAN, NATALIYA; NURLYBAYEV, AKBAR
To: NTHGEN SOFTWARE INC.
Reel/Frame 047382/0622 →
Continuity (2)
Provisional Application 62647457 · Mar 23, 2018
Related Publication 20190294878A1 · Sep 26, 2019
Cited By (1)
US 12,236,733