IP Library › Granted Patent US 11,557,030
Granted Patent B2
US 11,557,030 · App. 15/733,875 · Granted Jan 17, 2023

Device, method, and system for displaying a combined image representing a position of sensor having defect and a vehicle

Inventors: Kota Yonezawa (Kanagawa, JP); Satoshi Nakayama (Kanagawa, JP)
Assignee: SONY SEMICONDUCTOR SOLUTIONS CORPORATION
G06T7/0008B60K35/00G06T5/50G06T7/73G06T7/97G06T19/006B60K2370/566G06T2207/30252G06T2219/004
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,557,030
App. No.
15/733,875
Granted
Jan 17, 2023
Kind
B2
Abstract

The present technology relates to an information processing device, an information processing method, and an information processing system by which a user can easily recognize a position of a sensor having a defect from among sensors mounted on a vehicle. Position-related information about a relative position or direction with respect to a vehicle is acquired, and a combined image obtained by combining a defect image representing a position of the sensor having a defect from among the sensors mounted on the vehicle with a vehicle image reflecting the vehicle is displayed in response to the position-related information. The present technology can be applied so that a user can recognize a sensor having a defect from among the sensors mounted on a vehicle.

Claims (63)

1. An information processing device, comprising:

a display screen; and

a processor configured to:

acquire position-related information about one of a relative position of the information processing device with respect to a vehicle or a direction of the information processing device with respect to the vehicle;

generate a defect image representing a position of a sensor having a defect from among a plurality of sensors that is mounted on the vehicle, wherein

the generation of the defect image is based on the position-related information;

combine the defect image and a vehicle image to obtain a combined image,

wherein the vehicle image displays the vehicle;

control the display screen to display the combined image;

control the display screen to switch among an augmented reality (AR) display, a bird's eye view display, and a virtual reality (VR) display for the display of the combined image;

control the display screen to perform the bird's eye view display in a case where the sensor having the defect is not viewed at a position of the information processing device; and

control the display screen to perform one of the VR display or the AR display in a case where the sensor having the defect is viewed at the position of the information processing device.

2. The information processing device according to claim 1 , wherein

the vehicle image further displays the vehicle from a viewpoint overlooking the vehicle, and

the processor is further configured to combine the defect image with the vehicle image to generate the bird's eye view display.

3. The information processing device according to claim 1 , wherein

the vehicle image further displays the vehicle viewed at a position of the information processing device, and

the processor is further configured to combine the defect image with the vehicle image to generate the VR display.

4. The information processing device according to claim 3 , wherein

the vehicle image is computer graphics (CG) of the vehicle viewed at the position of the information processing device.

5. The information processing device according to claim 1 , wherein

the vehicle image further displays the vehicle from a viewpoint overlooking the vehicle,

the bird's eye view display combines the defect image with the vehicle image,

the (VR) VR display combines the defect image with computer graphics (CG) of the vehicle viewed at a position of the information processing device, the AR display combines the defect image with a captured image from a camera at the position of the information processing device, and

the captured image corresponds to the vehicle image.

6. The information processing device according to claim 1 , further comprising a camera configured to capture an image, wherein

the processor is further configured to control the display screen to perform one of the VR display or the AR display based on a brightness of the captured image.

7. The information processing device according to claim 1 , wherein

the vehicle causes at least one light-emitting part associated with the sensor having the defect to be turned on.

8. The information processing device according to claim 7 , wherein

the vehicle causes the at least one light-emitting part to be turned off based on elimination of the defect of the sensor.

9. The information processing device according to claim 7 , wherein

the sensor includes the at least one light-emitting part.

10. The information processing device according to claim 1 , wherein

the sensor is a camera.

11. The information processing device according to claim 1 , wherein

the defect is dirt on the sensor.

12. The information processing device according to claim 11 , wherein

the defect image includes an image representing a position of the dirt on the sensor having the dirt.

13. The information processing device according to claim 1 , wherein

the information processing device is a portable terminal.

14. An information processing method, comprising:

acquiring position-related information about one of a relative position of an information processing device or a direction of the information processing device with respect to a vehicle;

generating a defect image representing a position of a sensor having a defect among a plurality of sensors that is mounted on the vehicle, wherein

the generation of the defect image is based on the position-related information;

combining the defect image and a vehicle image to obtain a combined image, wherein the vehicle image displays the vehicle;

controlling a display screen to display the combined image;

controlling the display screen to switch among an augmented reality (AR) display, a bird's eye view display, and a virtual reality (VR) display for the display of the combined image;

controlling the display screen to perform the bird's eye view display in a case where the sensor having the defect is not viewed at a position of the information processing device; and

controlling the display screen to perform one of the VR display or the AR display in a case where the sensor having the defect is viewed at the position of the information processing device.

15. An information processing system, comprising:

a sensor configured to be mounted on a vehicle;

a first processor configured to detect a defect of the sensor;

a display screen configured to display an image;

a second processor configured to:

acquire position-related information about one of a relative position of an information processing device with respect to the vehicle or a direction of the information processing device with respect to the vehicle;

generate a defect image representing a position of the sensor having the defect, wherein the generation of the defect image is based on the position-related information;

combine the defect image and a vehicle image to obtain a combined image,

wherein the vehicle image displays the vehicle;

control the display screen to display the combined image;

control the display screen to switch among an augmented reality (AR) display, a bird's eye view display, and a virtual reality (VR) display for the display of the combined image;

control the display screen to perform the bird's eye view display in a case where the sensor having the defect is not viewed at a position of the information processing device; and

control the display screen to perform one of the VR display or the AR display in a case where the sensor having the defect is viewed at the position of the information processing device.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 4, 2021
From: YONEZAWA, KOTA; NAKAYAMA, SATOSHI
To: SONY SEMICONDUCTOR SOLUTIONS CORPORATION
Reel/Frame 057693/0612 →
Priority Claims (1)
JP JP2018-109523 · Jun 7, 2018 · national
Continuity (1)
Related Publication 20210224976A1 · Jul 22, 2021
Cited By (1)
US 1,063,958