Granted Patent B2

US 12,485,756 · App. 18/229,356 · Granted Dec 2, 2025

Inventor: Danghee Park (Seoul, KR)

Assignees: Hyundai Motor Company; Kia Corporation

B60K35/29G06V20/59G06V40/10G06V40/28B60K35/10B60K2360/115B60K2360/1464B60K2360/148B60K2360/191

Abstract

Claims (61)

1 . A vehicle comprising:

a microphone provided inside the vehicle and configured to receive a voice uttered by an occupant of the vehicle;

a camera configured to obtain an image inside the vehicle;

a display device configured to display a menu screen comprising a plurality of menu items; and

a controller configured to control the display device,

wherein the controller is further configured to:

recognize, based on the image obtained via the camera, a hand gesture of the occupant;

determine, based on the hand gesture, a menu item, of the plurality of menu items on the menu screen, pointed to by the occupant; and

execute, based on the voice uttered by the occupant, a command corresponding to the pointed menu item.

2 . The vehicle of claim 1 , wherein the display device comprises at least one of:

a display provided inside the vehicle and configured to display the menu screen, or

an image projector configured to project the menu screen onto a space or an interior surface in the vehicle.

3 . The vehicle of claim 1 , wherein the controller is further configured to, based on a determination that the hand gesture is a screen display motion for displaying the menu screen, control the display device to display the menu screen.

4 . The vehicle of claim 1 , wherein the controller is configured to determine the menu item by:

determining a position, on the menu screen, that is pointed to by the hand gesture; and

determining the menu item, among the plurality of menu items, based on the determined position.

5 . The vehicle of claim 4 , wherein the controller is configured to determine the position by:

estimating a skeletal structure of a hand of the occupant by applying feature points of the hand of the occupant to a skeletal model; and

determining the position based on the estimated skeletal structure of the hand of the occupant.

6 . The vehicle of claim 5 , wherein the controller is configured to determine the position by:

generating an extension line extending from a fingertip of the occupant to the menu screen based on the estimated skeletal structure of the hand of the occupant; and

determining the position further based on an intersection between the extension line and a plane on which the menu screen is located.

7 . The vehicle of claim 1 , wherein the controller is further configured to enlarge or highlight the menu item, pointed to by the occupant, relative to other menu items of the plurality of menu items.

8 . The vehicle of claim 7 , wherein the controller is further configured to, based on the menu item being enlarged or highlighted, activate the microphone.

9 . The vehicle of claim 1 , wherein the microphone comprises:

a first microphone configured to receive a first voice signal inside the vehicle; and

a second microphone directed at the occupant to receive a second voice signal associated with the occupant, and

wherein the controller is configured to execute the command by:

executing the command based on a sound pressure level difference, between the first voice signal received via the first microphone and the second voice signal received via the second microphone, being greater than a threshold sound pressure level.

10 . The vehicle of claim 1 , wherein the microphone comprises a single microphone, and

wherein the controller is configured to execute the command based on at least one of:

a sound pressure level difference, between a first voice signal corresponding to the voice uttered by the occupant and a second voice signal received before the occupant utters the voice, being greater than a threshold sound pressure level, or

an amplitude difference, between the first voice signal corresponding to the voice uttered by the occupant and the second voice signal received before the occupant utters the voice, being greater than a threshold amplitude.

11 . A method comprising:

obtaining, via a camera of a vehicle, an image inside the vehicle, wherein the image obtained via the camera comprises an image of an occupant of the vehicle;

extracting, based on the image obtained via the camera, a hand region of the occupant of the vehicle;

recognizing, based on the hand region, a hand gesture of the occupant of the vehicle;

determining, based on the hand gesture, a menu item, of a plurality of menu items on a menu screen displayed on a display device of the vehicle, pointed to by the occupant;

receiving a voice uttered by the occupant; and

executing, based on the voice, a command corresponding to the pointed menu item.

12 . The method of claim 11 , further comprising causing the display device to perform at least one of:

displaying the menu screen on a display provided in the vehicle, or

displaying the menu screen onto a space in the vehicle through an image projector provided in the vehicle.

13 . The method of claim 11 , further comprising, based on a determination that the hand gesture is a screen display motion for displaying the menu screen, controlling the display device to display the menu screen.

14 . The method of claim 11 , wherein the determining of the menu item comprises:

determining a position, on the menu screen, that is pointed to by the hand gesture; and

determining the menu item, among the plurality of menu items, based on the determined position.

15 . The method of claim 14 , wherein the determining of the position comprises estimating a skeletal structure of a hand of the occupant by applying feature points of the hand of the occupant to a skeletal model; and

determining the position based on the estimated skeletal structure of the hand of the occupant.

16 . The method of claim 15 , wherein the determining of the position comprises:

generating an extension line extending from a fingertip of the occupant to the menu screen based on the estimated skeletal structure of the hand of the occupant; and

determining the position further based on an intersection between the extension line and a plane on which the menu screen is located.

17 . The method of claim 11 , further comprising enlarging or highlighting the menu item, pointed to by the occupant, relative to other menu items of the plurality of menu items.

18 . The method of claim 17 , further comprising, based on the menu item being enlarged or highlighted, activating a microphone in the vehicle.

19 . The method of claim 11 , wherein the receiving of the voice uttered by the occupant comprises:

receiving a first voice signal inside the vehicle via a first microphone; and

receiving a second voice signal associated with the occupant via a second microphone directed at the occupant, and

wherein the executing of the command comprises executing the command based on a sound pressure level difference, between the first voice signal received via the first microphone and the second voice signal received via the second microphone, being greater than a threshold sound pressure level.

20 . The method of claim 11 , wherein the executing of the command comprises executing the command based on at least one of:

a sound pressure level difference, between a first voice signal corresponding to the voice uttered by the occupant and a second voice signal received before the occupant utters the voice, being greater than a threshold sound pressure level, or

an amplitude difference, between the first voice signal corresponding to the voice uttered by the occupant and the second voice signal received before the occupant utters the voice, being greater than a threshold amplitude.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (13)