Learning device, estimating device, estimating system, learning method, estimating method, and storage medium to estimate a state of vehicle-occupant with respect to vehicle equipment

1. A learning device configured to form a model to be used in an estimating device configured to estimate a state of a vehicle-occupant with respect to equipment of a vehicle, the learning device comprising:

a memory storing instructions; and

a processor, when executing the instructions stored in the memory, that performs operations including:

obtaining training data for a deep-learning that uses a neural network, the training data including an image and two labels to be learned in the deep-learning attached to the image, the image including a vicinity of a steering wheel, one of the two labels attached to the image being first information indicating two dimensional coordinates of a skeleton location of a hand of the vehicle-occupant in the image, another of the two labels attached to the image being second information including “ON” or “OFF” indicating the state of the vehicle-occupant with respect to the steering wheel, the “ON” indicating that the hand of the vehicle-occupant touches the steering wheel and the “OFF” indicating that the hand of the vehicle-occupant is separated from the steering wheel; and

forming the model through the deep-learning that uses the neural network such that the model learns and outputs both of the two labels of the first information and the second information attached to the image including the vicinity of the steering wheel.

2. An estimating device configured to estimate a state of a vehicle-occupant with respect to equipment of a vehicle, the estimating device comprising:

a storage configured to store the model formed by the learning device as defined in claim 1 ;

a memory storing instructions; and

a processor, when executing the instructions stored in the memory of the estimating device, that performs operations including:

obtaining an image including a vicinity of the steering wheel,

estimating the state of the vehicle-occupant based on the obtained image including the vicinity of the steering wheel and the model, and

outputting the two labels of the first information and the second information as an estimating result.

3. The estimating device according to claim 2 , wherein the processor of the estimating device recognizes at least one of a steering wheel.

4. The estimating device according to claim 3 , wherein the processor of the estimating device further recognizes at least one of a touch panel of a car navigation system, a window, a door handle, a control panel for an air-conditioner, a rear-view mirror, a dashboard, a seat, an arm rest, a center compartment, and a glove compartment as the equipment.

5. An estimating system configured to estimate a state of a vehicle-occupant with respect to equipment of a vehicle, the estimating system comprising:

a camera configured to shoot an image including a vicinity of a steering wheel; and

the estimating device as defined in claim 3 ,

wherein the processor of the estimating device obtains the image including the vicinity of the steering wheel from the camera.

6. The estimating system according to claim 5 , wherein the camera is configured to shoot an image including the vicinity of the steering wheel disposed within a given region in an interior of the vehicle.

7. The learning device according to claim 1 , wherein the processor forms the model by:

inputting the image including a vicinity of a steering wheel in the training data to the model;

obtaining the two labels of the first information and the second information output by the model;

modifying and optimizing the model such that the two labels of the first information and the second information output by the model respectively match the two labels of the first information and the second information included in the training data.

8. A learning method for forming a model to be used in an estimating method for estimating a state of a vehicle-occupant with respect to equipment of a vehicle, the learning method comprising:

obtaining training data for a deep-learning that uses a neural network, the training data including an image and two labels to be learned in the deep-learning attached to the image, the image including a vicinity of a steering wheel, one of the two labels attached to the image being first information indicating two dimensional coordinates of a skeleton location of a hand of the vehicle-occupant in the image, another of the two labels attached to the image being second information including “ON” or “OFF” indicating a state of the vehicle-occupant with respect to the steering wheel, the “ON” indicating that the hand of the vehicle-occupant touches the steering wheel and the “OFF” indicating that the hand of the vehicle-occupant is separated from the steering wheel; and

forming the model through the deep-learning that uses the neural network such that both of the two labels of the first information and the second information attached to the image including the vicinity of the steering wheel are learned by and output from the model in the estimating method.

9. An estimating method for estimating a state of a vehicle-occupant with respect to equipment of a vehicle, the estimating method comprising:

obtaining an image including a vicinity of the steering wheel;

estimating the state of the vehicle-occupant with respect to the steering wheel by using the obtained image including the vicinity of the steering wheel and the model formed by the learning method as defined in claim 7 ; and

outputting two labels of the first information and the second information as an estimating result.

10. A learning-program storage medium storing a learning program to be executed by a computer of a learning device for forming a model to be used in an estimating device for estimating a state of a vehicle-occupant with respect to equipment of a vehicle, and the storage medium being a non-transitory storage medium,

wherein the learning program causes the computer to execute:

a process of obtaining training data for a deep-learning that uses a neural network, the training data including an image and two labels to be learned in the deep-learning attached to the image, the image including a vicinity of a steering wheel, one of the two labels attached to the image being first information indicating two dimensional coordinates of a skeleton location of a hand of the vehicle-occupant in the image, another of the two labels attached to the image being second information including “ON” or “OFF” indicating a state of the vehicle-occupant with respect to the steering wheel, the “ON” indicating that the hand of the vehicle-occupant touches the steering wheel and the “OFF” indicating that the hand of the vehicle-occupant is separated from the steering wheel, and

a process of forming the model through the deep-learning that uses the neural network such that the model learns and outputs both of the two labels of the first information and the second information attached to the image including the vicinity of the steering wheel.

11. An estimating-program storage medium for storing an estimating program to be executed by a computer of an estimating device for estimating a state of a vehicle-occupant with respect to equipment of a vehicle, and the storage medium being a non-transitory storage medium,

wherein the estimating program causes the computer to execute:

a process of obtaining an image including a vicinity of the steering wheel,

a process of estimating the state of the vehicle-occupant with respect to the hand by using the obtained image including the vicinity of the steering wheel and the model formed by the learning program stored in the learning-program storage medium as defined in claim 9 , and

a process of outputting the two labels, the two labels being the first information indicating two dimensional coordinates of the skeleton location of the hand of the vehicle-occupant and the second information including “ON” or “OFF” indicating the state of the vehicle-occupant with respect to the equipment steering wheel as an estimating result.