Computer-implemented method of alerting driver of vehicle, apparatus for alerting driver of vehicle, vehicle, and computer-program product

1. A computer-implemented method of alerting a driver of a vehicle, comprising:

obtaining a plurality of consecutive input images during a time interval using a three-dimensional depth camera, a respective one of the plurality of consecutive input images comprising a user sub-image and an object sub-image;

deriving three-dimensional coordinates of user joint points in the user sub-image of the respective one of the plurality of consecutive input images using a first classifier;

deriving three-dimensional coordinates of object feature points in the object sub-image of the respective one of the plurality of consecutive input images using the first classifier;

classifying user posture in the respective one of the plurality of consecutive input images by analyzing the three-dimensional coordinates of user joint points and the three-dimensional coordinates of object feature points using a second classifier;

determining presence or absence of a gaze position by using three-dimensional coordinates of a sub-set of the user joint points comprising a right shoulder joint point and a left shoulder joint point to define a sub-region and searching for the gaze position in the sub-region; and

generating a driver alert signal based on one or a combination of (1) classification of the user posture and (2) the presence or absence of the gaze position.

2. The computer-implemented method of claim 1 , wherein classifying the user posture comprises classifying the user posture into a first type and a second type; and

wherein generating the driver alert signal is based on a determination that a first percentage of the plurality of consecutive input images obtained during the time interval determined to be the second type is greater than a first threshold value.

3. The computer-implemented method of claim 1 , wherein generating the driver alert signal is based on a determination that the gaze position is absent in a second percentage of the plurality of consecutive input images obtained during the time interval, the second percentage being greater than a second threshold value.

4. The computer-implemented method of claim 1 , wherein classifying the user posture comprises classifying the user posture into a first type and a second type;

generating the driver alert signal is based on a determination that (1) a first percentage of the plurality of consecutive input images obtained during the time interval determined to be the second type is greater than a first threshold value; and (2) the gaze position is absent in a second percentage of the plurality of consecutive input images obtained during the time interval, the second percentage being greater than a second threshold value.

5. The computer-implemented method of claim 1 , wherein a boundary of the sub-region is defined by a line connecting the left shoulder joint point and right shoulder joint point of the user sub-image; and

a height of the sub-region is in a range of 0.6 times to 1.2 times of a length of the line connecting the left shoulder joint point and right shoulder joint point of the user sub-image.

6. The computer-implemented method of claim 1 , further comprising pre-training a first initial classifier to obtain the first classifier;

wherein pre-training the first initial classifier comprises:

inputting a plurality of first training images into the first initial classifier, a respective one of the plurality of first training images comprising a user sub-image and an object sub-image;

inputting training three-dimensional coordinates of user joint points of user sub-images of the plurality of first training images into the first initial classifier;

inputting training three-dimensional coordinates of object feature points of object sub-images of the plurality of first training images into the first initial classifier;

deriving three-dimensional coordinates of user joint points in the user sub-images of the plurality of first training images using the first initial classifier;

deriving three-dimensional coordinates of object feature points in the object sub-images of the plurality of first training images using the first initial classifier;

determining a first degree of match between the training three-dimensional coordinates of the user joint points and the three-dimensional coordinates of the user joint points determined by the first initial classifier;

determining a second degree of match between the training three-dimensional coordinates of the object feature points and the three-dimensional coordinates of the object feature points determined by the first initial classifier; and

tuning the first initial classifier based on the first degree of match and the second degree of match.

7. The computer-implemented method of claim 1 , further comprising pre-training a second initial classifier to obtain the second classifier;

wherein pre-training the second initial classifier comprises:

inputting a plurality of second training images into the second initial classifier, a respective one of the plurality of second training images comprising a user sub-image and an object sub-image;

inputting training classified types respectively for the plurality of second training images into the second initial classifier, the classified types comprising a first type and a second type;

inputting training three-dimensional coordinates of user joint points of user sub-images of the plurality of second training images into the second initial classifier;

inputting training three-dimensional coordinates of object feature points of object sub-images of the plurality of second training images into the second initial classifier;

classifying user postures in the plurality of second training images by analyzing the training three-dimensional coordinates of the user joint points and the three-dimensional coordinates of object feature points using the second initial classifier;

determining a degree of match between the training classified types and results of classifying the user postures using the second initial classifier; and

tuning the second initial classifier based on the degree of match.

8. The computer-implemented method of claim 1 , wherein the first classifier is selected from a group consisting of a random forest classifier, a convolutional neural network classifier, an Adaboost classifier, and an SVM classifier.

9. The computer-implemented method of claim 1 , wherein the second classifier is selected from a group consisting of a random forest classifier, a convolutional neural network classifier, Adaboost classifier, and an SVM classifier.

10. The computer-implemented method of claim 1 , wherein the user sub-image comprises an image of an upper body of the driver and the object sub-image comprises an image of a driving wheel of the vehicle.

11. An apparatus for alerting a driver of a vehicle, comprising:

an image capturing device configured to obtaining a plurality of consecutive input images during a time interval using a three-dimensional depth camera, a respective one of the plurality of consecutive input images comprising a user sub-image and an object sub-image;

a memory;

one or more processors;

wherein the memory and the one or more processors are connected with each other; and

the memory stores computer-executable instructions for controlling the one or more processors to:

derive three-dimensional coordinates of user joint points in the user sub-image of the respective one of the plurality of consecutive input images using a first classifier;

derive three-dimensional coordinates of object feature points in the object sub-image of the respective one of the plurality of consecutive input images using the first classifier;

classify user posture in the respective one of the plurality of consecutive input images by analyzing the three-dimensional coordinates of user joint points and the three-dimensional coordinates of object feature points using a second classifier;

determine presence or absence of a gaze position by using three-dimensional coordinates of a sub-set of the user joint points from a group consisting of a head feature point, a right shoulder joint point, a left shoulder joint point to define a sub-region and searching for the gaze position in the sub-region; and

generate a driver alert signal based on one or a combination of (1) classification of the user posture and (2) the presence or absence of the gaze position.

12. The apparatus of claim 11 , wherein the memory stores computer-executable instructions for controlling the one or more processors to classify the user posture into a first type and a second type; and

generate the driver alert signal is based on a determination that a first percentage of the plurality of consecutive input images obtained during the time interval determined to be the second type is greater than a first threshold value.

13. The apparatus of claim 11 , wherein the memory stores computer-executable instructions for controlling the one or more processors to generate the driver alert signal based on a determination that the gaze position is absent in a second percentage of the plurality of consecutive input images obtained during the time interval, the second percentage being greater than a second threshold value.

14. The apparatus of claim 11 , wherein the memory stores computer-executable instructions for controlling the one or more processors to classify the user posture into a first type and a second type; and

generate the driver alert signal is based on a determination that (1) a first percentage of the plurality of consecutive input images obtained during the time interval determined to be the second type is greater than a first threshold value; and (2) the gaze position is absent in a second percentage of the plurality of consecutive input images obtained during the time interval, the second percentage being greater than a second threshold value.

15. The apparatus of claim 11 , wherein a boundary of the sub-region is defined by a line connecting the left shoulder joint point and right shoulder joint point of the user sub-image; and

a height of the sub-region is in a range of 0.6 times to 1.2 times of a length of the line connecting the left shoulder joint point and right shoulder joint point of the user sub-image.

16. The apparatus of claim 11 , wherein the first classifier is selected from a group consisting of a random forest classifier, a convolutional neural network classifier, an Adaboost classifier, and an SVM classifier.

17. The apparatus of claim 11 , wherein the second classifier is selected from a group consisting of a random forest classifier, a convolutional neural network classifier, an Adaboost classifier, and an SVM classifier.

18. The apparatus of claim 11 , wherein the user sub-image comprises an image of an upper body of the driver and the object sub-image comprises an image of a driving wheel of the vehicle.

19. A vehicle, comprising the apparatus of claim 11 .

20. A computer-program product comprising a non-transitory tangible computer-readable medium having computer-readable instructions thereon, the computer-readable instructions being executable by a processor to cause the processor to perform:

deriving three-dimensional coordinates of user joint points in a user sub-image of a respective one of a plurality of consecutive input images using a first classifier, the plurality of consecutive input images being obtained during a time interval using a three-dimensional depth camera, the respective one of the plurality of consecutive input images comprising the user sub-image and an object sub-image;

deriving three-dimensional coordinates of object feature points in the object sub-image of the respective one of the plurality of consecutive input images using the first classifier;

classifying user posture in the respective one of the plurality of consecutive input images by analyzing the three-dimensional coordinates of user joint points and the three-dimensional coordinates of object feature points using a second classifier;

determining presence or absence of a gaze position by using three-dimensional coordinates of a sub-set of the user joint points comprising a right shoulder joint point and a left shoulder joint point to define a sub-region and searching for the gaze position in the sub-region; and

generating a driver alert signal based on one or a combination of (1) classification of the user posture and (2) the presence or absence of the gaze position.