IP Library › Granted Patent US 11,769,056
Granted Patent B2
US 11,769,056 · App. 17/136,083 · Granted Sep 26, 2023

Synthetic data for neural network training using vectors

Inventors: Sandipan Banerjee (Boston, MA); Rana el Kaliouby (Milton, MA); Ajjen Das Joshi (Arlington, MA); Survi Kyal (Chestnut Hill, MA); Taniya Mishra (New York, NY)
Assignee: Affectiva, Inc.
G06N3/084G06F18/214G06N3/045G06N3/08G06V10/774G06V10/82G06V40/16G06V40/165G06V40/169G06V40/174G06V40/179
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,769,056
App. No.
17/136,083
Filed
Dec 29, 2020
Granted
Sep 26, 2023
Kind
B2
Art Unit
2666
USPC
382/156
Abstract

Machine learning is performed using synthetic data for neural network training using vectors. Facial images are obtained for a neural network training dataset. Facial elements from the facial images are encoded into vector representations of the facial elements. A generative adversarial network (GAN) generator is trained to provide one or more synthetic vectors based on the one or more vector representations, wherein the one or more synthetic vectors enable avoidance of discriminator detection in the GAN. The training a GAN further comprises determining a generator accuracy using the discriminator. The generator accuracy can enable a classifier, where the classifier comprises a multi-layer perceptron. Additional synthetic vectors are generated in the GAN, wherein the additional synthetic vectors avoid discriminator detection. A machine learning neural network is trained using the additional synthetic vectors. The training a machine learning neural network further includes using the one or more synthetic vectors.

Claims (42)

1. A computer-implemented method for machine learning comprising:

obtaining facial images for a neural network training dataset;

encoding facial elements from the facial images into one or more vector representations of the facial elements using a machine learning neural network;

training a generative adversarial network (GAN) generator to provide one or more synthetic vectors based on the one or more vector representations, wherein a feature selector selects values from the one or more vector representations, the values being used as an input to the GAN, and wherein the one or more synthetic vectors enable avoidance of discriminator detection in the GAN;

generating additional synthetic vectors in the GAN, wherein the additional synthetic vectors avoid discriminator detection; and

further training the machine learning neural network, using the additional synthetic vectors that were generated by the GAN based on the selected values of the feature selector.

2. The method of claim 1 wherein the training a machine learning neural network further comprises using the one or more synthetic vectors.

3. The method of claim 1 wherein the training a GAN further comprises determining a generator accuracy using the discriminator.

4. The method of claim 3 wherein the generator accuracy is a criterion for the generating additional synthetic vectors.

5. The method of claim 3 wherein the generator accuracy enables a classifier.

6. The method of claim 1 further comprising back-propagating an error function into the discriminator.

7. The method of claim 6 wherein the error function is determined on a real/fake basis.

8. The method of claim 1 wherein the facial elements comprise human drowsiness features.

9. The method of claim 8 wherein the human drowsiness features are distilled into n-element vectors.

10. The method of claim 9 wherein n is equal to a number of facial element features times a number of statistics for each feature.

11. The method of claim 10 wherein the number of facial element features is 18 and the number of statistics is 6, and thus n is equal to 108.

12. The method of claim 8 wherein the training a GAN and generating additional synthetic vectors enable a drowsiness detection neural network.

13. The method of claim 1 further comprising filtering the synthetic vectors by rejecting at least one training dataset vector generated by the GAN.

14. The method of claim 1 further comprising converting the additional synthetic vectors into image data.

15. The method of claim 14 further comprising augmenting the neural network training dataset using the image data converted from the additional synthetic vectors.

16. The method of claim 15 wherein filtering is performed before the augmenting.

17. The method of claim 1 wherein the GAN is seeded by a latent random variable.

18. The method of claim 17 wherein the latent random variable is targeted for a sparse facial expression within the training dataset.

19. The method of claim 18 wherein the sparse facial expression comprises a drowsiness state.

20. The method of claim 19 wherein the drowsiness state comprises one of a slightly drowsy state, a moderately drowsy state, or an extremely drowsy state.

21. The method of claim 19 wherein the drowsiness state comprises a non-alert state.

22. The method of claim 1 further comprising tagging the synthetic vectors with metadata.

23. The method of claim 22 further comprising propagating the metadata to a training dataset.

24. A computer-implemented method for machine learning comprising:

obtaining a facial expression training dataset for a first neural network implemented on a machine learning system;

identifying one or more sparse classes of facial expressions within the training dataset;

generating synthetic data to augment the sparse classes using at least one generative adversarial network (GAN), wherein a feature selector selects values from one or more vector representations, the values being used as an input to the GAN;

augmenting the training dataset using the synthetic data from the at least one GAN based on the selected values of the feature selector; and

training the machine learning system, using the training dataset that was augmented.

25. A computer system for machine learning comprising:

a memory which stores instructions;

one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to:

obtain facial images for a neural network training dataset;

encode facial elements from the facial images into one or more vector representations of the facial elements using a machine learning neural network;

train a generative adversarial network (GAN) generator to provide synthetic vectors based on the one or more vector representations, wherein a feature selector selects values from the one or more vector representations, the values being used as an input to the GAN, and wherein the synthetic vectors enable avoidance of discriminator detection in the GAN;

generate additional synthetic vectors in the GAN, wherein the additional synthetic vectors avoid discriminator detection; and

train the machine learning neural network, using the additional synthetic vectors that were generated by the GAN based on the selected values of the feature selector.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 19, 2021
From: BANERJEE, SANDIPAN; EL KALIOUBY, RANA; JOSHI, AJJEN DAS; KYAL, SURVI; MISHRA, TANIYA
To: AFFECTIVA, INC.
Reel/Frame 055649/0320 →
Continuity (6)
Provisional Application 63083136 · Sep 25, 2020
Provisional Application 63071401 · Aug 28, 2020
Provisional Application 62955493 · Dec 31, 2019
Provisional Application 62954819 · Dec 30, 2019
Provisional Application 62954833 · Dec 30, 2019
Related Publication 20210201003A1 · Jul 1, 2021