Context embedding for capturing image dynamics
Systems and methods capture image dynamics and use those captured image dynamics for image feature recognition and classification. Other methods and systems train a neural network to capture image dynamics. An image vector representing image dynamics is extracted from an image of an image stream using a first neural network. A second neural network, predicts a previous and/or subsequent image in the image stream from the image vector. The predicted previous and/or subsequent image is compared with an actual previous and/or subsequent image from the image stream. The first and second neural networks are trained using the result of the comparison.
1. A method for training a neural network to generate image vectors representing image dynamics, the method comprising:
a) generating an image vector representing image dynamics from a single image from an image stream using only the single image as an image input to a first neural network;
b) predicting a previous image and a subsequent image in the image stream from the image vector using a second neural network;
c) comparing the predicted previous image and subsequent image with an actual previous and/or subsequent image from the image stream;
d1) training the first and second neural networks using a result of the comparison; and
e) using the image vector from the fully trained first neural network to train a third neural network to recognize and classify features from image dynamics and classifying features in the single image from the image dynamics.
2. The method of claim 1 further comprising:
d2) repeating a)-d1) until the first and second neural networks are fully trained before using the image vector from the fully trained first neural network to train the third neural network.
3. The method of claim 1 wherein the image vector is a 512 dimensional floating point number.
4. The method of claim 1 wherein the single image and the image stream are medical images.
5. The method of claim 4 wherein the medical images are from specialized medical imaging devices.
6. The method of claim 1 wherein the single image has certain areas selected for extraction of the image vector and the predicting the previous image and the subsequent image.
7. The method of claim 1 wherein the single image and the stream of images have undergone video encoding before being provided to the first neural network.
8. The method of claim 1 wherein the single image dynamic is a small change within the single image.
9. The method of claim 8 wherein the small change is a change in facial expression.
10. A system for training a neural network for recognition of image dynamic, comprising:
a processor;
a memory;
instructions embodied within the memory which when executed cause the processor to carry out a method for training a neural network to generate image vectors representing image dynamics, the method comprising:
a) generating an image vector representing image dynamics from an image from a single image stream using only the single image as an image input to a first neural network;
b) predicting a previous image and a subsequent image in the image stream from the image vector using a second neural network;
c) comparing the predicted previous image and subsequent image with an actual previous and/or subsequent image from the image stream;
d) training the first and second neural networks using a result of the comparison; and
e) using the image vector from the fully trained first neural network to train a third neural network to recognize and classify features from image dynamics and classifying features in the single image from the image dynamics.
11. A system for recognition of image dynamics comprising:
a processor;
a memory;
the fully trained first neural network from claim 1 embodied in a non-transitory medium and configured to generate image vectors representing image dynamics using only a single image taken from an image stream as an image input; and
the fully trained third neural network from claim 1 configured to recognize and classify features from image dynamics.
12. A method for recognition of image dynamics:
a) predicting image vectors representing image dynamics of a single image using only the single image as an image input to the fully trained first neural network from claim 1 ; and
b) recognizing and classifying image features from the image dynamics represented by image vectors using the fully trained third neural network from claim 1 configured to recognize and classify features from the image dynamics.
13. The system of claim 12 wherein the image vector is a 512-bit floating point number.
14. The system of claim 12 wherein the single image is a medical image.
15. The system of claim 14 wherein the medical image is from a specialized medical imaging device.
16. The system of claim 12 wherein the single image has certain areas selected for extraction of the image vector and the predicting the previous image and the subsequent image.
17. The system of claim 12 wherein the single image has undergone video encoding before being provided to the first neural network.
18. The system of claim 12 wherein the third neural network is further configured to recognize and classify expressions from the single image.
19. The system of claim 12 wherein the third neural network is further configured to recognize and classify medical injuries from the single image.