Virtual object machine learning
A machine learning scheme can be trained on a set of labeled training images of a subject in different poses, with different textures, and with different background environments. The label or marker data of the subject may be stored as metadata to a 3D model of the subject or rendered images of the subject. The machine learning scheme may be implemented as a supervised learning scheme that can automatically identify the labeled data to create a classification model. The classification model can classify a depicted subject in many different environments and arrangements (e.g., poses).
1. A method comprising:
storing, on a client device, a convolutional neural network trained to detect different gestures of an object, the convolutional neural network trained on a plurality of images of a three-dimensional model of the object, each of the plurality of images being a render of the three-dimensional model of the object in a different gesture;
generating, using an image sensor of the client device, an image depicting a physical object;
classifying, using the convolutional neural network, the physical object as being in a gesture by applying the convolution neural network to the generated image depicting the physical object;
displaying with the image, on a display device of the client device, additional content associated with the identified gesture of the physical object; and
publishing, to a network platform, the additional content and the image as an ephemeral message.
2. The method of claim 1 , wherein the additional content is overlay content associated with the identified gesture of the physical object in the image.
3. The method of claim 1 , wherein the convolutional neural network is trained on a network platform.
4. The method of claim 3 , further comprising:
receiving, from the network platform, the convolutional neural network that is trained on the plurality of images.
5. The method of claim 1 , wherein the plurality of images are rendered from a plurality of virtual cameras that view the three-dimensional model of the object from different perspectives.
6. The method of claim 1 , wherein the physical object is a human hand and the different gestures depicted in the plurality of images used to train the convolutional neural network depict different human hand gestures.
7. The method of claim 1 , wherein the physical object is a human head and the different gestures depicted in the plurality of images used to train the convolutional neural network depict different human head gestures.
8. The method of claim 1 , wherein the physical object is an animal body and the different gestures depicted in the plurality of images used to train the convolutional neural network depict different animal body gestures.
9. The method of claim 8 , wherein the object has a fur texture of an animal.
10. The method of claim 1 , wherein the object has a skin shade texture.
11. The method of claim 10 , wherein the plurality of images include a set of images depicting the three-dimensional model of the object with a light skin shade texture arranged in the different gestures.
12. The method of claim 11 , wherein the plurality of images include another set of images depicting the three-dimensional model of the object with a dark skin shade texture arranged in the different gestures.
13. A client device comprising:
one or more processors;
an image sensor;
a display device; and
a memory storing instructions that, when executed by the one or more processors, cause the client device to perform operations comprising:
store a convolutional neural network trained to detect different gestures of an object, the convolutional neural network trained on a plurality of images of a three-dimensional model of the object, each of the plurality of images being a render of the three-dimensional model of the object in a different gesture;
generate, using the image sensor, an image depicting a physical object;
classify, using the convolutional neural network, the physical object as being in a gesture by applying the convolution neural network to the generated image depicting the physical object;
display with the image, on the display device, additional content associated with the identified gesture of the physical object; and
provide, to a network platform, the additional content and the image as an ephemeral message.
14. The client device of claim 13 , wherein the additional content is overlay content associated with the identified gesture of the physical object in the image.
15. The client device of claim 13 , wherein the convolutional neural network is trained on a network platform.
16. The client device of claim 15 , further comprising:
receiving, from the network platform, the convolutional neural network that is trained on the plurality of images.
17. The client device of claim 13 , wherein the plurality of images are rendered from a plurality of virtual cameras that view the three-dimensional model of the object from different perspectives.
18. A non-transitory machine-readable storage device embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
store a convolutional neural network trained to detect different gestures of an object, the convolutional neural network trained on a plurality of images of a three-dimensional model of the object, each of the plurality of images being a render of the three-dimensional model of the object in a different gesture;
generate, using the image sensor, an image depicting a physical object;
classify, using the convolutional neural network, the physical object as being in a gesture by applying the convolution neural network to the generated image depicting the physical object;
display with the image, on the display device, additional content associated with the identified gesture of the physical object; and
provide, to a network platform, the additional content and the image as an ephemeral message.