Systems and methods for stream recognition
The present disclosure provides systems and methods for providing augmented reality experiences. Consistent with disclosed embodiments, one or more machine-learning models can be trained to selectively process image data. A pre-processor can be configured to receive image data provided by a user device and trained to automatically determine whether to select and apply a preprocessing technique to the image data. A classifier can be trained to identify whether the image data received from the pre-processor includes a match to one of a plurality of triggers. A selection engine can be trained to select, based on a matched trigger and in response to the identification of the match, a processing engine. The processing engine can be configured to generate an output using the image data, and store the output or provide the output to the user device or a client system.
1. A system comprising:
a classifier, comprising a secondary classifier and a hierarchical classifier, the classifier configured to:
generate a segment of image data;
associate the segment with one of a plurality of triggers; and
determine that the image data includes a match to the one of the plurality of triggers, the match comprising an object, when the association between the segment and the one of the plurality of triggers satisfies a match condition; and
a trainer configured to train the hierarchical classifier using training data in response to a determination by the secondary classifier that the image data includes the match and a determination by the hierarchical classifier that the image data does not include the match, wherein:
the training data is generated, using a pose estimation model corresponding to the classifier, by tracking the object in a video stream; and
the video stream includes the image data.
2. The system of claim 1 , wherein the classifier comprises a convolutional neural network.
3. The system of claim 1 , wherein the classifier is configured to use keypoint matching to determine whether the image data includes the match.
4. The system of claim 1 , wherein:
the hierarchical classifier comprises specific classifiers; and
the trainer is configured to train the specific classifiers using a reinforcement learning model, the reinforcement learning model trained to generate classifier hyperparameters using a reward function based on classifier accuracy.
5. The system of claim 1 , wherein the training data is generated using the pose estimation model, at least in part, by segmenting the object in a frame of the image data.
6. The system of claim 1 , wherein:
the system further comprises a pre-processor configured to automatically determine whether to select and apply a preprocessing technique to the image data before providing the image data to the classifier.
7. The system of claim 6 , wherein:
the pre-processor comprises a reinforcement learning model; and
the trainer is further configured to train the pre-processor using a reward function based on a success or failure of the classifier in identifying the match.
8. The system of claim 7 , wherein the reward function is further based on a time required to identify the match.
9. The system of claim 1 , wherein the system further comprises a selection engine configured to select, based on the match and in response to the determination that the image data includes the match, a processing engine to generate an output from the image data.
10. The system of claim 9 , wherein the trainer is further configured to train the selection engine using a reward function based on a degree of engagement with provided outputs.
11. The system of claim 1 , wherein the system is configured to store the image data when the classifier determines that the image data includes the match.
12. A non-transitory, computer-readable medium containing instructions that, when executed by at least one processor of a system, cause the system to perform operations comprising:
generating a segment of image data;
associating the segment with one of a plurality of triggers;
determining, using a secondary classifier and a hierarchical classifier, that the image data includes a match to the one of the plurality of triggers, the match comprising an object, when the association between the segment and the one of the plurality of triggers satisfies a match condition; and
training the hierarchical classifier using training data in response to a determination by the secondary classifier that the image data includes the match and a determination by the hierarchical classifier that the image data does not include the match, wherein:
the training data is generated, using a pose estimation model, by tracking the object in a video stream; and
the video stream includes the image data.
13. The non-transitory, computer-readable medium of claim 12 , wherein the operations further comprise:
determining, using keypoint matching, whether the image data includes the match; and
storing the image data in response to the determination that the image data includes the match.
14. The non-transitory, computer-readable medium of claim 12 , wherein the hierarchical classifier is trained using a reinforcement learning model, the reinforcement learning model trained to generate classifier hyperparameters using a reward function based on classifier accuracy.
15. The non-transitory, computer-readable medium of claim 12 , wherein:
the operations further comprise automatically determining whether to select and apply a preprocessing technique to the image data; and
the system is trained using a reward function, the reward function based on:
a success or failure in identifying the match, and
a time required to identify the match.
16. The non-transitory, computer-readable medium of claim 12 , wherein the operations further comprise:
generating, based on the match and in response to the determination that the image data includes the match, an output using the image data;
providing the output to a user device; and
additionally training the system using a reward function based on a degree of engagement of with the provided output.