Computer-based platforms and methods for efficient AI-based digital video shot indexing

1. A method comprising:

receiving, by at least one processor, a digital video comprising a sequence of a plurality of video frames;

inputting, by the at least one processor, at least one video frame into a video frame encoder to output at least one video frame vector for the at least one video frame;

determining, by the at least one processor, a similarity value between the at least one video frame and at least one adjacent video frame in the sequence based at least in part on a similarity between the at least one video frame vector and at least one adjacent video frame vector of the at least one adjacent video frame;

determining, by the at least one processor, at least one scene within the sequence of the plurality of video frames based at least in part on:

the similarity value, and

a similarity threshold value;

wherein the at least one scene comprises at least one sub-sequence of adjacent video frames;

inputting, by the at least one processor, each video frame of the at least one scene into a plurality of expert machine learning models to output a plurality of expert machine learning model-specific labels associated with the at least one scene;

determining, by the at least one processor, a plurality of expert machine learning model-specific markup tags associated with the plurality of expert machine learning models; and

generating, by the at least one processor, at least one scene text-based markup for the at least one scene comprising the plurality of expert machine learning-specific markup tags and the plurality of expert machine learning-specific labels associated with the at least one scene.

2. The method of claim 1 , further comprising:

inputting, by the at least one processor, a plurality of video frames into a video frame encoder to output a plurality of video frame vectors;

generating, by the at least one processor, an aggregate video frame vector for the plurality of video frame vectors;

determining, by the at least one processor, a shot similarity value between the aggregate video frame vector and at least one adjacent aggregate video frame vector of an adjacent plurality of video frames in the sequence; and

determining, by the at least one processor, a scene comprising the plurality of video frames and the adjacent plurality of video frames based at least in part on the shot similarity value exceeding a threshold value.

3. The method of claim 2 , further comprising:

inputting, by the at least one processor, the scene into a scene classifier neural network to output at least one shot type based at least in part on a plurality of trained neural network parameters.

4. The method of claim 1 , further comprising:

indexing, by the at least one processor, the at least one sub-sequence of video frames of the at least one scene using the at least one scene markup.

5. The method of claim 4 , further comprising:

searching, by the at least one processor, the at least one scene markup, using the index, based on a search query comprising plain text.

6. The method of claim 4 , further comprising:

receiving, by at least one processor, a search query comprising plain text;

encoding, by the at least one processor, the search query into a search vector using at least one semantic embedding model;

encoding, by the at least one processor, the at least one scene text-based markup into a destination vector using the at least one semantic embedding model; and

searching, by the at least one processor, the at least one destination vector with the search vector based at least in part on a measure of similarity between the search vector and the destination vector.

7. The method of claim 1 , further comprising:

increasing, by the at least one processor, upon determining that a first expert machine learning model-specific label of the plurality of expert machine learning model-specific labels matches a second expert machine learning model-specific label of the plurality of expert machine learning model-specific labels, an expert machine learning model-specific label confidence score of at least one of at least one of the first expert machine learning model-specific label or the second expert machine learning model-specific label by at least one rule; and

confirming, by the at least one processor, the at least one of at least one of the first expert machine learning model-specific label or the second expert machine learning model-specific label by at least one rule based at least in part on the expert machine learning model-specific label confidence score exceeding a threshold.

8. The method of claim 7 , wherein the at least one rule is user configurable.

9. The method of claim 1 , further comprising:

querying, by the at least one processor, at least one external data source with at least one expert machine learning model-specific label of the plurality of expert machine learning model-specific labels;

receiving, by the at least one processor, property data associated with the at least one expert machine learning model-specific label from the at least one external data source in response; and

modifying, by the at least one processor, the at least one scene text-based markup to include metadata comprising the property data.

10. The method of claim 1 , wherein the video is live-streamed and the indexing is in real-time.

11. A system comprising:

At least one processor that is configured to:

receive a digital video comprising a sequence of a plurality of video frames;

input at least one video frame into a video frame encoder to output at least one video frame vector for the at least one video frame;

determine a similarity value between the at least one video frame and at least one adjacent video frame in the sequence based at least in part on a similarity between the at least one video frame vector and at least one adjacent video frame vector of the at least one adjacent video frame;

determine at least one scene within the sequence of the plurality of video frames based at least in part on:

the similarity value, and

a similarity threshold value;

wherein the at least one scene comprises at least one sub-sequence of adjacent video frames;

input each video frame of the at least one scene into a plurality of expert machine learning models to output a plurality of expert machine learning model-specific labels associated with the at least one scene;

determine a plurality of expert machine learning model-specific markup tags associated with the plurality of expert machine learning models; and

generate at least one scene text-based markup for the at least one scene comprising the plurality of expert machine learning-specific markup tags and the plurality of expert machine learning-specific labels associated with the at least one scene.

12. The system of claim 11 , wherein the at least one processor is further configured to:

input a plurality of video frames into a video frame encoder to output a plurality of video frame vectors;

generate an aggregate video frame vector for the plurality of video frame vectors;

determine a shot similarity value between the aggregate video frame vector and at least one adjacent aggregate video frame vector of an adjacent plurality of video frames in the sequence; and

determine a scene comprising the plurality of video frames and the adjacent plurality of video frames based at least in part on the shot similarity value exceeding a threshold value.

13. The system of claim 12 , wherein the at least one processor is further configured to:

input the scene into a scene classifier neural network to output at least one shot type based at least in part on a plurality of trained neural network parameters.

14. The system of claim 11 , wherein the at least one processor is further configured to:

index the at least one sub-sequence of video frames of the at least one scene using the at least one scene markup.

15. The system of claim 14 , wherein the at least one processor is further configured to:

search the at least one scene markup, using the index, based on a search query comprising plain text.

16. The system of claim 14 , wherein the at least one processor is further configured to:

receiving, by at least one processor, a search query comprising plain text;

encode the search query into a search vector using at least one semantic embedding model;

encode the at least one scene text-based markup into a destination vector using the at least one semantic embedding model; and

search the at least one destination vector with the search vector based at least in part on a measure of similarity between the search vector and the destination vector.

17. The system of claim 11 , wherein the at least one processor is further configured to:

increase upon determining that a first expert machine learning model-specific label of the plurality of expert machine learning model-specific labels matches a second expert machine learning model-specific label of the plurality of expert machine learning model-specific labels, an expert machine learning model-specific label confidence score of at least one of at least one of the first expert machine learning model-specific label or the second expert machine learning model-specific label by at least one rule; and

confirm the at least one of at least one of the first expert machine learning model-specific label or the second expert machine learning model-specific label by at least one rule based at least in part on the expert machine learning model-specific label confidence score exceeding a threshold.

18. The system of claim 17 , wherein the at least one rule is user configurable.

19. The system of claim 11 , wherein the at least one processor is further configured to:

query at least one external data source with at least one expert machine learning model-specific label of the plurality of expert machine learning model-specific labels;

receive property data associated with the at least one expert machine learning model-specific label from the at least one external data source in response; and

modify the at least one scene text-based markup to include metadata comprising the property data.

20. The system of claim 11 , wherein the video is live-streamed and the indexing is in real-time.