Dynamically altering preprocessing of streaming video data by a large language model dependent upon review by the large language model
A method of preprocessing incoming video data can include preprocessing incoming video data, by a computer processor, according to preprocessing parameters, wherein the preprocessing includes formatting the incoming video data to create first video data of a first region of interest. The method can also include providing the first video data to a first transformer model along with a first prompt requesting the first transformer model to describe the first video data, describing the first video data by the first transformer model to generate at least one description of the first video data, providing the at least one description to a second transformer model with a second prompt requesting the second transformer model to review the at least one description and determine altered preprocessing parameters for preprocessing the incoming video data, and generating, by the second transformer model, altered preprocessing parameters based upon the at least one description.
1 . A method of preprocessing incoming video data having at least one region of interest, the method comprising:
accessing the incoming video;
preprocessing the incoming video data, by a computer processor, according to preprocessing parameters, wherein the preprocessing includes formatting the incoming video data to create first video data of a first region of interest;
providing the first video data to a first transformer model along with a first prompt requesting the first transformer model to describe the first video data;
describing the first video data by the first transformer model to generate at least one description of the first video data;
providing the at least one description to a second transformer model with a second prompt requesting the second transformer model to review the at least one description and determine altered preprocessing parameters for preprocessing the incoming video data;
generating, by the second transformer model, altered preprocessing parameters based upon the at least one description;
providing the altered preprocessing parameters to the computer processor; and
preprocessing the incoming video data according to the altered preprocessing parameters to create second video data.
2 . The method of claim 1 , wherein the altered preprocessing parameters change at least one of the following video edits of the incoming video data as compared to the unaltered preprocessing parameters: crop, grayscale, contrast, brightness, color threshold, resize, blur, hue saturation value, sharpen, erosion, dilation, Laplacian image processing, Sobel image processing, pyramid up, and pyramid down.
3 . The method of claim 1 , further comprising:
formatting, by the second transformer model, the altered preprocessing parameters so as to be accepted and applied by the computer processor.
4 . The method of claim 1 , further comprising:
providing the altered preprocessing parameters to a formatting module; and
formatting the altered preprocessing parameters, by the formatting module, into a format that is acceptable by the computer processor,
wherein the formatting module provides the altered preprocessing parameters in an acceptable format to the computer processor.
5 . The method of claim 1 , wherein the first transformer model and the second transformer model each include a large language model.
6 . The method of claim 1 , wherein the first transformer model is different from the second transformer model.
7 . The method of claim 1 , wherein the first transformer model and the second transformer model are the same transformer model.
8 . The method of claim 1 , further comprising:
publishing the first video data to an endpoint,
wherein accessing the first video data includes subscribing to the endpoint.
9 . The method of claim 8 , wherein the endpoint is hosted by a gateway.
10 . The method of claim 1 , wherein the incoming video data is received from a camera.
11 . A method of preprocessing incoming video data having at least one region of interest, the method comprising:
preprocessing the incoming video data, by a computer processor, according to preprocessing parameters, wherein the preprocessing includes formatting the incoming video data to create first video data of a first region of interest;
accessing the first video data by an AI model;
processing the first video data by the AI model to determine a first output that is indicative of a first inference dependent upon the first video data;
providing the first video data, the first output, and a first prompt to a first transformer model with the first prompt requesting the first transformer model to describe the first video data;
generating, by the first transformer model, at least one description of the first video data from the first video data;
providing the at least one description and a second prompt to a second transformer model with the second prompt requesting the second transformer model to determine altered preprocessing parameters that alter the incoming video data to create second video data; and
determining, by the second transformer model, the altered preprocessing parameters dependent upon the at least one description with the altered preprocessing parameters altering the incoming video data to create the second video data.
12 . The method of claim 11 , further comprising:
changing the preprocessing parameters in a configuration file to be the altered preprocessing parameters;
accessing the altered preprocessing parameters by the computer processor; and
preprocessing the incoming video data according to the altered preprocessing parameters to create the second video data.
13 . The method of claim 12 , further comprising:
providing the second video data and a third prompt to the first transformer model with the third prompt requesting the first transformer model to describe the second video data; and
describing, by the first transformer model, the second video data to create at least one description of the second video data.
14 . The method of claim 13 , further comprising:
compiling the at least one description of the first video data and the at least one description of the second video data into an overall video data description.
15 . The method of claim 11 , further comprising:
providing the altered preprocessing parameters to a formatting module;
formatting the altered preprocessing parameters, by the formatting module, into a format that is acceptable by the computer processor; and
providing the altered preprocessing parameters in an acceptable format to the computer processor.
16 . The method of claim 11 , wherein the first transformer model and the second transformer model each include a large language model.
17 . The method of claim 11 , wherein the first transformer model and the second transformer model are the same transformer model.
18 . The method of claim 11 , further comprising:
generating multiple vector embeddings corresponding to multiple descriptions of the at least one description of the first video data.
19 . The method of claim 18 , wherein the at least one description of the first video data includes multiple descriptions with each description being generated by the first transformer model and corresponding to one frame of multiple frames that form the first video data.
20 . The method of claim 19 , further comprising:
searching the multiple descriptions to find at least one relevant frame of the first video data.