System and method for camera handling in live environments

1 . A processor-implemented method, comprising:

determining, by at least one processor, that at least one audio frame of a live version of a musical score being performed in a live environment corresponds to a recorded audio frame of a recorded version of the musical score; and

enabling, by at least one camera, a camera movement that is according to a time-based reference between the at least one audio frame and the recorded audio frame and that is according at least one predominant effect comprising audio and video of the recorded version or the live version.

2 . The processor-implemented method of claim 1 wherein the predominant effect is in a form of an instruction that is stored in a memory with the recorded version of the musical score and that is to be executed by the at least one processor, and wherein the instruction comprises, for the at least one audio frame, at least one of a musical performer to be captured by the at least one camera using the camera movement or a type of immersive experience to be generated and captured by the at least one camera, and wherein the processor-implemented method further comprises:

enabling the camera movement to capture the musical performer or the immersive experience; and

including the camera movement with the live version of the live environment.

3 . The processor-implemented method of claim 2 , further comprising:

enabling the instruction to be provided in a computer-based input and to be based on the recorded version of the musical score.

4 . The processor-implemented method of claim 2 , wherein the immersive experience is generated for the at least one audio frame of the live version and wherein the type of the immersive experience is at least one of: a graphical visualization being displayed during the live version or text being displayed to an audience of the live environment.

5 . The processor-implemented method of claim 4 , further comprising:

adjusting, dependent on features derived from the audio and video of the live environment, the camera movement or the immersive experience, wherein the features comprise at least one of a volume or a level of expression of emotions of the musical performer, and wherein the adjusting comprises at least one of camera zooming or tuning of colors of the graphical visualization or the text.

6 . The processor-implemented method of claim 5 , further comprising:

determining the level of expression of emotions of the musical performer based at least in part on a machine learning algorithm tracking a facial matrix of the musical performer in the live version of the musical score.

7 . The processor-implemented method of claim 5 , wherein the adjusting of the camera movement or the immersive experience is further based in part on object tracking preformed for the live version of the musical score.

8 . The processor-implemented method of claim 1 , further comprising:

enabling the camera movement using a plurality of cameras of the at least one camera, wherein the camera movement comprises particular actions performed on particular ones of the plurality of cameras.

9 . The processor-implemented method of claim 8 , further comprising:

modifying, in real time, the camera movement based at least in part on determination of a problem with the at least one camera or a further camera associated with the at least one camera.

10 . The processor-implemented method of claim 1 further comprising:

determining, for the at least one audio frame of the live version of the musical score, the recorded audio frame of the recorded version of the musical score based on a machine learning algorithm, the machine learning algorithm to extract features describing the at least one audio frame and the record audio frame and to use online dynamic time warping (ODTW) to determine associations between the features.

11 . The processor-implemented method of claim 10 , further comprising:

continuously improving the machine learning algorithm based at least in part on feedback provided by users, wherein the feedback is collected via a computer-based input and wherein the feedback comprises mistakes in the frame matching; and

the feedback being added to a training data set for the machine learning algorithm.

12 . The processor-implemented method of claim 10 , further comprising:

enabling the machine learning algorithm to use outcomes from previous live environments of the musical score, wherein the recorded audio frame determined using the at least one audio frame enables consecutive audio frames matches between the live version and the recorded version.

13 . The processor-implemented method of claim 10 further comprising:

enabling the machine learning algorithm to use a degree of confidence, wherein a first predetermined value of the degree of confidence allows the machine learning algorithm to influence the camera movement by one or more of switching to other types or settings of predominant effects or keeping the at least one camera or a plurality of cameras in a switched mode until the machine learning algorithm regain a second predetermined value of the degree of confidence for next audio frames of the live version or the recorded version of the musical score.

14 . The processor-implemented method of claim 13 , wherein the switching comprises at least one of redirecting the at least one camera or a plurality of cameras to capture a conductor of the musical score or reducing effects associated with the at least one predominant effect based on an immersive experience associated with the live version of the musical score.

15 . The processor-implemented method of claim 1 , further comprising:

storing a plurality of recorded versions that are different versions of the musical score, along with associated predominant effects; and

determining the recorded version from a plurality of recorded versions to be used with the live version at a beginning of the musical score based in part on a difference between the at least one audio frame and individual ones of a plurality of recorded audio frames of the plurality of recorded versions.

16 . The processor-implemented method of claim 1 , further comprising:

determining the recorded version of a plurality of recorded versions to be used with the live version of the musical score based in part on a degree of confidence with which a machine learning algorithm, which is responsible for determination of the recorded audio frame associated with the at least one audio frame, provides an output using the at least one audio frame against a plurality of recorded audio frames of the plurality of recorded versions.

17 . The processor-implemented method of claim 16 , further comprising:

enabling the camera movement for the at least one camera or a plurality of cameras using preliminary instructions until the recorded version is determined with a pre-determined value of a degree of confidence, wherein the determination of the recorded version is performed in real time or in near real time using a scalable cloud environment and using a plurality of recorded versions; and

sending information associated with the recorded version to the at least one camera or to a plurality of cameras; and

enabling further real time determination of another recorded audio frame to be used with the live version using an edge environment that is associated with the scalable cloud environment.

18 . The processor-implemented method of claim 1 , further comprising:

capturing the live version using the at least one camera or a plurality of cameras, wherein the capturing is synchronized in real time with live audio and used in at least one of an online broadcasting, a television-based live entertainment production, or a production of a musical score recording.

19 . The processor-implemented method of claim 18 , further comprising:

enabling the camera movement to obtain a plurality of live versions of the musical score for different ones of the online broadcasting, the television-based live entertainment production, or the production of the musical score recording.

20 . The processor-implemented method of claim 1 , further comprising:

enabling turning on or off at least one function associated with the processor-implemented method using a computer-based input that reports at least one aspect of the musical score, the at least one function to control a level of automation of camera movement.