Robotic camera software and controller
A robotic camera system comprising: a robot head ( 45 ), for carrying and orienting a camera ( 48 ), a video capture unit ( 30 ), operatively arranged to capture video and/audio recording from the camera and storing in a frame buffer area ( 260 ), a processor unit ( 40 ), having access to the frame buffer area ( 260 ) and operatively arranged for generating a reference camera trajectory ( 130 ) based on directives from a director, optimizing ( 140 ) said camera trajectory based on a real-time projection of objects of interest in the video recording in the frame buffer area ( 260 ), driving the robot head ( 45 ) to follow the optimized trajectory.
1. A robotic camera system comprising:
a robot head that carries and orients a camera;
a video capture unit that captures a video recording from the camera and stores the video recording in a frame buffer area;
an inference layer that detects objects within the video recording; and
a processor unit having access to the frame buffer area, the processor unit performing the following operations:
generating a reference camera path of the detected objects based on directives, the directives specify target positions of the detected objects in a frame of the video recording constraining the reference camera path in order to satisfy the directives;
optimizing said camera path based on a real-time projection of the detected objects in the video recording; and
driving the robot head of the camera to follow the optimized camera path.
2. The robotic camera system of claim 1 , wherein the robot head is operable to move, orient, or set the camera in at least some of: space coordinates (x, y, z); pan angle, tilt angle, roll angle, focus, angle of view, diaphragm.
3. The robotic camera system of claim 1 , comprising a robotic interface arranged for controlling the camera head in position and speed.
4. The robotic camera system of claim 1 , comprising a projector unit that generates said real-time projection by an optical flow algorithm.
5. The robotic camera system of claim 1 , the inference layer including a convolutional neural network.
6. The robotic camera system of claim 1 , including an object modeler unit that builds a movement model for objects of interest captured in the frame buffer based on a time series of frames.
7. A method of directing a camera along an optimized camera path comprising:
capturing and storing a video recording from the camera,
detecting objects within the video recording,
generating a reference camera path of the detected objects based on directives, the directives specify target positions of the detected objects in a frame of the video recording constraining the reference camera path to satisfy the directives,
optimizing said camera path based on a real-time projection of the detected objects in the video recording, and
driving a robot head carrying and orienting the camera to follow the optimized camera path.
8. The method of claim 7 , wherein said real-time projection is generated by an optical flow algorithm.
9. The method of claim 7 , wherein the detection of object is performed by a convolutional neural network.
10. The method of claim 7 , further comprising:
building a movement model for objects of interest based on a time series of frames in the captured video recording,
passing the movement model to a 3D scene estimation unit that builds an estimate of a scene currently in front of the camera expressed in 3D world coordinates and
passing the estimate of the scene to the projection unit that builds the real-time projection of the detected objects.
11. The method of claim 7 , wherein the optimized camera path comprises moving, orienting or setting the camera in at least some of: space coordinates (x, y, z), pan angle, tilt angle, roll angle, focus, angle of view, diaphragm.
12. The method of claim 7 , comprising a step of controlling the position and the speed of the robotic head.
13. A non-transitory computer program product comprising a computer usable medium having recorded therein a program executable by a computer to perform method steps for directing a camera along an optimized camera path, the method steps comprising:
capturing and storing a video recording from the camera,
detecting objects within the video recording,
generating a reference camera path of the detected objects based on directives, the directives specify target positions of the detected objects in a frame of the video recording constraining the reference camera path to satisfy the directives,
optimizing said camera path based on a real-time projection of the detected objects in the video recording, and
driving a robot head carrying and orienting the camera to follow the optimized camera path.
14. The non-transitory computer program product of claim 13 ,
wherein said real-time projection is generated by an optical flow algorithm.
15. The non-transitory computer program product of claim 13 ,
wherein the detection of object is performed by a convolutional neural network.
16. The non-transitory computer program product of claim 13 , further comprising:
building a movement model for objects of interest based on a time series of frames in the captured video recording,
passing the movement model to a 3D scene estimation unit that builds an estimate of a scene currently in front of the camera expressed in 3D world coordinates and
passing the estimate of the scene to the projection unit that builds the real-time projection of the detected objects.
17. The non-transitory computer program product of claim 13 , wherein the optimized camera path comprises moving, orienting or setting the camera in at least some of: space coordinates (x, y, z), pan angle, tilt angle, roll angle, focus, angle of view, diaphragm.
18. The non-transitory computer program product of claim 13 , comprising a step of controlling the position and the speed of the robotic head.