Pipelined video interface for remote controlled aerial vehicle with camera
Disclosed is a system and method for reducing the total latency for transferring a frame from the low latency camera system mounted on an aerial vehicle to the display of the remote controller. The method includes reducing the latency through each of the modules of the system, i.e. through a camera module, an encoder module, a wireless interface transmission, wireless interface receiver module, a decoder module and a display module. To reduce the latency across the modules, methods such as overclocking the image processor, pipelining the frame, squashing the processed frame, using a fast hardware encoder that can perform slice based encoding, tuning the wireless medium using queue sizing, queue flushing, bitrate feedback, physical medium rate feedback, dynamic encoder parameter tuning and wireless radio parameter adjustment, using a fast hardware decoder that can perform slice based decoding and overclocking the display module are used.
1. A method for pipelining a frame over a low latency camera system, the method comprising:
receiving one or more unprocessed frames captured by the low latency camera system;
overclocking an image processor pipeline, the overclocking to increase an operating frequency of an image processor of the low latency camera system such that there is idle image processor time between frames;
performing one or more image processing operations by the image processor on the at least one frame during the idle image processor time between frames, the one or more image processing operations including at least one of sharpening, compression, or color conversion;
forwarding the processed frame from a first pipeline stage to a second pipeline stage of the image processor that includes a plurality of pipeline stages;
squashing image data of the processed frame, the squashing to reduce the size of the image data without losing image information, wherein the squashing comprises aligning an external interface in the image processor pipeline to when data is ready from the image processor pipeline to avoid buffering;
transmitting the processed frame to a fast encoder, the fast encoder comprising a hardware encoder;
performing one or more preprocessing operations by the image processor on the processed frame, the one or more preprocessing operations including at least one of scaling, resizing, or cropping;
performing one or more image encoding operations on the preprocessed frame by the fast encoder, the one or more image encoding operations including slice based encoding that encodes input frames when a slice of the preprocessed frame is available to reduce the latency associated with an input buffer; and
transmitting the encoded frame to a wireless interface for transmission.
2. The method of claim 1 , wherein the one or more image processing operations further includes at least one of performing a Bayer transformation, demosaicing, noise reduction, image stabilization, or rolling shutter artifact reduction.
3. The method of claim 1 , wherein the overclocking of the image processor pipeline and the squashing of the image data is performed to reduce the latency by at least one frame.
4. The method of claim 1 , wherein the squashing is performed in response to detecting an input/output interface, further wherein a clock of the input/output interface is distinct from a clock of the image processor.
5. The method of claim 1 , further comprising:
performing the one or more image processing operations on the one or more unprocessed frames in each of the plurality of pipeline stages.
6. The method of claim 1 , further comprising:
storing the processed frame in memory of the low latency camera system.
7. A non-transitory computer readable medium configured to store instructions, the instructions when executed by a processor cause the processor to:
receive one or more unprocessed frames captured by the low latency camera system;
overclock an image processor pipeline, the overclocking to increase an operating frequency of an image processor of the low latency camera system such that there is idle image processor time between frames;
perform one or more image processing operations by the image processor on the at least one frame during the idle image processor time between frames, the one or more image processing operations including at least one of sharpening, compression, or color conversion;
forward the processed frame from a first pipeline stage to a second pipeline stage of the image processor that includes a plurality of pipeline stages;
squash image data of the processed frame, the squashing to reduce the size of the image data without losing image information, wherein the squashing comprises aligning an external interface in the image processor pipeline to when data is ready from the image processor pipeline to avoid buffering;
transmit the processed frame to a fast encoder, the fast encoder comprising a hardware encoder;
perform one or more preprocessing operations by the image processor on the processed frame, the one or more preprocessing operations including at least one of scaling, resizing, or cropping;
perform one or more image encoding operations on the preprocessed frame by the fast encoder, the one or more image encoding operations including slice based encoding that encodes input frames when a slice of the preprocessed frame is available to reduce the latency associated with an input buffer; and
transmit the encoded frame to a wireless interface for transmission.
8. The non-transitory computer readable storage medium of claim 7 , wherein the one or more image processing operations further includes at least one of performing a Bayer transformation, demosaicing, noise reduction, image stabilization, or rolling shutter artifact reduction.
9. The non-transitory computer readable storage medium of claim 7 , wherein the overclocking of the image processor pipeline and the squashing of the image data is performed to reduce the latency by at least one frame.
10. The non-transitory computer readable storage medium of claim 7 , wherein the squashing is performed in response to detecting an input/output interface, further wherein a clock of the input/output interface is different from a clock of the image processor.
11. The non-transitory computer readable storage medium of claim 7 , the instructions when executed by the processor further causing the processor to:
perform the one or more image processing operations on the one or more unprocessed frames in each of the plurality of pipeline stages.
12. The non-transitory computer readable storage medium of claim 7 , the instructions when executed by the processor further causing the processor to:
store the processed frame in memory of the low latency camera system.
13. The non-transitory computer readable storage medium of claim 7 , wherein the overclocking drops a frame of the one or more unprocessed frames to increase the operating frequency.
14. The non-transitory computer readable storage medium of claim 7 , wherein the overlocking increases idle time of the image processor.
15. A low latency video pipelining system, comprising:
a memory; and
a processor configured to execute instructions stored on the memory to:
receive one or more unprocessed frames captured by an imaging device of the low latency video pipelining system;
overclock an image processor pipeline, the overclocking to increase an operating frequency of an image processor of the low latency video pipelining system such that there is idle image processor time between frames;
perform one or more image processing operations by the image processor on the at least one frame during the idle image processor time between frames, the one or more image processing operations including at least one of sharpening, compression, or color conversion;
forward the processed frame from a first pipeline stage to a second pipeline stage of the image processor that includes a plurality of pipeline stages;
squash image data of the processed frame, the squashing to reduce the size of the image data without losing image information, wherein the squashing comprises aligning an external interface in the image processor pipeline to when data is ready from the image processor pipeline to avoid buffering;
transmit the processed frame to a fast encoder, the fast encoder comprising a hardware encoder;
perform one or more preprocessing operations by the image processor on the processed frame, the one or more preprocessing operations including at least one of scaling, resizing, or cropping;
perform one or more image encoding operations on the preprocessed frame by the fast encoder, the one or more image encoding operations including slice based encoding that encodes input frames when a slice of the preprocessed frame is available to reduce the latency associated with an input buffer; and
transmit the encoded frame to a wireless interface for transmission.
16. The low latency video pipelining system of claim 15 , wherein the one or more image processing operations further includes at least one of performing a Bayer transformation, demosaicing, noise reduction, image stabilization, or rolling shutter artifact reduction.
17. The low latency video pipelining system of claim 15 , wherein the overclocking of the image processor pipeline and the squashing of the image data is performed to reduce the latency by at least one frame.
18. The low latency video pipelining system of claim 15 , wherein the squashing is performed in response to detecting an input/output interface, further wherein a clock of the input/output interface is distinct from a clock of the image processor.
19. The low latency video pipelining system of claim 15 , wherein the overclocking drops a frame of the one or more unprocessed frames to increase the operating frequency.
20. The pipelining system of claim 18 , wherein the squashing includes overclocking the clock of the input/output interface, synchronization of the input/output interface, and data forwarding.