NP-micro for video artificial intelligence (AI) hardware (HW) pipeline
Aspects of the present disclosure provide a video playback system. For example, the video playback system can include a video controller configured to receive one or more video frames and partition each of the video frames into a plurality of video tiles, a video decoder configured to decode one of the video tiles partitioned from each of the video frames, an artificial intelligence (AI) accelerator configured to execute an executable AI model on the decoded video tile, a display configured to display the processed video tile, and a pipeline manager installed in the video controller. The pipeline manager can be configured to control the video decoder to decode the video tile, control the AI accelerator to execute the executable AI model on the decoded video tile, and control the display to display the processed video tile.
1 . A video playback system, comprising:
a hardware video controller configured to receive one or more video frames and partition each of the video frames into a plurality of video tiles;
a hardware video decoder coupled to the video controller, the video decoder configured to decode a video tile of the video tiles partitioned from each of the video frames to generate a decoded video tile;
a hardware artificial intelligence (AI) accelerator coupled to the video controller and the video decoder, the AI accelerator configured to receive the decoded video tile and execute an executable AI model on the decoded video tile to generate a processed video tile;
a hardware display coupled to the video controller and the AI accelerator, the display configured to receive and display the processed video tile; and
a pipeline manager installed in the video controller, the pipeline manager configured to directly control the video decoder to decode the video tile to generate the decoded video tile, directly control the AI accelerator to execute the executable AI model on the decoded video tile to generate the processed video tile, and directly control the display to display the processed video tile by generating and receiving a plurality of synchronization signals.
2 . The video playback system of claim 1 , wherein the pipeline manager controls the AI accelerator to execute the executable AI model on the decoded video tile by issuing a synchronization signal of the plurality of synchronization signals to the AI accelerator when the video decoder generates the decoded video tile and controls the display to display the processed video tile by receiving another synchronization signal of the plurality of synchronization signals, wherein the other synchronization signal is issued by the AI accelerator when the AI accelerator generates the processed video tile.
3 . The video playback system of claim 1 , further comprising:
an AI engine installed in the video controller and coupled to the pipeline manager and the AI accelerator,
wherein the pipeline manager is further configured to instruct the AI engine to trigger the AI accelerator to execute the executable AI model on the decoded video tile.
4 . The video playback system of claim 3 , wherein the AI engine is configured for the executable AI model to be loaded thereinto.
5 . The video playback system of claim 4 , wherein the executable AI model is prepared based on an AI model and generated in the AI engine.
6 . The video playback system of claim 1 , further comprising:
a memory coupled to the AI accelerator, the memory configured for the executable AI model to be loaded thereinto.
7 . The video playback system of claim 6 , wherein the memory is protected by a first firewall.
8 . The video playback system of claim 7 , wherein the AI accelerator is protected by a second firewall.
9 . The video playback system of claim 8 , wherein the second firewall is a same as the first firewall.
10 . The video playback system of claim 8 , wherein the executable AI model is prepared based on an AI model and generated within a trusted execution environment (TEE) provided by a secure operating system (OS).
11 . The video playback system of claim 6 , wherein the memory includes a dynamic random access memory (DRAM).
12 . The video playback system of claim 6 , wherein the memory includes an on-bus SRAM.
13 . The video playback system of claim 6 , wherein the memory includes a serial flash memory.
14 . The video playback system of claim 1 , wherein the AI accelerator includes a deep learning accelerator (DLA).
15 . The video playback system of claim 14 , wherein the DLA includes an accelerated processing unit (APU).
16 . The video playback system of claim 1 , wherein the video controller includes a central processing unit (CPU).
17 . The video playback system of claim 1 , which is installed on a smart phone.