Differential signaling for video coding for machines
An example device for processing video data includes a memory configured to store video data; and a processing system implemented in circuitry, the processing system being configured to: receive data representing a plurality of neural networks associated with a video bitstream, each of the plurality of neural networks having a different type; receive data representing an update to at least one of the neural networks, the data including a type corresponding to the at least one of the neural networks and a neural network structure for the update; update the neural network according to the data representing the update to generate an updated neural network; and provide video data from the video bitstream to the updated neural network to cause the updated neural network to process the video data.
1 . A method of processing video data, the method comprising:
receiving data representing a plurality of neural networks associated with a video bitstream, each of the plurality of neural networks having a different type;
receiving data representing an update to at least one of the neural networks, the data including a type corresponding to the at least one of the neural networks and a neural network structure for the update, the neural network structure for the update including one or more inner droppable structures to be removed from the at least one of the neural networks;
updating the at least one of the neural networks according to the data representing the update to remove the one or more inner droppable structures to generate an updated neural network; and
providing video data from the video bitstream to the plurality of neural networks including the updated neural network to cause the plurality of neural networks including the updated neural network to process the video data.
2 . The method of claim 1 , wherein the type represents a task to which the data representing the update corresponds, and wherein each of the neural networks is configured to perform a respective task.
3 . The method of claim 2 , further comprising determining the at least one of the neural networks that performs the task indicated in the data representing the update.
4 . The method of claim 2 , wherein the plurality of neural networks are configured to perform at least one of region of interest (ROI) based coding, neural network based intra-prediction coding, frame-level spatial resampling, temporal resampling, or post filtering.
5 . The method of claim 1 , wherein the video bitstream comprises an encoded video bitstream, the method further comprising decoding the video bitstream to form decoded video data, wherein providing the video data to the updated neural network comprises providing the decoded video data to the updated neural network.
6 . The method of claim 1 , wherein the video bitstream comprises an encoded video bitstream, and wherein providing the video data to the updated neural network comprises providing the encoded video bitstream to the updated neural network.
7 . The method of claim 1 , wherein the data representing the neural network includes one or more weight values for the neural network, bias values for the neural network, or an identifier for the neural network.
8 . The method of claim 7 , wherein the data representing the neural network includes the identifier, and wherein the identifier comprises a sequence number.
9 . The method of claim 1 , wherein the data representing the plurality of neural networks include the structures for the plurality of neural networks expressed in one of Open Neural Network Exchange (ONNX) format, Neural Network Exchange Format (NNEF), or a format defined at a universal resource locator (URL), universal resource identifier (URI), or universal resource name (URN).
10 . The method of claim 1 , wherein the data representing the update includes one or more of an identifier for the at least one of the neural networks, a change to a structure of the at least one of the neural networks, a change to weights of the at least one of the neural networks, a change to biases of the at least one of the neural networks, a new value for a configuration parameter, or a change to the configuration parameter.
11 . The method of claim 10 , wherein the configuration parameter comprises one of a quantization parameter, a spatial downsampling ratio, or a temporal downsampling ratio.
12 . The method of claim 1 , wherein receiving the data representing the update comprises receiving the data representing the update in at least one of a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), or in a supplemental enhancement information (SEI) message.
13 . The method of claim 1 , wherein receiving the data representing the update comprises receiving the data representing the update via Transmission Control Protocol (TCP), Real-time Transport Protocol (RTP), or User Datagram Protocol (UDP).
14 . The method of claim 13 , further comprising performing a session initiation for TCP, RTP, or UDP.
15 . The method of claim 13 , wherein receiving the data representing the update comprises receiving the data representing update in a payload of an RTP packet, further comprising extracting data from a header of the RTP packet indicating that the payload includes the data representing the update.
16 . The method of claim 13 , wherein receiving the data representing the update comprises receiving the data representing update in an RTP packet header extension.
17 . The method of claim 1 , wherein receiving the data representing the update comprises receiving the data representing the update in a data channel of one of an IP Multimedia Subsystem (IMS) or Web Real-Time Communications (WebRTC).
18 . A device for processing video data, the device comprising:
a memory configured to store video data; and
a processing system implemented in circuitry, the processing system being configured to:
receive data representing a plurality of neural networks associated with a video bitstream, each of the plurality of neural networks having a different type;
receive data representing an update to at least one of the neural networks, the data including a type corresponding to the at least one of the neural networks and a neural network structure for the update, the neural network structure for the update including one or more inner droppable structures to be removed from the at least one of the neural networks;
update the at least one of the neural networks according to the data representing the update to remove the one or more inner droppable structures to generate an updated neural network; and
provide video data from the video bitstream to the plurality of neural networks including the updated neural network to cause the plurality of neural networks including the updated neural network to process the video data.
19 . The device of claim 18 , wherein the type represents a task to which the data representing the update corresponds, and wherein each of the neural networks is configured to perform a respective task, wherein the processing system is further configured to determine the at least one of the neural networks that performs the task indicated in the data representing the update.
20 . A device for processing video data, the device comprising:
means for receiving data representing a plurality of neural networks associated with a video bitstream, each of the plurality of neural networks having a different type;
means for receiving data representing an update to at least one of the neural networks, the data including a type corresponding to the at least one of the neural networks and a neural network structure for the update, the neural network structure for the update including one or more inner droppable structures to be removed from the at least one of the neural networks;
means for updating the at least one of the neural networks according to the data representing the update to remove the one or more inner droppable structures to generate an updated neural network; and
means for providing video data from the video bitstream to the plurality of neural networks including the updated neural network to cause the plurality of neural networks including the updated neural network to process the video data.