Data processing using accelerators
In various examples, systems and methods are disclosed that relate to performing inter-accelerator data transfers. For example, an accelerator such as a pixel processing engine (PPE) can include multiple processing engines (PEs). The PEs can be arranged in a two-dimensional array and each PE can be configured to receive data in the registers of the PEs. The PEs can transfer data between registers of the same or different PEs. The PEs can also be configured to perform transfers and operations in sequence to perform complex functions such as filtering.
1 . One or more processors comprising processing circuitry to:
receive, using one or more circuits of a first processing element of a two-dimensional array of processing elements, first sensor data associated with a first pixel of an image and instruction data associated with one or more instructions;
determine, using the one or more circuits of the first processing element, an updated first pixel value based at least on the first sensor data associated with the first pixel, second sensor data associated with one or more other pixels received using the two-dimensional array of processing elements, and the instruction data associated with the one or more instructions; and
provide, using the one or more circuits of the first processing element, output data associated with the updated first pixel value as output based at least on determining the updated first pixel value.
2 . The one or more processors of claim 1 , wherein the first sensor data associated with the first pixel is associated with a first block,
wherein the two-dimensional array of processing elements are configured to receive the second sensor data, the second sensor data associated with at least one second block, and
wherein the one or more circuits of the first processing element that are to receive the first sensor data associated with a first pixel are to: receive the first sensor data associated with the first pixel and the second sensor data associated with at least one second pixel.
3 . The one or more processors of claim 2 , wherein the one or more circuits of the first processing element are to:
store the first sensor data associated with the first block in a first register;
store the second sensor data associated with the at least one second block in at least one second register; and
transfer the first sensor data associated with the first block from the first register to a first destination register; or
transfer the second sensor data associated with the second block from the second register to a second destination register,
wherein the first destination register or the second destination register comprise registers corresponding to the first processing element or one or more second processing elements.
4 . The one or more processors of claim 1 , wherein the one or more circuits of the first processing element are to:
obtain the second sensor data associated with at least one second pixel based at least on the instruction data associated with the one or more instructions; and
determine the output data associated with the updated first pixel value based at least on the first sensor data associated with the first pixel and the second sensor data associated with the at least one second pixel.
5 . The one or more processors of claim 4 , wherein the one or more circuits of the first processing element that obtain the second sensor data associated with the at least one second pixel are to:
obtain the second sensor data associated with the at least one second pixel from at least one different processing element of a plurality of processing elements.
6 . The one or more processors of claim 4 , wherein the one or more circuits of the first processing element that obtain the second sensor data associated with the at least one second pixel are to:
obtain the second sensor data associated with the at least one second pixel from at least one register of the first processing element.
7 . The one or more processors of claim 4 , wherein the one or more instructions are associated with a sequence of operations that represent one or more of: a matrix addition function, a matrix subtraction function, a matrix multiplication function, a filter operation, or a band pass filter operation.
8 . The one or more processors of claim 1 , wherein the one or more processors are comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system implemented using a robot;
an aerial system;
a medical system;
a boating system;
a smart area monitoring system;
a system for performing deep learning operations;
a system for performing simulation operations;
a system for generating or presenting virtual reality (VR) content, augmented reality (AR) content, or mixed reality (MR) content;
a system for performing digital twin operations;
a system implemented using an edge device;
a system incorporating one or more virtual machines (VMs);
a system for generating synthetic data;
a system implemented at least partially in a data center;
a system for performing conversational artificial intelligence (AI) operations;
a system for performing generative AI operations;
a system implementing language models;
a system for performing generative AI operations;
a system for implementing vision language models (VLMs);
a system implementing large language models (LLMs);
a system for implementing multi-modal language models;
a system for hosting one or more real-time streaming applications;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets; or
a system implemented at least partially using cloud computing resources.
9 . A system, comprising:
a plurality of processing elements of a two-dimensional array of processing elements, each processing element of the plurality of processing elements operatively coupled with one or more different processing elements of the plurality of processing elements;
wherein each processing element comprises one or more circuits to:
receive, using one or more circuits of a first processing element of a two-dimensional array of processing elements, first sensor data associated with a first pixel of an image and instruction data associated with one or more instructions;
determine, using the one or more circuits of the first processing element, an updated first pixel value based at least on the first sensor data associated with the first pixel, second sensor data associated with one or more other pixels received using the two-dimensional array of processing elements, and the instruction data associated with the one or more instructions; and
provide, using the one or more circuits of the first processing element, output data associated with the updated first pixel value as output based at least on determining the updated first pixel value.
10 . The system of claim 9 , wherein the first sensor data associated with the first pixel is associated with a first block,
wherein the plurality of processing elements are configured to receive the second sensor data, the second sensor data associated with at least one second block, and
wherein the one or more circuits of the first processing element that are to receive the first sensor data associated with the first pixel are to: receive the first sensor data associated with the first pixel and the second sensor data associated with at least one second pixel.
11 . The system of claim 10 , wherein the one or more circuits of the first processing element are to:
store the first sensor data associated with the first block in a first register;
store the second sensor data associated with the at least one second block in at least one second register; and
transfer the first sensor data associated with the first block from the first register to a first destination register; or
transfer the second sensor data associated with the second block from the second register to a second destination register,
wherein the first destination register or the second destination register comprise registers corresponding to the first processing element or one or more second processing elements.
12 . The system of claim 9 , wherein the one or more circuits of the first processing element are to:
obtain the second sensor data associated with at least one second pixel based at least on the instruction data associated with the one or more instructions; and
determine the output data associated with the updated first pixel value based at least on the first sensor data associated with the first pixel and the second sensor data associated with the at least one second pixel.
13 . The system of claim 12 , wherein the one or more circuits of the first processing element that obtain the second sensor data associated with the at least one second pixel are to:
obtain the second sensor data associated with the at least one second pixel from at least one different processing element of a plurality of processing elements.
14 . The system of claim 12 , wherein the one or more circuits of the first processing element that obtain the second sensor data associated with the at least one second pixel are to:
obtain the second sensor data associated with the at least one second pixel from at least one register of the first processing element.
15 . The system of claim 12 , wherein the one or more instructions are associated with a sequence of operations that represent one or more of: a matrix addition function, a matrix subtraction function, a matrix multiplication function, a filter operation, or a band pass filter operation.
16 . The system of claim 9 , wherein the plurality of processing elements are comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system implemented using a robot;
an aerial system;
a medical system;
a boating system;
a smart area monitoring system;
a system for performing deep learning operations;
a system for performing simulation operations;
a system for generating or presenting virtual reality (VR) content, augmented reality (AR) content, or mixed reality (MR) content;
a system for performing digital twin operations;
a system implemented using an edge device;
a system incorporating one or more virtual machines (VMs);
a system for generating synthetic data;
a system implemented at least partially in a data center;
a system for performing conversational artificial intelligence (AI) operations;
a system for performing generative AI operations;
a system implementing language models;
a system for performing generative AI operations;
a system for implementing vision language models (VLMs);
a system for implementing large language models (LLMs);
a system for implementing multi-modal language models;
a system for hosting one or more real-time streaming applications;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets; or
a system implemented at least partially using cloud computing resources.
17 . A method comprising:
receiving, using one or more circuits of a first processing element of an accelerator including a two-dimensional array of processing elements, data associated with a first pixel and data associated with one or more instructions;
determining, using the one or more circuits of the first processing element, an updated first pixel value based at least on the first pixel and the one or more instructions; and
providing, using the one or more circuits of the first processing element, data associated with the updated first pixel value as output based at least on determining the updated first pixel value.
18 . The method of claim 17 , wherein the data associated with the first pixel is associated with a first block,
wherein the accelerator comprises a plurality of processing elements that are configured to receive data associated with at least one second block, and
wherein the receiving the data associated with the first pixel comprises: receiving the data associated with the first pixel and at least one second pixel, the at least one second pixel corresponding to the at least one second block.
19 . The method of claim 18 , further comprising:
storing the data associated with the first block in a first register;
storing the data associated with the at least one second block in at least one second register; and
transferring the data associated with the first block from the first register to a first destination register; or
transferring the data associated with the second block from the second register to a second destination register,
wherein the first destination register or the second destination register comprise registers corresponding to the first processing element or one or more second processing elements.
20 . The method of claim 17 , further comprising:
obtaining data associated with at least one second pixel based at least on the one or more instructions; and
determining the updated first pixel value based at least on the first pixel and the at least one second pixel.