Systems and methods for accelerating memory transfers and computation efficiency using a computation-informed partitioning of an on-chip data buffer and implementing computation-aware data transfer operations to the on-chip data buffer
Systems and methods for implementing accelerated memory transfers in an integrated circuit includes configuring a region of memory of an on-chip data buffer based on a neural network computation graph, wherein configuring the region of memory includes: partitioning the region of memory of the on-chip data buffer to include a first distinct sub-region of memory and a second distinct sub-region of memory; initializing a plurality of distinct memory transfer operations from the off-chip main memory to the on-chip data buffer; executing a first set of memory transfer operations that includes writing a first set of computational components to the first distinct sub-region of memory, and while executing, using the integrated circuit, a leading computation based on the first set of computational components, executing a second set of memory transfer operations to the second distinct sub-region of memory for an impending computation.
1 . A method comprising:
performing, via an integrated circuit, a plurality of memory transfer operations that write computational components of a computational operation to a first region of an on-chip data buffer, the first region being allocated for active consumption by the computational operation;
executing, via the integrated circuit, the computational operation by reading the computational components from the first region of the on-chip data buffer; and
while executing the computational operation, and concurrently with the execution of the computational operation, performing a second plurality of memory transfer operations that write computational components of a succeeding computational operation to a second region of the on-chip data buffer that (a) is distinct from the first region and (b) is not read by the computational operation during execution,
wherein the first region and the second region are simultaneously active regions of the on-chip data buffer during execution of the computational operation.
2 . The method of claim 1 , wherein:
when output data resulting from an execution of the computational operation is required for the succeeding computational operation, writing the output data of the computational operation to the second region of the on-chip data buffer.
3 . The method of claim 1 , wherein:
the computational operation comprises an operation of a first layer within a neural network,
the succeeding computational operation comprises another operation of a second layer within the neural network,
an output of the first layer is computed based on one or more computations with a set of weights of the first layer and a tensor provided to the first layer,
an output of the second layer depends on one or more computations involving the output of the first layer and a set of weights of the second layer, and
performing the plurality of memory transfer operations and the second plurality of memory transfer operations include:
writing the tensor to a first memory slice of the first region,
writing the set of weights of the first layer to a second memory slice of the first region, and
writing the set of weights of the second layer to a first memory slice of the second region.
4 . The method of claim 1 , wherein:
at least part of the computational components of the succeeding computational operation is stored within the second region of the on-chip data buffer prior to a completion of the computational operation, and
storing the at least part of the computational components prior to the completion of the computational operation minimizes a latency between the completion of the computational operation and a start of an execution of the succeeding computational operation.
5 . The method of claim 1 , further comprising:
during an execution of the succeeding computational operation based on the computational components written to the second region of the on-chip data buffer, performing a third plurality of memory transfer operations that write computational components of a new computational operation that is subsequent to the succeeding computational operation to the first region of the on-chip data buffer.
6 . The method of claim 1 , wherein the plurality of memory transfer operations and the second plurality of memory transfer operations are created based on a configuration of a neural network computation graph.
7 . The method of claim 1 , wherein:
the computational components of the computational operation include (i) an input data component and (ii) a distinct sets of coefficients data of the computational operation, and
executing the computational operation includes:
accessing the input data component and the distinct sets of coefficients data from the first region of the on-chip data buffer, and
performing one or more computations with the distinct set of coefficients and the input data component accessed from the first region of the on-chip data buffer.
8 . The method of claim 1 , wherein:
the integrated circuit comprises the on-chip data buffer and a main memory,
the computational components of the computational operation and the computational components of the succeeding computational operation are stored in the main memory, and
performing the plurality and the second plurality of memory transfer operations cause the computational components of the computational operation and the succeeding computational operation to be fetched from the main memory and stored in the on-chip data buffer.
9 . The method of claim 1 , wherein:
a neural network computation graph defines a plurality of computational operations including the computational operation and the succeeding computational operation,
each distinct computational operation of the neural network computation graph corresponds to a distinct operation of a distinct layer within a neural network, and
a configuration of the neural network computation graph that identifies graphical edges of flows of input into each distinct computational operation define distinct computational components required by the distinct layer of the neural network.
10 . The method of claim 1 , wherein:
the first region of the on-chip data buffer includes a plurality of distinct memory slices, wherein each of the plurality of distinct memory slices relates to a subdivision of the first region of the on-chip data buffer, and
the plurality of memory transfer operations write each of the computational components of the computational operation into a distinct memory slice of the plurality of distinct memory slices.
11 . A method of mitigating computational latency by controlling a flow of data within an integrated circuit, the method comprising:
partitioning an on-chip data buffer into a plurality of distinct memory regions, wherein:
a first memory region of the plurality of distinct memory regions is configured to store computational components for a target computational operation, and
a second memory region of the plurality of distinct memory regions is configured to store computational components for a computational operation that is subsequent to the target computational operation;
performing a first set of memory transfer operations based on a configuration of a neural network computation graph that writes at least a subset of a plurality of computational components for a first computational operation to the first memory region;
executing the first computational operation by reading the subset of the plurality of computational components from the first memory region of the on-chip data buffer; and
concurrently with executing the target computational operation, performing a second set of memory transfer operations based on the configuration of a neural network computation graph that writes at least a subset of a plurality of computational components for a second computational operation to the second memory region,
wherein the first memory region and the second memory region are spatially distinct regions of the on-chip data buffer that are simultaneously active during execution of the target computational operation, and wherein the second memory region is not read during execution of the target computational operation.
12 . The method of claim 11 , wherein
writing the subset of computational components for the second computational operation to the second memory region minimizes a latency between a completion of the first computational operation and a start of the second computational operation.
13 . The method of claim 11 , wherein:
the first computational operation comprises an operation of a first layer in a neural network and the second computational operation comprises another operation of relates to a second layer in the neural network,
a computational output of the second layer depends on one or more computational outputs of the first layer that define input data into the second layer and a set of weights of the second layer, and
performing the second set of memory transfer operations include:
writing the set of weights of the second layer to a first memory slice of the second memory region.
14 . The method of claim 13 , wherein the first computational operation computes the output of the first layer, the method further comprising:
after executing the first computational operation:
writing the computational output of the first layer to a second memory slice of the second memory region; and
executing the second computational operation based on the plurality of computational components for the second computational operation, including the set of weights stored in the first memory slice of the second memory region and the computational output of the first layer stored in the second memory slice of the second memory region.
15 . A method of reducing latency in an operation of an integrated circuit, the method comprising:
configuring an on-chip data buffer based on a neural network computation graph, wherein the configuring of the on-chip data buffer includes:
partitioning the on-chip data buffer to include a first memory region that is configured to receive requisite computational components for a target computation associated with the neural network computation graph; and
partitioning the on-chip data buffer to include a second memory region that is configured to receive requisite computational components for an impending computation succeeding the target computation in the neural network computation graph;
concurrently:
(a) executing the target computation by reading the requisite computational components from the first memory region and
(b) writing, via one or more memory transfer operations, requisite computational components associated with the impending computation to the second memory region based on the configuration of the on-chip data buffer,
wherein the first memory region and the second memory region are spatially distinct regions of the on-chip data buffer that are simultaneously active during execution of the target computation, and wherein the second memory region is not read during execution of the target computation.
16 . The method of claim 15 , further comprising:
writing, into the first memory region, a set of requisite computational components for the target computation; and
executing, by at least one processing core of an array of processing cores, the target computation based at least on the set of requisite computational components stored in the first memory region.
17 . The method of claim 16 , further comprising:
while executing the target computation:
writing, to the second memory region, a set of requisite computational components for an impending, non-active computation.
18 . The method according to claim 17 , wherein:
the writing the set of requisite computational components to the second memory region occurs minimizes a latency between the target computation and the impending computation.
19 . The method according to claim 17 , wherein:
the set of requisite computational components for the target computation includes (i) an input component and (ii) a distinct set of weights associated with a target layer of the neural network computation graph, and
the target computation is a result of one or more operations with the input component and the distinct set of weights.
20 . The method according to claim 19 , wherein the target computation computes an output data tensor, the method further comprising:
based on computing the output data tensor, writing the output data tensor to the second memory region of the on-chip data buffer; and
based on writing the output data tensor to the second memory region of the on-chip data buffer, executing, via the at least one processing core of the array of processing cores, the impending, non-active computation based on the output data tensor and the set of requisite computational components associated with the impending, non-active computation.