Granted Patent B2

US 12,657,130 · App. 18/494,696 · Granted Jun 16, 2026

Inventors: Jinwook Oh (Seongnam-si, KR); Jinseok Kim (Seongnam-si, KR); Donghan Kim (Seongnam-si, KR); Kyeongryeol Bong (Seongnam-si, KR)

Assignee: Rebellions Inc.

G06F12/0875G06F9/30043G06F9/30047G06F9/383G06F15/8023G06N3/04G06F2212/1016

Abstract

Claims (59)

1 . A neural processing device comprising:

a processing unit configured to receive an input activation and a weight, and perform a two-dimensional matrix calculation with the input activation and the weight to generate an output activation;

a first memory;

a load-store unit (LSU) configured to perform memory access operations between the first memory and a second memory;

an activation buffer configured to provide the input activation to the processing unit, receive the output activation from the processing unit, and temporarily store the input activation and the output activation; and

an activation LSU configured to transmit the input activation from the first memory to the activation buffer and transmit the output activation from the activation buffer to the first memory,

wherein the memory access operations include a main memory access operation for a current processing operation that is performed by the processing unit, and a standby memory access operation for a standby processing operation that is performed by the processing unit after the current processing operation, and

wherein the standby memory access operation is performed by using a bandwidth that is not used by the main memory access operation within a bandwidth of a data channel.

2 . The neural processing device of claim 1 , wherein the first memory is configured to:

temporarily store program, the input activation and the weight used for calculation which is performed by the processing unit;

transmit the stored program, input activation and weight to the processing unit; and

temporarily store the output activation received from the processing unit.

3 . The neural processing device of claim 1 , wherein a level of the first memory is equal to a level of the processing unit, and a level of the second memory is different from the level of the first memory.

4 . The neural processing device of claim 1 , wherein the LSU is further configured to:

receive a standby load instruction or a standby store instruction from a task controller;

fetch and issue the received standby load instruction or the received standby store instruction;

load data for the standby load instruction from the second memory to the first memory; and

store data for the standby store instruction from the first memory to the second memory.

5 . The neural processing device of claim 1 , wherein the LSU is further configured to:

fetch and issue a main load instruction or a main store instruction;

load data for the main load instruction from the second memory to the first memory; and

store data for the main store instruction from the first memory to the second memory.

6 . The neural processing device of claim 1 , wherein data transmitted for the main memory access has a higher priority than data transmitted for the standby memory access.

7 . The neural processing device of claim 6 , wherein the priority is tagged to data between the first memory and the second memory.

8 . The neural processing device of claim 1 , wherein the LSU is further configured to transmit data between the first memory and the second memory in a round-robin manner.

9 . A neural processing device comprising:

at least one neural processor, each of which comprising at least one neural core;

a first shared memory shared between the at least one neural processor; and

a global interconnection comprising a data channel configured to transmit data between the at least one neural processor and the shared memory,

wherein each of the neural core comprises:

a processing unit configured to perform a two-dimensional matrix calculation;

a private memory; and

a load-store unit (LSU) configured to:

perform memory access operations between the private memory and at least one shared memory,

load data from the second memory to the first memory for a load instruction or a standby load instruction, and

store data from the first memory to the second memory for the store instruction or the standby store instruction,

wherein the memory access operations include a main memory access operation for a current processing operation that is performed by the processing unit, and a standby memory access operation for a standby processing operation that is performed by the processing unit after the current processing operation,

wherein the standby memory access operation is performed by using a bandwidth that is not used by the main memory access operation within a bandwidth of the data channel,

wherein the main memory access operation includes a main load instruction and a main store instruction, and

wherein the standby memory access operation includes the standby load instruction and the standby store instruction.

10 . The neural processing device of claim 9 , further comprising a local interconnection configured to transmit data between the at least one neural cores,

wherein the LSU performs the standby memory access operation by using a bandwidth that is not used by the main memory access operation within a bandwidth of the local interconnection.

11 . The neural processing device of claim 9 , wherein the main memory access operation has a higher priority than the standby memory access operation.

12 . The neural processing device of claim 9 , wherein the at least one shared memory comprises the first shared memory and a second shared memory, and

the second shared memory is configured to be disposed outside the at least one neural processor.

13 . The neural processing device of claim 12 , wherein the LSU is operable coupled with at least one of the first shared memory or the second shared memory through the data channel.

14 . The neural processing device of claim 12 , wherein a level of the private memory is different from a level of the first shared memory and a level of the second shared memory.

15 . A memory access method of a neural processing device including a load-store unit (LSU), an activation LSU, a processing unit and an activation buffer, comprising:

performing, by the LSU, a main memory access operation to load a first data for a first work from a second memory to a first memory, the first data including a first input activation and a first weight and the first work being a two-dimensional matrix calculation of a first layer of the neural network;

temporarily storing, by the LSU, the first input activation in the activation buffer;

receiving, by the processing unit, the first input activation from the activation buffer and the first weight;

generating, by the processing unit, a first output activation by performing the two-dimensional matrix calculation of the first work with the first input activation and the first weight;

storing the first output activation in the activation buffer;

transmitting, by the activation LSU, the first input activation from the first memory to the activation buffer and transmitting the first output activation from the activation buffer to the first memory;

performing, by the LSU, a standby memory access operation to load a second data for a second work to be executed after the first work, the second data including a second input activation and a second weight and the second work being a two-dimensional matrix calculation of a second layer of the neural network;

temporarily storing, by the LSU, the second input activation in the activation buffer;

receiving, by the processing unit, the second input activation from the activation buffer and the second weight; and

generating, by the processing unit, a second output activation by performing the two-dimensional matrix calculation of the second work with the second input activation and the second weight,

wherein the standby memory access operation is performed by using a bandwidth that is not used by the main memory access operation within a bandwidth of a data channel.

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (13)