Multi-node cluster-based inference method and system through GPU separate allocation of pre-trained layer and fine-tuning layer of multiple deep learning models

1 . A multi-node cluster-based inference method using GPU-separated allocation of a pre-trained layer and a fine-tuning layer of multiple deep learning models, comprising:

receiving, by an endpoint unit of an inference server, an input value from a client;

distributing the input value received by the endpoint unit of the inference server, and transmitting a first input value to a first computation node including a first container for loading a neural network bundle of a first step;

performing an operation through a neural network layer of a GPU by using the first input value received by the first container of the first computation node as an input, and generating a first output value;

selecting, by the first container, a next container in which a neural network bundle of a next step is loaded, and transferring the first output value to the next container in which the next step is executed or a next computation node including the next container in which the next step is executed;

if a second computation node loads the neural network bundle of the next step, performing computation through the neural network layer of the GPU by using the first output value received by a second container of the second computations node as an input, and generating a second output value;

repeating until an n th output value is generated from an n th container of an n th computation node including the n th container that carries the neural network bundle of the last step, wherein n is an integer greater than two;

taking, by the nth container, the nth output value as a final output value and delivering the final output value to the endpoint unit; and

delivering, the endpoint unit, the final output value to the client,

wherein the GPU is included in each computation node as a split virtualized GPU fragment,

wherein at least one pre-trained layer and at least one fine-tuning layer of a model to which fine-tuning is applied through a split virtualization technique are separately stored in the GPU;

wherein the each computation node includes an inference framework part, and the inference framework part includes an additional metadata management module,

wherein the additional metadata management module is configured to load a model parameter set into a memory of the GPU, the model parameter set is stored in a form of a snapshot of the model parameter set after being divided into a certain capacity based on a GPU capacity configuration in a unit of the neural network layer of the GPU, and a recognized additional metadata included in each layer of the model which is more than one model is divided and stored in each GPU, and

wherein the GPU uses a shared GPU as pre-trained layers and separates only fine-tuning layers and loads each of the fine-tuning layers into individual GPU memories.

2 . The method of claim 1 , wherein in the step of selecting, by the first container, the next container in which the neural network bundle of the next step is loaded, and transferring the first output value to the next container in which the next step is to be executed or the next computation node including the next container in which to execute the next step,

the first container tracks a number of operations currently waiting and being processed and an expected time required for each GPU of a first GPU unit in the first container through a performance metric and a performance measurement mechanism of an inference framework constituting the first container, stores the number of operations and the expected time as a scheduling hint, receives a list of containers capable of executing the next step from a server control unit of the inference server, and selects the next container for executing the next step based on the scheduling hint and the list of containers.

3 . The method of claim 1 , wherein

the pre-trained layer and the fine-tuning layer include the additional metadata, and

the additional metadata: is a data in a form of a constant list or a binary flag to determine whether each neural network layer is the pre-trained or fine-tuning layers, the data includes at least one of format information of an output value and an input value including a size of a matrix or a vector and a representation format of individual elements, a version of each layer set, and a unique ID, so that the pre-trained layer and the fine-tuning layer are independently updated and distributed.

4 . The method of claim 1 , wherein the first container includes a first inference framework part, and

the first inference framework part recognizes the additional metadata, selects the pre-trained layer and the fine-tuning layer from a single model, and loads the selected pre-trained layers and fine-tuning layers into the GPU.

5 . A multi-node cluster-based inference system using GPU separation allocation of a pre-trained layer and a fine-tuning layer of multiple deep learning models, comprising:

an endpoint unit that receives an input value from a client, distribute the input value, deliver the distributed input value to at least one computation node including at least one container in which a neural network bundle of a first stage is loaded, receive an output value from the at least one container, and deliver a final output value to the client;

at least one computation node that receives the input value or the output value, performs computation through a neural network layer of a GPU to generate an output value, selects a next container in which a neural network bundle of a next step is loaded, forwards the output value to the next container in which the next step is to be executed or a next computation node including the next container in which to execute the next step, and when the next container in which a network bundle of a last step is loaded generates the output value, takes the output value as a final output value, and forwards the final output value to the endpoint unit; and

a server control unit that delivers a list of containers capable of executing next step to the at least one computation node,

wherein the GPU is included in each computation node as a split virtualized GPU fragment,

wherein at least one pre-trained layer and at least one fine-tuning layer of a model to which fine-tuning is applied through a split virtualization technique are separately stored in the GPU,

wherein the at least one computation node includes an inference framework part, and the inference framework part includes an additional metadata management module,

wherein the additional metadata management module is configured to load a model parameter set into a memory of the GPU,

the model parameter is stored in a form of a snapshot of the model parameter set by dividing the model parameter set to a constant capacity based on a GPU capacity configuration in a unit of the neural network layer of the GPU, a recognized additional metadata included in each layer of the model which is more than one model is divided and stored in each GPU, and

wherein the GPU uses a shared GPU for the pre-trained layers, and only the fine-tuning layers are separated and loaded into individual GPU memories respectively.

6 . The system of claim 5 ,:

wherein the at least one container tracks a number of operations currently waiting and being processed and an expected time required for each GPU of GPU units in the at least one container through a performance metric and a performance measurement mechanism of an inference framework constituting the at least one container, stores the number of operations and the expected time as a scheduling hint, receives a list of containers capable of executing the next step from the server control unit of the inference server, and selects the next container for executing the next step based on the scheduling hint and the list of containers.

7 . The system of claim 5 , wherein

the pre-trained layer and the fine-tuning layer include additional metadata, and the additional metadata:

is a data in a form of a constant list or a binary flag capable of determining whether each neural network layer is the pre-trained or fine-tuning layers,

wherein the data comprises at least one of format information of an output value and an input value including a size of a matrix or a vector and a representation format of individual elements, a version of each layer set, and a unique ID, so that the pre-trained layer and the fine-tuning layer are updated and distributed independently.

8 . The system of claim 5 , wherein

the at least one container includes an inference framework part, and

the inference framework part recognizes the additional metadata, selects the pre-trained layer and the fine-tuning layer from a single model, and loads the selected pre-trained layers and fine-tuning layers into the GPU.