Methods and devices for lossless switching between multiple parameter sets for serving model

1 . A method for processing data in a multi-model single-engine (MMSE) system, comprising:

receiving, by a graphic processing engine in the MMSE system, a first input query, wherein the first input query is associated with a first category identification;

obtaining, by the graphic processing engine in the MMSE system, a first set of model parameters by switching between multiple sets of model parameters based on the first input query, wherein obtaining the first set of model parameters comprises accessing, by the graphic processing engine in the MMSE system, the first set of model parameters from a graphic processing unit (GPU) memory in the graphic processing engine based on the first category identification, wherein the GPU memory stores the multiple sets of model parameters associated with multiple categories comprising a first category; and

inferring, by the graphic processing engine in the MMSE system, a first output for the first input query based on the first set of model parameters;

and wherein accessing the first set of model parameters comprises:

pre-defining a mapping between the first category identification and a first physical region in the GPU memory, wherein the mapping is included in a compute kernel that is executed on the graphic processing engine;

obtaining a first offset based on the first category identification and a first layer index of the compute kernel;

obtaining a first physical location in the first physical region based on the first offset; and

directly accessing the first set of model parameters based on the first physical location in the GPU memory.

2 . The method of claim 1 , further comprising:

accessing, by the graphic processing engine in the MMSE system, the first set of model parameters from the GPU memory in the graphic processing engine based on the first input query, wherein the GPU memory stores the multiple sets of model parameters.

3 . The method of claim 2 , wherein the first input query is associated with a first category identification that indicates the first set of model parameters that are obtained by training a plurality of first training data associated with the first category,

wherein the GPU memory stores the multiple sets of model parameters associated with multiple categories comprising the first category, and

wherein the multiple sets of model parameters are obtained by training a plurality of training data associated with the multiple categories, and

wherein the method further comprises:

accessing, by the graphic processing engine in the MMSE system, the first set of model parameters from the GPU memory based on the first category identification; and

inferring, by the graphic processing engine in the MMSE system, the first output for the first input query based on the first set of model parameters that are obtained from the GPU memory.

4 . The method of claim 3 , further comprising:

receiving, by the graphic processing engine, a second input query associated with a second category, wherein the second input query comprises a second category identification that indicates a second set of model parameters that are obtained by training a plurality of second training data associated with the second category;

accessing, by the graphic processing engine, the second set of model parameters from the GPU memory based on the second category identification to switch the graphic processing engine from the first set of model parameters to the second set of model parameters, wherein the multiple categories further comprise the second category; and

inferring, by the graphic processing engine, a second output for the second input query based on the second set of model parameters that are obtained from the GPU memory.

5 . The method of claim 4 , wherein the mapping further comprises the second category identification and a second physical region in the GPU memory, and

wherein accessing the second set of model parameters from the GPU memory based on the second category identification comprises:

obtaining a second offset based on the second category identification and a second layer index of the compute kernel;

obtaining a second physical location in the second physical region based on the second offset; and

directly accessing the second set of model parameters based on the second physical location in the GPU memory.

6 . The method of claim 1 , further comprising:

obtaining, by the graphic processing engine in the MMSE system, multiple sets of compressed model parameters by compressing the multiple sets of model parameters; and

storing, by the graphic processing engine in the MMSE system, the multiple sets of compressed model parameters in the GPU memory.

7 . The method of claim 6 , wherein compressing the multiple sets of model parameters associated with the multiple categories comprises:

quantizing and compressing the multiple sets of model parameters using mix-precision quantization.

8 . An apparatus for processing data, comprising:

one or more processors; and

a memory coupled to the one or more processors and configured to store instructions executable by the one or more processors,

wherein the one or more processors, upon execution of the instructions, are configured to perform acts comprising:

receiving, by a graphic processing engine in the one or more processors, a first input query, wherein the first input query is associated with a first category identification;

obtaining, by the graphic processing engine, a first set of model parameters by switching between multiple sets of model parameters based on the first input query, wherein obtaining the first set of model parameters comprises accessing, by the graphic processing engine in the MMSE system, the first set of model parameters from a graphic processing unit (GPU) memory in the graphic processing engine based on the first category identification, wherein the GPU memory stores the multiple sets of model parameters associated with multiple categories comprising a first category; and

inferring, by the graphic processing engine, a first output for the first input query based on the first set of model parameters;

and wherein accessing the first set of model parameters comprises:

pre-defining a mapping between the first category identification and a first physical region in the GPU memory, wherein the mapping is included in a compute kernel that is executed on the graphic processing engine;

obtaining a first offset based on the first category identification and a first layer index of the compute kernel;

obtaining a first physical location in the first physical region based on the first offset; and

directly accessing the first set of model parameters based on the first physical location in the GPU memory.

9 . The apparatus of claim 8 , wherein the one or more processors are configured to perform acts further comprising:

accessing, by the graphic processing engine, the first set of model parameters from the GPU memory in the graphic processing engine based on the first input query, wherein the GPU memory stores the multiple sets of model parameters.

10 . The apparatus of claim 9 , wherein the first input query is associated with a first category identification that indicates the first set of model parameters that are obtained by training a plurality of first training data associated with the first category,

wherein the GPU memory stores the multiple sets of model parameters associated with multiple categories comprising the first category, and

wherein the multiple sets of model parameters are obtained by training a plurality of training data associated with the multiple categories, and

wherein the one or more processors are configured to perform acts further comprising:

accessing, by the graphic processing engine, the first set of model parameters from the GPU memory based on the first category identification; and

inferring, by the graphic processing engines, the first output for the first input query based on the first set of model parameters that are obtained from the GPU memory.

11 . The apparatus of claim 10 , wherein the one or more processors are configured to perform acts further comprising:

receiving, by the graphic processing engine, a second input query associated with a second category, wherein the second input query comprises a second category identification that indicates a second set of model parameters that are obtained by training a plurality of second training data associated with the second category;

accessing, by the graphic processing engine, the second set of model parameters from the GPU memory based on the second category identification to switch the graphic processing engine from the first set of model parameters to the second set of model parameters, wherein the multiple categories further comprise the second category; and

inferring, by the graphic processing engine, a second output for the second input query based on the second set of model parameters that are obtained from the GPU memory.

12 . The apparatus of claim 11 , wherein the mapping further comprises the second category identification and a second physical region in the GPU memory, and

wherein accessing the second set of model parameters from the GPU memory based on the second category identification comprises:

obtaining a second offset based on the second category identification and a second layer index of the compute kernel;

obtaining a second physical location in the second physical region based on the second offset; and

directly accessing the second set of model parameters based on the second physical location in the GPU memory.

13 . The apparatus of claim 8 , wherein the one or more processors are configured to perform acts further comprising:

obtaining, by the graphic processing engine, multiple sets of compressed model parameters by compressing the multiple sets of model parameters; and

storing, by the graphic processing engine, the multiple sets of compressed model parameters in the GPU memory.

14 . The apparatus of claim 13 , wherein compressing the multiple sets of model parameters associated with the multiple categories comprises:

quantizing and compressing the multiple sets of model parameters using mix-precision quantization.

15 . A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform acts comprising:

receiving, by a graphic processing engine in the one or more processors, a first input query, wherein the first input query is associated with a first category identification;

obtaining, by the graphic processing engine, a first set of model parameters by switching between multiple sets of model parameters based on the first input query, wherein obtaining the first set of model parameters comprises accessing, by the graphic processing engine in the MMSE system, the first set of model parameters from a graphic processing unit (GPU) memory in the graphic processing engine based on the first category identification; and

inferring, by the graphic processing engine, a first output for the first input query based on the first set of model parameters;

and wherein the GPU memory stores the multiple sets of model parameters associated with multiple categories comprising a first category, and wherein accessing the first set of model parameters comprises:

pre-defining a mapping between the first category identification and a first physical region in the GPU memory, wherein the mapping is included in a compute kernel that is executed on the graphic processing engine;

obtaining a first offset based on the first category identification and a first layer index of the compute kernel;

obtaining a first physical location in the first physical region based on the first offset; and

directly accessing the first set of model parameters based on the first physical location in the GPU memory.

16 . The non-transitory computer-readable storage medium of claim 15 , wherein the one or more processors are caused to perform acts further comprising:

accessing, by the graphic processing engine, the first set of model parameters from the GPU memory in the graphic processing engine based on the first input query, wherein the GPU memory stores the multiple sets of model parameters.

17 . The non-transitory computer-readable storage medium of claim 15 , wherein the first input query is associated with a first category identification that indicates the first set of model parameters that are obtained by training a plurality of first training data associated with the first category,

wherein the GPU memory stores the multiple sets of model parameters associated with multiple categories comprising the first category, and

wherein the multiple sets of model parameters are obtained by training a plurality of training data associated with the multiple categories, and

wherein the one or more processors are caused to perform acts further comprising:

accessing, by the graphic processing engine, the first set of model parameters from the GPU memory based on the first category identification; and

inferring, by the graphic processing engine, the first output for the first input query based on the first set of model parameters that are obtained from the GPU memory.

18 . The non-transitory computer-readable storage medium of claim 15 , wherein the one or more processors are caused to perform acts further comprising:

obtaining, by the graphic processing engine, multiple sets of compressed model parameters by compressing the multiple sets of model parameters; and

storing, by the graphic processing engine, the multiple sets of compressed model parameters in the GPU memory.

19 . The non-transitory computer-readable storage medium of claim 18 , wherein compressing the multiple sets of model parameters associated with the multiple categories comprises:

quantizing and compressing the multiple sets of model parameters using mix-precision quantization.

20 . The non-transitory computer-readable storage medium of claim 15 , wherein the one or more processors are configured to perform acts further comprising:

receiving, by the graphic processing engine, a second input query associated with a second category, wherein the second input query comprises a second category identification that indicates a second set of model parameters that are obtained by training a plurality of second training data associated with the second category;

accessing, by the graphic processing engine, the second set of model parameters from the GPU memory based on the second category identification to switch the graphic processing engine from the first set of model parameters to the second set of model parameters, wherein the multiple categories further comprise the second category; and

inferring, by the graphic processing engine, a second output for the second input query based on the second set of model parameters that are obtained from the GPU memory.