Data labeling system and method, and data labeling manager
View Patent ↗Embodiments of this application disclose a data labeling system and method, and a data labeling manager. An example system includes a data labeling manager, a labeling model storage repository, and a basic computing unit storage repository. The data labeling manager receives a data labeling request, obtains a target basic computing unit, allocates a hardware resource to the target basic computing unit, establishes a target computing unit, obtains first storage path information of basic parameter data of a first labeling model, and sends the first storage path information to the target computing unit. The target computing unit obtains the basic parameter data of the to-be-used labeling model by using the first storage path information, combines a target model inference framework and the basic parameter data of the first labeling model to obtain the first labeling model, and labels to-be-labeled data by using the first labeling model.
1 . A data labeling system comprising at least one processor and at least one memory coupled to the at least one processor, wherein the at least one memory stores programming instructions for execution by the at least one processor to:
receive a data labeling request from a client, wherein:
the data labeling request comprises a first model identifier of a first labeling model and hardware resource allocation information, and
the hardware resource allocation information comprises at least one of a quantity of central processing units (CPUs) or a quantity of graphics processing units (GPUs);
obtain, from a basic computing unit storage repository, a target basic computing unit stored in the basic computing unit storage repository, wherein:
the target basic computing unit is one of a plurality of basic computing units stored in the basic computing unit storage repository,
the target basic computing unit comprises a target model inference framework corresponding to the first labeling model,
the target basic computing unit is established as a target computing unit based on an allocation of hardware resources to the target basic computing unit, and
the hardware resources of the target computing unit are based on the hardware resource allocation information;
send first storage path information to the target computing unit;
store a correspondence between a first model identifier and the first storage path information, wherein:
the first storage path information indicates a storage path of first basic parameter data of the first labeling model,
the first basic parameter data comprises a trained value of a trainable parameter of the first labeling model, and
the trained value results from training the first labeling model;
obtain, from a labeling model storage repository and using the target computing unit, the first basic parameter data of the first labeling model stored in the labeling model storage repository based on the first storage path information, wherein the first labeling model is based on the target model inference framework and the first basic parameter data;
receive to-be-labeled data; and
input the to-be-labeled data into the first labeling model to label the to-be-labeled data.
2 . The data labeling system according to claim 1 , wherein the programming instructions are for execution by the at least one processor further to:
receive the first model identifier and the first basic parameter data from the client;
store the first basic parameter data in the labeling model storage repository; and
correspondingly store the first model identifier and the first storage path information.
3 . The data labeling system according to claim 1 , wherein:
the data labeling request further comprises a first data identifier of the to-be-labeled data,
second storage path information corresponds to the first data identifier of the to-be-labeled data, and
the to-be-labeled data is based on the second storage path information.
4 . The data labeling system according to claim 1 , wherein the data labeling request further comprises a framework identifier of the target model inference framework; and the programming instructions are for execution by the at least one processor to:
obtain, from the basic computing unit storage repository based on the framework identifier of the target model inference framework, the target basic computing unit comprising the target model inference framework.
5 . The data labeling system according to claim 1 , wherein the target basic computing unit corresponds to the first model identifier.
6 . The data labeling system according to claim 1 , wherein the programming instructions are for execution by the at least one processor further to:
receive, in a process in which the to-be-labeled data is labeled by using the first labeling model, a labeling model replacement request from the client, wherein:
the labeling model replacement request comprises a second model identifier of a second labeling model,
third storage path information corresponds to the second model identifier,
the third storage path information indicates a storage path of second basic parameter data of the second labeling model, and
the second basic parameter data of the second labeling model is based on the third storage path information;
replace the first basic parameter data with the second basic parameter data in the target model inference framework, to obtain the second labeling model; and
input data that has not been labeled in the to-be-labeled data into the second labeling model to label the data that has not been labeled.
7 . A method comprising:
receiving a data labeling request from a client, wherein:
the data labeling request comprises a first model identifier of a first labeling model and hardware resource allocation information, and
the hardware resource allocation information comprises at least one of a quantity of central processing units (CPUs) or a quantity of graphics processing units (GPUs), and
a target model inference framework corresponds to the first labeling model;
storing a correspondence between the first model identifier and first storage path information, wherein:
a target basic computing unit comprises the target model inference framework,
the target basic computing unit is one of a plurality of basic computing units stored in a basic computing unit storage repository,
the target basic computing unit is established as a target computing unit based on an allocation of hardware resources to the target basic computing unit, and
the hardware resources of the target computing unit are based on the hardware resource allocation information,
the first storage path information indicates a storage path of first basic parameter data of the first labeling model,
the first basic parameter data comprises a trained value of a trainable parameter of the first labeling model, and
the trained value results from training the first labeling model; and
sending the first storage path information to the target computing unit, wherein:
the first storage path information is associated with the first basic parameter data, and
the target model inference framework and the first basic parameter data are associated with the first labeling model.
8 . The method according to claim 7 , wherein the first basic parameter data is stored in a labeling model storage repository, and the target model inference framework is stored in the basic computing unit storage repository.
9 . The method according to claim 8 , further comprising:
obtaining the target basic computing unit from the basic computing unit storage repository.
10 . The method according to claim 9 , wherein;
the data labeling request further comprises a framework identifier of the target model inference framework, and
the target basic computing unit comprising the target model inference framework is based on the framework identifier of the target model inference framework.
11 . The method according to claim 9 , wherein the obtaining the target basic computing unit from the basic computing unit storage repository comprises:
obtaining, based on a stored correspondence between a model identifier and a basic computing unit, the target basic computing unit corresponding to the first model identifier.
12 . The method according to claim 7 , wherein the method further comprises:
receiving the first model identifier and the first basic parameter data from the client;
storing the first basic parameter data in a labeling model storage repository; and
correspondingly storing the first model identifier and the first storage path information.
13 . The method according to claim 7 , wherein the data labeling request further comprises a first data identifier of the to-be-labeled data; and the method further comprises:
sending second storage path information, wherein:
the second storage path information is associated with the to-be-labeled data, and
the second storage path information corresponds to the first data identifier of the to-be-labeled data.
14 . The method according to claim 7 , wherein the method further comprises:
receiving a labeling model replacement request from the client, wherein the labeling model replacement request comprises a second model identifier of a second labeling model, wherein:
third storage path information corresponds to the second model identifier, and
the third storage path information indicates a storage path of second basic parameter data of the second labeling model; and
sending a model replacement instruction, wherein:
the model replacement instruction comprises the third storage path information, and
the second basic parameter data of the second labeling model replaces the first basic parameter data in the target model inference framework to obtain the second labeling model.
15 . The method according to claim 7 , wherein the first basic parameter data is a basic parameter data of a public labeling model, or the first basic parameter data is a basic parameter data of an uploaded labeling model.
16 . The method according to claim 7 , wherein the trainable parameter comprises one or more parameters of weight.
17 . An apparatus, comprising at least one processor and at least one memory coupled to the at least one processor, wherein the at least one memory stores programming instructions for execution by the at least one processor to:
receive a data labeling request from a client, wherein:
the data labeling request comprises a first model identifier of a first labeling model and hardware resource allocation information,
the hardware resource allocation information comprises at least one of a quantity of central processing units (CPUs) or a quantity of graphics processing units (GPUs),
a target model inference framework corresponds to the first labeling model,
a target basic computing unit is one of a plurality of basic computing units stored in a basic computing unit storage repository,
the target basic computing unit comprises the target model inference framework,
the target basic computing unit is established as a target computing unit based on an allocation of hardware resources to the target basic computing unit, and
the hardware resources of the target computing unit are based on the hardware resource allocation information;
store a correspondence between a first model identifier and first storage path information, wherein:
the first storage path information corresponds to the first model identifier of the first labeling model,
the first storage path information indicates a storage path of first basic parameter data of the first labeling model,
the first basic parameter data comprises a trained value of a trainable parameter of the first labeling model, and
the trained value results from training the first labeling model; and
send the first storage path information to the target computing unit, wherein:
the first storage path information is associated with the first basic parameter data, and
the target model inference framework and the first basic parameter data are associated with the first labeling model.
18 . The apparatus according to claim 17 , wherein the first basic parameter data is stored in a labeling model storage repository, and the target model inference framework is stored in the basic computing unit storage repository.
19 . The apparatus according to claim 18 , wherein the programming instructions are for execution by the at least one processor to:
obtain the target basic computing unit from the basic computing unit storage repository.
20 . The apparatus according to claim 19 , wherein the programming instructions are for execution by the at least one processor to:
obtain, based on a framework identifier of the target model inference framework, the target basic computing unit comprising the target model inference framework from the basic computing unit storage repository.
21 . The apparatus according to claim 19 , wherein the target basic computing unit corresponds to the first model identifier.
22 . The apparatus according to claim 18 , wherein the programming instructions are for execution by the at least one processor to:
send second storage path information, wherein:
the second storage path information is associated with the to-be-labeled data, and
the second storage path information corresponds to the first data identifier of the to-be-labeled data.
23 . The apparatus according to claim 18 , wherein the programming instructions are for execution by the at least one processor to:
receive a labeling model replacement request from a client, wherein:
the labeling model replacement request comprises a second model identifier of a second labeling model,
third storage path information corresponds to the second model identifier, and
the third storage path information indicates a storage path of second basic parameter data of the second labeling model; and
send a model replacement instruction, wherein;
the model replacement instruction comprises the third storage path information,
the third storage path information is associated with second basic parameter data of the second labeling model, and
the second basic parameter data of the second labeling model replaces the first basic parameter data in the target model inference framework to obtain the second labeling model.
24 . The apparatus according to claim 17 , wherein the programming instructions are for execution by the at least one processor to:
receive the first model identifier and the first basic parameter data from a client;
store the first basic parameter data in a labeling model storage repository; and
correspondingly store the first model identifier and the first storage path information.