General machine learning model, and model file generation and parsing method
Disclosed are a general machine learning model generation method and apparatus, and a computer device and a storage medium. The method comprises: acquiring task parameters of a machine learning task (S 1201 ); performing classification processing on the task parameters to obtain task instructions and model parameters (S 1202 ); aggregating the task instructions and the model parameters according to a data type to obtain stack data and heap data (S 1203 ); and integrating the stack data and the heap data to obtain a general machine learning model (S 1204 ). By means of the method, compiled results of a corresponding general model in the running of an algorithm can be directly executed, which avoids repetitive compilation, thus greatly improving the efficiency of machine learning algorithm implementation and shortening the time from compilation to obtaining execution results.
1. A method for generating a machine learning secondary model file, wherein the machine learning secondary model file comprises a secondary model and a secondary model directory, and the generation method comprises:
obtaining a general-purpose machine learning model; wherein, the general-purpose machine learning model is generated by integrating a stack data and a heap data, the stack data and the heap data are respectively generated by aggregating the task instructions and the model parameters of task parameters of a machine learning task;
performing storage optimization on the general-purpose machine learning model to generate the secondary model;
calculating a storage offset of the secondary model;
generating a secondary model directory according to the secondary model and the storage offset of the secondary model; and
generating a machine learning secondary model file according to the secondary model and the secondary model directory;
obtaining a file header and a file tailer of the machine learning secondary model file; and
generating the machine learning secondary model file according to the file header, the secondary model directory, the general-purpose machine learning model, and the file tail;
wherein calculating a storage offset of the secondary model further comprising:
obtaining a size of a storage space required for each general-purpose machine learning model and a count of the secondary models,
obtaining a storage order of the secondary models, and
calculating a storage offset of each secondary model according to the size of the storage space required for each secondary model, the count of the secondary models, and the storage order of the secondary models.
2. The method of claim 1 , wherein the step of performing storage optimization on the general-purpose machine learning model to generate the secondary model includes: compressing or encrypting the general-purpose machine learning model to generate the secondary model.
3. The method of claim 1 , wherein the step of generating a machine learning secondary model file according to the general-purpose machine learning model and the model directory includes:
creating an identification code of a machine learning secondary model file, and
generating a machine learning secondary model file according to the identification code of the model file, the secondary model, and the secondary model directory.
4. The method of claim 1 , wherein the step of generating a machine learning secondary model file according to the secondary model and the model directory includes:
creating a check code or an error correction code of the machine learning secondary model file, generating the machine learning secondary model file according to the check code or the error correction code of the machine learning secondary model file, the secondary model, and the secondary model directory.
5. A system for generating a machine learning secondary model file, wherein the machine learning secondary model file comprises a secondary model and a secondary model directory, comprising:
a model filler configured to obtain a general-purpose machine learning model; wherein, the general-purpose machine learning model is generated by integrating a stack data and a heap data, the stack data and the heap data are respectively generated by aggregating the task instructions and the model parameters of task parameters of a machine learning task;
a model storage optimizer configured to perform storage optimization on the general-purpose machine learning model to generate the secondary model;
a storage offset calculating module configured to calculate a storage offset of the secondary model;
a directory generator configured to generate a secondary model directory according to the secondary model and the storage offset of the secondary model; and
a file generator configured to generate a machine learning secondary model file according to the secondary model and the secondary model directory;
a file generator further configured to obtain a file header and a file tailer of the machine learning secondary model file and generate the machine learning secondary model file according to the file header, the secondary model directory, the general-purpose machine learning model, and the file tail;
wherein the storage offset calculating module is further configured to:
obtain a size of a storage space required for each general-purpose machine learning model and a count of the secondary models,
obtain a storage order of the secondary models, and
calculate a storage offset of each secondary model according to the size of the storage space required for each secondary model, the count of the secondary models, and the storage order of the secondary models.
6. The system of claim 5 , wherein the model storage optimizer is further configured to compress or encrypting the general-purpose machine learning model to generate the secondary model.
7. The system of claim 5 , wherein the file generator is further configured to:
create an identification code of a machine learning secondary model file, and
generate a machine learning secondary model file according to the identification code of the model file, the secondary model, and the secondary model directory.
8. The system of claim 5 , wherein the file generator is further configured to:
create a check code or an error correction code of the machine learning secondary model file, generate the machine learning secondary model file according to the check code or the error correction code of the machine learning secondary model file, the secondary model, and the secondary model directory.
9. A non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform operations for generating a machine learning secondary model file, comprising:
obtaining a general-purpose machine learning model; wherein, the general-purpose machine learning model is generated by integrating a stack data and a heap data, the stack data and the heap data are respectively generated by aggregating the task instructions and the model parameters of task parameters of a machine learning task;
performing storage optimization on the general-purpose machine learning model to generate the secondary model;
calculating a storage offset of the secondary model;
generating a secondary model directory according to the secondary model and the storage offset of the secondary model; and
generating a machine learning secondary model file according to the secondary model and the secondary model directory;
obtaining a file header and a file tailer of the machine learning secondary model file; and
generating the machine learning secondary model file according to the file header, the secondary model directory, the general-purpose machine learning model, and the file tail;
wherein calculating a storage offset of the secondary model further comprising:
obtaining a size of a storage space required for each general-purpose machine learning model and a count of the secondary models,
obtaining a storage order of the secondary models, and
calculating a storage offset of each secondary model according to the size of the storage space required for each secondary model, the count of the secondary models, and the storage order of the secondary models.
10. The non-transitory computer-readable medium of claim 9 , wherein the performing storage optimization on the general-purpose machine learning model to generate the secondary model includes: compressing or encrypting the general-purpose machine learning model to generate the secondary model.
11. The non-transitory computer-readable medium of claim 9 , wherein the generating a machine learning secondary model file according to the general-purpose machine learning model and the model directory includes:
creating an identification code of a machine learning secondary model file, and
generating a machine learning secondary model file according to the identification code of the model file, the secondary model, and the secondary model directory.
12. The non-transitory computer-readable medium of claim 9 , wherein the generating a machine learning secondary model file according to the secondary model and the model directory includes:
creating a check code or an error correction code of the machine learning secondary model file, generating the machine learning secondary model file according to the check code or the error correction code of the machine learning secondary model file, the secondary model, and the secondary model directory.