General machine learning model, and model file generation and parsing method
Disclosed are a general machine learning model generation method and apparatus, and a computer device and a storage medium. The method comprises: acquiring task parameters of a machine learning task (S 1201 ); performing classification processing on the task parameters to obtain task instructions and model parameters (S 1202 ); aggregating the task instructions and the model parameters according to a data type to obtain stack data and heap data (S 1203 ); and integrating the stack data and the heap data to obtain a general machine learning model (S 1204 ). By means of the method, compiled results of a corresponding general model in the running of an algorithm can be directly executed, which avoids repetitive compilation, thus greatly improving the efficiency of machine learning algorithm implementation and shortening the time from compilation to obtaining execution results.
1. A method for generating a general-purpose machine learning model file, wherein the general-purpose machine learning model file includes a general-purpose machine learning model and a model directory, and the generation method comprises:
obtaining, by an external interface module, task parameters of a machine learning task;
classifying, by a classification processing module, the task parameters to obtain task instructions and model parameters;
aggregating, by a parameter aggregating module, the task instructions and the model parameters according to data types to generate stack data and heap data;
integrating, by a model generation module, the stack data and the heap data to obtain a general-purpose machine learning model;
calculating, by a storage offset calculation module, a storage offset of the general-purpose machine learning model;
obtaining, by the storage offset calculation module, a size of a storage space occupied by each general-purpose machine learning model and a count of the general-purpose machine learning models;
obtaining, by the storage offset calculation module, a storage order of the general-purpose machine learning models;
calculating, by the storage offset calculation module, a storage offset of each general-purpose machine learning model according to the size of the storage space occupied by each general-purpose machine learning model, the count of the general-purpose machine learning models, and the storage order of the general-purpose machine learning models;
generating, by a model directory generation module, the model directory according to the general-purpose machine learning model and the storage offset of the general-purpose machine learning model;
generating, by a model file generation module, the general-purpose machine learning model file according to the general-purpose machine learning model and the model directory;
creating, by the model file generation module, a check code or an error correction code of the general-purpose machine learning model file;
generating, by the model file generation module, the general-purpose machine learning model file according to the check code or the error correction code of the general-purpose machine learning model file, the general-purpose machine learning model, and the model directory;
calculating, by the model file generation module, a size of a storage space required for the general-purpose machine learning model file; and
generating the general-purpose machine learning model file according to the general-purpose machine learning model, the size of the storage space required for the general-purpose machine learning model file, and the model directory.
2. The method of claim 1 , wherein the step of generating the general-purpose machine learning model file according to the general-purpose machine learning model and the model directory includes:
obtaining a file header and a file tail of the general-purpose machine learning model file; and
generating the general-purpose machine learning model file according to the file header, the model directory, the general-purpose machine learning model, and the file tail.
3. The method of claim 1 , wherein the step of generating the general-purpose machine learning model file according to the general-purpose machine learning model and the model directory includes:
creating an identification code of the general-purpose machine learning model file; and
generating the general-purpose machine learning model file according to the identification code, the general-purpose machine learning model, and the model directory.
4. A device for generating a general-purpose machine learning model file, comprising:
a hardware central processing unit (CPU); a memory physically connected to the (CPU);
an external interface module configured to obtain task parameters of a machine learning task;
a classification processing module configured to classify the task parameters to obtain task instructions and model parameters;
a parameter aggregating module configured to aggregate the task instructions and the model parameters according to data types to generate stack data and heap data;
a model generation module configured to integrate the stack data and the heap data to obtain a general-purpose machine learning model;
a storage offset calculation module configured to calculate a storage offset of the general-purpose machine learning model, wherein the storage offset calculation module is further configured to:
obtain a size of a storage space occupied by each general-purpose machine learning model and a count of the general-purpose machine learning models;
obtain a storage order of the general-purpose machine learning models; and
calculate a storage offset of each general-purpose machine learning model according to the size of the storage space occupied by each general-purpose machine learning model, the count of the general-purpose machine learning models, and the storage order of the general-purpose machine learning models;
a model directory generation module configured to generate the model directory according to the general-purpose machine learning model and the storage offset of the general-purpose machine learning model; and
a model file generation module configured to generate the general-purpose machine learning model file according to the general-purpose machine learning model and the model directory, wherein the model file generation module is further configured to:
create a check code or an error correction code of the general-purpose machine learning model file;
generate the general-purpose machine learning model file according to the check code or the error correction code of the general-purpose machine learning model file, the general-purpose machine learning model, and the model directory;
calculate a size of a storage space required for the general-purpose machine learning model file; and
generate the general-purpose machine learning model file according to the general-purpose machine learning model, the size of the storage space required for the general-purpose machine learning model file, and the model directory.
5. The device of claim 4 , wherein the file generator further includes a file header generator and a file tailer generator, wherein the file header generator is connected to the directory generator, and the file tailer generator is connected to the model filler.
6. The device of claim 5 , wherein the file header generator is further configured to create an identification code of a general-purpose machine learning model file, and generate the general-purpose machine learning model file according to the identification code of the general-purpose machine learning model file, the general-purpose machine learning model, and the model directory.
7. The device of claim 4 , wherein the model filler is further configured to sequentially store the general-purpose machine learning model into the file generator.