Method, product, and apparatus for variable precision weight management for neural networks
An approach includes identification of a machine learning model for processing and generating an ordered set of weights with varying precisions and metadata that specifies where those values can be found in order to allow the identification of weights needed during processing. In a first embodiment, the variable precision weights are separated into different memory segments where each segment has weights of only a single precision. In a second embodiment, the variable precision weights are provided in a memory where weights of different precisions are intermingled, and those weights are identified using a sequence of pairs of data representing a number of weights with the same precision and the precision of those weights. In some embodiments, both the first and second embodiments are combined, where some segments contain weights with only a single precision and at least one segment stores weights with different precisions within a respective segment.
1. A method, comprising:
identifying a machine learning model having a plurality of weights of a single precision;
generating an ordered set of variable precision weights for the machine learning model from the plurality of weights, wherein:
a first subset of the ordered set of variable precision weights having a first precision are stored in a first memory segment and a second subset of the ordered set of variable precision weights having a second precision are stored in a second memory segment different from the first memory segment; or
at least a portion of the ordered set of variable precision weights are stored in a single memory segment having weights of different precisions intermingled together in a sequence; and
generating management data specifying an arrangement of the ordered set of variable precision weights, wherein the management data comprises at least:
a plurality of values for determining a memory segment used for storage of respective variable precision weights based on at least a corresponding precision; or
a plurality of sequence value pairs corresponding to the portion of the ordered set of variable precision weights stored in the single memory segment having the weights of different precisions intermingled together in a sequence, each sequence value pair of the plurality of sequence value pairs identifies a precision and a number of sequential weights having the corresponding precision.
2. The method of claim 1 , further comprising:
receiving a machine learning processing job having input data for processing;
identifying the management data specifying the arrangement of the ordered set of variable precision weights; and
identifying variable precision weights corresponding to input data for processing the machine learning processing job.
3. The method of claim 1 , wherein each weight of the plurality of weights is analyzed to determine a precision for each weight and at least a first weight and a second weight have different precision.
4. The method of claim 2 , wherein the ordered set of variable precision weights and the management data are generated prior to receipt of the machine learning processing.
5. The method of claim 1 , wherein the management data is maintained in a relational database table having a first plurality of entries for the first memory segment and a second plurality of entries for the second memory segment.
6. The method of claim 1 , wherein:
the first subset of the ordered set of variable precision weights having the first precision are stored in the first memory segment,
the second subset of the ordered set of variable precision weights having the second precision are stored in the second memory segment,
the portion of the ordered set of variable precision weights are stored in the single memory segment comprising a third memory segment having weights of different precisions intermingled together in the sequence, and
the first, second, and third memory segments are different memory segments.
7. The method of claim 1 , wherein the ordered set of variable precision weights and the management data are generated for a machine learning processing job and in response to receipt of the machine learning processing job.
8. A non-transitory computer readable medium, having stored thereon a set of instructions which when executed by a processor causes a set of acts, the set of acts comprising:
identifying a machine learning model having a plurality of weights of a single precision;
generating variable precision weights for the machine learning model from the plurality of weights for the machine learning model, wherein:
a first subset of the variable precision weights having a first precision are stored in a first memory segment and a second subset of the variable precision weights having a second precision are stored in a second memory segment different from the first memory segment; or
at least a portion of the variable precision weights are stored in a single memory segment having weights of different precisions intermingled together in a sequence; and
generating management data specifying an arrangement of the variable precision weights, wherein the management data comprises at least:
a plurality of values for determining a memory segment used for storage of respective variable precision weights based on at least a corresponding precision; or
a plurality of sequence value pairs corresponding to the portion of the variable precision weights stored in the single memory segment having the weights of different precisions intermingled together in a sequence, each sequence value pair of the plurality of sequence value pairs identifies a precision and a number of sequential weights having the corresponding precision.
9. The computer readable medium of claim 8 , wherein the set of acts further comprise:
receiving a machine learning processing job having input data for processing;
identifying the management data specifying the arrangement of the variable precision weights; and
identifying variable precision weights corresponding to input data for processing the machine learning processing job.
10. The computer readable medium of claim 8 , wherein each weight of the plurality of weights is analyzed to determine a precision for each weight and at least a first weight and a second weight have different precision.
11. The computer readable medium of claim 9 , wherein the variable precision weights and the management data are generated prior to receipt of the machine learning processing job.
12. The computer readable medium of claim 8 , wherein the management data is maintained in a relational database table having a first plurality of entries for the first memory segment and a second plurality of entries for the second memory segment.
13. The computer readable medium of claim 8 , wherein:
the first subset of the variable precision weights having the first precision are stored in the first memory segment,
the second subset of the variable precision weights having the second precision are stored in the second memory segment,
the portion of the variable precision weights are stored in the single memory segment comprising a third memory segment having weights of different precisions intermingled together in the sequence, and
the first, second, and third memory segments are different memory segments.
14. The computer readable medium of claim 13 , wherein the variable precision weights and the management data are generated for a machine learning processing job and in response to receipt of the machine learning processing job.
15. A computer system, comprising:
a memory comprising a set of instructions; and
a processor that executes the set of instructions to perform a set of acts comprising:
identifying a machine learning model having a plurality of weights of a single precision;
generating, from a plurality of weights corresponding to a machine learning model, a set of weights having variable precision, wherein:
a first subset of the set of weights having a first precision are stored in a first memory segment and a second subset of the of the set of weights having a second precision are stored in a second memory segment different from the first memory segment; or
at least a portion of the set of weights having variable precision are stored in a single memory segment having weights of different precisions intermingled together in a sequence; and
generating management data specifying an arrangement of the set of weights having variable precision, wherein the management data comprises at least:
a plurality of values for determining a memory segment used for storage of respective variable precision weights based on at least a corresponding precision; or
a plurality of sequence value pairs corresponding to the portion of the set of the weights having variable precision stored in the single memory segment having the weights of different precisions intermingled together in a sequence, each sequence value pair of the plurality of sequence value pairs identifies a precision and a number of sequential weights having the corresponding precision.
16. The computer system of claim 15 , wherein the set of acts further comprise:
receiving a machine learning processing job having input data for processing;
identifying the management data specifying the arrangement of the set of weights having variable precision; and
identifying variable precision weights corresponding to input data for processing the machine learning processing job.
17. The computer system of claim 15 , wherein each weight of the plurality of weights is analyzed to determine a precision for each weight and at least a first weight and a second weight have different precision.
18. The computer system of claim 16 , wherein the weights having variable precision and the management data are generated prior to receipt of the machine learning processing job.
19. The computer system of claim 15 , wherein the management data is maintained in a relational database table having a first plurality of entries for the first memory segment and a second plurality of entries for the second memory segment.
20. The computer system of claim 15 , wherein:
the first subset of the set of weights having the first precision are stored in the first memory segment,
the second subset of the of the set of weights having the second precision are stored in the second memory segment, and
the portion of the set of weights having variable precision are stored in the single memory segment comprising a third memory segment having weights of different precisions intermingled together in a sequence,
the first, second, and third memory segments are different memory segments, and the weights having variable precision and the management data are generated for a machine learning processing job and in response to receipt of the machine learning processing job.