High-speed scanning parser for scalable collection of statistics and use in preparing data for machine learning
A parser is deployed early in a machine learning pipeline to read raw data and collect useful statistics about the raw data's content to determine which items of raw data exhibit a proxy for feature importance for the machine learning model. The parser operates at high speeds that approach the disk's absolute throughput while utilizing a small memory footprint. Utilization of the parser enables the machine learning pipeline to receive a fraction of the total raw data that would otherwise be available. Several scans through the data are performed, by which proxies for feature importance are indicated and irrelevant features may be discarded and thereby not forwarded to the machine learning pipeline. This reduces the amount of memory and other hardware resources used at the server and also expedites the machine learning process.
1. A method performed by a computing device for optimization of a machine learning pipeline, comprising:
ingesting raw data from a file, in which the raw data has demarcations between items of data;
performing one or more scans on the file, in which the one or more scans are used to:
allocate a memory footprint for a catalogue, wherein the memory footprint indicates a maximum amount of memory utilized during a machine learning pipeline optimization process to avoid performance of additional memory allocation operations during subsequent processing,
create a pre-allocated buffer having a size that is based on the largest sized item of the items of data,
individually transfer an item of the raw data into the created pre-allocated buffer, and
parse the transferred item of the raw data into the catalogue responsive to the item of the raw data being transferred into the pre-allocated buffer, wherein each item of the raw data is individually and sequentially transferred into the pre-allocated buffer and parsed into the catalogue;
load the transferred items of raw data into the machine learning pipeline for utilization by using the parsed catalogue as a reference; and
training a machine learning model in the machine learning pipeline based on the transferred items of raw data.
2. The method of claim 1 , in which the raw data in the file is in CSV (comma-separated values) format.
3. The method of claim 1 , in which the one or more scans on the file are performed on bytes of the raw data.
4. The method of claim 1 , in which the items of data are demarcated by a delimiter.
5. The method of claim 4 , in which the delimiter is a comma, tab, or pipe.
6. The method of claim 1 , in which the items of the raw data are transferred in byte form into the pre-allocated buffer.
7. The method of claim 1 , in which the memory footprint is allocated based on a size of the catalogue, wherein a construction of the catalogue includes holding online statistic objects for each column of the raw data, in which the online statistic objects accept data from a respective column's population.
8. The method of claim 7 , in which parsing the items of the raw data includes:
assembling the items within the pre-allocated buffer into a number, and
pushing the number to a corresponding online statistic object inside the catalogue.
9. The method of claim 8 , in which the allocated memory footprint is comprised of pre-allocated arrays for individual objects, including the online statistic objects and flags identified for each column.
10. A computing device configured to parse raw data for use by a machine learning pipeline, comprising:
one or more processors; and
one or more hardware-based memory devices having instructions which, when executed by the one or more processors, cause the computing device to:
determine a size of a memory footprint for a catalogue;
allocate the determined size of the memory footprint for the catalogue, in which the allocated memory footprint is utilized during processing of raw data to reduce a number of times the memory footprint is re-allocated in memory;
individually transfer a demarcated item from a plurality of demarcated items of the raw data into a buffer wherein a size of the buffer is based on the largest sized item of the demarcated items of data;
parse the item of the raw data that is transferred into the buffer into the catalogue, wherein each demarcated item of the plurality of demarcated items of the raw data is individually and sequentially transferred into the buffer and parsed into the catalogue;
load the transferred items of raw data into a machine learning pipeline using the catalogue with the parsed items as a reference; and
train a machine learning model in the machine learning pipeline based on the transferred items of raw data.
11. The computing device of claim 10 , in which the buffer is a pre-allocated buffer.
12. The computing device of claim 11 , in which the executed instructions further cause the computing device to:
parse for delimiters in the raw data that demarcate the items;
identify a largest item between delimiters; and
create the pre-allocated buffer using a size based on the identified largest item.
13. The computing device of claim 12 , in which the parsing for delimiters and the identification of the largest item are performed on the raw data in byte format.
14. The computing device of claim 12 , in which the executed instructions further cause the computing device to parse the raw data for label distribution, in which the label distribution includes designating user-specified rows of raw data as a training set or a testing set for utilization by the machine learning pipeline,
wherein the parsing is performed during a scan in which the computing device parses the raw data for the delimiters and identifies the largest item in the raw data.
15. The computing device of claim 10 , in which the parsed individual items are assembled into a number and the items or parts of the items are pushed to a relevant online statistic object in the catalogue, and wherein the online statistic object is utilized to determine which raw data from the catalogue to load into the machine learning pipeline.
16. The computing device of claim 10 , in which the catalogue is loaded into the machine learning pipeline in a tabular data structure.
17. One or more hardware-based non-transitory computer-readable memory devices storing instructions which, when executed by one or more processors disposed in computing device, cause the computing device to:
ingest raw data from a file;
perform a first scan on the raw data in the file, by which a memory footprint is allocated for a catalogue;
responsive to completion of the first scan, perform a second scan on the raw data in the file by which a pre-allocated buffer is created, wherein a size of the preallocated buffer is based on the largest item of items of raw data in the file, in which the items within the raw data are demarcated by delimiters;
responsive to completion of the second scan, perform a third scan on the raw data in the file, by which each of the items of the raw data is individually transferred into the created pre-allocated buffer and parsed into the catalogue;
responsive to completion of the third scan, load the transferred items of the raw data into a machine learning pipeline for utilization, in which the catalogue is used as a reference for which pieces of raw data to load; and
train a machine learning model in the machine learning pipeline based on the transferred items of raw data.
18. The one or more hardware-based non-transitory computer-readable memory devices of claim 17 , in which each scan of the first, second and third scans is performed on the raw data as bytes.
19. The one or more hardware-based non-transitory computer-readable memory devices of claim 17 , in which items are assembled and pushed into an online statistics object within the catalogue which informs which data within the catalogue is to be loaded into the machine learning pipeline.
20. The one or more hardware-based non-transitory computer-readable memory devices of claim 19 , in which the online statistics object includes one or more of variance or prevalence.