CONVERTING UNSTRUCTURED DATASETS INTO STRUCTURED DATASETS
Executing a machine learning model in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: receiving, by a graphical processing unit (‘GPU’) server, a dataset transformed by a storage system that is external to the GPU server; and executing, by the GPU server, one or more machine learning algorithms using the transformed dataset as input.
1 . A method comprising:
receiving an unstructured dataset produced by a data producer;
converting, by a processing device, the unstructured dataset into a structured dataset;
organizing the structured dataset into an indexed directory stored within one or more storage devices; and
storing the structured dataset within the indexed directory for ingestion by an analytics application.
2 . The method of claim 1 , wherein converting the unstructured dataset comprises extracting metadata from the unstructured dataset.
3 . The method of claim 1 , wherein converting the unstructured dataset comprises applying natural language processing to identify entities contained within the unstructured dataset.
4 . The method of claim 1 , wherein converting the unstructured dataset comprises assigning the structured dataset to a schema inferred from the unstructured dataset.
5 . The method of claim 1 , wherein organizing the structured dataset into the indexed directory comprises generating directory keys based on metadata contained within the structured dataset.
6 . The method of claim 1 , wherein the indexed directory comprises a plurality of directory levels that correspond to portions of the structured dataset.
7 . The method of claim 1 , wherein receiving the unstructured dataset comprises receiving log files that describe one or more execution states of a computing system.
8 . The method of claim 1 , wherein converting the unstructured dataset comprises removing duplicate entries contained within the unstructured dataset.
9 . The method of claim 1 , wherein organizing the structured dataset comprises generating a metadata index that identifies locations of portions of the structured dataset within the indexed directory.
10 . The method of claim 1 , wherein the indexed directory comprises storage distributed across the one or more storage devices.
11 . A system comprising:
one or more storage devices; and
a processing device, operatively coupled to the one or more storage devices, configured to:
receive an unstructured dataset produced by a data producer;
convert the unstructured dataset into a structured dataset;
organize the structured dataset into an indexed directory stored within the one or more storage devices; and
store the structured dataset within the indexed directory for ingestion by an analytics application.
12 . The system of claim 11 , wherein converting the unstructured dataset comprises extracting metadata from the unstructured dataset.
13 . The system of claim 11 , wherein converting the unstructured dataset comprises applying natural language processing to identify entities contained within the unstructured dataset.
14 . The system of claim 11 , wherein organizing the structured dataset into the indexed directory comprises generating directory keys based on metadata contained within the structured dataset.
15 . The system of claim 11 , wherein the indexed directory comprises storage distributed across the one or more storage devices.
16 . A non-transitory computer readable storage medium storing instructions that, when executed, cause a processing device to:
receive an unstructured dataset produced by a data producer;
convert the unstructured dataset into a structured dataset;
organize the structured dataset into an indexed directory stored within one or more storage devices; and
store the structured dataset within the indexed directory for ingestion by an analytics application.
17 . The non-transitory computer readable storage medium of claim 16 , wherein converting the unstructured dataset comprises extracting metadata from the unstructured dataset.
18 . The non-transitory computer readable storage medium of claim 16 , wherein converting the unstructured dataset comprises applying natural language processing to identify entities contained within the unstructured dataset.
19 . The non-transitory computer readable storage medium of claim 16 , wherein organizing the structured dataset comprises generating a metadata index that identifies locations of portions of the structured dataset within the indexed directory.
20 . The non-transitory computer readable storage medium of claim 16 , wherein the indexed directory comprises a plurality of directory levels that correspond to portions of the structured dataset.