Executing a big data analytics pipeline using shared storage resources

1. A method comprising:

receiving a dataset by polling a location to which a data producer writes the dataset;

converting, by a storage system, the dataset for use by an analytics application;

storing, by the storage system, the dataset within shared storage resources of the storage system at a particular location specific to the data producer;

executing the analytics application on first compute resources of the storage system to ingest a first portion of the dataset that is read from the particular location of the shared storage resources;

detecting that the analytics application has ceased executing properly on the first compute resources; and

in response to detecting that the analytics application has ceased executing properly on the first compute resources, executing the analytics application on second compute resources of the storage system to ingest a second portion of the dataset from the particular location of the shared storage resources that is specific to the data producer while bypassing copying the second portion of the dataset to a different location of the shared storage resources.

2. The method of claim 1 , wherein the first compute resources and the second compute resources are located within a physical storage system.

3. The method of claim 1 , further comprising:

allocating additional processing resources to a real-time analytics application; and

executing the real-time analytics application on the additional processing resources, including ingesting at least a portion of the dataset.

4. The method of claim 1 , further comprising:

receiving, by a storage system, an unstructured dataset;

converting, by the storage system, the unstructured dataset into a structured dataset; and

storing the structured dataset within the storage system.

5. The method of claim 1 , further comprising:

detecting that the analytics application needs additional processing resources;

allocating additional processing resources to the analytics application; and

executing the analytics application on the additional processing resources.

6. The method of claim 1 , wherein the dataset includes log files describing one or more execution states of a computing system and wherein executing the analytics application on the compute resources further comprises:

evaluating the log files to identify one or more execution patterns associated with the computing system.

7. The method of claim 6 , wherein evaluating the log files to identify one or more execution patterns associated with the computing system further comprises:

comparing fingerprints associated with known execution patterns to information contained in the log files.

8. An apparatus comprising:

a memory; and

a processing device, operatively coupled to the memory, configured to:

receive a dataset by polling a location to which a data producer writes the dataset;

convert, by a storage system, the dataset for use by an analytics application;

storing, by the storage system, the dataset within shared storage resources of the storage system at a particular location specific to the data producer

execute the analytics application on first compute resources of the storage system to ingest a first portion of the dataset that is read from the particular location of the shared storage resources;

detect that the analytics application has ceased executing properly on the first compute resources; and

in response to detecting that the analytics application has ceased executing properly, execute the analytics application on second compute resources of the storage system to ingest a second portion of the dataset from the particular location that is specific to the data producer while bypassing copying the second portion of the dataset to a different location of the shared storage resources.

9. The apparatus of claim 8 , wherein the first compute resources and the second compute resources are located within a physical storage system.

10. The apparatus of claim 8 , wherein the processing device is further configured to:

allocate additional processing resources to a real-time analytics application; and

execute the real-time analytics application on the additional processing resources, including ingesting at least a portion of the dataset.

11. The apparatus of claim 8 , wherein the processing device is further configured to:

receive, by a storage system, an unstructured dataset;

convert, by the storage system, the unstructured dataset into a structured dataset; and

store the structured dataset within the storage system.

12. The apparatus of claim 8 , wherein the processing device is further configured to:

detect that the analytics application needs additional processing resources;

allocate additional processing resources to the analytics application; and

execute the analytics application on the additional processing resources.

13. The apparatus of claim 8 , wherein the dataset includes log files describing one or more execution states of a computing system and wherein to execute the analytics application on the compute resources, the processing device is further configured to:

evaluate the log files to identify one or more execution patterns associated with the computing system.

14. The apparatus of claim 13 , wherein to evaluate the log files to identify one or more execution patterns associated with the computing system, the processing device is further configured to:

compare fingerprints associated with known execution patterns to information contained in the log files.

15. A non-transitory computer readable storage medium storing instructions which, when executed, cause a processing device to:

receive a dataset by polling a location to which a data producer writes the dataset;

convert, by a storage system, the dataset for use by an analytics application;

store, by the storage system, the dataset within shared storage resources of the storage system at a particular location specific to the data producer;

execute the analytics application on first compute resources of the storage system to ingest a first portion of the dataset that is read from the particular location of the shared storage resources;

detect that the analytics application has ceased executing properly on the first compute resources; and

in response to detecting that the analytics application has ceased executing properly on the first compute resources, execute the analytics application on second compute resources of the storage system to ingest a second portion of the dataset from the particular location of the shared storage resources that is specific to the data producer while bypassing copying the second portion of the dataset to a different location of the shared storage resources.

16. The non-transitory computer readable storage medium of claim 15 , wherein the first compute resources and the second compute resources are located within a physical storage system.

17. The non-transitory computer readable storage medium of claim 15 , wherein the processing device is further configured to:

allocate additional processing resources to a real-time analytics application; and

execute the real-time analytics application on the additional processing resources, including ingesting at least a portion of the dataset.

18. The non-transitory computer readable storage medium of claim 15 , wherein the processing device is further configured to:

receive, by a storage system, an unstructured dataset;

convert, by the storage system, the unstructured dataset into a structured dataset; and

store the structured dataset within the storage system.

19. The non-transitory computer readable storage medium of claim 15 , wherein the processing device is further configured to:

detect that the analytics application needs additional processing resources;

allocate additional processing resources to the analytics application; and

execute the analytics application on the additional processing resources.

20. The non-transitory computer readable storage medium of claim 15 , wherein the dataset includes log files describing one or more execution states of a computing system and wherein to execute the analytics application on the compute resources the processing device is further configured to:

evaluate the log files to identify one or more execution patterns associated with the computing system.