LOCK ON READ TECHNIQUES FOR IMPROVED FILE SYSTEM PERFORMANCE

1 . A method for storing data in a datastore, the method comprising:

storing first data from a first data producer in a first file maintained in the datastore;

receiving a read input/output (I/O) request to read the first data stored in the first file;

in response to receiving the read I/O request, locking data in the first file to prevent further data from being stored in the first file;

processing the read I/O request;

creating a second file for storing at least second data from the first data producer; and

advertising the second file.

2 . The method of claim 1 , further comprising:

prior to storing the first data from the first data producer in the first file:

receiving third data from the first data producer;

determining a file for storing data received from the first data producer has not been previously created;

in response to determining the file for storing the data received from the first data producer has not been previously created, creating the first file; and

storing third data from the first data producer in the first file.

3 . The method of claim 2 , wherein:

subsequent to storing first data from the first data producer in the first file, the data in the first file comprises the first data and the third data.

4 . The method of claim 1 , wherein advertising the second file comprises:

performing, by the data store, a callback to a plurality of query engines in response to creation of the second file; and

reducing data growth by artificially delaying, by the data store, advertising of the second file.

5 . The method of claim 1 , wherein:

the first data comprises data generated at the first data producer, wherein the first data producer is configured to automatically transmit the first data to the datastore when the first data is generated.

6 . The method of claim 1 , wherein processing the read I/O request comprises:

retrieving the first data from the first file; and

returning the first data to a requestor which initiated the read I/O request.

7 . The method of claim 1 , wherein the datastore comprises a distributed file system or object storage.

8 . The method of claim 1 , wherein, prior to receiving the read I/O request, the first file is maintained in a cache, and further comprising, in response to receiving the read I/O request, storing the first file in a storage.

9 . A system comprising:

one or more processors; and

at least one memory, the one or more processors and the at least one memory configured to:

store first data from a first data producer in a first file maintained in a datastore;

receive a read input/output (I/O) request to read the first data stored in the first file;

in response to receiving the read I/O request, lock data in the first file to prevent further data from being stored in the first file;

process the read I/O request;

create a second file for storing at least second data from the first data producer; and

advertise the second file.

10 . The system of claim 9 , wherein the one or more processors and the at least one memory are further configured to:

prior to storing the first data from the first data producer in the first file:

receive third data from the first data producer;

determine a file for storing data received from the first data producer has not been previously created;

in response to determining the file for storing the data received from the first data producer has not been previously created, create the first file; and

store third data from the first data producer in the first file.

11 . The system of claim 10 , wherein:

subsequent to storing first data from the first data producer in the first file, the data in the first file comprises the first data and the third data.

12 . The system of claim 9 , wherein advertising the second file comprises:

performing, by the data store, a callback to a plurality of query engines in response to creation of the second file; and

reducing data growth by artificially delaying, by the data store, advertising of the second file.

13 . The system of claim 9 , wherein:

the first data comprises data generated at the first data producer, wherein the first data producer is configured to automatically transmit the first data to the datastore when the first data is generated.

14 . The system of claim 9 , wherein to process the read I/O request comprises to:

retrieve the first data from the first file; and

return the first data to a requestor which initiated the read I/O request.

15 . The system of claim 9 , wherein the datastore comprises a distributed file system or object storage.

16 . The system of claim 9 , wherein, prior to receiving the read I/O request, the first file is maintained in a cache, and the one or more processors and the at least one memory are further configured to, in response to receiving the read I/O request, store the first file in a storage.

17 . A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations for storing data in a datastore, the operations comprising:

storing first data from a first data producer in a first file maintained in the datastore;

receiving a read input/output (I/O) request to read the first data stored in the first file;

in response to receiving the read I/O request, locking data in the first file to prevent further data from being stored in the first file;

processing the read I/O request;

creating a second file for storing at least second data from the first data producer; and

advertising the second file.

18 . The non-transitory computer-readable medium of claim 17 , wherein the operations further comprise:

prior to storing the first data from the first data producer in the first file:

receiving third data from the first data producer;

determining a file for storing data received from the first data producer has not been previously created;

in response to determining the file for storing the data received from the first data producer has not been previously created, creating the first file; and

storing third data from the first data producer in the first file.

19 . The non-transitory computer-readable medium of claim 18 , wherein:

subsequent to storing first data from the first data producer in the first file, the data in the first file comprises the first data and the third data.

20 . The non-transitory computer-readable medium of claim 17 , wherein advertising the second file comprises:

performing, by the data store, a callback to a plurality of query engines in response to creation of the second file; and

reducing data growth by artificially delaying, by the data store, advertising of the second file.