Queue optimization in cloud computing

1. A computer-implemented method for object-based data storage, comprising:

instantiating, via one or more processors, a virtual machine, the virtual machine including at least one ingestion processor;

maintaining, with the virtual machine, a queue of operations relating to a plurality of documents operable to be maintained at an object-based data storage;

identifying, with the virtual machine, at least one independent operation in the queue that must be processed prior to processing at least one dependent operation to thereby enable parallelization of processing of operations in the queue, wherein the at least one independent operation was placed in the queue subsequent to the at least one dependent operation;

processing, with the at least one ingestion processor, the identified at least one independent operation; and

subsequent to processing the at least one independent operation, processing, in parallel, with the virtual machine, operations in the queue.

2. The computer-implemented method of claim 1 , wherein the identifying of the at least one independent operation in the queue that must be processed prior to processing at least one dependent operation includes identifying a group of operations in the queue according to at least one of: (i) a document, (ii) a file path, or (iii) a type of operation.

3. The computer-implemented method of claim 2 , wherein the group of operations comprises a create operation for a particular directory, and the at least one dependent operation includes operations to store two or more documents in the created directory.

4. The computer-implemented method of claim 3 , wherein:

the operations in the queue include a rename operation set to be processed prior to a create operation;

the identifying comprises identifying the at least one independent operation to be the create operation, and the at least one dependent operation to be the rename operation; and

the processing the at least one independent operation comprises processing the create operation.

5. The computer-implemented method of claim 2 , wherein the group of operations include a write or modify operation to the object-based data storage operation, and a read operation from the object-based data storage operation.

6. The computer-implemented method of claim 1 , wherein the identifying of the at least one independent operation in the queue that must be processed prior to processing at least one dependent operation comprises analyzing a decision directory.

7. The computer-implemented method of claim 1 , further comprising:

slicing documents of the plurality of documents into individual slices, wherein each individual slice of the individual slices has a memory size of less than a predetermined individual slice memory size.

8. The computer-implemented method of claim 1 , further comprising, with the one or more processors:

identifying, in the queue, an operation to create a document of the plurality of documents;

identifying, in the queue, an operation to delete the document of the plurality of documents; and

in response to identifying the operation to delete the document of the plurality of documents, removing, from the queue, both the operation to create the document, and the operation to delete the document.

9. A system for object-based data storage, the system comprising:

one or more processors; and

a non-transitory memory storing instructions that, when executed by the one or more processors, cause the system to:

instantiate, via the one or more processors, a virtual machine, the virtual machine including at least one ingestion processor;

maintain, with the virtual machine, a queue of operations relating to a plurality of documents operable to be maintained at an object-based data storage;

identify, with the virtual machine, at least one independent operation in the queue that must be processed prior to processing at least one dependent operation to thereby enable parallelization of processing of operations in the queue, wherein the at least one independent operation was placed in the queue subsequent to the at least one dependent operation;

process, with the at least one ingestion processor, the identified at least one independent operation; and

subsequent to processing the at least one independent operation, process, in parallel, with the virtual machine, operations in the queue.

10. The system of claim 9 , wherein the identification of the at least one independent operation in the queue that must be processed prior to processing at least one dependent operation includes identifying a group of operations in the queue according to at least one of: (i) a document, (ii) a file path, or (iii) a type of operation.

11. The system of claim 10 , wherein the group of operations comprises a create operation for a particular directory, and the at least one dependent operation includes operations to store two or more documents in the created directory.

12. The system of claim 9 , wherein the instructions, when executed by the one or more processors, further cause the system to:

slice documents of the plurality of documents into individual slices, wherein each individual slice of the individual slices has a memory size of less than a predetermined individual slice memory size.

13. A non-transitory computer readable medium storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to:

instantiate a virtual machine, the virtual machine including at least one ingestion processor;

maintain, with the virtual machine, a queue of operations relating to a plurality of documents operable to be maintained at an object-based data storage;

identify, with the virtual machine, at least one independent operation in the queue that must be processed prior to processing at least one dependent operation to thereby enable parallelization of processing of operations in the queue, wherein the at least one independent operation was placed in the queue subsequent to the at least one dependent operation;

process, with the at least one ingestion processor, the identified at least one independent operation; and

subsequent to processing the at least one independent operation, process, in parallel, with the virtual machine, operations in the queue.

14. The non-transitory computer readable medium of claim 13 , wherein the identification of the at least one independent operation in the queue that must be processed prior to processing at least one dependent operation includes identifying a group of operations in the queue according to: (i) a document, (ii) a file path, or (iii) a type of operation.

15. The non-transitory computer readable medium of claim 14 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:

rearrange the group of operations in the queue according to decision directory.

16. The non-transitory computer readable medium of claim 13 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:

slice documents of the plurality of documents into individual slices, wherein each individual slice of the individual slices has a memory size of less than a predetermined individual slice memory size.

17. The non-transitory computer readable medium of claim 13 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:

identify, in the queue, an operation to create a document of the plurality of documents;

identify, in the queue, an operation to delete the document of the plurality of documents; and

in response to the identification of the operation to delete the document of the plurality of documents, remove, from the queue, both the operation to create the document, and the operation to delete the document.