Data volume management
Embodiments are directed towards a system and method for a cloud-based front end that may abstract and enable access to the underlying cloud-hosted elements and objects that may be part of a multi-tenant application, such as a search application. Search objects may be employed to access indexed objects. An amount of indexed data accessible to a user may be based on an index storage limit selected by the user, such that data that exceeds the index storage limit may continue to be indexed. Also, one or more projects can be elastically scaled for a user to provide resources that may meet the specific needs of each project.
1. A computer-implemented method, comprising:
receiving, on a computing device, raw data;
indexing the raw data, wherein indexing includes dividing the raw data into time stamped searchable events;
storing the indexed data in an indexed data store;
establishing a data volume limit on an amount of storage for a project in the indexed data store;
receiving additional raw data;
determining that the data volume limit has been reached for the project; and
declining to store in the indexed data store additional indexed data resulting from indexing the additional raw data, wherein additional indexed data is declined for storage until after instructions have been received to increase the data volume limit for the project.
2. The method of claim 1 , wherein declining to store includes discarding the additional data.
3. The method of claim 2 , further comprising:
discarding the additional data using a first in first out operation.
4. The method of claim 1 , wherein declining to store includes storing the additional data in overflow memory in a manner not accessible by a search query.
5. The method of claim 4 , further comprising:
copying at least a portion of data in the overflow memory to archival storage.
6. The method of claim 1 , wherein declining to store includes storing the additional data in overflow memory not accessible by a search query, and further comprising:
receiving instructions to increase the data volume limit for the project;
indexing the additional data to produce additional indexed data; and
storing the additional indexed data in the indexed data store.
7. The method of claim 6 , further comprising:
determining an amount of the additional data to index and store in the indexed data store based on an amount of the increase to the data volume limit for the project.
8. The method of claim 1 , wherein declining to store includes:
indexing the additional data to produce additional indexed data; and
storing the additional indexed data in overflow memory in a manner not accessible by a search query.
9. The method of claim 1 , further comprising:
determining one or more roles that establish permissions for interacting with the project.
10. A system for managing resources, comprising:
a processor; and
a non-transitory computer-readable storage medium containing instructions configured to cause the processor to perform operations including:
receiving raw data;
indexing the raw data, wherein indexing includes dividing the raw data into time stamped searchable events;
storing the indexed data in an indexed data store;
establishing a data volume limit on an amount of storage for a project in the indexed data store;
receiving additional raw data;
determining that the data volume limit has been reached for the project; and
declining to store in the indexed data store additional indexed data resulting from indexing the additional raw data, wherein additional indexed data is declined for storage until after instructions have been received to increase the data volume limit for the project.
11. The system of claim 10 , wherein declining to store includes discarding the additional data.
12. The system of claim 11 , wherein the non-transitory computer-readable storage medium contains further instructions configured to cause the processor to perform operations including:
discarding the additional data using a first in first out operation.
13. The system of claim 10 , wherein declining to store includes storing the additional data in overflow memory in a manner not accessible by a search query.
14. The system of claim 13 , wherein the non-transitory computer-readable storage medium contains further instructions configured to cause the processor to perform operations including:
copying at least a portion of data in the overflow memory to archival storage.
15. The system of claim 10 , wherein declining to store includes storing the additional data in overflow memory not accessible by a search query, and wherein the non-transitory computer-readable storage medium contains further instructions configured to cause the processor to perform operations including:
receiving instructions to increase the data volume limit for the project;
indexing the additional data to produce additional indexed data; and
storing the additional indexed data in the indexed data store.
16. The system of claim 15 , wherein the non-transitory computer-readable storage medium contains further instructions configured to cause the processor to perform operations including:
determining an amount of the additional data to index and store in the indexed data store based on an amount of the increase to the data volume limit for the project.
17. The system of claim 10 , wherein declining to store includes:
indexing the additional data to produce additional indexed data; and
storing the additional indexed data in overflow memory in a manner not accessible by a search query.
18. The system of claim 10 , wherein the non-transitory computer-readable storage medium contains further instructions configured to cause the processor to perform operations including:
determining one or more roles that establish permissions for interacting with the project.
19. A computer-program product, tangibly embodied in a non-transitory machine-readable medium, including instructions configured to cause a data processing apparatus to:
receive raw data;
index the raw data, wherein indexing includes dividing the raw data into time stamped searchable events;
store the indexed data in an indexed data store;
establish a data volume limit on an amount of storage for a project in the indexed data store;
receive additional raw data;
determine that the data volume limit has been reached for the project; and
decline to store in the indexed data store additional indexed data resulting from indexing the additional raw data, wherein additional indexed data is declined for storage until after instructions have been received to increase the data volume limit for the project.
20. The computer-program product of claim 19 , wherein declining to store includes discarding the additional data.
21. The computer-program product of claim 20 , further including instructions configured to cause the data processing apparatus to:
discarding the additional data using a first in first out operation.
22. The computer-program product of claim 19 , wherein declining to store includes storing the additional data in overflow memory in a manner not accessible by a search query.
23. The computer-program product of claim 22 , further including instructions configured to cause the data processing apparatus to:
copying at least a portion of data in the overflow memory to archival storage.
24. The computer-program product of claim 19 , wherein declining to store includes storing the additional data in overflow memory not accessible by a search query, and further including instructions configured to cause the data processing apparatus to:
receiving instructions to increase the data volume limit for the project;
indexing the additional data to produce additional indexed data; and
storing the additional indexed data in the indexed data store.
25. The computer-program product of claim 24 , further including instructions configured to cause the data processing apparatus to:
determining an amount of the additional data to index and store in the indexed data store based on an amount of the increase to the data volume limit for the project.
26. The computer-program product of claim 19 , wherein declining to store includes:
indexing the additional data to produce additional indexed data; and
storing the additional indexed data in overflow memory in a manner not accessible by a search query.
27. The computer-program product of claim 19 , further including instructions configured to cause the data processing apparatus to:
determining one or more roles that establish permissions for interacting with the project.