Assigning raw data size of source data to storage consumption of an account
Provided are systems and methods for managing storage of machine data. In one embodiment, a method can be provided. The method can include receiving, from one or more data sources, raw machine data; processing the raw machine data to generate processed machine data; storing the processed machine data in a data store; and determining an allocated data size associated with the processed machine data stored in the data store, wherein the allocated data size is the size of the raw machine data corresponding to the processed machine data stored in the data store.
1. A computer-implemented method comprising:
receiving source data associated with an account, the source data having a raw data size;
processing the source data into searchable events and storing the searchable events in a plurality of searchable indices, the searchable events having a storage footprint that is different than the raw data size;
monitoring the plurality of searchable indices to track the raw data size of the source data corresponding to the searchable events maintained in the plurality of searchable indices and assigning the raw data size of the source data corresponding to the storage footprint of the searchable events in the plurality of searchable indices to an allocated storage consumption of the account; and
based on the allocated storage consumption, instead of the storage footprint of the searchable events, for the account exceeding a storage consumption threshold, executing one or more actions comprising at least one of: deleting at least some of the searchable events from the plurality of searchable indices, inhibiting access to the plurality of searchable indices, or inhibiting processing of additional source data for the account, charge an overage fee to the account, or increase the storage consumption threshold for the account.
2. The computer-implemented method of claim 1 , further comprising causing a graphical index management user interface configured to manage the plurality of searchable indices to present a representation of the allocated storage consumption of the account and corresponding portions of the allocated storage consumption consumed by the plurality of searchable indices.
3. The computer-implemented method of claim 1 , further comprising, in response to receiving the additional source data associated with the account, increasing the allocated storage consumption of the account by a corresponding raw data size of the additional source data, independent of a corresponding storage footprint of the additional source data in the plurality of searchable indices.
4. The computer-implemented method of claim 1 , further comprising, in response to removing stored data associated with the account from the plurality of searchable indices, decreasing the allocated storage consumption of the account by a corresponding raw data size of tracked source data that corresponds to the stored data removed from the plurality of searchable indices.
5. The computer-implemented method of claim 1 , further comprising indexing the source data, wherein the indexing comprises the processing and the storing.
6. The computer-implemented method of claim 1 , wherein the processing of the source data comprises metadata addition, compression of the source data, or replication of the source data.
7. The computer-implemented method of claim 1 , further comprising performing further processing on the searchable events, thereby changing the storage footprint of the searchable events in the plurality of searchable indices, without changing the allocated storage consumption of the account.
8. One or more non-transitory computer storage media storing computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:
receiving source data associated with an account, the source data having a raw data size;
processing the source data into searchable events and storing the searchable events in a plurality of searchable indices, the searchable events having a storage footprint that is different than the raw data size;
monitoring the plurality of searchable indices to track the raw data size of the source data corresponding to the searchable events maintained in the plurality of searchable indices and assigning the raw data size of the source data corresponding to the storage footprint of the searchable events in the plurality of searchable indices to an allocated storage consumption of the account; and
based on the allocated storage consumption, instead of the storage footprint of the searchable events, for the account exceeding a storage consumption threshold, executing one or more actions comprising at least one of: deleting at least some of the searchable events from the plurality of searchable indices, inhibiting access to the plurality of searchable indices, or inhibiting processing of additional source data for the account, charge an overage fee to the account, or increase the storage consumption threshold for the account.
9. The one or more non-transitory computer storage media of claim 8 , the operations further comprising adjusting the allocated storage consumption of the account in response to receiving the additional source data associated with the account or removing stored data associated with the account from the plurality of searchable indices.
10. The one or more non-transitory computer storage media of claim 8 , the operations further comprising, in response to receiving the additional source data associated with the account, increasing the allocated storage consumption of the account by a corresponding raw data size of the additional source data, independent of a corresponding storage footprint of the additional source data in the plurality of searchable indices.
11. The one or more non-transitory computer storage media of claim 8 , the operations further comprising, in response to removing stored data associated with the account from the plurality of searchable indices, decreasing the allocated storage consumption of the account by a corresponding raw data size of tracked source data that corresponds to the stored data removed from the plurality of searchable indices.
12. The one or more non-transitory computer storage media of claim 8 , the operations further comprising indexing the source data, wherein the indexing comprises the processing and the storing.
13. The one or more non-transitory computer storage media of claim 8 , wherein the processing of the source data comprises metadata addition, compression of the source data, or replication of the source data.
14. The non-transitory one or more computer storage media of claim 8 , the operations further comprising performing further processing on the searchable events, thereby changing the storage footprint of the searchable events in the plurality of searchable indices, without changing the allocated storage consumption of the account.
15. A computer system comprising:
one or more processors and one or more computer-readable storage media containing instructions which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving source data associated with an account, the source data having a raw data size;
processing the source data into searchable events and storing the searchable events in a plurality of searchable indices, the searchable events having a storage footprint that is different than the raw data size;
monitoring the plurality of searchable indices to track the raw data size of the source data corresponding to the searchable events maintained in the plurality of searchable indices and assigning the raw data size of the source data corresponding to the storage footprint of the searchable events in the plurality of searchable indices to an allocated storage consumption of the account; and
based on the allocated storage consumption, instead of the storage footprint of the searchable events, for the account exceeding a storage consumption threshold, executing one or more actions comprising at least one of: deleting at least some of the searchable events from the plurality of searchable indices, inhibiting access to the plurality of searchable indices, or inhibiting processing of additional source data for the account, charge an overage fee to the account, or increase the storage consumption threshold for the account.
16. The computer system of claim 15 , the operations further comprising, in response to receiving the additional source data associated with the account, increasing the allocated storage consumption of the account by a corresponding raw data size of the additional source data, independent of a corresponding storage footprint of the additional source data in the plurality of searchable indices.
17. The computer system of claim 15 , the operations further comprising, in response to removing stored data associated with the account from the plurality of searchable indices, decreasing the allocated storage consumption of the account by a corresponding raw data size of tracked source data that corresponds to the stored data removed from the plurality of searchable indices.
18. The computer system of claim 15 , the operations further comprising indexing the source data, wherein the indexing comprises the processing and the storing.
19. The computer system of claim 15 , wherein the processing of the source data comprises metadata addition, compression of the source data, or replication of the source data.
20. The computer system of claim 15 , the operations further comprising performing further processing on the searchable events, thereby changing the storage footprint of the searchable events in the plurality of searchable indices, without changing the allocated storage consumption of the account.