Storage estimate generation
Disclosed herein is a data estimation technique for a data intake and query system. The system receives user inputs indicative that a first data source is to be the subject of a storage related estimate. The system receives a first plurality of events generated by the first data source. The system indexes only a sample of the received first plurality of events, based on a sampling criterion, where the sample is fewer than all of the first plurality of events. The system generates the storage related estimate based on at least some of the first plurality of events, and causes an indication of the estimate to be output to a user.
1. A method comprising:
receiving, by a data intake and query system, first data generated by a first data source;
processing, by the data intake and query system, the first data to produce second data corresponding to the first data;
generating a storage related estimate, by the data intake and query system, based on the second data;
causing an indication of the storage related estimate to be output to a user of the data intake and query system; and
completing, by the data intake and query system, indexing of only a sample of the second data based on a sampling criterion, the sample being less than all of the second data, and not completing indexing of a remainder of the second data in the absence of a user input indicative that the first data source should be indexed, wherein completing indexing includes committing data being indexed or to be indexed to persistent storage.
2. The method of claim 1 , further comprising:
parsing, by the data intake and query system, the first data into a plurality of events to produce the second data.
3. The method of claim 1 , wherein generating the storage related estimate is based on a prioritization of the first data source in relation to at least one data source that is being fully indexed or that is to be fully indexed.
4. The method of claim 1 , wherein the storage related estimate comprises a storage indication of an amount of storage space that would be needed for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period;
the method further comprising:
causing to be output to the user, in association with the storage indication, an indication of an amount of storage space that would be needed if the data intake and query system were not to index or store any data from the first data source for the time period.
5. The method of claim 1 , wherein the storage related estimate comprises a license indication of a license requirement for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period.
6. The method of claim 1 , wherein the storage related estimate comprises a license indication of a license requirement for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period;
the method further comprising:
causing to be output to the user, in association with the license indication, an indication of a current license associated with the user.
7. The method of claim 1 , wherein the storage related estimate comprises:
a storage indication indicative of an amount of storage space that would be needed for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period; and
a license indication indicative of a license requirement for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for the time period;
the method further comprising:
causing to be output to the user, in association with the storage indication,
an indication of an amount of storage space that would be needed if the data intake and query system were not to index or store any data from the first data source for the time period, and
an indication of a current license associated with the user.
8. A system comprising:
a communication device through which to communicate on a computer network; and
at least one processor operatively coupled to the communication device and configured to perform operations including
receiving first data generated by a first data source;
processing the first data to produce second data corresponding to the first data;
generating a storage related estimate based on the second data;
causing an indication of the storage related estimate to be output to a user of the data intake and query system; and
completing indexing of only a sample of the second data based on a sampling criterion, the sample being less than all of the second data, and not completing indexing of a remainder of the second data in the absence of a user input indicative that the first data source should be indexed, wherein completing indexing includes committing data being indexed or to be indexed to persistent storage.
9. The system of claim 8 , wherein said operations further comprise:
parsing, by the data intake and query system, the first data into a plurality of events to produce the second data.
10. The system of claim 8 , wherein generating the storage related estimate is based on a prioritization of the first data source in relation to at least one data source that is being fully indexed or that is to be fully indexed.
11. The system of claim 8 , wherein the storage related estimate comprises a storage indication of an amount of storage space that would be needed for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period;
the operations further comprising:
causing to be output to the user, in association with the storage indication, an indication of an amount of storage space that would be needed if the data intake and query system were not to index or store any data from the first data source for the time period.
12. The system of claim 8 , wherein the storage related estimate comprises a license indication of a license requirement for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period.
13. The system of claim 8 , wherein the storage related estimate comprises a license indication of a license requirement for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period;
the operations further comprising:
causing to be output to the user, in association with the license indication, an indication of a current license associated with the user.
14. A non-transitory machine-readable storage medium for use in a processing system of a data intake and query system, the non-transitory machine-readable storage medium storing instructions, execution of which in the processing system causes the processing system to perform operations comprising:
receiving first data generated by a first data source;
processing the first data to produce second data corresponding to the first data;
generating a storage related estimate based on the second data;
causing an indication of the storage related estimate to be output to a user of the data intake and query system; and
completing indexing of only a sample of the second data based on a sampling criterion, the sample being less than all of the second data, and not completing indexing of a remainder of the second data in the absence of a user input indicative that the first data source should be indexed, wherein completing indexing includes committing data being indexed or to be indexed to persistent storage.
15. The non-transitory machine-readable storage medium of claim 14 , such that said operations further comprise:
parsing, by the data intake and query system, the first data into a plurality of events to produce the second data.
16. The non-transitory machine-readable storage medium of claim 14 , such that generating the storage related estimate is based on a prioritization of the first data source in relation to at least one data source that is being fully indexed or that is to be fully indexed.
17. The non-transitory machine-readable storage medium of claim 14 , such that the storage related estimate comprises a storage indication of an amount of storage space that would be needed for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period;
the operations further comprising:
causing to be output to the user, in association with the storage indication, an indication of an amount of storage space that would be needed if the data intake and query system were not to index or store any data from the first data source for the time period.
18. The non-transitory machine-readable storage medium of claim 14 , such that the storage related estimate comprises a license indication of a license requirement for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period.
19. The non-transitory machine-readable storage medium of claim 14 , such that the storage related estimate comprises a license indication of a license requirement for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period;
the operations further comprising:
causing to be output to the user, in association with the license indication, an indication of a current license associated with the user.
20. The non-transitory machine-readable storage medium of claim 14 , such that the storage related estimate comprises:
a storage indication indicative of an amount of storage space that would be needed for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period; and
a license indication indicative of a license requirement for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for the time period;
the operations further comprising:
causing to be output to the user, in association with the storage indication,
an indication of an amount of storage space that would be needed if the data intake and query system were not to index or store any data from the first data source for the time period, and
an indication of a current license associated with the user.