Authenticating data associated with a data intake and query system using a distributed ledger system
Systems and methods are disclosed for authenticating a chunk of data identified in a query received by a data intake and query system. The data intake and query system receives a query that identifies a set of data and manner for processing the set of data, and identifies a chunk of data that is part of the set of data. The system generates a content identifier, such as a hash, of the chunk of data. The system further authenticates the chunk of data based on the generated content identifier and a content identifier stored by a distributed ledger system.
1. A method, comprising:
receiving, at a data intake and query system, a query that identifies a set of data and a manner of processing the set of data;
identifying a plurality of buckets containing data that satisfies at least a portion of the query, wherein a bucket is a data storage structure for storing files on a digital storage medium;
identifying a particular file of a particular bucket of the plurality of buckets to be used to execute the query;
generating a first content identifier for the particular file based on content of the particular file; and
authenticating the particular file for the query based on a comparison of the first content identifier with a second content identifier for the particular file and a comparison of the second content identifier with a third content identifier for the particular file, wherein the second content identifier is stored by the data intake and query system and the third content identifier is stored by a distributed ledger system, wherein the distributed ledger system stores a plurality of blocks of a blockchain, wherein a block of the blockchain comprises a plurality of content identifiers and a time range, wherein the time range is associated with chunks of data, wherein the chunks of data are associated with the plurality of content identifiers.
2. The method of claim 1 , wherein authenticating the particular file comprises communicating an identifier of the particular file to the distributed ledger system, and receiving the third content identifier from the distributed ledger system based on the identifier of the particular file.
3. The method of claim 1 , wherein authenticating the particular file comprises communicating a file identifier of the particular file to the distributed ledger system and receiving the third content identifier from the distributed ledger system, wherein the distributed ledger system uses the file identifier of the particular file to identify a block of a blockchain that stores the third content identifier.
4. The method of claim 1 , further comprising:
identifying one or more query parameters based on the query;
communicating the one or more query parameters to the distributed ledger system; and
receiving a plurality of content identifiers from the distributed ledger system including the third content identifier, wherein the distributed ledger system uses the one or more query parameters to identify block entries that satisfy the one or more query parameters.
5. The method of claim 1 , further comprising executing the query.
6. The method of claim 1 , further comprising obtaining the set of data and processing the set of data based on the query.
7. The method of claim 1 , further comprising processing the query to identify the particular file.
8. The method of claim 1 , wherein the first content identifier comprises a hash of the particular file.
9. The method of claim 1 , wherein the first content identifier comprises a hash of a group of related files including the particular file.
10. The method of claim 1 , wherein the first content identifier comprises a hash of a plurality of files of the particular bucket.
11. The method of claim 1 , wherein the first content identifier comprises a hash of a plurality of hashes of a plurality of files of the particular bucket.
12. The method of claim 1 , further comprising obtaining the particular file from an external data system.
13. The method of claim 1 , further comprising obtaining the particular file from a data store of the data intake and query system.
14. The method of claim 1 , wherein the particular file comprises raw machine data.
15. The method of claim 1 , wherein the third content identifier corresponds to a content identifier generated by the data intake and query system prior to the first content identifier.
16. The method of claim 1 , wherein the second content identifier is stored in association with the particular file.
17. The method of claim 1 , wherein the distributed ledger system stores the third content identifier with other content identifiers based on a time associated with the third content identifier and the other content identifiers.
18. The method of claim 1 , wherein the distributed ledger system stores the third content identifier with other content identifiers associated with buckets of data associated with a same field-value pair.
19. The method of claim 1 , wherein at least one block of the blockchain comprises a plurality of block entries including at least one block entry corresponding to the particular file.
20. The method of claim 1 , wherein at least one block of the blockchain comprises a plurality of block entries, each block entry corresponding to a bucket of data managed by the data intake and query system.
21. The method of claim 1 , wherein at least one block of the blockchain comprises a plurality of block entries, at least one block entry of the plurality of block entries comprising at least one content identifier.
22. The method of claim 1 , wherein at least one block of the blockchain comprises a plurality of block entries, at least one block entry of the plurality of block entries comprising the plurality of content identifiers.
23. The method of claim 1 , wherein at least one block of the blockchain comprises a plurality of block entries associated with a same field-value pair.
24. The method of claim 1 , wherein the distributed ledger system stores a plurality of blockchains, each blockchain comprising a plurality of blocks that store one or more content identifiers, wherein the plurality of blocks of a particular blockchain are associated with a same field-value pair, wherein the plurality of blockchains includes the blockchain.
25. A computing system of a data intake and query system, the computing system comprising:
memory; and
one or more processing devices coupled to the memory and configured to:
receive a query that identifies a set of data and a manner of processing the set of data;
identify a plurality of buckets containing data that satisfies at least a portion of the query, wherein a bucket is a data storage structure for storing files on a digital storage medium;
identify a particular file of a particular bucket of the plurality of buckets to be used to execute the query;
generate a first content identifier for the particular file based on content of the particular file; and
authenticate the particular file for the query based on a comparison of the first content identifier with a second content identifier for the particular file and a comparison of the second content identifier with a third content identifier for the particular file, wherein the second content identifier is stored by the data intake and query system and the third content identifier is stored by a distributed ledger system, wherein the distributed ledger system stores a plurality of blocks of a blockchain, wherein a block of the blockchain comprises a plurality of content identifiers and a time range, wherein the time range is associated with chunks of data, wherein the chunks of data are associated with the plurality of content identifiers.
26. Non-transitory computer readable media comprising computer-executable instructions that, when executed by a computing system of a data intake and query system, cause the computing system to:
receive a query that identifies a set of data and a manner of processing the set of data;
identify a plurality of buckets containing data that satisfies at least a portion of the query, wherein a bucket is a data storage structure for storing files on a digital storage medium;
identify a particular file of a particular bucket of the plurality of buckets to be used to execute the query;
generate a first content identifier for the particular file based on content of the particular file; and
authenticate the particular file for the query based on a comparison of the first content identifier with a second content identifier for the particular file and a comparison of the second content identifier with a third content identifier for the particular file, wherein the second content identifier is stored by the data intake and query system and the third content identifier is stored by a distributed ledger system, wherein the distributed ledger system stores a plurality of blocks of a blockchain, wherein a block of the blockchain comprises a plurality of content identifiers and a time range, wherein the time range is associated with chunks of data, wherein the chunks of data are associated with the plurality of content identifiers.