Data edge platform for improved storage and analytics
Disclosed are system and methods for processing and storing data files, using a data edge file format. The data edge file separates information about what symbols are in a data file and information about the corresponding location of those symbols in the data file. The described technique for converting a source file comprising symbols into a data edge file includes: generating a locality file of symbol location from the source file to identify locations of the symbols in the source file, generating a symbol file to identify symbols in the source file, and then modifying the locality file of symbol location to associate each symbol from the symbol file with a location in the source file.
1. A computer-implemented method for storing a file in object storage, comprising:
receiving, from an object storage system, a source file containing raw data;
determining a plurality of symbols contained in the raw data of the source file;
generating a symbol file associated with the source file, the symbol file comprising a sorted unique set of the plurality of symbols from the source file, wherein each symbol of the plurality of symbols is stored at a corresponding location within the symbol file;
generating a locality file associated with the source file, the locality file comprising a plurality of location values referencing the symbol file, wherein a location value at a respective position within the locality file represents an occurrence in the source file of a corresponding symbol identified by a respective location value;
storing a converted file including the symbol file and the locality file associated with the source file in the object storage system, the converted file providing lossless compression of the source file;
receiving a search query regarding stored files in the object storage system; and
displaying a result for the search query, the result for the search query returned based on querying at least one of the symbol file and the locality file of the converted file, the result for the search query reproducing the source file without accessing the source file.
2. The method of claim 1 , further comprising:
generating the converted file comprising the symbol file concatenated with the locality file.
3. The method of claim 1 , wherein the source file comprises structured data, and wherein the plurality of location values are ordered within the locality file by one of a row orientation or a column orientation.
4. The method of claim 3 , wherein the source file comprises at least one of a comma-separated values (CSV) file having a plurality of records, each record of the plurality of records comprising a plurality of fields separated by a delimiter character.
5. The method of claim 1 , further comprising:
responsive to determining that the source file comprises unstructured text data:
generating a plurality of fingerprint values based on the plurality of symbols contained in the source file; and
generating the locality file comprising the plurality of fingerprint values, wherein a fingerprint value based on the corresponding symbol and at the respective position within the locality file represents the occurrence in the source file of the corresponding symbol.
6. The method of claim 1 , further comprising:
responsive to determining that the source file comprises semi-structured data, transforming the semi-structured data into a two-dimensional structure prior to generating the locality file.
7. The method of claim 6 , wherein the source file comprising the semi-structured data is formatted as at least one of a JavaScript Object Notation (JSON) file, and an extensible markup language (XML) file.
8. The method of claim 1 , wherein returning the result of the search query includes:
querying the locality file of the converted file associated with the source file.
9. The method of claim 1 , wherein returning the result of the search query includes:
searching the symbol file of the converted file associated with the source file; and
determining an associated location from the locality file associated with the symbol file of the converted file.
10. The method of claim 1 , wherein the search query comprises one of a relational construct and a text search.
11. The method of claim 1 , wherein the symbol file and the locality file are stored in a same bucket as the source file in the object storage system.
12. The method of claim 1 , wherein the sorted unique set of the plurality of symbols in the symbol file are concatenated together with a delimiter.
13. The method of claim 1 , further comprising:
selecting a compression algorithm based on a composition of at least one of the symbol file and the locality file; and
compressing the locality file using the compression algorithm.
14. The method of claim 13 , wherein the compression algorithm is selected to achieve maximum compression based on organized symbols.
15. The method of claim 13 , wherein the compression algorithm is selected based on pattern recognition.
16. The method of claim 1 , further comprising:
receiving a command to transform a shape of the source file; and
modifying the locality file to transform a file format of the source file based on the command.
17. The method of claim 1 , further comprising:
receiving a command to reorganize the source file; and
modifying the locality file such that a decompressed symbol file comprises a representation of the source file reorganized according to the command.
18. The method of claim 17 , wherein the command to reorganize comprises a reorder command to reorder the plurality of symbols contained in the source file.
19. The method of claim 17 , wherein the source file comprises a plurality of columns, and wherein the command to reorganize the source file comprises a reshape command to reshape at least one column of the plurality of columns of the source file.
20. The method of claim 19 , wherein the command to reshape the at least one column of the plurality of columns of the source file comprises removing a column from the source file.
21. The method of claim 19 , wherein the command to reshape the at least one column of the plurality of columns of the source file comprises concatenating a second plurality of columns from a second source file with the plurality of columns from the source file.
22. The method of claim 17 , further comprising:
receiving a query to search the source file; and
using the modified locality file and the symbol file to return a query result based on respective locations of the plurality of symbols in the source file.
23. The method of claim 17 , wherein the symbol file is unchanged.
24. The method of claim 1 , wherein the source file is a first source file, the method further comprising aggregating the first source file and a second source file.
25. The method of claim 1 , wherein returning the result without accessing the source file includes without extracting, transforming, and loading the raw data of the source file.
26. A computer apparatus for storing a file in object storage, comprising:
memory; and
at least one processor coupled to the memory and configured to:
receive, from an object storage system, a source file containing raw data;
determine a plurality of symbols contained in the raw data of the source file;
generate a symbol file associated with the source file, the symbol file comprising a sorted unique set of the plurality of symbols from the source file, wherein each symbol of the plurality of symbols is stored at a corresponding location within the symbol file;
generate a locality file associated with the source file, the locality file comprising a plurality of location values referencing the symbol file, wherein a location value at a respective position within the locality file represents an occurrence in the source file of a corresponding symbol identified by a respective location value;
store a converted file including the symbol file and the locality file associated with the source file in the object storage system, the converted file providing lossless compression of the source file;
receive a search query regarding stored files in the object storage system; and
display a result for the search query, the result for the search query returned based on querying at least one of the symbol file and the locality file of the converted file, the result of the search query reproducing the source file without accessing the source file.
27. A non-transitory computer-readable medium storing computer executable code for storing a file in object storage, comprising code to:
receive, from an object storage system, a source file containing raw data;
determine a plurality of symbols contained in the raw data of the source file;
generate a symbol file associated with the source file, the symbol file comprising a sorted unique set of the plurality of symbols from the source file, wherein each symbol of the plurality of symbols is stored at a corresponding location within the symbol file;
generate a locality file associated with the source file, the locality file comprising a plurality of location values referencing the symbol file, wherein a location value at a respective position within the locality file represents an occurrence in the source file of a corresponding symbol identified by a respective location value;
store a converted file including the symbol file and the locality file associated with the source file in the object storage system, the converted file providing lossless compression of the source file;
receive a search query regarding stored files in the object storage system; and
display a result for the search query, the result for the search query returned based on querying at least one of the symbol file and the locality file of the converted file, the result of the search query reproducing the source file without accessing the source file.