Consistent structured data hash value generation across formats and platforms
Methods, information handling systems and computer readable media are disclosed for generation of consistent structured data hash values across platforms and structured data formats. In one embodiment, a method includes identifying, at an information handling system, a set of data segments within a set of structured data and expressing the set of structured data in a standard format. This embodiment of the method further includes, for each of the data segments within the set of data segments, applying a hash algorithm to produce a data segment hash value and obtaining a standardized identifier for the data segment. The method continues with assembling an aggregate hash code comprising the standardized identifier and the data segment hash value for each of the data segments, and confirming an association of the aggregate hash code with the set of structured data.
1. A method, comprising:
identifying, at an information handling system, a first set and a second set of data segments within a first set and a second set of structured data;
for each of the data segments within each set of data segments,
converting the data segment into a normalized data segment that is expressed in a standardized data format,
applying a hash algorithm to the normalized data segment to produce a data segment hash value,
obtaining a standardized identifier for the normalized data segment, and
generating a pair corresponding to the data segment, the pair including a first value and a second value, wherein the first value includes the standardized identifier and the second value includes the data segment hash value;
using the standardized identifier for each normalized data segment to sort each pair corresponding to each data segment within each set of data segments into a standardized order;
assembling an aggregate hash code for each set of data segments to produce a first and a second aggregate hash code, each aggregate hash code comprising each pair sorted according to the standardized order for each of the normalized data segments in the corresponding first or second set of data segments;
comparing corresponding data segment hash values within the first and second aggregate hash codes to determine a similarity coefficient between the first set and the second set of structured data; and
conveying the similarity coefficient to a storage location at the information handling system or to an application, module, service, or system requesting the similarity coefficient.
2. The method of claim 1 , further comprising receiving the set of structured data at the information handling system.
3. The method of claim 1 , wherein converting the data segment comprises removing one or more extraneous spaces from the data segment.
4. An information handling system, comprising:
one or more processors;
a non-transitory computer-readable storage medium coupled to the one or more processors; and
a plurality of instructions, encoded in the computer-readable storage medium and configured to cause the one or more processors to
identify a first set and a second set of data segments within a first set and a second set of structured data,
for each of the data segments within each set of data segments,
converting the data segment into a normalized data segment that is expressed in a standardized data format,
apply a hash algorithm to the normalized data segment to produce a data segment hash value,
obtain a standardized identifier for the normalized data segment, and
generate a pair corresponding to the data segment, the pair including a first value and a second value, wherein the first value includes the standardized identifier and the second value includes the data segment hash value,
use the standardized identifier for each normalized data segment to sort each pair corresponding to each data segment within each set of data segments in a standardized order,
assemble an aggregate hash code for each set of data segments to produce a first and a second aggregate hash code, each aggregate hash code comprising each pair sorted according to the standardized order for each of the normalized data segments in the corresponding first or second set of data segments,
compare corresponding data segment hash values within the first and second aggregate hash codes to determine a similarity coefficient between the first set and the second set of structured data; and
convey the similarity coefficient to a storage location at the information handling system or to an application, module, service, or system requesting the similarity coefficient.
5. The information handling system of claim 4 , wherein the plurality of instructions is further configured to cause the one or more processors to receive the set of structured data at the information handling system.
6. The apparatus of claim 4 , wherein the plurality of instructions is further configured to remove one or more extraneous line breaks from the data segment, as a part of converting the data segment.
7. A non-transitory computer readable storage medium having program instructions encoded therein, wherein the program instructions are executable to:
identify, at an information handling system, a first set and a second set of data segments within a first set and a second set of structured data;
for each of the data segments within each set of data segments,
convert the data segment into a normalized data segment that is expressed in a standardized data format,
apply a hash algorithm to the normalized data segment to produce a data segment hash value,
obtain a standardized identifier for the normalized data segment, and
generate a pair corresponding to the data segment, the pair including a first value and a second value, wherein the first value includes the standardized identifier and the second value includes the data segment hash value;
use the standardized identifier for each normalized data segment to sort each pair corresponding to each data segment within each set of data segments in a standardized order;
assemble an aggregate hash code for each set of data segments to produce a first and a second aggregate hash code, each aggregate hash code comprising each pair sorted according to the standardized order for each of the normalized data segments in the corresponding first or second set of data segments;
compare corresponding data segment hash values within the first and second aggregate hash codes to determine a similarity coefficient between the first set and the second set of structured data; and
convey the similarity coefficient to a storage location at the information handling system or to an application, module, service, or system requesting the similarity coefficient.
8. The non-transitory computer readable storage medium of claim 7 , wherein the program instructions are further executable to receive the set of structured data at the information handling system.
9. The non-transitory computer readable storage medium of claim 7 , wherein the aggregate hash code includes a header.
10. The non-transitory computer readable storage medium of claim 9 , wherein the header includes information regarding an application program configured to use the aggregate hash code.
11. The non-transitory computer readable storage medium of claim 9 , wherein the header includes information regarding a source or a destination of the set of structured data.