Method and system for authoritative name analysis of true origin of a file
A computer system, method, or non-transitory computer-readable medium provides an authoritative name source for files within an ecosystem. Files in the ecosystem which have identical contends and similar contents to each other are merged into the same supercluster, to capture possibly incremental changes to the files over time in one of the superclusters. For each supercluster which has files with identical and similar contents, the supercluster is broken down into package clusters, based on packages to which the files belong, each of the package clusters has the files from a same package. The package cluster which has most change frequency across versions, is identified as the authoritative package. The authoritative name for the files is resolved, based on the authoritative packages that are determined, across the plurality of superclusters which have files with identical and similar contents, and the authoritative name is generated. Any authoritative name collision is resolved.
1. A method for providing an authoritative name source for files within an ecosystem, comprising the following performed by at least one processor:
clustering the files in the ecosystem into a plurality of superclusters, in which the files in each supercluster of the plurality of superclusters have identical contents; then
determining, of the files in the ecosystem which are clustered into the plurality of superclusters, which of the files have similar contents to each other, and merging the files which have similar contents to each other into the same supercluster, to capture possibly incremental changes to the files over time in one of the superclusters which has the files with identical contents and similar contents;
for each supercluster which has the files with identical and similar contents:
breaking the each supercluster down into package clusters, based on packages to which the files belong, each of the package clusters has the files from a same package; and
determining which of the package clusters has most change frequency across versions of the files within the same package, as the authoritative package, wherein change frequency refers to how frequently the version is changed in relation to how frequently the package is released; then
resolving an authoritative name for the files, based on the authoritative packages that are determined, across the plurality of superclusters which have files with identical and similar contents, and generating the authoritative name; and
resolving any authoritative name collision.
2. The method of claim 1 , wherein
a hash of file contents of each of the plurality of files is used to determine which of the files have identical contents,
the clustering of the files in the superclusters with identical contents is determined based on the hashes of file contents,
for the each supercluster, the each supercluster includes an indication of a source within the ecosystem for each file of the files,
the breaking down into the package clusters is based on the indication of the source of the each file.
3. The method of claim 1 , wherein
an abstract syntax tree of file contents of each of the plurality of files is used to determine whether contents of files are similar,
the determining of files with similar contents into the same supercluster is based on the abstract syntax tree of file contents,
for the each supercluster, the each supercluster includes an indication of a source within the ecosystem for each file of the files.
4. The method of claim 1 , wherein breaking each supercluster down into package clusters includes
generating package names and versions for members of the supercluster,
aggregating package clusters on package name and counting unique changes to the package cluster, and
determining, for each package cluster, the most change frequency among the package clusters.
5. The method of claim 1 , wherein
the authoritative name is resolved by
merging authoritative package outliers into groups which are merged into package cluster based on file name,
the resolved authoritative name and source are identified.
6. The method of claim 1 , further comprising
determining whether the authoritative name source is correct based on ecosystem in which authoritative name source originated.
7. The method of claim 1 , wherein the ecosystem, in which the files are located which are clustered, includes at least one software repository in which the files are located.
8. A non-transitory computer readable medium comprising instructions for execution by a computer, the instructions including a computer-implemented method for providing an authoritative name source for files within an ecosystem, the instructions for implementing:
clustering the files in the ecosystem into a plurality of superclusters, in which the files in each supercluster of the plurality of superclusters have identical contents; then
determining, of the files in the ecosystem which are clustered into the plurality of superclusters, which of the files have similar contents to each other, and merging the files which have similar contents to each other into the same supercluster, to capture possibly incremental changes to the files over time in one of the superclusters which has the files with identical contents and similar contents;
for each supercluster which has the files with identical and similar contents:
breaking the each supercluster down into package clusters, based on packages to which the files belong, each of the package clusters has the files from a same package; and
determining which of the package clusters has most change frequency across versions of the files within the same package, as the authoritative package, wherein change frequency refers to how frequently the version is changed in relation to how frequently the package is released; then
resolving an authoritative name for the files, based on the authoritative packages that are determined, across the plurality of superclusters which have files with identical and similar contents, and generating the authoritative name; and
resolving any authoritative name collision.
9. The non-transitory computer-readable medium of claim 8 , further comprising instructions for implementing:
a hash of file contents of each of the plurality of files is used to determine which of the files have identical contents,
the clustering of the files in the superclusters with identical contents is determined based on the hashes of file contents,
for the each supercluster, the each supercluster includes an indication of a source within the ecosystem for each file of the files,
the breaking down into the package clusters is based on the indication of the source of the each file.
10. The non-transitory computer-readable medium of claim 8 , wherein
an abstract syntax tree of file contents of each of the plurality of files is used to determine whether contents of files are similar,
the determining of files with similar contents into the same supercluster is based on the abstract syntax tree of file contents,
for the each supercluster, the each supercluster includes an indication of a source within the ecosystem for each file of the files.
11. The non-transitory computer-readable medium of claim 8 , wherein breaking each supercluster down into package clusters includes instructions for implementing:
generating package names and versions for members of the supercluster,
aggregating package clusters on package name and counting unique changes to the package cluster, and
determining, for each package cluster, the most change frequency among the package clusters.
12. The non-transitory computer-readable medium of claim 8 , wherein
the authoritative name is resolved by instructions for implementing:
merging authoritative package outliers into groups which are merged into package cluster based on file name,
identifying the resolved authoritative name and source.
13. The non-transitory computer-readable medium of claim 8 , further comprising further comprising instructions for implementing:
determining whether the authoritative name source is correct based on ecosystem in which authoritative name source originated.
14. The non-transitory computer-readable medium of claim 8 , wherein the ecosystem, in which the files are located which are clustered, includes at least one software repository in which the files are located.
15. A computer system that provides an authoritative name source for files within an ecosystem, comprising:
at least one processor, the at least one processor is configured to:
identify the ecosystem;
cluster the files in the ecosystem into a plurality of superclusters, in which the files in each supercluster of the plurality of superclusters have identical contents; then
determine, of the files in the ecosystem which are clustered into the plurality of superclusters, which of the files have similar contents to each other, and merge the files which have similar contents to each other into the same supercluster, to capture possibly incremental changes to the files over time in one of the superclusters which has the files with identical contents and similar contents;
for each supercluster which has the files with identical and similar contents:
break the each supercluster down into package clusters, based on packages to which the files belong, each of the package clusters has the files from a same package; and
determine which of the package clusters has most change frequency across versions of the files within the same package, as the authoritative package, wherein change frequency refers to how frequently the version is changed in relation to how frequently the package is released; then
resolve an authoritative name for the files, based on the authoritative packages that are determined, across the plurality of superclusters which have files with identical and similar contents, and generate the authoritative name; and
resolve any authoritative name collision.
16. The computer system of claim 15 , wherein
a hash of file contents of each of the plurality of files is used to determine which of the files have identical contents,
the clustering of the files in the superclusters with identical contents is determined based on the hashes of file contents,
for the each supercluster, the each supercluster includes an indication of a source within the ecosystem for each file of the files,
the processor is further configured to
break down the each supercluster into the package clusters based on the indication of the source of the each file.
17. The computer system of claim 15 , wherein
an abstract syntax tree of file contents of each of the plurality of files is used to determine whether contents of files are similar,
the determining of files with similar contents into the same supercluster is based on the abstract syntax tree of file contents,
for the each supercluster, the each supercluster includes an indication of a source within the ecosystem for each file of the files.
18. The computer system of claim 15 , wherein the processor is further configured to break down the each supercluster into the package clusters, to:
generate package names and versions for members of the supercluster,
aggregate package clusters on package name and counting unique changes to the package cluster, and
determine, for each package cluster, the most change frequency among the package clusters.
19. The computer system of claim 15 , wherein the processor is further configured to
resolve the authoritative name, to:
merge authoritative package outliers into groups which are merged into package cluster based on file name,
identify the resolved authoritative name and source.
20. The computer system of claim 15 , wherein the processor is further configured to
determine whether the authoritative name source is correct based on ecosystem in which authoritative name source originated.