Method, system, apparatus, program code and means for determining a redundancy of information
Some embodiments include a system, method, apparatus and means for determining that first information in an input data packet is not redundant with second information previously stored in a database system, includes receiving the input data packet, generating a database query based on one or more tagged portions in the input data packet, comparing second information retrieved by the database query with the first information to identify at least a first portion of the first information that is different than the second information, and causing storage of the at least a first portion of the first information in the database system.
1. A system, comprising:
an information feed;
a first device to store information; and
a second device in communication with the information feed and the first device, the second device configured to:
receive first information comprising one or more portions from the information feed,
determine a redundancy portion score associated with each of the one or more portions, wherein the redundancy portion score associated with the each of the one or more portions indicates a degree of redundancy of the each of the one or more portions with second information,
determine a first score comprising a total of the redundancy portion scores associated with the each of the one or more portions,
compare the first score with a redundancy limit, and
cause non-redundant portions of the first information to be stored in the first device.
2. The system of claim 1 , wherein the second device includes:
a component to extract the second information from the first device.
3. The system of claim 2 , wherein the second device further includes:
a component to compare the first information to the second information to identify the non-redundant portions of the first information.
4. The system of claim 3 , wherein the second device further includes: a component to insert the identified portions of the first information into the first device.
5. The system of claim 1 , wherein the first device includes a data structure arranged to facilitate identification of risk.
6. The system of claim 1 , wherein the second device is further configured to spell check the first information.
7. The system of claim 1 , wherein the second device is further configured to cause redundant portions of the first information to be stored in the first device store.
8. The system of claim 1 , wherein the second device is further configured to tag the each of the one or more portions with the corresponding redundancy portion score.
9. A method, comprising:
receiving first information comprising one or more portions from an information feed;
determining, by a processor, a redundancy portion score associated with each of the one or more portions, wherein the redundancy portion score associated with the each of the one or more portions indicates a degree of redundancy of the each of the one or more portions with second information;
determining, by the processor, a first score comprising a total of the redundancy portion scores associated with the each of the one or more portions;
comparing, by the processor, the first score with a redundancy limit; and
storing non-redundant portions of the first information in a database.
10. The method of claim 9 , further comprising extracting the second information from the database.
11. The method of claim 10 , further comprising:
comparing the first information to the second information to identify the non-redundant portions of the first information that are not redundant.
12. The method of claim 11 , further comprising inserting the identified portions of the first information into the database.
13. The method of claim 9 , further comprising arranging a data structure to facilitate identification of risk.
14. The method of claim 9 , further comprising spell checking the first information.
15. A non-transitory, computer-readable medium comprising computer-readable instructions which when executed by a processor program the processor determine redundant information, the instructions comprising instructions for:
receiving first information comprising one or more portions from an information feed;
determining a redundancy portion score associated with each of the one or more portions, wherein the redundancy portion score associated with the each of the one or more portions indicates a degree of redundancy of the each of the one or more portions with second information;
determining a first score comprising a total of the redundancy portion scores associated with the each of the one or more portions;
comparing the first score with a redundancy limit; and
storing non-redundant portions of the first information in a database.
16. The non-transitory, computer-readable medium of claim 15 , wherein the instructions further comprise instructions for extracting the second information from the database.
17. The non-transitory, computer-readable medium of claim 16 , wherein the instructions further comprise instructions for comparing the first information to the second information to identify the non-redundant portions of the first information.
18. The non-transitory, computer-readable medium of claim 17 , wherein the instructions further comprise instructions for inserting the identified portions of the first information into the database.
19. The non-transitory, computer-readable medium of claim 15 , wherein the instructions further comprise instructions for arranging a data structure to facilitate identification of risk.
20. The non-transitory, computer-readable medium of claim 15 , wherein the instructions further comprise instructions for spell checking the first information.