Deferred data storage
A method for deferred storage of data is disclosed. A packet comprising instructions and literal data for rebuilding packet data from a network device is received. The packet data is rebuilt according to the instructions and the literal data. The rebuilt packet data is rewritten to a temporary page in memory. A page utilization metric is calculated based on the contents of the temporary page, the contents comprising the rebuilt data packet. Whether the contents of the temporary page sufficiently match data in data storage is determined. The contents of the temporary page are discarded or stored based on the determination.
1. A method comprising:
receiving a packet comprising instructions and literal data for rebuilding packet data from a network device;
rebuilding the packet data according to the instructions and the literal data;
writing the rebuilt packet data to a temporary page in memory;
calculating a page utilization metric based on the contents of the temporary page, the contents comprising the rebuilt data packet, wherein the page utilization metric is a quantitative comparison of the matching of the contents of the temporary page to content in the database;
determining if the contents of the temporary page sufficiently match data in a database;
and discarding or storing the contents of the temporary page based on the determination, the temporary page discarded if the level of matching between the contents of the temporary page and the data in the database satisfies a threshold.
2. The method of claim 1 , wherein discarding or storing the contents of the temporary page based on the determination comprises storing the contents of the temporary page in a location in the database based on the determination.
3. The method of claim 2 , further comprising sending a proposed location to the network device based on the determination.
4. The method of claim 2 , further comprising receiving a preferred location from the network device.
5. The method of claim 1 , wherein the page utilization metric comprises a total match count.
6. The method of claim 1 , further comprising transmitting an indication that the contents of the temporary page are stored at a location in the database.
7. The method of claim 6 , wherein the location in the database is sequential to previously stored data.
8. The method of claim 1 , further comprising sending integrity check information based on contents of the temporary page to the network device.
9. The method of claim 1 , further comprising indexing the temporary page based on the determination.
10. A system comprising:
a processor configured to receive a packet comprising instructions and literal data for rebuilding packet data from a network device, rebuild the packet data according to the instructions and the literal data, write the rebuilt packet data to a temporary page in memory, calculate a page utilization metric based on the contents of the temporary page, the page utilization metric calculated as a quantitative comparison of the matching of the contents of the temporary page to content in the database, the contents comprising the rebuilt data packet, determine if the contents of the temporary page sufficiently match data in a database, and discard or store the contents of the temporary page based on the determination, the temporary page discarded if the level of matching between the contents of the temporary page and the data in the database satisfies a threshold;
and data storage configured to store the contents of the temporary page based on the determination.
11. The system of claim 10 , wherein the processor is further configured to store the contents of the temporary page in a location in the database based on the determination.
12. The system of claim 11 , wherein the processor is further configured to send a proposed location to the network device based on the determination.
13. The system of claim 11 , wherein the processor is further configured to receive a preferred location from the network device.
14. The system of claim 10 , wherein the page utilization metric comprises a total match count.
15. The system of claim 10 , wherein the processor is further configured to transmit an indication that the contents of the temporary page are stored at a location in the database to the network device.
16. The system of claim 15 , wherein the location in the database is sequential to previously stored data.
17. The systems of claim 10 , wherein the processor is further configured to index the temporary page based on the determination.
18. A non-transitory computer readable medium having embodied thereon instructions executable by a processor for performing a method for deferred storing of data, the method comprising:
receiving a packet comprising instructions and literal data for rebuilding packet data from a network device;
rebuilding the packet data according to the instructions and the literal data;
writing the rebuilt packet data to a temporary page in memory;
calculating a page utilization metric based on the contents of the temporary page, the contents comprising the rebuilt data packet, wherein the page utilization metric is a quantitative comparison of the matching of the contents of the temporary page to content in the database;
determining if the contents of the temporary page sufficiently match data in a database;
and discarding or storing the contents of the temporary page based on the determination, the temporary page discarded if the level of matching between the contents of the temporary page and the data in the database satisfies a threshold.
19. The non-transitory computer readable medium of claim 18 , wherein discarding or storing the contents of the temporary page based on the determination comprises storing the contents of the temporary page in the database based on the determination.
20. The non-transitory computer readable medium of claim 19 , wherein the method further comprises sending a proposed location to the network device based on the determination.
21. The non-transitory computer readable medium of claim 19 , wherein the method further comprises receiving a preferred location from the network device.
22. The non-transitory computer readable medium of claim 18 , wherein the page utilization metric comprises a total match count.
23. The non-transitory computer readable medium of claim 18 , wherein the method further comprises transmitting an indication that the contents of the temporary page are stored at a location in the database.
24. The non-transitory computer readable medium of claim 23 , wherein the location in the database is sequential to previously stored data.
25. The non-transitory computer readable medium of claim 18 , wherein the method further comprises sending integrity check information based on contents of the temporary page to the network device.
26. The non-transitory computer readable medium of claim 18 , wherein the method further comprises indexing the temporary page based on the determination.