Network crawling prioritization
In some embodiments, an entry management module may receive a set of entries, each entry of the set of entries including a time-stamp value and a path identifier to a data object having content that may be changed. An entry selection module may select a first entry from the set of entries based on a first time-stamp value of the first entry, the first time-stamp value identifying a first time being no later than other times identified by time-stamp values of other entries. A crawler module may retrieve first content from a first data object using a first path identifier of the first entry. A freshness module may calculate a first updated timestamp value using a first freshness value, the first freshness value indicating a priority to retrieve possibly changed content from the first data object, the first updated time-stamp value replacing the first time-stamp value.
1. A method comprising:
receiving a set of entries, each entry of the set of entries including a time-stamp value, a path identifier to a data object having content that may be changed, and a freshness value indicating a frequency to retrieve possibly changed content from the data object, wherein a time-stamp value of an entry is updated periodically based on a freshness value of the entry, wherein a plurality of entries in the set of entries are ordered based on time-stamp values of the entries;
selecting a first entry to be retrieved from the plurality of entries, in order of the plurality of entries, wherein the selected first entry includes a first time-stamp value identifying a first time being no later than other times identified by time-stamp values of other entries of the set of entries;
retrieving first content from a first data object using a first path identifier of the selected first entry; and
after the retrieving the first content, updating a first time-stamp value of the selected first entry based on a freshness value of the selected first entry so as to re-order the plurality of entries in the set of entries,
wherein a first entry and a second entry in the set of the entries include different freshness values, and
wherein first content of the first entry is retrieved at a higher frequency than second content of the second entry, based on a difference of the freshness values.
2. The method of claim 1 , further comprising:
determining that the selected first entry is a mature entry based on whether the first time-stamp value is later than a time when the first entry was selected;
retrieving first content from a first data object using a first path identifier of the selected first entry, upon a determination that the selected first entry is the mature entry;
waiting a predetermined amount of time, upon a determination that the selected first entry is a non-mature entry; and
after the predetermined amount of time has passed, re-selecting a first entry to be retrieved.
3. The method of claim 1 , wherein the first time-stamp value of the first entry is updated by adding a first freshness value of the first entry to a time when the first content was retrieved.
4. The method of claim 1 , further comprising:
storing the first content retrieved from the first data object.
5. The method of claim 1 , further comprising:
analyzing the first content for a new path identifier not included in any entry in the set of entries.
6. The method of claim 5 , further comprising:
upon a determination that the first content includes a new path identifier, adding a new entry to the set of entries, the new entry including the new path identifier and a second time-stamp value.
7. The method of claim 6 , wherein the second time-stamp value is set based on a time when the new entry is created.
8. The method of claim 1 , further comprising:
receiving a first freshness value of the first entry;
storing the first freshness value in memory; and
prior to updating the first time-stamp value of the first entry, retrieving the first freshness value from memory.
9. A system comprising:
one or more processors; and
a memory that stores instructions for causing, when executed, the one or more processors to:
receive a set of entries, each entry of the set of entries including a time-stamp value, a path identifier to a data object having content that may be changed, and a freshness value indicating a frequency to retrieve possibly changed content from the data object, wherein a time-stamp value of an entry is updated periodically based on a freshness value of the entry, wherein a plurality of entries in the set of entries are ordered based on time-stamp values of the entries,
select a first entry to be retrieved from the plurality of entries, in order of the plurality of entries, wherein the selected first entry includes a first time-stamp value identifying a first time being no later than other times identified by time-stamp values of other entries of the set of entries,
retrieve first content from a first data object using a first path identifier of the selected first entry, and
after the retrieving the first content, update a first time-stamp value of the selected first entry based on a freshness value of the selected first entry so as to re-order the plurality of entries in the set of entries,
wherein a first entry and a second entry in the set of the entries include different freshness values, and
wherein first content of the first entry is retrieved at a higher frequency than second content of the second entry, based on a difference of the freshness values.
10. The system of claim 9 , wherein the instructions for causing, when executed, the one or more processors to:
determine that the selected first entry is a mature entry based on whether the first time-stamp value is later than a time when the first entry was selected;
retrieve first content from a first data object using a first path identifier of the selected first entry, upon a determination that the selected first entry is the mature entry;
wait a predetermined amount of time, upon a determination that the selected first entry is a non-mature entry; and
after the predetermined amount of time has passed, re-select a first entry to be retrieved.
11. The system of claim 9 , wherein the first time-stamp value of the first entry is updated by adding a first freshness value of the first entry to a time when the first content was retrieved.
12. The system of claim 9 , wherein the instructions for causing, when executed, the one or more processors to store the first content retrieved from the first data object.
13. The system of claim 9 , wherein the instructions for causing, when executed, the one or more processors to analyze the first content for a new path identifier not included in any entry in the set of entries.
14. The system of claim 13 , wherein the instructions for causing, when executed, the one or more processors to add a new entry to the set of entries upon a determination that the first content includes a new path identifier, the new entry including the new path identifier and a second time-stamp value.
15. The system of claim 14 , wherein the second time-stamp value is set based on a time when the new entry is created.
16. The system of claim 9 , wherein the instructions for causing, when executed, the one or more processors to:
receive a first freshness value of the first entry;
store the first freshness value in memory; and
prior to updating the first time-stamp value of the first entry, retrieve the first freshness value from memory.
17. A non-transitory computer readable medium comprising executable instructions, the instructions being executable by a data processing device to perform a method, the method comprising:
receiving a set of entries, each entry of the set of entries including a time-stamp value, a path identifier to a data object having content that may be changed, and a freshness value indicating a frequency to retrieve possibly changed content from the data object, wherein a time-stamp value of an entry is updated periodically based on a freshness value of the entry, wherein a plurality of entries in the set of entries are ordered based on time-stamp values of the entries;
selecting a first entry to be retrieved from the plurality of entries, in order of the plurality of entries, wherein the selected first entry includes a first time-stamp value identifying a first time being no later than other times identified by time-stamp values of other entries of the set of entries;
retrieving first content from a first data object using a first path identifier of the selected first entry; and
after the retrieving the first content, updating a first time-stamp value of the selected first entry based on a freshness value of the selected first entry so as to re-order the plurality of entries in the set of entries,
wherein a first entry and a second entry in the set of the entries include different freshness values, and
wherein first content of the first entry is retrieved at a higher frequency than second content of the second entry, based on a difference of the freshness values.