System and method for reducing crawl frequency and memory usage for an autonomous internet crawler
The system which dynamically determines crawl frequency. Crawl frequency is computed by determining the proportional integral derivative (PID) contribution of certain data to the system. Alternatively, the system determines the next time to crawl based on both the rate of change and displacement of data. Maximum and minimum wait times between crawls are taken into account for the computation. In another embodiment, crawl frequency is calculated based on the rate of change. The system determines the next time to crawl based on the rate of change of data within set parameters, such as maximum and minimum wait times between crawls. Alternatively, the system implements a recurrent neural network (RNN) using long short-term memory (LSTM) units for a future data prediction. With this information, the system determines the next time to crawl.
1. A system for reducing memory consumption and increasing bandwidth usage efficiency comprising:
a web crawler server, having a first memory, operatively connected to a network;
a database server, having a second memory, operatively connected to the network;
a dynamic crawl rate server, having a third memory, operatively connected to the network; and
a set of instructions, resident in the first memory, the second memory and the third memory, that when executed cause the system to:
instantiate an artificial neural network (ANN) having:
an ANN input layer;
an ANN hidden layer set, operatively connected to the ANN input layer;
an ANN dense hidden layer, operatively connected to the ANN hidden layer set; and
an ANN output layer, operatively connected to the ANN dense hidden layer;
receive a minimum crawl frequency value;
receive a maximum crawl frequency value;
receive a data lower bound value;
receive a current data point;
introducing the current data point into the ANN input layer;
activating the ANN hidden layer set and the ANN dense hidden layer to generate a future data point related to the current data point;
if the future data point is less than the data lower bound value; or, the future data point is less than or equal to about 75% of the current data point; or, if the future data point is greater than or equal to about 125% of the current data point; then, set a next time to crawl variable to the minimum crawl frequency value;
if the future data point is greater than or equal to the data lower bound value; and, if the future data point is greater than about 75% of the current data point; and, if the future data point is less than about 125% of the current data point; then, set the next time to crawl variable to the maximum crawl frequency value;
wait until a time to next crawl occurs based on the next time to crawl variable;
request a set of webpages, related to the current data point and the next time to crawl variable;
receive the set of webpages; and,
store the set of webpages.
2. The system of claim 1 wherein the set of instructions further comprises training the ANN by:
receiving a set of programming data sequences, including at least a current training data point and a set of past training data points;
populating the ANN hidden layer set and the ANN dense hidden layer with a set of random weight values;
executing back propagation through time for each set of programming data sequences; and,
returning a set of trained weight matrices for the ANN hidden layer set and the ANN dense hidden layer, thereby providing a trained ANN.
3. The system of claim 2 wherein the set of instructions further comprises:
deriving a performance measure from the trained ANN.
4. The system of claim 2 wherein the set of programming data sequences is standardized.
5. The system of claim 1 wherein the ANN hidden layer set is further comprised of a plurality of LSTM cells.
6. The system of claim 1 wherein the ANN hidden layer set further comprises:
a first LSTM hidden layer operatively connected to a second LSTM hidden layer through a set of synapses.