System and method for processing data feeds

1. A method for processing a plurality of data feeds, the method comprising:

acquiring said plurality of data feeds from one or more content servers via a network;

adding a respective task of a plurality of tasks to each of said plurality of data feeds, wherein each respective task of said plurality of tasks is a request for processing a data feed of said plurality of data feeds;

storing and queuing said plurality of tasks in a persistent queue at a queuing database;

fetching each of said plurality of tasks queued at said queuing database by one or more worker servers;

asynchronously processing said plurality of tasks by said one or more worker servers; and

engaging additional worker servers when traffic load of the tasks increases, and disengaging one or more of the worker servers when the traffic load of the tasks decreases, to maintain a ratio of traffic load of the tasks to number of engaged worker servers;

wherein a push mechanism enables acquiring said plurality of data feeds from said one or more content servers;

wherein a pull mechanism enables acquiring said plurality of data feeds from said one or more content servers;

wherein the persistent queue is created by taking a snapshot of data stored in the queuing database for recovery in event of the queuing database being inaccessible.

2. The method as recited in claim 1 , wherein queuing said plurality of tasks is based on a priority.

3. The method as recited in claim 2 , wherein said priority is based on a predetermined criterion, the predetermined criteria including at least one of a frequency with which the content servers publish new web content and a service-level agreement relating to the content servers and the feed processing platform.

4. The method as recited in claim 1 , wherein asynchronously processing said plurality of tasks by a worker server of said one or more worker servers comprises requesting for additional tasks irrespective of status of said plurality of tasks running on other worker servers of said one or more worker servers.

5. A method for processing a plurality of data feeds, the method comprising:

acquiring said plurality of data feeds from one or more content servers via a network;

adding a respective task of a plurality of tasks to each of said plurality of data feeds, wherein each respective task of said plurality of tasks is a request for processing a data feed of said plurality of data feeds;

storing and queuing said plurality of tasks in a persistent queue at a queuing database based on a priority;

asynchronously processing said plurality of tasks by one or more worker servers, wherein said one or more worker server fetches each of said plurality of tasks; and

engaging additional worker servers when traffic load of the tasks increases, and disengaging one or more of the worker servers when the traffic load of the tasks decreases, to maintain a ratio of traffic load of the tasks to number of engaged worker servers

wherein a push mechanism enables acquiring said plurality of data feeds from said one or more content servers;

wherein a pull mechanism enables acquiring said plurality of data feeds from said one or more content servers.

6. The method as recited in claim 5 , wherein said priority is based on a predetermined criterion.

7. The method as recited in claim 5 , wherein asynchronously processing said plurality of tasks by a worker server of said one or more worker servers comprises requesting for additional tasks irrespective of status of tasks running on other worker servers of said one or more worker servers.

8. A system for processing a plurality of data feeds, the system comprising:

a feed processing platform, wherein said feed processing platform comprises:

a queuing data base; and

a plurality of worker servers,

wherein said feed processing platform is configured to:

communicate with a plurality of content servers to source a plurality of data feeds from said one or more content servers via a network;

add a respective task of a plurality of tasks to each of said plurality of data feeds, wherein each respective task of said plurality of tasks is a request for processing a data feed of said plurality of data feeds;

queue said plurality of tasks at said queuing database;

fetch each of said plurality of tasks queued at said queuing database by said one or more worker servers;

asynchronously process said plurality of tasks by said one or more worker servers; and

engage additional worker servers when traffic load of the tasks increases, and disengaging one or more of the worker servers when the traffic load of the tasks decreases, to maintain a ratio of traffic load of the tasks to number of engaged worker servers;

wherein a push mechanism enables sourcing said plurality of data feeds from said one or more content servers;

wherein a pull mechanism enables sourcing said plurality of data feeds from said one or more content servers;

wherein queuing said plurality of tasks is based on a priority based on a predetermined criterion.

9. The system as recited in claim 8 , wherein asynchronously processing said plurality of tasks by a worker server of said one or more worker servers comprises requesting for additional tasks irrespective of status of said plurality of tasks running on other worker servers of said one or more worker servers.

10. The system as recited in claim 8 , wherein said feed processing platform is configured to disengage a first set of workers servers of said one or more worker servers when traffic load of said plurality of tasks is decreased.

11. The system as recited in claim 8 , wherein said feed processing platform is configured to engage a first set of workers servers in said one or more worker servers when traffic load of said plurality of tasks is increased.