Dynamic gathering of social media content
Methods, techniques, and systems for gathering social media content are provided. Some embodiments provide a Social Intelligence System (“SIS”) configured to provide dynamic search capability of a content source by using a proxy server system as an intermediary between the SIS and the content source. The SIS may then dynamically determine a rate at which it searches for content based on a rate of change or predicted change of a particular content source. Dynamically determining a rate allows the SIS to track a particular topic or series of topics over time, while only searching for content on the topic at the most optimal time periods to reduce overall cost.
1. A computer implemented method for retrieving social media content from at least one content source over a network, the method comprising the steps of:
collecting a first content sample from the at least one content source;
after a first collection time interval of a first length, collecting a second content sample from the at least one content source;
comparing the first and second samples to determine a quantity of content common to the first and second samples;
determining, based upon the determined quantity of content common to the first and second samples, a second collection time interval of a second length different from the first length; and
after the second collection time interval, collecting a third content sample from the at least one content source.
2. The method of claim 1 , wherein the difference between the first length and the second length is proportional to the determined quantity of content common to the first and second samples.
3. The method of claim 1 , further comprising receiving a request for content including an indication of desired content and an indication of at least one content source.
4. The method of claim 3 , further comprising generating a collection request based on an interface of the indicated at least one content source.
5. The method of claim 4 , further comprising storing the generated collection request and associated collection time interval associated with the collection request within a data structure.
6. The method of claim 1 wherein the first collection time interval is an indication of at least one of a clock time, an interval, absolute time, relative time, and a frequency.
7. The method of claim 1 wherein the determining a second collection time interval further comprises setting at least one of a minimum time period and a maximum time period.
8. The method of claim 1 wherein the comparing the first and second samples to determine a quantity of content common to the first and second samples further comprises comparing duplication of permalinks, time, date and/or author.
9. The method of claim 1 , further comprising determining a proxy server service for a collection request based on at least one of an associated collection time interval, a received indication of a desired geographic location to be searched, a cost of the proxy server service and a quantity of searches conducted by the proxy server service.
10. The method of claim 1 , further comprising transmitting a collection request to at least one third party content service to collect content when an associated collection time interval of the collection request has elapsed.
11. The method of claim 1 wherein the first collection time interval is based on a predetermined matrix of overlap values and prior collection time intervals.
12. A computing system configured to gather social media content, comprising:
a memory;
a content collection and ingestion system, stored in the memory and configured, when executed on a computer processor, to communicate with one or more computing systems over a network, collect a first content sample from the at least one content source, and, after a first collection time interval of a first length, collect a second content sample from the at least one content source; and
a content collection determination engine, stored in the memory, and configured, when executed on a computer processor, to compare the first and second samples to determine a quantity of content common to the first and second samples, and, determine, based upon the determined quantity of content common to the first and second samples, a second collection time interval of a second length different from the first length, the content collection and ingestion system further configured to collect a third content sample from the at least one content source after the second collection time interval.
13. The computing system of claim 12 wherein the content collection and ingestion system further comprises a Collection Request queue, stored in the memory and configured, when accessed by the content collection and ingestion system, to provide a pending collection request based on the second collection time interval.
14. The computing system of claim 13 wherein the content collection and ingestion system further comprises a collection engine, stored in the memory and configured, when executed on a computer processor, to manage a provided collection request from the Collection Request queue based on the second collection time interval.
15. The computing system of claim 13 wherein the content collection determination engine is further configured to determine the quantity of content common to the first and second samples by comparing permalinks, time, date and/or author.
16. The computing system of claim 12 wherein the one or more computing systems comprise at least one of proxy server services and third party content providers.
17. The computing system of claim 16 wherein the content collection determination engine further comprises a data collection adjuster, stored in the memory and configured, when executed on a computer processor, to determine the proxy server service for the received collection request based on the time value, a received indication of a desired location to be searched, a cost of the proxy server service and/or a quantity of searches conducted by the proxy server service.
18. A non-transitory computer readable medium whose contents, when executed, cause a computing system to retrieve social media content from at least one content source over a network by performing a method comprising:
collecting a first content sample from the at least one content source;
after a first collection time interval of a first length, collecting a second content sample from the at least one content source;
comparing the first and second samples to determine a quantity of content common to the first and second samples;
determining, based upon the determined quantity of content common to the first and second samples, a second collection time interval of a second length different from the first length; and
after the second collection time interval, collecting a third content sample from the at least one content source.
19. The computer-readable medium of claim 18 wherein the computer-readable medium is a memory in a computing device.
20. The computer-readable medium of claim 19 wherein the contents are instructions that when executed cause a computing system to perform the method.