Dynamic gathering of social media content
Methods, techniques, and systems for gathering social media content are provided. Some embodiments provide a Social Intelligence System (“SIS”) configured to provide dynamic search capability of a content source by using a proxy server system as an intermediary between the SIS and the content source. The SIS may then dynamically determine a rate at which it searches for content based on a rate of change or predicted change of a particular content source. Dynamically determining a rate allows the SIS to track a particular topic or series of topics over time, while only searching for content on the topic at the most optimal time periods to reduce overall cost.
1. A computer implemented method for retrieving social media content from at least one content source over a network, the method comprising the steps:
collecting a first content sample from the at least one content source;
after a first collection time interval of a first length, collecting a second content sample from the at least one content source;
comparing the first and second samples to determine a quantity of content common to the first and second samples, wherein the comparing the first and second samples to determine a quantity of content common to the first and second samples further comprises comparing duplication of at least one of permalinks, time, date and author;
determining, based upon the determined quantity of content common to the first and second samples, a second time interval of a second length different from the first length, wherein the second length is derived from predicting a new rate of content being posted to the content source and the second time length is dynamically adjusted based on the quantity of common content determined in the comparing step;
after the adjusted collection time interval, collecting a third content sample from the at least one content source; and
determining a proxy server service for a collection request based on at least one of an association time interval, a received indication of a desired geographic location to be searched, a cost of the proxy server service and a quantity of searches conduct by the proxy server service, wherein the first, second and third content samples comprise social media content.
2. The method of claim 1 , wherein the difference between the first length and the second length is proportional to the determined quantity of content common to the first and second samples.
3. The method of claim 1 , further comprising receiving a request for content including an indication of desired content and an indication of at least one content source.
4. The method of claim 3 , further comprising generating a collection request based on an interface of the indicated at least one content source.
5. The method of claim 4 , further comprising storing the generated collection request and associated collection time interval associated with the collection request within a data structure.
6. The method of claim 1 wherein the first collection time interval is an indication of at least one of a clock time, an interval, absolute time, relative time, and a frequency.
7. The method of claim 1 wherein the determining a second collection time interval further comprises setting at least one of a minimum time period and a maximum time period.
8. The method of claim 1 , further comprising transmitting a collection request to at least one third party content service to collect content when an associated collection time interval of the collection request has elapsed.
9. The method of claim 1 wherein the first collection time interval is based on a predetermined matrix of overlap values and prior collection time intervals.
10. a computing system configured to gather social media content, comprising:
a memory;
a content collection and ingestion system, stored in the memory and configured, when executed on a computer processor, to communicate with one or more computing systems over a network, collect a first content sample from the at least one content source, and after a first time interval of a first length, collect a second content sample from the at least one content source; and
a content collection determination engine, stored in the memory, and configured, when executed on a computer processor, to compare the first and second samples to determine a quantity of content common to the first and second samples, and, determine, based upon the determined quantity of content common to the first and second samples, a second collection time interval of the second length different from the first length, wherein the comparing the first and second samples to determine a quantity of content common to the first and second samples further comprises comparing duplication of at least one of permalinks, time, date and author, the content collection and ingestion system further configured to collect a third content sample from the at least one content source after the second collection time interval, the first, second and third content sample is the social media content, wherein the second time length is derived from predicting a new rate of content being posted to the content source and the second time length is dynamically adjusted based on the quantity of common content determined in the comparing step; and wherein the content collection determination engine further comprises a data collector adjuster, stored in the memory and configured, when executed on a computer processor, to determine a proxy server service for a collection request based on at least one of an association time interval, a received indication of a desired geographic location to be searched, a cost of the proxy server service and a quantity of searches conducted by the proxy server service.
11. The computing system of claim 10 wherein the content collection and ingestion system further comprises a Collection Request queue, stored in the memory and configured, when accessed by the content collection and ingestion system, to provide a pending collection request based on the second collection time interval.
12. The computing system of claim 11 wherein the content collection and ingestion system further comprises a collection engine, stored in the memory and configured, when executed on a computer processor, to manage a provided collection request from the Collection Request queue based on the second collection time interval.
13. The computing system of claim 10 wherein the one or more computing systems comprise at least one of proxy server services and third party content providers.
14. A non-transitory computer readable medium whose contents, when executed, cause a computing system to retrieve social media content from at least one content source over a network by performing a method comprising:
collecting a first content sample from the at least one content source;
after a first collection time interval of a first length, collecting a second content sample from the at least one content source;
comparing the first and second samples to determine a quantity of content common to the first and second samples, wherein the comparing the first and second samples to determine a quantity of content common to the first and second samples further comprises comparing duplication of at least one of permalinks, time, date and author;
determining, based upon the determined quantity of content common to the first and second samples, a second time interval of a second length different from the first length, wherein the second length is derived from predicting a new rate of content being posted to the content source and the second time length is dynamically adjusted based on the quantity of common content determined in the comparing step;
after the adjusted collection time interval, collecting a third content sample from the at least one content source; and
determining a proxy server service for a collection request based on at least one of an association time interval, a received indication of a desired geographic location to be searched, a cost of the proxy server service and a quantity of searches conduct by the proxy server service, wherein the first, second and third content samples comprise social media content.
15. The non-transitory computer-readable medium of claim 14 wherein the computer-readable medium is a memory in a computing device.
16. The non-transitory computer-readable medium of claim 15 wherein the contents are instructions that when executed cause a computing system to perform the method.