SYSTEM AND APPARATUS FOR EFFECTIVE COORDINATION AND SCHEDULING OF ACCESSES TO RATE LIMITED ONLINE SITES TO OBTAIN DATA FOR USE IN ASSESSING SOCIAL METRICS BASED ON DOMAIN TAILORED EVALUATION OF SOCIAL MEDIA EXPOSURE
Embodiments described herein can be employed in a social media analytics platform to monitor which data has been acquired for entities from online sites at which entities may maintain an account, including social media platforms by continually moderating, coordinating or scheduling requests for data retrieval across the online sites to substantially maximize the data pulled from each site, and the freshness of that data, given any imposed rate limits associated with those online sites without slowing or idling the social media analytics platform. This results in increased efficiencies in time and expense and a simultaneous reduction in the number of errors received from the online sites, as the analytics platform can spread out requests over time and fully utilize parallel update processes in a way that respects the rate limiting while still improving freshness of data given that rate limiting.
1 . A system, comprising:
a social media analytics platform coupled to each of one of more online sites over a network, wherein the one or more online sites include at least one social media site and each online site provides a proprietary interface for requesting data from that online site,
the social media analytics platform configured with a plurality of entities, each entity having an account at each of the online sites, wherein the social media analytics platform is further configured such that each of the one of more online sites is associated with rate limiting data defining a number of requests and a time period for that online site, and the social media analytics platform comprises:
a data store storing a historical snapshot for each account for each entity, wherein each historical snapshot is associated with a timestamp;
a database manager operable to determine a final number of accounts that were least recently updated by:
determining at least one online site of the one or more online sites for which a rate limit has not been reached based on the rate limiting data associated with each online site;
identifying a number of accounts associated with each of the at least one online site that were least recently updated;
identifying a final number of accounts associated from the number of accounts associated with each of the at least one online site;
an update process module operable to update the data store with values associated with accounts of entities by:
determining a plurality of accounts to update from the final number of accounts, where each account of the plurality of accounts is associated with an entity of the plurality of entities and one of the online sites;
for each account of the plurality of accounts:
forming a request for a plurality of data points associated with the online site associated with the account, wherein the request conforms to the proprietary interface provided by that online site and requests values for the plurality of data points associated with the online site for the entity associated with the account;
sending the request to the online site associated with the account over the network;
receiving a response to the request, where the response includes a value returned by the associated online site for each of the plurality of data points associated with the online site associated with the account; and
storing the value for each of the plurality of data points received from the online site as a historical snapshot in the data store in association with the entity and the account.
2 . The system of claim 1 , wherein the data store includes a management module operable for determining the at least one online site for which a rate limit has not been reached by:
for each of the at least one online sites:
accessing the rate limiting data associated with the online site;
accessing a plurality of snapshots of historical data obtained from that online site in the data store;
determining, based on the timestamps associated with the plurality of snapshots from that online site, if the number of requests defined by the rate limiting data for that online site exceeds the number of snapshots from that online site during a previous time period equal to the time period defined by the rate limiting data; and
identifying the online site as one of the at least one online sites for which a rate limit has not been reached if the number of requests defined by the rate limiting data exceeds the number of snapshots from that online site during the previous time period.
3 . The system of claim 2 , wherein the management module is operable to determine the final number of accounts by:
for each of the identified online sites for which a rate limit has not been reached, identifying a number of accounts in the data store associated with that online site that have been least recently updated; and
sorting all the accounts identified for all the online sites according to the timestamp associated with each account to determine the final number of accounts from all the accounts identified for all the online sites that were least recently updated.
4 . The system of claim 3 , wherein the number of accounts associated with that online site are identified by:
accessing all the accounts associated with that online site in the data store;
determining, based on the timestamps associated with the snapshots of historical data associated with accounts of that online site, the number of accounts that were least recently updated.
5 . The system of claim 4 , wherein the plurality of accounts are selected randomly from the final number of accounts.
6 . The system of claim 5 , wherein the number of the plurality of accounts to update selected is based on a number of available slots for a dispatch queue maintained by the update process module.
7 . The system of claim 1 , wherein the update process module is one of a plurality of update process modules operating simultaneously to update the data store with values associated with the accounts of entities.
8 . The system of claim 7 , wherein each update process module is operable to reserve the plurality of accounts to update by marking each of the plurality of accounts in the data store.
9 . The system of claim 1 , wherein the update process module is operable to update the data store with values for the plurality of data points for the plurality of entities by:
determining that the online site associated with the account is a search provider site;
accessing the data store to determine a set of search terms associated with each of a set of conversation categories; and
accessing the data store to determine a name associated with the entity, wherein forming the request for the plurality of data points associated with the online site associated with the account includes a request for values for each conversation category based on the search terms for the conversation category and the name of the entity.
10 . A method, comprising:
in a social media analytics platform coupled to each of one of more online sites over a network, wherein the one or more online sites include at least one social media site and each online site provides a proprietary interface for requesting data from that online site, and the social media analytics platform configured with a plurality of entities, each entity having an account at each of the online sites, wherein the social media analytics platform is further configured such that each of the one of more online sites is associated with rate limiting data defining a number of requests and a time period for that online site, and the social media analytics platform:
storing a historical snapshot for each account for each entity, wherein each historical snapshot is associated with a timestamp;
determining a final number of accounts that were least recently updated, wherein determining the final number of accounts comprises:
determining at least one online site of the one or more online sites for which a rate limit has not been reached based on the rate limiting data associated with each online site;
identifying a number of accounts associated with each of the at least one online site that were least recently updated; and
identifying a final number of accounts associated from the number of accounts associated with each of the at least one online site;
updating a data store with values associated with accounts of entities, wherein updating the data store comprises:
determining a plurality of accounts to update from the final number of accounts, where each account of the plurality of accounts is associated with an entity of the plurality of entities and one of the online sites;
for each account of the plurality of accounts:
forming a request for a plurality of data points associated with the online site associated with the account, wherein the request conforms to the proprietary interface provided by that online site and requests values for the plurality of data points associated with the online site for the entity associated with the account;
sending the request to the online site associated with the account over the network;
receiving a response to the request, where the response includes a value returned by the associated online site for each of the plurality of data points associated with the online site associated with the account; and
storing the value for each of the plurality of data points received from the online site as a historical snapshot in the data store in association with the entity and the account.
11 . The method of claim 10 , wherein determining the at least one online site for which a rate limit has not been reached comprises:
for each of the at least one online sites:
accessing the rate limiting data associated with the online site;
accessing a plurality of snapshots of historical data obtained from that online site in the data store;
determining, based on the timestamps associated with the plurality of snapshots from that online site, if the number of requests defined by the rate limiting data for that online site exceeds the number of snapshots from that online site during a previous time period equal to the time period defined by the rate limiting data; and
identifying the online site as one of the at least one online sites for which a rate limit has not been reached if the number of requests defined by the rate limiting data exceeds the number of snapshots from that online site during the previous time period.
12 . The method of claim 11 , wherein determining the final number of accounts comprises:
for each of the identified online sites for which a rate limit has not been reached, identifying a number of accounts in the data store associated with that online site that have been least recently updated; and
sorting all the accounts identified for all the online sites according to the timestamp associated with each account to determine the final number of accounts from all the accounts identified for all the online sites that were least recently updated.
13 . The method of claim 12 , wherein the number of accounts associated with that online site are identified by:
accessing all the accounts associated with that online site in the data store;
determining, based on the timestamps associated with the snapshots of historical data associated with accounts of that online site, the number of accounts that were least recently updated.
14 . The method of claim 13 , wherein the plurality of accounts are selected randomly from the final number of accounts.
15 . The method of claim 14 , wherein the number of the plurality of accounts to update selected is based on a number of available slots for a dispatch queue maintained by the update process module.
16 . The method of claim 10 , wherein updating a data store is done by a plurality of update process modules operating simultaneously to update the data store with values associated with the accounts of entities.
17 . The method of claim 16 , wherein each update process module is operable to reserve the plurality of accounts to update by marking each of the plurality of accounts in the data store.
18 . The method of claim 10 , wherein updating the data store with values for the plurality of data points for the plurality of entities comprises:
determining that the online site associated with the account is a search provider site;
accessing the data store to determine a set of search terms associated with each of a set of conversation categories; and
accessing the data store to determine a name associated with the entity, wherein forming the request for the plurality of data points associated with the online site associated with the account includes a request for values for each conversation category based on the search terms for the conversation category and the name of the entity.