Cloud-Enabled Caching Architecture for On-Demand Native Application Crawling

1 . A search system comprising:

a query receiver configured to receive a first query related to a first application state of a first application, wherein the first query is requested by a user device;

a live scrape system configured to:

operate a device including an operating system in which an instance of the first application executes; and

responsive to the first query, and within the device,

(i) navigate to the first application state in the instance of the first application and (ii) extract content from the instance of the first application; and

a cache manager configured to:

store information based on the extracted content; and

in response to the query receiver receiving a second query matching the first query, selectively provide the stored information to the user device for presentation to a user.

2 . The search system of claim 1 wherein the device comprises one of:

an emulator configured to execute the operating system on emulated hardware; and

a simulator configured to simulate interaction of the operating system with the first application.

3 . The search system of claim 1 wherein the cache manager is configured to monitor freshness of the stored information.

4 . The search system of claim 3 wherein:

the cache manager is configured to determine a frequency at which content within the first application state changes; and

the freshness of the stored information is (i) inversely proportional to the frequency at which the content of the first application state changes and (ii) inversely proportional to an age of the information stored by the cache manager.

5 . The search system of claim 1 further comprising a cache builder configured to:

after a delay period following the first query, provide a refresh query based on the first query to the live scrape system; and

update the information stored by the cache manager from results returned in response to the refresh query.

6 . The search system of claim 5 wherein:

the cache builder is further configured to predict a user query and provide the predicted user query to the live scrape system; and

the cache manager is configured to (i) store information returned by the live scrape system responsive to the predicted user query and (ii) in response to the query receiver receiving a third query matching the predicted user query, provide the information related to the predicted user query to the user device.

7 . The search system of claim 1 further comprising a query frequency analyzer configured to determine relative frequencies of occurrence of a plurality of queries, wherein the cache manager is configured to preferentially retain information for queries having higher relative frequencies and discard information for queries having lower relative frequencies.

8 . The search system of claim 1 further comprising a query frequency analyzer configured to determine relative frequencies of occurrence of a plurality of queries, wherein the cache manager is configured to store information only for queries having a relative frequency greater than a threshold.

9 . The search system of claim 8 wherein the threshold is adaptive, and wherein the query frequency analyzer increases the threshold as an average of the relative frequencies increases.

10 . The search system of claim 8 further comprising:

a cache builder configured to (i) provide a refresh query based on a common query to the live scrape system and (ii) update information stored by the cache manager related to the common query based on results returned by the live scrape system in response to the refresh query,

wherein the query frequency analyzer identifies a query as the common query in response to the relative frequency of the query exceeding a second threshold, and

wherein the second threshold is higher than the threshold.

11 . The search system of claim 1 further comprising a query router configured to, responsive to the query receiver receiving a third query:

determine whether the cache manager has information stored for the third query;

in response to the cache manager having information stored for the third query, selectively provide the information to the user device; and

in response to the cache manager not having information stored for the third query, selectively provide the third query to the live scrape system.

12 . A computer-implemented method of operating search system, the method comprising:

receiving a first query related to a first application state of a first application, wherein the first query is requested by a user device;

operating a device including an operating system in which an instance of the first application executes;

responsive to the first query, and within the device, (i) navigating to the first application state in the instance of the first application and (ii) extracting content from the instance of the first application;

caching information based on the extracted content; and

in response to receiving a second query matching the first query, selectively providing the cached information to the user device for presentation to a user.

13 . The method of claim 12 wherein operating the device comprises one of:

operating an emulator configured to execute the operating system on emulated hardware; and

operating a simulator configured to simulate interaction of the operating system with the first application.

14 . The method of claim 12 further comprising:

monitoring freshness of the cached information; and

determining a frequency at which content within the first application state changes,

wherein the freshness of the cached information is (i) inversely proportional to the frequency at which the content of the first application state changes and (ii) inversely proportional to an age of the cached information.

15 . The method of claim 12 further comprising:

after a delay period following the first query, servicing a refresh query based on the first query by navigating to the first application state in a device and extracting content; and

updating the cached information from results returned in response to the refresh query.

16 . The method of claim 15 further comprising:

predicting a user query;

servicing the predicted user query by navigating to an application state indicated by the predicted user query and extracting content;

caching the extracted content generated in response to the predicted user query; and

in response to receiving a third query matching the predicted user query, providing the cached information related to the predicted user query.

17 . The method of claim 12 further comprising:

determine relative frequencies of occurrence of a plurality of queries; and

preferentially retaining cached information for queries having higher relative frequencies and discarding cached information for queries having lower relative frequencies.

18 . The method of claim 12 further comprising:

determining relative frequencies of occurrence of a plurality of queries; and

caching information only for queries having a relative frequency greater than a threshold.

19 . The method of claim 18 further comprising increasing the threshold as an average of the relative frequencies increases.

20 . The method of claim 18 further comprising:

servicing a refresh query based on a common query by navigating to an application state indicated by the refresh query and extracting content;

updating cached information related to the common query based on results returned in response to the refresh query; and

identifying a query as the common query in response to the relative frequency of the query exceeding a second threshold,

wherein the second threshold is higher than the threshold.

21 . The method of claim 12 further comprising, in response to receiving a third query:

determining whether information for the third query is cached;

in response to the third query information being cached, selectively providing the cached information to a source of the third query; and

in response to the third query information not being cached, selectively servicing the third query by navigating to an application state indicated by the third query and scraping content to return to the source of the third query.