Method and approach for pagination over data stream with sliding window
An example method includes receiving, at a server, a request for a list of a first group of index records that correspond to stored data, creating a data streaming session with a client, reading an index file and obtaining a list of the first group of index records from the index file, populating a content cache with a signature that corresponds to the first group of index records, creating a sliding window and populating it with the signature and a pointer to a next group of index records, populating an attribute cache with data streaming session information and the sliding window, creating a continuation token which, when received by the server from the client, indicates to the server that the list of the first group of index records has been received by the client.
1. A method, comprising:
receiving at a server, from a client, a request for a list of a first group of index records that correspond to stored data;
creating a data streaming session with the client;
reading an index file and obtaining a list of the first group of index records from the index file;
populating a content cache with a signature that corresponds to the first group of index records;
creating a sliding window and populating the sliding window with the signature and a pointer to a next group of index records;
populating an attribute cache with data streaming session information and the sliding window; and
creating a continuation token which, when received by the server from the client, provides an indication to the server that the list of the first group of index records has been received by the client.
2. The method as recited in claim 1 , wherein the streaming session is only created when a header of the request does not include a continuation token.
3. The method as recited in claim 1 , wherein the continuation token is creating by encoding session information together with the pointer to the next group of index records.
4. The method as recited in claim 1 , wherein the continuation token comprises a tuple that includes a signature that corresponds to data in the content cache, and that also includes the pointer to the next group of index records.
5. The method as recited in claim 1 , further comprising sending a response to the client, wherein the response comprises a list of the first group of index records and an updated continuation token that corresponds to a second group of index records.
6. The method as recited in claim 1 , wherein in response to receipt of the continuation token, the server transmits, to the client, a list of the next group of index records.
7. The method as recited in claim 1 , further comprising:
performing a check to determine if all records requested by the client have been fetched; and,
when it has been determined that all records requested by the client have been fetched, terminating the session.
8. The method as recited in claim 7 , wherein when it has been determined that all records requested by the client have been fetched, the method further comprises sending a response to the client indicating that all records have been fetched.
9. The method as recited in claim 1 , further comprising creating a second continuation token, and the second continuation token includes a first pointer that is also included in the continuation token, and the second continuation token further includes a second pointer that is not included in the continuation token.
10. A system, comprising:
one or more hardware processors; and
a non-transitory storage medium having stored therein instructions which are executable by the one or more hardware processors to perform operations comprising:
receiving at a server, from a client, a request for a list of a first group of index records that correspond to stored data;
creating a data streaming session with the client;
reading an index file and obtaining a list of the first group of index records from the index file;
populating a content cache with a signature that corresponds to the first group of index records;
creating a sliding window and populating the sliding window with the signature and a pointer to a next group of index records;
populating an attribute cache with data streaming session information and the sliding window; and
creating a continuation token which, when received by the server from the client, provides an indication to the server that the list of the first group of index records has been received by the client.
11. The system as recited in claim 10 , wherein the streaming session is only created when a header of the request does not include a continuation token.
12. The system as recited in claim 10 , wherein the continuation token is creating by encoding session information together with the pointer to the next group of index records.
13. The system as recited in claim 10 , wherein the continuation token comprises a tuple that includes a signature that corresponds to data in the content cache, and that also includes the pointer to the next group of index records.
14. The system as recited in claim 10 , wherein the operations further comprise sending a response to the client, wherein the response comprises a list of the first group of index records and an updated continuation token that corresponds to a second group of index records.
15. The system as recited in claim 10 , wherein in response to receipt of the continuation token, the server transmits, to the client, a list of the next group of index records.
16. The system as recited in claim 10 , wherein the operations further comprise:
performing a check to determine if all records requested by the client have been fetched; and,
when it has been determined that all records requested by the client have been fetched, terminating the session.
17. The system as recited in claim 16 , wherein when it has been determined that all records requested by the client have been fetched, the operations further comprise sending a response to the client indicating that all records have been fetched.
18. The system as recited in claim 10 , wherein the operations further comprise creating a second continuation token, and the second continuation token includes a first pointer that is also included in the continuation token, and the second continuation token further includes a second pointer that is not included in the continuation token.