Efficient storage for segment store

1. A method comprising:

analyzing, by a computing device, segments for segment lists associated with a plurality of user identifiers to generate an index of segments;

determining, by the computing device, a frequency of respective segments, wherein the frequency is based on a segment being included in one or more segment lists;

calculating, by the computing device, a first threshold for an array storage type based on at least the frequency of respective segments;

calculating, by the computing device, a second threshold for a run-length encoding storage type based on removing x segments from a bitmap storage type based on the x segments being less than or equal to the (first threshold/2)-1;

analyzing, by the computing device, a characteristic of a segment list for a user identifier;

comparing, by the computing device, the characteristic for the segment list to the first threshold and the second threshold, wherein the first threshold and the second threshold are used to determine which of the array storage type, the run-length encoding storage type, and the bitmap storage type to use to store a segment value for the user identifier;

selecting, by the computing device, a storage type from the array storage type, the run-length encoding storage type, and a bitmap storage type for the segment list based on the comparing to the first threshold and the second threshold, wherein the bitmap storage type is selected when the first threshold and the second threshold are not met;

storing, by the computing device, the segment value for the segment list and a type value in a value formate for a key in storage, wherein the segment value is stored using the storage type and the key is associated with the user identifier;

retrieving, by the computing device, the value format using the key for a request that is associated with the user identifier;

determining, by the computing device, a type value stored in the value format, wherein the type value indicates the storage type that was used to store the segment value;

calculating, by the computing device, the segment list using the storage type from the segment value of the value format; and

outputting, by the computing device, the segment list to respond to the request.

2. The method of claim 1 , further comprising:

determining the key for the user identifier; and

storing the key in association with the segment value and the type value in the storage.

3. The method of claim 1 , wherein comparing the characteristic for the segment list to the first threshold and the second threshold comprises comparing a length of the segment list to the first threshold and the second threshold.

4. The method of claim 3 , wherein,

the segment list includes a first number of segments that are associated with the user identifier from a second number of segments, and

the length is based on the first number of the segments that are associated with the user identifier.

5. The method of claim 1 , wherein comparing the characteristic for the segment list to the first threshold and the second threshold comprises:

using the array storage type when the characteristic meets the first threshold; and

using run-length encoding storage type when the characteristic meets the second threshold.

6. The method of claim 5 , wherein comparing the characteristic for the segment list to the first threshold and the second threshold comprises:

using the bitmap storage type when the characteristic does not meet the first threshold and the second threshold.

7. The method of claim 1 , further comprising:

determining an array of segments found in the storage; and

determining indices for segments in the segment list based on a position of the segments in the segment list in the array of segments.

8. The method of claim 7 , further comprising:

determining the value for the segment list by storing the indices using the array.

9. The method of claim 7 , wherein:

the array storage type stores an array the indices,

the run-length encoding storage type uses the indices to store runs, and

the bitmap storage type store bits for the array of segments in a bitmap, wherein bits are set based on the indices for the segments.

10. The method of claim 1 , wherein calculating the (x+1)×IndexAverageBytes≤2≤M is true, wherein x is the x segments, IndexAverageBytes is an average bytes for the index of the segments, and M is a most bytes that can be used to store a record with the bitmap storage type.

11. The method of claim 10 , wherein calculating the first threshold comprises:

calculating a sum of the frequency of segments for the segment lists associated with the plurality of user identifiers;

calculating a weight for each segment based on a frequency of each segment and the sum;

calculating an amount of storage to store an index for each segment;

calculating an average amount of storage for each segment based on the amount of storage to store the index and the respective weight for the index; and

calculating the first threshold based on the average amount of storage.

12. The method of claim 1 , wherein calculating the second threshold comprises:

determining a maximum number of segments that can be taken from the segments for the segment lists associated with the plurality of user identifiers based on an average bytes in an index used in the array storage type and a value for the most bytes that can be used to store a record with the bitmap storage type; and

calculating the second threshold based on the maximum number.

13. The method of claim 1 , further comprising:

receiving a request for segments for the user identifier, wherein the response to the request is subject to a time constraint to respond.

14. A non-transitory computer-readable storage medium having stored thereon computer executable instructions, which when executed by a computing device, cause the computing device to be operable for:

analyzing segments for segment lists associated with a plurality of user identifiers to generate an index of segments;

determining a frequency of respective segments, wherein the frequency is based on a segment being included in one or more segment lists;

calculating a first threshold for an array storage type based on the frequency of respective segments;

calculating a second threshold for a run-length encoding storage type based on removing x segments from a bitmap storage type based on the x segments being less than or equal to the (first threshold/2)−1;

analyzing a characteristic of a segment list for a user identifier;

comparing the characteristic for the segment list to the first threshold and the second threshold, wherein the first threshold and the second threshold are used to determine which of the array storage type, the run-length encoding storage type, and the bitmap storage type to use to store a segment value for the user identifier;

selecting a storage type from the array storage type, the run-length encoding storage type, and a bitmap storage type for the segment list based on the comparing to the first threshold and the second threshold, wherein the bitmap storage type is selected when the first threshold and the second threshold are not met; and

storing the segment value for the segment list and a type value in a value format for a key in storage, wherein the segment value is stored using the storage type and the key is associated with the user identifier;

retrieving the value format using the key for a request that is associated with the user identifier;

determining a type value stored in the value format, wherein the type value indicates the storage type that was used to store the segment value;

calculating the segment list using the storage type from the segment value of the value formate; and

outputting, by the computing device, the segment list to respond to the request.

15. A method comprising:

analyzing, by a computing device, segments for segment lists associated with a plurality of user identifiers to generate an index of segments;

determining, by the computing device, a frequency of respective segments, wherein the frequency is based on a segment being included in one or more segment lists;

calculating, by the computing device, a first threshold in two or more thresholds for a first storage type based on the frequency of respective segments;

calculating, by the computing device, a second threshold in the two or more thresholds for a second storage type, wherein the first threshold and the second threshold are used to determine which of the first storage type, the second storage type, and a third storage type to use to store a value for the user identifier, wherein calculating the second threshold comprises:

determining a maximum number of segments that can be taken from the segments for the segment lists associated with the plurality of user identifiers based on removing x segments from a third storage type based on the x segments being less than or equal to the (first threshold/2)−1;

calculating the second threshold based on the maximum number;

storing, by the computing device, the first threshold and the second threshold;

storing, by the computing device, a segment value for a segment list and a type value in a value format for a key in storage, wherein the segment value is stored using a storage type that is determined based on the first threshold or the second threshold, and the key is associated with a user identifier;

retrieving, by the computing device, the value format using the key for a request that is associated with the user identifier;

determining a type value stored in the value format, wherein the type value indicates the storage type that was used to store the segment value;

calculating the segment list using the storage type from the segment value of the value format; and

outputting, by the computing device, the segment list to respond to the request.

16. The method of claim 15 , wherein calculating the first threshold comprises:

calculating a sum of the frequency of segments for the segment lists associated with the plurality of user identifiers;

calculating a weight for each segment based on a frequency of each segment and the sum;

calculating an amount of storage to store an index for each segment;

calculating an average amount of storage for each segment based on the amount of storage to store the index and the respective weight for the index; and

calculating the first threshold based on the average amount of storage.

17. The method of claim 15 , further comprising:

analyzing a characteristic of a segment list for a user identifier;

comparing the characteristic for the segment list to the first threshold or the second threshold;

determining a storage type for the segment list based on the comparing, wherein a value for the segment list is stored using a storage type from a plurality of storage types based on the characteristic for the segment list meeting the first threshold, the second threshold, or not meeting the first threshold and the second threshold; and

storing, by the computing device, the value for the segment list in storage, wherein the value is stored based on the storage type.

18. The method of claim 15 , wherein:

the first storage type comprises an array storage type,

the second storage type comprises a run-length encoding storage type, and

the third storage type comprises a bitmap.

19. The method of claim 15 , wherein the maximum number of segments is determined based on:

dividing the value for the most bytes by the average bytes in the index.

20. The method of claim 15 , wherein the average bytes in the index is determined based on:

determining a weight for each index based on the frequency for each segment; and

multiplying the weight for each index by a respective index length.