REGULATING DATA STORAGE BASED ON POPULARITY
Technology is disclosed for regulating data storage based on a popularity of data files (“the technology”). Various embodiments of the technology includes maintaining a fixed durability level of data files stored in a storage system by regulating a number of copies of the data files stored in the storage system. One embodiment includes regulating the number of copies of a particular data file based on popularity of the particular data file among various users using the storage system. The number of copies stored in the storage system is increased or decreased, including from/to zero, based on the popularity of the particular data file. The popularity is determined based on at least one of: a number of computing devices of various users having the particular data file, a latency, network bandwidth and/or availability with the computing devices for reading the particular data file, or access pattern of the particular data file.
1 . A method of regulating data storage, the method comprising:
receiving, at a server, a data file from one or more users to generate multiple copies of the data file;
storing, by the server, the copies of the data file at a storage system;
determining, by the server, a popularity value of the data file, the popularity value indicating a popularity of the data file among the one or more users;
determining, by the server, a number of copies of the data file to be stored at the storage system based on the popularity value; and
adjusting, by the server, the number of copies of the data file stored at the storage system based on the popularity value.
2 . The method of claim 1 , wherein adjusting the number of copies stored at the storage system includes increasing the number of copies stored at the storage system according to a popularity value range the popularity value of the data file corresponds to.
3 . The method of claim 1 , wherein adjusting the number of copies stored at the storage system includes decreasing the number of copies stored at the storage system according to a popularity value range the popularity value of the data file corresponds to.
4 . The method of claim 1 , wherein adjusting the number of copies stored at the storage system includes not storing any of the copies of the data file at the storage system if the popularity value exceeds a threshold popularity value.
5 . The method of claim 1 , wherein receiving the data file from one or more users includes receiving the data file from one or more computing devices associated with each of the one or more users.
6 . The method of claim 1 , wherein the copies of the data file include copies of a portion of the data file.
7 . The method of claim 1 , wherein determining the popularity value of the data file includes determining the popularity value as a function of a number of computing devices associated with the one or more users that contain the data file.
8 . The method of claim 7 , wherein determining the popularity value of the data file includes determining the popularity value as a function of a latency associated with reading the data file from one or more of the computing devices that contain the data file.
9 . The method of claim 7 , wherein determining the popularity value of the data file includes determining the popularity value as a function of a network bandwidth available for reading the data file from one or more of the computing devices that contain the data file.
10 . The method of claim 7 , wherein determining the popularity value of the data file includes determining the popularity value as a function of availability of a network connection with one or more of the computing devices that contain the data file for reading the data file.
11 . The method of claim 1 , wherein determining the popularity value of the data file includes determining the popularity value as a function of a number of the one or more users requiring storage for the same data file at the storage system.
12 . The method of claim 1 , wherein determining the popularity value of the data file includes determining the popularity value as a function of access pattern of the data file for a specific user of the one or more users.
13 . The method of claim 1 , wherein determining the popularity value of the data file includes determining the popularity value as a function of access pattern of the data file for a subset of the one or more users.
14 . A method comprising:
receiving, at a server and from a first computing device associated with a first user, a request to retrieve a first data file of the first user from a storage system, the storage system configured to store a plurality of data files of a plurality of users based on a plurality of popularity values of the corresponding data files;
determining, by the server, whether storage system has the first data file;
responsive to a determination that the storage system does not have the copy of the first data file, determining a plurality of computing devices associated with the users that have a copy of the first data file;
retrieving, by the server, the copy of the first data file from one of the computing devices; and
serving, by the server, the copy of the first data file to the first user.
15 . The method of claim 14 , wherein storing the data files of the users in the storage system includes
determining, by the server and for a data file of the data files, a popularity value of the data file, the popularity value indicating a popularity of the data file among the users,
determining, by the server and based on the popularity value, a number of copies of the data file to be stored at the storage system, and
adjusting, by the server, the number of copies of the data file stored at the storage system based on the popularity value.
16 . The method of claim 15 , wherein determining the popularity value of the data file includes determining the popularity value as a function of a number of computing devices associated with the users that contain the data file.
17 . The method of claim 16 , wherein determining the popularity value of the data file includes determining the popularity value as a function of at least one of: (a) a latency associated with reading the data file from the computing devices that contain the data file, (b) a network bandwidth available for reading the data file from the computing devices that contain the data file or (c) availability of a network connection with the computing devices that contain the data file for reading the data file.
18 . The method of claim 14 , wherein adjusting the number of copies stored at the storage system includes at least one of: (a) not storing any of the copies of the data file at the storage system if the popularity value exceeds a first threshold, (b) increasing the number of copies stored at the storage according to a popularity value range the popularity value of the data file corresponds to or (c) decreasing the number of copies stored at the storage system according to a popularity value range the popularity value of the data file corresponds to.
19 . The method of claim 14 , wherein determining a plurality of computing devices associated with the users that have a copy of the first data file includes
determining, by the server, a checksum of each of the data files received from the users to generate a plurality of checksums,
storing the checksums of the data files and identifications of the computing devices having the copy of the data files at the storage system, and
comparing a first checksum of the first data file with the checksums of the data files to determine if any of the computing devices has the copy of the first data file.
20 . An apparatus comprising:
a storage system configured to store a plurality of data files received from a plurality of users based on a popularity value of each of the data files;
a popularity value determination module to determine the popularity value for each of the data files, the popularity value indicating a popularity of the corresponding data file among the users; and
a data file replication management module to
determine, based on the popularity value, a number of copies of the corresponding data file to be stored at the storage system, and
adjusting, based on the popularity value, the number of copies of the corresponding data file stored at the storage system.
21 . The apparatus of claim 20 further comprising:
a request receiving module to receive from a first computing device associated with a first user a request to retrieve a first data file of the first user from the storage system, the first data file being one of the data files; and
a data file serving module to
determine whether storage system has a copy of the first data file,
responsive to a determination that the storage system does not have the copy of the first data file, determine a plurality of computing devices associated with the users that have the copy of the first data file,
retrieve the copy of the first data file from one of the computing devices, and
serve the copy of the first data file to the first user.
22 . A method comprising:
receiving, at a server and from a first computing device associated with a first user, a request to retrieve a first data file of the first user from a storage system, the storage system configured to store a plurality of data files of a plurality of users based on a plurality of popularity values of the corresponding data files, the storage system configured to store portions of the data files;
determining, by the server, whether storage system has entire first data file or a portion of the first data file;
responsive to a determination that the storage system has the portion of the first data file, determining a plurality of computing devices associated with the users that have a copy of remaining portions of the first data file;
retrieving, by the server, the copy of the remaining portions of the first data file from one of the computing devices; and
serving, by the server, the copy of the entire first data file to the first user, the entire first data file generated using the portion retrieved from the storage system and the remaining portions retrieved from the one of the computing devices.