Data distribution system
Distributing data is disclosed. A first request for data associated with a first file is received at a first node. It is determined, at the first node, that the first file is located on a second node. A second request for data associated with the first file is received at the first node. It is determined that a third node should have an instance of the first file. The third node is caused to have an instance of the first file.
1. A system for distributing data, comprising:
a processor configured to:
receive at a first node of a plurality of nodes a first request for data associated with a first file;
determine at the first node that the first file is located on a second node;
receive at the first node a second request for data associated with the first file;
determine that a third node should have an instance of the first file;
cause the third node to have the instance of the first file; and
receive an indication that the third node has received the instance of the first file from a sending node, wherein in response to the indication, the first file is demoted at the sending node to a cached copy and an association to the third node is generated at the sending node to cause the sending node to redirect a subsequent request associated with the first file received at the sending node to the third node; and
a memory coupled to the processor and configured to provide the processor with instructions.
2. The system recited in claim 1 , wherein receiving the first request includes collecting a piece of statistical information at the first node.
3. The system recited in claim 1 , wherein determining that the third node should have the instance of the first file includes evaluating statistical information associated with the first file.
4. The system recited in claim 3 , wherein the statistical information includes information pertaining to an identity of a first requestor of the first request.
5. The system recited in claim 3 , wherein the statistical information includes information pertaining to a location of a first requestor of the first request.
6. The system recited in claim 3 , wherein the statistical information includes information pertaining to how frequently a location of the first file is requested.
7. The system recited in claim 3 , wherein the statistical information includes information pertaining to how frequently the first file is downloaded.
8. The system recited in claim 1 , wherein determining that the third node should have the instance of the first file includes determining whether the third node is in possession of the cached copy.
9. The system recited in claim 8 , wherein the processor is further configured to verify a checksum of the cached copy if it is determined that the third node is in possession of the cached copy.
10. The system recited in claim 1 , wherein the processor is further configured to confirm that the third node is able to have the instance of the first file prior to causing the third node to have the instance of the first file.
11. The system recited in claim 1 , wherein an instance of the first file is demoted at the sending node to a cache and a cache time-out policy is applied to that instance of the first file in the cache.
12. The system recited in claim 1 , wherein it is indicated by the third node to the sending node that the third node is able to receive the instance of the first file based at least in part on not having a cached instance of the first file already being stored at the third node.
13. The system recited in claim 1 , wherein it is indicated by the third node to the sending node that the third node is able to receive the instance of the first file based at least in part on having enough space at the third node for the first file.
14. The system recited in claim 1 , wherein the processor is further configured to determine that the association to the third node is removed from the sending node.
15. The system recited in claim 1 , wherein the first node and the sending node comprise a same node.
16. The system recited in claim 1 , wherein the first node and the sending node comprise different nodes.
17. A method for distributing data, comprising:
receiving at a first node of a plurality of nodes a first request for data associated with a first file;
determining at the first node that the first file is located on a second node;
receiving at the first node a second request for data associated with the first file;
determining that a third node should have an instance of the first file;
causing the third node to have the instance of the first file; and
receiving an indication that the third node has received the instance of the first file from a sending node, wherein in response to the indication, the first file is demoted at the sending node to a cached copy and an association to the third node is generated at the sending node to cause the sending node to redirect a subsequent request associated with the first file received at the sending node to the third node.
18. The method recited in claim 17 , wherein receiving the first request includes collecting a piece of statistical information at the first node.
19. The method recited in claim 17 , wherein determining that the third node should have the instance of the first file includes evaluating statistical information associated with the first file.
20. The method recited in claim 19 , wherein the statistical information includes information pertaining to an identity of a first requestor of the first request.
21. The method recited in claim 19 , wherein the statistical information includes information pertaining to a location of a first requestor of the first request.
22. The method recited in claim 19 , wherein the statistical information includes information pertaining to how frequently a location of the first file is requested.
23. The method recited in claim 19 , wherein the statistical information includes information pertaining to how frequently the first file is downloaded.
24. The method recited in claim 17 , further comprising confirming that the third node is able to have the instance of the first file prior to causing the third node to have the instance of the first file.
25. The method recited in claim 17 , wherein an instance of the first file is demoted at the sending node to a cache and a cache time-out policy is applied to that instance of the first file in the cache.
26. A computer program product for distributing data, the computer program product being embodied in a computer readable medium and comprising computer instructions for:
receiving at a first node of a plurality of nodes a first request for data associated with a first file;
determining at the first node that the first file is located on a second node;
receiving at the first node a second request for data associated with the first file;
determining that a third node should have an instance of the first file;
causing the third node to have the instance of the first file; and
receiving an indication that the third node has received the instance of the first file from a sending node, wherein in response to the indication, the first file is demoted at the sending node to a cached copy and an association to the third node is generated at the sending node to cause the sending node to redirect a subsequent request associated with the first file received at the sending node to the third node.