RETURN-LINK OPTIMIZATION FOR FILE-SHARING TRAFFIC
Methods, apparatuses, and systems for return-link optimization are provided. Embodiments identify upload-after-download content (e.g., file sharing content) upon download, and generate one or more identifiers characterizing the content (e.g., a digest). The identifiers are stored in a client-side server dictionary model reflecting a presumption that the content is stored in a server-side dictionary. When content is later uploaded, the server dictionary model is used to identify when the upload content matches previously downloaded content. When a match is detected, the stored identifiers are used to generate a highly compressed version of the upload content, which is then uploaded to the server instead of uploading the full content data. In some embodiments, similar techniques are used to optimize return link bandwidth usage for upload-after-upload transactions.
1 . A method of compressing file uploads comprising:
receiving, at a server-side proxy device, a portion of a piecewise transfer of an object to a first user device as part of a peer-to-peer file transfer protocol, wherein the portion of the piecewise transfer of the object comprises an object identifier as part of the peer-to-peer file transfer protocol, and, wherein the object comprises a plurality of pieces that are communicated as part of the piecewise transfer using the peer-to-peer file transfer protocol;
identifying, by the server-side proxy device, the peer-to-peer file transfer protocol and the object identifier from the portion of the piecewise transfer of the object;
storing as separate files by the server-side proxy device, each piece of the object received via the peer-to-peer file transfer protocol, including at least a first piece of the plurality of pieces of the object identified by the server-side proxy device;
receiving, at the server-side proxy device, a notification associated with a transfer, using the peer-to-peer file transfer protocol, of the first piece of the object from the first user device to a second user device via the server-side proxy device;
receiving, from the second user device via the server-side proxy device at a first client-side proxy associated with the first user device, a request for the first piece of the object as part of the piecewise transfer of the object to the second user device;
communicating from the first client-side proxy to a file transfer client of the first user device, the request for the first piece of the object, wherein the file transfer client is associated with the peer-to-peer file transfer protocol;
communicating from the server-side proxy device to the first client-side proxy, a confirmation that the first piece of the object is stored at the server-side proxy device;
receiving at the first client-side proxy from the server-side proxy device, the confirmation that the first piece of the object is stored at the server-side proxy device;
receiving at the first client-side proxy from the file transfer client of the first user device, the first piece of the object for communication to the second user device via the server-side proxy device;
compressing the first piece of the object at the first client-side proxy to create a compressed copy of the first piece of the object;
communicating the first piece of the object from the first client-side proxy to the second user device by communicating the compressed copy of the first piece of the object to the server-side proxy device for decompression using the first piece of the object stored at the server-side proxy device;
receiving, from the first client-side proxy, the compressed copy of the first piece of the object;
decompressing the compressed copy of the first piece of the object using the first piece of the object stored at the server-side proxy device; and
communicating from the server-side proxy device to the second user device, the first piece of the object created by decompressing the compressed copy of the first piece of the object using the first piece of the object stored at the server-side proxy device.
2 . The method of claim 1 wherein the notification associated with the transfer of the first piece of the object from the first user device to the second user device is a request message from the second user device to the first user device for the first piece of the object as part of a file transfer protocol structuring the piecewise transfer.
3 . The method of claim 2 further comprising:
in response to receiving the request message, queuing, by the server-side proxy device, the first piece of the object for communication to the second user device prior to receiving the compressed copy of the first piece from the first client-side proxy.
4 . The method of claim 1 wherein the compressed copy of the first piece of the object is a checksum of the first piece of the object.
5 . The method of claim 1 wherein the notification associated with the transfer of the first piece comprises the object identifier, an index value associated with the first piece of the object, and a checksum of the first piece of the object.
6 . The method of claim 1 further comprising:
storing by the server-side proxy device, a second piece of the plurality of pieces of the object identified by the server-side proxy device as part of the portion of the piecewise transfer.
7 . The method of claim 6 wherein the object consists of the plurality of pieces, and the plurality of pieces consists of (1) a plurality of object length pieces each having a same object length set as a piece length, and (2) an ending object having an object length less than the piece length.
8 . The method of claim 7 further comprising determining the piece length by:
identifying a first size of the first piece of the object and a second size of the second piece of the object; and
setting a larger size or an equal size of the first size and the second size as the piece length;
wherein the piece length is stored on the server-side proxy device.
9 . The method of claim 1 wherein each piece of the plurality of pieces of the object is associated with a different index value identifying a relative location of each piece within the object.
10 . The method of claim 9 wherein storing the first piece of the object at the server-side proxy device comprises storing the first piece of the object with the object identifier and an index value associated with the first piece of the object that identifies the relative location of the first piece within the object.
11 . The method of claim 10 wherein the object identifier is a 20 byte hash value.
12 . The method of claim 11 wherein storing the first piece of the object at the server-side proxy device comprising creating a file structure with a first level associated with the object identifier and a second level below the first level, wherein the second level is associated with the index value associated with the first piece of the object.
13 . The method of claim 1 wherein identifying, by the server-side proxy device, the portion of the piecewise transfer of the object comprises:
identifying a peer-to-peer handshake in a communication between two peer devices.
14 . The method of claim 13 further comprising:
identifying the object identifier, the first piece of the object, and an index value associated with the first piece of the object as part of the communication between two peer systems in a TCP connection.
15 . A method of compressing file uploads comprising:
receiving, at a server-side proxy device, a portion of a piecewise transfer of an object to a first user device as part of a peer-to-peer file transfer protocol, wherein the portion of the piecewise transfer of the object comprises an object identifier as part of the peer-to-peer file transfer protocol, and, wherein the object comprises a plurality of pieces that are communicated as part of the piecewise transfer using the peer-to-peer file transfer protocol;
identifying, by the server-side proxy device, the peer-to-peer file transfer protocol and the object identifier from the portion of the piecewise transfer of the object;
storing as separate files by the server-side proxy device, each piece of the object received via the peer-to-peer file transfer protocol, including at least a first piece of the plurality of pieces of the object identified by the server-side proxy device;
receiving, at the server-side proxy device, a notification associated with a transfer, using the peer-to-peer file transfer protocol, of the first piece of the object from the first user device to a second user device via the server-side proxy device;
communicating from the server-side proxy device to a first client-side proxy associated with the first user device, a confirmation that the first piece of the object is stored at the server-side proxy device;
receiving, from the first client-side proxy, a compressed copy of the first piece of the object; and
communicating from the server-side proxy device to the second user device, the first piece of the object as stored by the server-side proxy device.
16 . The method of claim 15 wherein each piece of the plurality of pieces of the object is associated with a different index value identifying a relative location of each piece within the object.
17 . The method of claim 16 wherein storing the first piece of the object at the server-side proxy device comprises storing the first piece of the object with the object identifier and an index value associated with the first piece of the object that identifies the relative location of the first piece within the object; and
wherein the object identifier is a 20 byte hash value.
18 . The method of claim 17 wherein storing the first piece of the object at the server-side proxy device comprising creating a file structure with a first level associated with the object identifier and a second level below the first level, wherein the second level is associated with the index value associated with the first piece of the object.
19 . A method of compressing file uploads comprising:
receiving, from a second user device via a server-side proxy device at a first client-side proxy associated with a first user device, a request for a first piece of an object as at least a part of a piecewise transfer of the object to the second user device, wherein the object is associated with an object identifier that is standardized by a peer-to-peer transfer protocol which is used for the piecewise transfer of the object;
communicating from the first client-side proxy to a file transfer client of the first user device, the request for the first piece of the object;
receiving at the first client-side proxy from the server-side proxy device, an indication that the first piece of the object is stored at the server-side proxy device;
receiving at the first client-side proxy from the file transfer client of the first user device, the first piece of the object for communication to the second user device via the server-side proxy device;
compressing the first piece of the object at the first client-side proxy to create a compressed copy of the first piece of the object; and
communicating the first piece of the object from the first client-side proxy to the second user device by communicating the compressed copy of the first piece of the object to the server-side proxy device for decompression using the first piece of the object stored at the server-side proxy device.
20 . The method of claim 19 wherein each piece of the plurality of pieces of the object is associated with a different index value identifying a relative location of each piece within the object; and
wherein storing the first piece of the object at the server-side proxy device comprises storing the first piece of the object with the object identifier and an index value associated with the first piece of the object that identifies the relative location of the first piece within the object.