Method and system of retrieving data in a data file

1. A method for retrieving at least a portion of a first data file, the method being implemented by a computer system that includes one or more processors configured to execute computer program modules, the first data file being stored in a storage device in communication with the computer system, the method comprising:

receiving, by the computer system from a client computer system in communication with the computer system, a request for the at least a portion of the first data file, the request including a start of the at least a portion of the first data file and a size of the at least a portion of the first data file;

computing, by the computer system, a data offset that is aligned to an integer multiple of a block size of the storage device and nearest to the start of the at least a portion of the first data file, wherein computing the data offset comprises computing a data offset O′ using the start O of the at least a portion of the first data file and using an integer multiple of the block size Bsz of the storage device using the following equation O′=O−(O % Bsz), where the operator % is the modulo operator;

computing, by the computer system, a byte-range that is an integer multiple of the block size of the storage device, the byte-range being greater than and encompasses the size of the at least a portion of the first data file;

reading, by the computer system, data within the byte-range and transferring the data within the byte-range starting at the data offset into memory of the computer system; and

transmitting, by the computer system to the client computer system, the at least a portion of the first data file from the memory starting at an offset relative to the data offset that is aligned to an integer multiple of the block size of the storage device.

2. The method according to claim 1 , wherein receiving the request includes receiving a HTTP GET request.

3. The method according to claim 1 , wherein the reading comprises reading using one or more Unbuffered Aligned Direct Input-Output (IO) read operations.

4. The method according to claim 1 , wherein computing the byte-range comprises computing a byte-range S′ using an integer multiple of the block size Bsz of the storage device, the data offset O′ and the start O of the at least a portion of first data file using the following equation S′=(|O−O′|+S)+(Bsz−((|O−O′|+S) % Bsz)) % Bsz, where the operator % is the modulo operator and where S is the size of the at least a portion of the first data file.

5. The method according to claim 4 , wherein reading the data within the byte-range comprises reading data within the size S′ starting at the data offset O′ and transferring into the memory of the computer system.

6. The method according to claim 5 , wherein transmitting the at least a portion of the first data file from the memory comprises transmitting the at least a portion of the first data file from the memory starting at offset |O−O′|.

7. A method for retrieving at least a portion of a second derived virtual data file, the at least a portion of the second derived virtual data file being derived from at least a portion of a first data file based on a first data file virtualization policy, the method being implemented by a computer system, the first data file being stored in a storage device in communication with the computer system, the method comprising:

receiving, by the computer system from a client computer system in communication with the computer system, a request for the at least a portion of the second derived virtual data file, the request including a virtual offset of the at least a portion of the second derived virtual data file and a size of the at least a portion of the second derived virtual data file;

computing, by the computer system, a data offset and a size of at least a portion of the first data file to derive the requested at least a portion of the second derived virtual data file, wherein computing the data offset comprises computing a data offset O′ using the start O of the at least a portion of the first data file and using an integer multiple of a block size Bsz of the storage device using the following equation O′=O−(O % Bsz), where the operator % is the modulo operator;

reading, by the computer system, data from the first data file based on the computed data offset and size of the at least a portion of the first data file;

generating, by the computer system, the requested at least a portion of the second derived data file using information in the first data file virtualization policy and the read data from the first data file; and

transmitting, by the computer system to the client computer system, the requested at least a portion of the second derived virtual data file.

8. The method according to claim 7 , wherein receiving the request includes receiving a HTTP GET request.

9. The method according to claim 7 , further comprising loading, by the computer system, the first data file virtualization policy.

10. The method according to claim 7 , further comprising validating, by the computer system, the request for the at least a portion of the second derived virtual data file.

11. The method according to claim 10 , wherein generating the requested at least a portion of the second derived data file comprises generating partial data from the second derived virtual data file starting from the virtual offset of the at least a portion of the second derived virtual data file.

12. A system for retrieving at least a portion of a first data file stored in a storage device in communication with a computer system, the computer system being configured to:

receive from a client computer system in communication with the computer system, a request for the at least a portion of the first data file, the request including a start of the at least a portion of the first data file and a size of the at least a portion of the first data file;

compute a data offset that is aligned to an integer multiple of a block size of the storage device and nearest to the start of the at least a portion of the first data file, wherein the data offset O′ is computed using the start offset O of the at least a portion of the first data file and using an integer multiple of the block size Bsz of the storage device using the following equation O′=O−(O % Bsz), where the operator % is the modulo operator;

compute a byte-range that is an integer multiple of the block size of the storage device, the byte-range being greater than and encompasses the size of the at least a portion of the first data file;

read data within the byte-range and transferring the data within the byte-range starting at the data offset into memory of the computer system; and

transmit to a client computer system the at least a portion of the first data file from the memory starting at an offset relative to the data offset that is aligned to an integer multiple of the block size of the storage device.

13. The system according to claim 12 , wherein the request includes a HTTP GET request.

14. The system according to claim 12 , wherein the computer system is further configured to read data using one or more Unbuffered Aligned Direct Input-Output (IO) read operations.

15. The system according to claim 12 , wherein the computer system is configured to compute a byte-range S′ using an integer multiple of the block size Bsz of the storage device, the data offset O′ and the start offset O of the at least a portion of first data file using the following equation S′=(|O−O′|+S)+(Bsz−((|O−O′|+S) % Bsz)) % Bsz, where the operator % is the modulo operator and where S is the size of the at least a portion of the first data file.

16. The system according to claim 15 , wherein the computer system is further configured to read the data within the byte-range comprises reading data within the size S′ starting at the data offset O′ and transfer into the memory of the computer system.

17. The system according to claim 16 , wherein the computer system is further configured to transmit the at least a portion of the first data file from the memory starting at offset |O−O′|.

18. A system for retrieving at least a portion of a second derived data file, the at least a portion of the second derived virtual data file being derived from at least a portion of a first data file based on a first data file virtualization policy, the first data file being stored in a storage device in communication with a computer system, the computer system being configured to:

receive from a client computer system in communication with the computer system, a request for the at least a portion of the second derived virtual data file, the request including a virtual offset of the at least a portion of the second derived virtual data file and a size of the at least a portion of the second derived virtual data file;

compute a data offset and a size of at least a portion of the first data file to derive the requested at least a portion of the second derived virtual data file, wherein the data offset O′ is computed using the start offset O of the at least a portion of the first data file and using an integer multiple of a block size Bsz of the storage device using the following equation O′=O−(O % Bsz), where the operator % is the modulo operator;

read data from the first data file based on the computed data offset and size of the at least a portion of the first data file;

generate the requested at least a portion of the second derived data file using information in the first data file virtualization policy and the read data from the first data file; and

transmit to the client computer system the requested at least a portion of the second derived virtual data file.

19. The system according to claim 18 , wherein the request includes a HTTP GET request.

20. The system according to claim 18 , wherein the computer system is further configured to load the first data file virtualization policy.

21. The system according to claim 20 , wherein the computer system is further configured to validate the request for the at least a portion of the second derived virtual data file.

22. The system according to claim 21 , wherein the computer system is further configured to generate partial data from the second derived virtual data file starting from the virtual offset of the at least a portion of the second derived virtual data file.