Random access file management

1. A method for modifying a file having a file size exceeding a size of available main memory, the file comprising existing data arranged in one or more existing data blocks stored in at least one persistent memory source,

by writing an array of new data to the file at a file offset within the file, to replace a portion of the existing data of the file,

the method comprising:

identifying a block size of the one more existing data blocks of the file;

determining one or more target data blocks of the file, the target data blocks being existing data blocks of the file which contain data to be replaced by a portion of the array of new data, said determining taking into consideration a size of the array of new data, the block size of the file and the file offset within the file;

determining one or more write byte arrays, each write byte array being a portion of the array of new data;

performing one or more write operations, said write operations comprising the steps of:

identifying a location of at least one of the one or more target data blocks within the file;

retrieving the at least one of the one or more target data blocks from the file into the available main memory;

creating at least one new data block comprised of at least one of the one or more write byte arrays; and

writing the at least one new data block to the file at the location of the at least one of the one or more target data blocks, wherein the new data block is created by writing a subset of the contents of the at least one of the one or more target data blocks into a first portion of the new data block, and writing the contents of the at least one of the one or more write byte arrays into a second portion of the new data block.

2. A method for modifying a file having a file size exceeding a size of available main memory, the file comprising existing data arranged in one or more existing data blocks stored in at least one persistent memory source,

by writing an array of new data to the file at a file offset within the file, to replace a portion of the existing data of the file,

the method comprising:

identifying a block size of the one more existing data blocks of the file;

determining one or more target data blocks of the file, the target data blocks being existing data blocks of the file which contain data to be replaced by a portion of the array of new data, said determining taking into consideration a size of the array of new data, the block size of the file and the file offset within the file;

determining one or more write byte arrays, each write byte array being a portion of the array of new data;

performing one or more write operations, said write operations comprising the steps of:

identifying a location of at least one of the one or more target data blocks within the file;

retrieving the at least one of the one or more target data blocks from the file into the available main memory;

creating at least one new data block comprised of at least one of the one or more write byte arrays; and

writing the at least one new data block to the file at the location of the at least one of the one or more target data blocks, wherein the one or more write operations includes a plurality of write operations, each write operation creating at least one new data block comprised of at least one of a plurality of write byte arrays.

3. The method of claim 2 , wherein each of the plurality of write byte arrays is a unique portion of the array of new data.

4. The method of claim 2 , wherein the one or more target data blocks of the file includes a plurality of target data blocks.

5. The method of claim 4 , wherein the plurality of target data blocks is read from the persistent memory source in one operation.

6. The method of claim 4 , wherein the plurality of target data blocks is read from the persistent memory source in a series of operations.

7. The method of claim 4 , wherein a plurality of new data blocks is created, each new data block being comprised of one of the plurality of write byte arrays and a subset of one of the plurality of target data blocks.

8. The method of claim 7 , wherein the plurality of new data blocks is written to the persistent memory source in one operation.

9. The method of claim 7 , wherein the plurality of new data blocks is written to the persistent memory source in a series of operations.

10. The method of claim 7 , wherein at least one of the plurality of new data blocks is written to the persistent memory source before all the target data blocks have been retrieved from the persistent memory source.

11. A system for modifying a file having a size exceeding a size of available main memory, the file comprising existing data arranged in one or more existing data blocks stored in at least one persistent memory source,

by writing an array of new data to the file at a file offset within the file, to replace a portion of the existing data of the file,

the system comprising:

a file controller, configured to interface with main memory; and

a memory interface, configured to provide an interface between the at least one persistent memory source and the file controller;

the file controller configured to:

identify a block size of the one more existing data blocks of the file;

determine one or more target data blocks of the file, the target data blocks being existing data blocks of the file which contain data to be replaced by a portion of the array of new data, said determining taking into consideration a size of the array of new data, the block size of the file and the file offset within the file;

determine one or more write byte arrays, each write byte array being a portion of the array of new data;

perform one or more write operations, said write operations comprising the steps of:

identifying a location of at least one of the one or more target data blocks within the file;

instructing the memory interface to retrieve the at least one of the one or more target data blocks from the file into available main memory;

creating at least one new data block comprised of at least one of the one or more write byte arrays; and

instructing the memory interface to write the at least one new data block to the file at the location of the at least one of the one or more target data blocks, wherein the new data block is created by writing a subset of the contents of the at least one of the one or more target data blocks into a first portion of the new data block, and writing the contents of the at least one of the one or more write byte arrays into a second portion of the new data block.

12. A system for modifying a file having a size exceeding a size of available main memory, the file comprising existing data arranged in one or more existing data blocks stored in at least one persistent memory source, by writing an array of new data to the file at a file offset within the file, to replace a portion of the existing data of the file,

the system comprising:

a file controller, configured to interface with main memory; and

a memory interface, configured to provide an interface between the at least one persistent memory source and the file controller;

the file controller configured to:

identify a block size of the one more existing data blocks of the file;

determine one or more target data blocks of the file, the target data blocks being existing data blocks of the file which contain data to be replaced by a portion of the array of new data, said determining taking into consideration a size of the array of new data, the block size of the file and the file offset within the file;

determine one or more write byte arrays, each write byte array being a portion of the array of new data;

perform one or more write operations, said write operations comprising the steps of:

identifying a location of at least one of the one or more target data blocks within the file;

instructing the memory interface to retrieve the at least one of the one or more target data blocks from the file into available main memory;

creating at least one new data block comprised of at least one of the one or more write byte arrays; and

instructing the memory interface to write the at least one new data block to the file at the location of the at least one of the one or more target data blocks, wherein the one or more write operations includes a plurality of write operations, each write operation creating at least one new data block comprised of at least one of a plurality of write byte arrays.