Data processing system with application-controlled allocation of file storage space
View Patent ↗A data processing system stores data in a data file by first identifying the data file as a large file type known to require an amount of storage space substantially greater than a system allocation unit size. As data is to be written to the data file beginning at a location identified by a file pointer, the system determines whether it is necessary to allocate storage space of the data processing system to store the data to be written, and if so then commands are issued to the file system including (i) one or more first commands to set an end of file marker for the data file to a new value greater than a current value of the end of file marker by at least an application allocation unit size which is at least twice the system allocation unit size, and (ii) one or more second commands to mark a region of the data file extending between the current and new values of the end of file marker as holding valid data. Subsequently the data is written to the data file at the location identified by the file pointer.
1. A method by which a data processing system stores data in a data file, comprising:
identifying the data file as a large file type known to require an amount of storage space substantially greater than a system allocation unit size employed by a file system of the data processing system;
as data is to be written to the data file beginning at a location identified by a file pointer, determining whether it is necessary to allocate storage space of the data processing system to store the data to be written, and if so then issuing commands to the file system including (i) one or more first commands to set an end of file marker for the data file to a new value greater than a current value of the end of file marker by at least an application allocation unit size which is at least twice the system allocation unit size, and (ii) one or more second commands to mark a region of the data file extending between the current and new values of the end of file marker as holding valid data; and
subsequently writing the data to the data file at the location identified by the file pointer.
2. A method according to claim 1 , wherein the one or more first commands include (1) a set file pointer command to advance the file pointer from a current value to an advanced value equal to the new value of the end of file marker, and (2) a set end of file command to set the end of file marker equal to the file pointer, and wherein the commands issued to the file system further include (iii) a third command to return the file pointer to the current value.
3. A method according to claim 1 , wherein data is written to the data file using a seek command paired with a write data command, the seek command indicating that the file pointer is to be moved to the location so that data written by the write data command is written beginning at the location, and wherein the determining and command-issuing is performed in response to the seek command when the location is beyond a current end of allocation for the data file.
4. A method according to claim 3 , wherein data is written to the data file in a multi-threaded fashion in which different processing threads write data to different sections of the data file, and wherein each processing thread uses the seek command to explicitly control the file pointer to avoid writing data to a section of the data file being used by a different processing thread.
5. A method according to claim 1 , wherein data processing system is a backup system which consolidates backup data from one or more client computers into the data file, and wherein the determining includes determining that the data file is a backup data type of file which stores the backup data from the client computers.
6. A method according to claim 5 , wherein the backup system employs a consistent file naming convention under which the backup data type of file has a file name distinct from other types of files, and wherein determining that the data file is a backup data type of file includes determining that the data file has the distinct file name.
7. A data processing system, comprising:
a storage device operative to store data files; and
a computer executing (i) an operating system including a file system, and (ii) an application operative to store data in a data file to be stored on the storage device by:
identifying the data file as a large file type known to require an amount of storage space substantially greater than a system allocation unit size employed by the file system;
as data is to be written to the data file beginning at a location identified by a file pointer, determining whether it is necessary to allocate storage space of the data processing system to store the data to be written, and if so then issuing commands to the file system including (i) one or more first commands to set an end of file marker for the data file to a new value greater than a current value of the end of file marker by at least an application allocation unit size which is at least twice the system allocation unit size, and (ii) one or more second commands to mark a region of the data file extending between the current and new values of the end of file marker as holding valid data; and
subsequently writing the data to the data file at the location identified by the file pointer.
8. A data processing system according to claim 7 , wherein the one or more first commands include (1) a set file pointer command to advance the file pointer from a current value to an advanced value equal to the new value of the end of file marker, and (2) a set end of file command to set the end of file marker equal to the file pointer, and wherein the commands issued to the file system further include (iii) a third command to return the file pointer to the current value.
9. A data processing system according to claim 7 , wherein data is written to the data file using a seek command paired with a write data command, the seek command indicating that the file pointer is to be moved to the location so that data written by the write data command is written beginning at the location, and wherein the determining and command-issuing is performed in response to the seek command when the location is beyond a current end of allocation for the data file.
10. A data processing system according to claim 9 , wherein data is written to the data file in a multi-threaded fashion in which different processing threads write data to different sections of the data file, and wherein each processing thread uses the seek command to explicitly control the file pointer to avoid writing data to a section of the data file being used by a different processing thread.
11. A data processing system according to claim 7 , wherein data processing system is a backup system which consolidates backup data from one or more client computers into the data file, and wherein the determining includes determining that the data file is a backup data type of file which stores the backup data from the client computers.
12. A data processing system according to claim 11 , wherein the backup system employs a consistent file naming convention under which the backup data type of file has a file name distinct from other types of files, and wherein determining that the data file is a backup data type of file includes determining that the data file has the distinct file name.