Low-latency method to replace SQL insert for bulk data transfer to relational database
View Patent ↗The invention is generally directed to a method for efficiently loading bulk data into a relational database with reduced load time. A database with record data loaded therein is provided in a computer's main memory. A coordinating program is invoked as well as a standard load utility program that issues record data input requests, opens record data from external media and loads record data to the database therefrom. With the coordinating program record data input requests made by the load utility from external media are intercepted. Record data input requests from external media are replaced with record data input requests from the computer's main memory. Record data from the computer's main memory is inserted directly into the database by the load utility. As a result, delays encountered by reading of input files on the external media by the load utility are avoided thereby freeing up computer resources.
1. A method of loading data into a database, comprising the steps of:
providing an operating system and access method for a record data;
providing a default input routine by the access method for the operating system;
providing a database capable of having record data loaded therein;
providing a computer's main memory;
providing an application data section in the computer's main memory;
providing an input buffer section in the computer's main memory;
providing record data for loading into the database; the record data residing in the computer's main memory;
invoking a coordinating program;
invoking a load utility program that issues record data input requests, opens record data from external media and loads record data to the database therefrom; the load utility having a required syntax;
with the coordinating program, intercepting record data input requests from external media made by the load utility program;
replacing the record data input request from external media with record data input requests from the computer's main memory;
formatting the record data to the syntax required by the load utility;
moving formatted record data from the application data memory section to the input buffer section, for later processing by the load utility;
inserting record data from computer's main memory directly into the database by the load utility; and
whereby delays encountered by reading of input files on the external media by the load utility is avoided.
2. The method of claim 1 , further comprising the steps of:
providing record data in the application data memory section;
moving record data in the application data section to the input buffer section of file computer's main memory by the load utility; and
inserting record data from the input buffer section of the computer's main memory directly into the database.
3. The method of claim 1 , further comprising the steps of:
providing record data in the application data memory section;
copying record data in the application data section to the input buffer section of the computer's main memory by the load utility; and
inserting record data from the input buffer section of the computer's main memory directly into file database.
4. A method of loading data into a database, comprising the steps of:
providing an operating system and access method for a record data;
providing a default input routine by the access method for the operating system;
providing a database capable of having record data loaded therein;
providing a computer's main memory;
providing an application data section in the computer's main memory;
providing an input buffer section in the computer's main memory;
providing record data for loading into the database; the record data residing in the computer's main memory;
invoking a coordinating program;
invoking a load utility program that issues record data input requests, opens record data from external media and loads record data to the database therefrom; the load utility having a required syntax;
with the coordinating program, intercepting record data input requests from external media made by the load utility program;
replacing the record data input request from external media with record data input requests from the computer's main memory;
formatting the record data to the syntax required by the load utility;
coving formatted record data from the application data memory section to the input buffer section, for later processing by the load utility;
inserting record data from computer's main memory directly into the database by the load utility; and
whereby delays encountered by reading of input files on the external media by the load utility is avoided.
5. The method of claim 4 , further comprising the steps of: providing record data in the application data memory section;
moving record data in the application data section to the input buffer section of file computer's main memory by the load utility; and
inserting record data from the input buffer section of the computer's main memory directly into the database.
6. The method of claim 4 , further comprising the steps of:
providing record data in the application data memory section;
copying record data in the application data section to the input buffer section of the computer's main memory by the load utility; and
inserting record data from the input buffer section of the computer's main memory directly into file database.