Remote data collection systems and methods using read only data extraction and dynamic data handling
Remote data collection systems and methods retrieve data including financial, sales, marketing, operational and the like data from a plurality of databases and database types remotely over a network in an automated, platform-agnostic manner. A remote data collection system includes a network interface, a connection to a data source, a processor communicatively coupled to the network interface and the connection, and memory storing instructions for remote data collection that, when executed, cause the processor to: receive a request to extract data from the data source; extract the data in a non-intrusive manner from the data source using a two phase process comprising a reconciliation phase and a collection phase; and transmit one of an entire set and a subset of the extracted data based on the request.
1. A remote data collection system, comprising:
a network interface;
a connection to a data source;
a processor communicatively coupled to the network interface and the connection; and
memory storing instructions for remote data collection that, when executed, cause the processor to:
receive a request to extract data from the data source;
extract the data in a non-intrusive manner from the data source using a two phase process comprising a reconciliation phase and a collection phase; and
transmit one of an entire set and a subset of the extracted data based on the request.
2. The remote data collection system of claim 1 , wherein the non-intrusive manner comprises the data source being read only to the processor during the remote data collection.
3. The remote data collection system of claim 1 , wherein the instructions, when executed, further cause the processor to:
copy the data from the data source to a shadow database; and
process the data in the shadow database to provide the entire set and the subset of the extracted data.
4. The remote data collection system of claim 3 , wherein the data source is read only to the processor during the remote data collection.
5. The remote data collection system of claim 3 , wherein the shadow database is separate from the data source, and wherein the non-intrusive manner utilizes the shadow database for any insertions, updates, or deletions thereby preventing any interference, corruption, or modification of the data source by the processor during the remote data collection.
6. The remote data collection system of claim 3 , wherein the instructions, when executed, further cause the processor to:
mark data stored in the shadow database with metadata to distinguish between existing data, modified data, deleted data, or new data, in order to facilitate the remote data collection process.
7. The remote data collection system of claim 3 , wherein the instructions, when executed, further cause the processor to:
create the shadow database based on database schema, tables, and columns defined by the current collection object for the data source;
wherein the shadow database is adapted to a type associated with the data source thereby enabling the remote data collection to operate on a plurality of software, systems, and data files for obtaining extractable data therefrom.
8. The remote data collection system of claim 7 , wherein the data source comprises one of a text file, a spreadsheet, a structured query language database, an Indexed Sequential Access Method file, and a flat file.
9. The remote data collection system of claim 7 , wherein the instructions, when executed, further cause the processor to:
perform peer-to-peer ad hoc dynamic data replication and synchronization with a remote data system; and
utilize a dynamic command language with the remote data system to determine what data sets to retrieve, how to extract the data sets, and compare extracted data to targeted or requested data.
10. The remote data collection system of claim 1 , wherein the instructions, when executed, further cause the processor to:
perform the reconciliation phase to determine what data to extract from the data source, to determine how to extract the data from the data source, and to define a current collection object for extracting the data from the data source; and
perform the collection phase to synchronize data between the data source and the shadow database, to process the data in the shadow database, and to send the processed data.
11. A remote data collection method, comprising:
receiving a request for remote data collection to extract data from a data source;
extracting the data in a non-intrusive manner from the data source using a two phase process comprising a reconciliation phase and a collection phase; and
transmitting one of an entire set and a subset of the extracted data based on the request.
12. The remote data collection method of claim 10 , further comprising:
copying the data from the data source to a shadow database; and
processing the data in the shadow database to provide the entire set and the subset of the extracted data, wherein the data source is read only during the remote data collection.
13. The remote data collection method of claim 11 , wherein the shadow database is separate from the data source, and wherein the non-intrusive manner utilizes the shadow database for any insertions, updates, or deletions thereby preventing any interference, corruption, or modification of the data source by the processor during the remote data collection.
14. The remote data collection method of claim 11 , further comprising:
marking data stored in the shadow database with metadata to distinguish between existing data, modified data, deleted data, or new data, in order to facilitate the remote data collection process.
15. The remote data collection method of claim 11 , further comprising:
creating the shadow database based on database schema, tables, and columns defined by the current collection object for the data source;
wherein the shadow database is adapted to a type associated with the data source thereby enabling the remote data collection to operate on a plurality of software, systems, and data files for obtaining extractable data therefrom.
16. The remote data collection method of claim 15 , wherein the data source comprises one of a text file, a spreadsheet, a structured query language database, an Indexed Sequential Access Method file, and a flat file.
17. The remote data collection method of claim 15 , further comprising:
performing peer-to-peer ad hoc dynamic data replication and synchronization with a remote data system; and
utilizing a dynamic command language with the remote data system to determine what data sets to retrieve, how to extract the data sets, and compare extracted data to targeted or requested data.
18. The remote data collection method of claim 10 , further comprising:
performing the reconciliation phase to determine what data to extract from the data source, to determine how to extract the data from the data source, and to define a current collection object for extracting the data from the data source; and
performing the collection phase to synchronize data between the data source and the shadow database, to process the data in the shadow database, and to send the processed data.
19. Software stored in a non-transitory computer readable medium and comprising instructions executable by a system, and in response to such execution causes the system to perform operations comprising:
receiving a request for remote data collection to extract data from a data source;
interfacing to the data source in a read-only manner;
performing a reconciliation phase to determine what data to extract from the data source, to determine how to extract the data from the data source, and to define a current collection object for extracting the data from the data source;
performing a collection phase to synchronize data between the data source and a shadow database, to process the data in the shadow database, and to send the processed data; and
transmitting one of an entire set and a subset of the extracted data based on the request.