Extract Transform Load (ETL) applications for job matching
Various of the disclosed embodiments present systems and methods for generating consolidated job postings from disparate originating sources and formats. Applying an Extraction Transform Load (ETL) framework to the incoming data, a parallel and asynchronous as well as scalable approach to distributing job posting information is presented. “Extraction” may involve the recognition of salient information in the disparate formats (e.g., in employment listings on company webpages). During “transformation”, the information may be reformatted into a universal format or into a format suitable for use at a given destination system. During “loading”, the reformatted data may be supplied to a suitable destination system, e.g., the Application Programming Interface (APIs) of a job board system. Applications in related domains and various optimizations are also discussed.
1. A computer-implemented method for presenting job information to a plurality of job posting systems, comprising:
receiving a source content;
responsive to receiving the source content, detecting a file structure or structure and formatting of the source content;
responsive to detecting the file structure or structure and formatting of the source content, selecting an appropriate extractor based on the detected structure of the source content;
receiving an indication of the plurality of systems on which to post job content;
extracting, using the selected extractor, transforming, and loading the source content to a first system of the plurality of job posting systems;
extracting, using the selected extractor, transforming and loading the source content to a second system of the plurality of job posting systems, wherein the extracting, transforming, and loading of the source content to the second system occurs in parallel with the extracting, transforming, and loading of the source content to first system;
presenting a progress indication for the extracting, transforming and loading for the first and second system at an interface;
monitoring in real-time, by a monitoring processor, the extraction, transforming, and loading processes for the source content at each of the first and second systems, and, based on the real-time monitoring results, reprioritizing operations of the extraction, transforming, and loading processes at any of or both of the first and second systems.
2. The computer-implemented method of claim 1 , wherein transforming source content comprises reformatting the content in a universal format.
3. The computer-implemented method of claim 1 , wherein transforming source content comprises applying a field concatenation operation to portions of the content.
4. The computer-implemented method of claim 1 , wherein transforming source content comprises converting a relative reference URL to an absolute reference URL.
5. The computer-implemented method of claim 1 , wherein loading comprises providing data to a job board API.
6. The computer-implemented method of claim 1 , the method further comprising receiving a prioritization command for one of the extracting, transforming and loading operations for the first system or deferring the loading to the first system in favor of loading to the second system.
7. A non-transitory computer-readable medium comprising instructions configured to cause at least on processor to perform a method comprising:
receiving a source content;
responsive to receiving the source content, detecting a file structure or structure and formatting of the source content;
responsive to detecting the file structure or structure and formatting of the source content, selecting an appropriate extractor based on the detected structure of the source content;
receiving an indication of the plurality of systems on which to post job content;
extracting, using the selected extractor, transforming, and loading the source content to a first system of the plurality of job posting systems;
extracting, using the selected extractor, transforming and loading the source content to a second system of the plurality of job posting systems, wherein the extracting, transforming, and loading of the source content to the second system occurs in parallel with the extracting, transforming, and loading of the source content to first system;
presenting a progress indication for the extracting, transforming and loading for the first and second system at an interface;
monitoring in real-time, by a monitoring processor, the extraction, transforming, and loading processes for the source content at each of the first and second systems, and, based on the real-time monitoring results, reprioritizing operations of the extraction, transforming, and loading processes at any of or both of the first and second systems.
8. The non-transitory computer-readable medium of claim 7 , wherein transforming source content comprises reformatting the content in a universal format.
9. The non-transitory computer-readable medium of claim 7 , wherein transforming source content comprises applying a field concatenation operation to portions of the content.
10. The non-transitory computer-readable medium of claim 7 , wherein transforming source content comprises converting a relative reference URL to an absolute reference URL.
11. The non-transitory computer-readable medium of claim 7 , wherein loading comprises providing data to a job board API.
12. The non-transitory computer-readable medium of claim 7 , the method further comprising receiving a prioritization command for one of the extracting, transforming and loading operations for the first system or deferring the loading to the first system in favor of loading to the second system.
13. A computer system comprising:
at least one processor;
at least one memory comprising instructions configured to cause the at least one processor to perform a method comprising:
receiving a source content;
responsive to receiving the source content, detecting a file structure or structure and formatting of the source content;
responsive to detecting the file structure or structure and formatting of the source content, selecting an appropriate extractor based on the detected structure of the source content;
receiving an indication of the plurality of systems on which to post job content;
extracting, using the selected extractor, transforming, and loading the source content to a first system of the plurality of job posting systems;
extracting, using the selected extractor, transforming and loading the source content to a second system of the plurality of job posting systems, wherein the extracting, transforming, and loading of the source content to the second system occurs in parallel with the extracting, transforming, and loading of the source content to first system;
presenting a progress indication for the extracting, transforming and loading for the first and second system at an interface;
monitoring in real-time, by a monitoring processor, the extraction, transforming, and loading processes for the source content at each of the first and second systems, and, based on the real-time monitoring results, reprioritizing operations of the extraction, transforming, and loading processes at any of or both of the first and second systems.
14. The computer system of claim 13 , wherein transforming source content comprises reformatting the content in a universal format.
15. The computer system of claim 13 , wherein transforming source content comprises applying a field concatenation operation to portions of the content.
16. The computer system of claim 13 , wherein transforming source content comprises converting a relative reference URL to an absolute reference URL.
17. The computer system of claim 13 , wherein loading comprises providing data to a job board API.
18. The computer system of claim 13 , the method further comprising receiving a prioritization command for one of the extracting, transforming and loading operations for the first system or deferring the loading to the first system in favor of loading to the second system.