Data delivery and validation in hybrid cloud environments
Data content delivery and validation in a computer environment may provide a file system in the computer environment, the file system subdivided into unique folder locations per content type, each of the unique folder locations representing a content type folder. The file system is monitored for changes to a content type folder. An occurrence of a manifest file in the content type folder may be detected, the occurrence of the manifest file ensuring that all files in a package of files associated with the manifest file have arrived. Content of the manifest file may be analyzed to check validity of the files. A content package registry may be queried to determine a base job for processing a given content type associated with the package of files, and the base job may be run to process the package of files.
1. A method of data content delivery and validation in a computer environment, the method comprising:
subdividing a file system in the computer environment into a plurality of content type folders, each content type folder comprising content of a certain type;
identifying an arrival of a manifest file associated with a package of files in at least one content type folder of the plurality of content type folders;
analyzing the manifest file to check validity of the package of files;
determining a base job for processing the type of content associated with the package of files; and
in response to determining that each file of the package of files that is indicated in the manifest file is received by the at least one content type folder, running the base job in the computer environment prior to processing the package of files.
2. The method of claim 1 , wherein the analyzing the manifest file comprises:
determining that each file of the package of files that is indicated in the manifest file is received by the at least one content type folder.
3. The method of claim 1 , further comprising:
determining a status of the running of the base job on an interval basis until a completion of the running of the base job prior to processing the package of files.
4. The method of claim 1 , further comprising:
responsive to determining that the base job has failed, reporting an error via a notification rule.
5. The method of claim 1 , further comprising:
responsive to determining a non-occurrence of at least one file of the package of files in the at least one content type folder within a latency window, reporting an error via a notification rule.
6. The method of claim 1 , wherein the analyzing the manifest file includes:
determining that a size of each of the package of files received by the at least one content type folder matches a size of each of the package of files indicated in the manifest file.
7. The method of claim 1 , wherein the package of files includes a content arrival script describing the base job for the package of files.
8. A system comprising:
one or more processors; and
a computer-readable medium comprising instructions stored therein, which when executed by the one or more processors, cause the one or more processors to:
subdivide a file system in a computer environment into a plurality of content type folders, each content type folder comprising content of a certain type;
identify an arrival of a manifest file associated with a package of files in at least one content type folder of the plurality of content type folders;
analyze the manifest file to check validity of the package of files;
determine a base job for processing the type of content associated with the package of files; and
in response to determining that each file of the package of files that is indicated in the manifest file is received by the at least one content type folder, run the base job in the computer environment prior to processing the package of files.
9. The system of claim 8 , wherein the instructions to analyze the manifest file include instructions to:
determine that each file of the package of files that is indicated in the manifest file is received by the at least one content type folder.
10. The system of claim 8 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
determine a status of the running of the base job on an interval basis until a completion of the running of the base job prior to processing the package of files.
11. The system of claim 8 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
responsive to determining that the base job has failed, report an error via a notification rule.
12. The system of claim 8 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
responsive to determining a non-occurrence of at least one file of the package of files in the at least one content type folder within a latency window, report an error via a notification rule.
13. The system of claim 8 , wherein the instructions to analyze the manifest file include instructions to:
determine that a size of each of the package of files received by the at least one content type folder matches a size of each of the package of files indicated in the manifest file.
14. The system of claim 8 , wherein the package of files includes a content arrival script describing the base job for the package of files.
15. A non-transitory computer-readable storage medium comprising computer-readable instructions, which when executed by one or more processors in a computing system, cause the computing system to:
subdivide a file system in a computer environment into a plurality of content type folders each content type folder comprising content of a certain type;
identify an arrival of a manifest file associated with a package of files in at least one content type folder of the plurality of content type folders;
analyze the manifest file to check validity of the package of files;
determine a base job for processing the type of content associated with the package of files; and
in response to determining that each file of the package of files that is indicated in the manifest file is received by the at least one content type folder, run the base job in the computer environment prior to processing the package of files.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the computer-readable instructions to analyze the manifest file include instructions to:
determine that each file of the package of files that is indicated in the manifest file is received by the at least one content type folder.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the computer-readable instructions, when executed by the one or more processors, further cause the computing system to:
determine a status of the running of the base job on an interval basis until a completion of the running of the base job prior to processing the package of files.
18. The non-transitory computer-readable storage medium of claim 15 , wherein the computer-readable instructions, when executed by the one or more processors, further cause the computing system to:
responsive to determining that the base job has failed, report an error via a notification rule.
19. The non-transitory computer-readable storage medium of claim 15 , wherein the computer-readable instructions, when executed by the one or more processors, further cause the computer system to:
responsive to determining a non-occurrence of at least one file of the package of files in the at least one content type folder within a latency window, report an error via a notification rule.
20. The non-transitory computer-readable storage medium of claim 15 , wherein the computer-readable instructions to analyze the manifest file include instructions to:
determine that a size of each of the package of files received by the at least one content type folder matches a size of each of the package of files indicated in the manifest file.