Data analytic systems
A method is disclosed, comprising receiving, at a build service, an external dataset and an adaptor associated with the external dataset, the external dataset being in a format which is not managed by the build server. The method may also comprise querying the adaptor, by the build service, for information related to the external dataset. The method may also comprise receiving, at the build service, the information related to the external dataset; and causing, by the build service, the external dataset to be input into a build pipeline using the information related to the external dataset.
1. A method comprising:
receiving, at a build service of a build server, an external dataset and an adaptor application module, the external dataset being in a specific format, the adaptor application module associated with the specific format of the external dataset, the external dataset and the adaptor application module stored in a first location of a distributed file system separate from a second location of the distributed file system where a second external dataset and a second adaptor application module are stored;
in response to the receiving the external dataset, retrieving from the adaptor application module, information relevant to a build pipeline for building an output dataset based, at least in part, on the external dataset, wherein the information includes changes to the external dataset since a previous build of the output dataset is performed and a data schema used in the previous build, wherein the information allows the external dataset to be incorporated into the build pipeline, wherein the build pipeline involves only in one or more formats other than the specific format;
incorporating the external dataset into the build pipeline using the information relevant to the building pipeline without the external dataset being reformatted in accordance with requirements of the build service, thereby performing data revision control on the external dataset by a set of interconnected processors that are interconnected by one or more communication networks; and
implementing the build pipeline as a current build following the previous build to generate the output dataset.
2. The method of claim 1 , the information further including a dataset title.
3. The method of claim 1 , further comprising the build service receiving a request from the adaptor application module for information relevant to a most recent data build run by the build service; and in response thereto, providing, by the build service to the adaptor application module, the most recent data build run by the build service.
4. A computer system comprising: one or more processors; one or more non-transitory computer-readable storage media coupled to the one or more processors and storing one or more sequences of instructions which, when executed using the one or more processors, cause the one or more processors to perform:
receiving, at a build service of a build server, an externa dataset and an adaptor application module, the external dataset being in a specific format, the adaptor application module associated with the specific format of the external dataset, the external dataset and the adaptor application module stored in a first location of a distributed the system separate from a second location of the distributed the system, where a second external dataset and a second adaptor application module are stored;
in response to the receiving the external dataset, retrieving from the adaptor application module the information relevant to a build pipeline for building an output dataset based, at least in part, on the external dataset, wherein the information includes changes to the external dataset since a previous build of the output dataset is performed and a data schema used in the previous build, wherein the information allows the external dataset to be incorporated into the build pipeline, wherein the build pipeline involves only in one or more formats other than the specific format;
incorporating the external dataset into the build pipeline using the information relevant to the building pipeline without the external dataset being reformatted in accordance with requirements of the build service, thereby performing data revision control on the external dataset by a set of interconnected processors that are interconnected by, one or more communication networks; and
implementing the build pipeline as a current build following the previous build to generate the output dataset.
5. The computer system of claim 4 , the information further including a dataset title.
6. The computer system of claim 4 , the one or more non-transitory computer-readable storage media further comprising instructions which when executed cause performing: the build service receiving a request from the adaptor application module for information relevant to a most recent data build run by the build service; and in response thereto, providing, by the build service to the adaptor application module, the most recent data build run by the build service.