MAP/REDUCE CONTROLLER SCHEDULER
The inventions related to Map/Reduce, a big data platform that helps in running large scale data processing jobs, e.g. spark jobs, in a batch pipeline used for building an Identity Graph and other data products.
1 . A big data platform for executing spark jobs in a batch pipeline used for building data products, comprising:
a scheduler service, comprising an app written in Scala, capable of starting and/or stopping jobs, where the scheduler service abstracts away the complexity of running a Spark job;
a controller, capable of communication with the scheduler service, the controller orchestrating and managing running Spark clusters, provisioning clusters, and managing cluster scaling;
where the scheduler service takes as input a job request including a json object;
where the schedule service maintains a queue of active jobs, and matches job configuration with capacity for the job to run on a cluster.
2 . The big data platform of claim 1 , where jobs are submitted through an http api, taking a json file as input, where the api response is used to extract error messages, job status, and job diagnostics.
3 . The big data platform of claim 1 , where the controller utilizes spot instances, and supports adding additional storage into a cluster when needed.
4 . The big data platform of claim 1 , including incident analysis logs including cluster level information, node level information, and job level information.
5 . The big data platform of claim 1 , including a metric and event monitoring layer.