DECLARATIVE API FOR MANAGING CONNECTORS AND SCHEMAS
A request to configure a connect cluster including one or more connectors for a cloud computing system is received. The request include a desired connector state. A connector specification file is automatically generated based on the desired connector state for the connect cluster via a declarative application programming interface (API). Application resources associated with the connect cluster are automatically configured based on the specification file.
1 . A method comprising:
receiving, at a declarative application programming interface (API), a request to configure a connect cluster comprising one or more connectors, the request comprising a particular state or a schema defined by a user for the connect cluster;
in response to receiving the request, automatically generating a connector specification file based on the particular state or schema defined by the user that triggers installation of one or more connector plugins, the connector specification file comprising a list of the one or more connector plugins; and
automatically configuring application resources associated with the connect cluster based on the connector specification file, the automatically configuring comprising causing download and installation of the one or more connector plugins.
2 . The method of claim 1 , wherein the automatically configuring the application resources comprises automatically assigning one or more tasks based on the particular state of the connect cluster specified by the request.
3 . The method of claim 1 , wherein the connector specification file further comprises an indication of a maximum amount of volume for storing the one or more connector plugins.
4 . The method of claim 1 , wherein the connector specification comprises a Yet Another Markup Language (YAML) file that defines class names and one or more configurations of the one or more connectors.
5 . The method of claim 1 , wherein the connect cluster comprises a plurality of stateful sets, each stateful set of the plurality of stateful sets having a plurality of pods associated therewith.
6 . The method of claim 5 , wherein the one or more connector plugins are downloaded and installed by an init container in a pod of the plurality of pods, the init container downloading the one or more connector plugins, storing the one or more connector plugins, and installing the one or more connector plugins before an application container in the pod runs.
7 . The method of claim 5 , wherein:
the connector specification file further comprises a secret name; and
the automatically configuring the application resources further comprises injecting data of a secret associated with the secret name into a pod of the plurality of pods.
8 . The method of claim 5 , further comprising:
mounting one or more certificates for communicating with external data systems into a pod of the plurality of pods based on the particular state.
9 . The method of claim 1 , further comprising;
determining that a connector of the one or more connectors is in a failed state; and
in response to determining that the connector is in the failed state, attempting to restart the connector via the declarative API.
10 . The method of claim 9 , wherein attempting to restart the connector is based on user input to configure the declarative API to automatically perform a restart attempt.
11 . The method of claim 9 , further comprising:
determining the connector remains in the failed state; and
in response to the determining that the connector remains in the failed state, repeatedly attempting to restart the connector until a specified number of restart attempts have been attempted.
12 . A system, comprising:
at least one processor; and
memory storing instructions that, when executed by the at least one processor cause the system to perform operations comprising:
receiving, at a declarative application programming interface (API), a request to configure a connect cluster comprising one or more connectors, the request comprising a particular state or a schema defined by a user for the connect cluster;
in response to receiving the request, automatically generating a connector specification file based on the particular state or schema defined by the user that triggers installation of one or more connector plugins, the connector specification file comprising a list of the one or more connector plugins; and
automatically configuring application resources associated with the connect cluster based on the connector specification file, the automatically configuring comprising causing download and installation of the one or more connector plugins.
13 . The system of claim 12 , wherein the automatically configuring the application resources comprises automatically assigning one or more tasks based on the particular state of the connect cluster specified by the request.
14 . The system of claim 12 , wherein the connector specification file further comprises an indication of a maximum amount of volume for storing the one or more connector plugins.
15 . The system of claim 12 , wherein the connect cluster comprises a plurality of stateful sets, each stateful set of the plurality of stateful sets having a plurality of pods associated therewith.
16 . The system of claim 15 , wherein the one or more connector plugins are downloaded and installed by an init container in a pod of the plurality of pods, the init container downloading the one or more connector plugins, storing the one or more connector plugins, and installing the one or more connector plugins before an application container in the pod runs.
17 . The system of claim 15 , wherein:
the connector specification file further comprises a secret name; and
the automatically configuring the application resources further comprises injecting data of a secret associated with the secret name into a pod of the plurality of pods.
18 . The system of claim 15 , wherein the operations further comprise:
mounting one or more certificates for communicating with external data systems into a pod of the plurality of pods based on the particular state.
19 . The system of claim 12 , wherein the operations further comprise;
determining that a connector of the one or more connectors is in a failed state; and
in response to determining that the connector is in the failed state, attempting to restart the connector via the declarative API.
20 . A machine-storage medium storing instructions which, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
receiving, at a declarative application programming interface (API), a request to configure a connect cluster comprising one or more connectors, the request comprising a particular state or a schema defined by a user for the connect cluster;
in response to receiving the request, automatically generating a connector specification file based on the particular state or schema defined by the user that triggers installation of one or more connector plugins, the connector specification file comprising a list of the one or more connector plugins; and
automatically configuring application resources associated with the connect cluster based on the connector specification file, the automatically configuring comprising causing download and installation of the one or more connector plugins.