Methods and systems to troubleshoot and localize storage failures for a multitenant application run in a distributed computing system

1. A process stored in one or more data-storage devices and executed using one or more processors of a computer system to troubleshoot and localize failures in data storage of a multitenant application run in a distributed computing system, the process comprising:

determining network connection status for each data plane node of a database management system for a multitenant database, each data plane node having one or more data cores;

determining availability status of the one or more data cores of each connected data plane node;

isolating the one or more data cores of each disconnected data plane node from data storage ingestion and data queries;

isolating unavailable data cores of each connected data plane node from data storage ingestion and data queries; and

executing recovery to restore the disconnected data plane nodes to receive data storage request and queries from control plane nodes and restore the unavailable data cores to ingest data and respond to data queries.

2. The process of claim 1 wherein determining the network connection status for each data plane node comprises:

sending pings that include echo requests from one or more control plane nodes to each of the data plane nodes;

for each data plane node, when the data plane node sends an echo reply to the sending control plane node in response to receiving the echo request within a reply time limit, identifying the data plane node as network connected; and

for each data plane node, when a data plane node fails to send an echo reply to the sending control plane node in response to receiving the echo request within the reply time limit, identifying the data plane node as network disconnected.

3. The process of claim 1 wherein determine availability status of the one or more data cores of each connected data plane node comprises:

sending pings that include echo requests from one or more control plane nodes to each data core of the connected data plane nodes;

when the data core sends an echo reply to the sending control plane node in response to receiving the echo request within a reply time limit, identifying the data core as available; and

when a data core fails to send an echo reply to the sending control plane node within the reply time limit, identifying the data core as unavailable.

4. The process of claim 1 wherein determining availability status of the one or more data cores of each connected data plane node comprises identifying the data cores as unavailable when data of the data core fails to load data into memory.

5. The process of claim 1 wherein determining availability status of the one or more data cores of each connected data plane node comprises:

for each data core mapping data of the data core to an index;

comparing the index of each data core with a previously generated index for the data core; and

for each data core identifying the data core as unavailable, when the index and the previously generated index disagree.

6. The process of claim 1 wherein executing recovery to restore the disconnected data plane nodes comprises at least one of restarting devices associated with the disconnected data plane nodes and destroying the data plane node followed by recreating the data plane nodes in another virtual object or server computer.

7. The process of claim 1 wherein executing recovery to restore the unavailable data cores comprises at least one of restarting devices associated with the disconnected data cores and destroying the data cores followed by recreating the data cores in one or more data plane nodes.

8. A computer system to troubleshoot and localize failures in data storage of a multitenant application run in a distributed computing system in a distributed computing system, the system comprising:

one or more processors;

one or more data-storage devices; and

machine-readable instructions stored in the one or more data-storage devices that when executed using the one or more processors controls the system to carry out

determining network connection status for each data plane node of a database management system for a multitenant database, each data plane node having one or more data cores;

determining availability status of the one or more data cores of each connected data plane node;

isolating the one or more data cores of each disconnected data plane node from data storage ingestion and data queries;

isolating unavailable data cores of each connected data plane node from data storage ingestion and data queries; and

executing recovery to restore the disconnected data plane nodes to receive data storage request and queries from control plane nodes and restore the unavailable data cores to ingest data and respond to data queries.

9. The system of claim 8 wherein determining the network connection status for each data plane node comprises:

sending pings that include echo requests from one or more control plane nodes to each of the data plane nodes;

for each data plane node, when the data plane node sends an echo reply to the sending control plane node in response to receiving the echo request within a reply time limit, identifying the data plane node as network connected; and

for each data plane node, when a data plane node fails to send an echo reply to the sending control plane node in response to receiving the echo request within the reply time limit, identifying the data plane node as network disconnected.

10. The system of claim 8 wherein determine availability status of the one or more data cores of each connected data plane node comprises:

sending pings that include echo requests from one or more control plane nodes to each data core of the connected data plane nodes;

when the data core sends an echo reply to the sending control plane node in response to receiving the echo request within a reply time limit, identifying the data core as available; and

when a data core fails to send an echo reply to the sending control plane node within the reply time limit, identifying the data core as unavailable.

11. The system of claim 8 wherein determining availability status of the one or more data cores of each connected data plane node comprises identifying the data cores as unavailable when data of the data core fails to load data into memory.

12. The system of claim 8 wherein determining availability status of the one or more data cores of each connected data plane node comprises:

for each data core mapping data of the data core to an index;

comparing the index of each data core with a previously generated index for the data core; and

for each data core identifying the data core as unavailable, when the index and the previously generated index disagree.

13. The system of claim 8 wherein executing recovery to restore the disconnected data plane nodes comprises at least one of restarting devices associated with the disconnected data plane nodes and destroying the data plane node followed by recreating the data plane nodes in another virtual object or server computer.

14. The system of claim 8 wherein executing recovery to restore the unavailable data cores comprises at least one of restarting devices associated with the disconnected data cores and destroying the data cores followed by recreating the data cores in one or more data plane nodes.

15. A non-transitory computer-readable medium encoded with machine-readable instructions that implement a method carried out by one or more processors of a computer system to perform the operations of

determining network connection status for each data plane node of a database management system for a multitenant database, each data plane node having one or more data cores;

determining availability status of the one or more data cores of each connected data plane node;

isolating the one or more data cores of each disconnected data plane node from data storage ingestion and data queries;

isolating unavailable data cores of each connected data plane node from data storage ingestion and data queries; and

executing recovery to restore the disconnected data plane nodes to receive data storage request and queries from control plane nodes and restore the unavailable data cores to ingest data and respond to data queries.

16. The medium of claim 15 wherein determining the network connection status for each data plane node comprises:

sending pings that include echo requests from one or more control plane nodes to each of the data plane nodes;

for each data plane node, when the data plane node sends an echo reply to the sending control plane node in response to receiving the echo request within a reply time limit, identifying the data plane node as network connected; and

for each data plane node, when a data plane node fails to send an echo reply to the sending control plane node in response to receiving the echo request within the reply time limit, identifying the data plane node as network disconnected.

17. The medium of claim 15 wherein determine availability status of the one or more data cores of each connected data plane node comprises:

sending pings that include echo requests from one or more control plane nodes to each data core of the connected data plane nodes;

when the data core sends an echo reply to the sending control plane node in response to receiving the echo request within a reply time limit, identifying the data core as available; and

when a data core fails to send an echo reply to the sending control plane node within the reply time limit, identifying the data core as unavailable.

18. The medium of claim 15 wherein determining availability status of the one or more data cores of each connected data plane node comprises identifying the data cores as unavailable when data of the data core fails to load data into memory.

19. The medium of claim 15 wherein determining availability status of the one or more data cores of each connected data plane node comprises:

for each data core mapping data of the data core to an index;

comparing the index of each data core with a previously generated index for the data core; and

for each data core identifying the data core as unavailable, when the index and the previously generated index disagree.

20. The medium of claim 15 wherein executing recover to restore the disconnected data plane nodes comprises at least one of restarting devices associated with the disconnected data plane nodes and destroying the data plane node followed by recreating the data plane nodes in another virtual object or server computer.

21. The medium of claim 15 wherein executing recovery to restore the unavailable data cores comprises at least one of restarting devices associated with the disconnected data cores and destroying the data cores followed by recreating the data cores in one or more data plane nodes.