Management of large-scale networks

1 . A method, comprising:

receiving, by a network orchestrator, a request for performance of a data operation, wherein the request identifies a specific data node of a network configuration tree representing configuration data of a computer network, the configuration data stored in a persistent storage;

performing, by the network orchestrator, a lookup operation for configuration data corresponding to the specific data node the lookup operation comprising:

satisfying the lookup operation using the configuration in a partially loaded network configuration tree resident in a memory of the network orchestrator when available, and

satisfying the lookup operation by retrieving the configuration data from the persistent storage for use by the partially loaded network configuration tree when the configuration data is not available in the partially loaded network configuration tree; and

enqueuing, by the network orchestrator and in response to initiation of the lookup operation, a background loading operation that loads from the persistent storage configuration data for one or more children nodes associated with the specific data node into the memory of the network orchestrator for use by the partially loaded network configuration tree.

2 . The method of claim 1 , wherein the background loading operation is performed to enqueue only configuration data for immediate children nodes associated with the specific data node.

3 . The method of claim 1 , wherein the partially loaded network configuration tree comprises an in-memory tree that represents a subset of configuration data stored in persistent storage.

4 . The method of claim 1 , further comprising:

performing, by the network orchestrator, a range scan of the persistent storage to retrieve the configuration data for the one or more children nodes associated with the specific data node.

5 . The method of claim 1 , further comprising:

evaluating, by the network orchestrator, a plurality of expressions to perform a validation operation involving a data model corresponding to nodes associated with the request to determine expressions among the plurality of expressions that are relevant to nodes associated with the request;

extracting, by the network orchestrator, sub-paths corresponding to the expressions among the plurality of expressions that are relevant to nodes associated with the request; and

using the sub-paths as hints to prefetch upcoming point lookups on sub-trees associated with the lookup operation by retrieving, from the persistent storage, configuration data for the sub-trees into the memory of the network orchestrator for use by the partially loaded network configuration tree.

6 . The method of claim 5 , wherein:

the plurality of expressions comprise XPath expressions, and

the data model is constructed using a YANG data modeling language.

7 . The method of claim 1 , further comprising:

loading, by the network orchestrator, configuration data corresponding to nodes associated with the request into the memory of the network orchestrator in background threads in parallel from the persistent storage; and

mapping, by the network orchestrator, the configuration data into a helper data structure where locations of nodes in the partially loaded network configuration tree are associated with corresponding nodes associated with the request in the helper data structure.

8 . The method of claim 7 , further comprising:

generating, by the network orchestrator, links in the partially loaded network configuration tree; and

removing, by the network orchestrator, entries in the helper data structure that are associated with corresponding to nodes associated with the request in response to generating the links in the partially loaded network configuration tree.

9 . The method of claim 7 , further comprising:

ensuring, by the network orchestrator, a schema order associated with nodes associated with the request while mapping the configuration data into the helper data structure.

10 . The method of claim 1 , further comprising:

monitoring, by the network orchestrator, memory consumption of the memory of the network orchestrator;

determining, by the network orchestrator, that a threshold amount of the memory consumption has been reached; and

issuing, by the network orchestrator, a data purge on nodes of the partially loaded network configuration tree in response to determining that the threshold amount of memory consumption has been reached.

11 . An apparatus, comprising:

one or more network interfaces to communicate with a network;

a processor coupled to the one or more network interfaces and configured to execute one or more processes; and

a memory configured to store a process that is executable by the processor, the process comprising:

receiving, by a network orchestrator, a request for performance of a data operation, wherein the request identifies a specific data node of a network configuration tree representing configuration data of a computer network, the configuration data stored in a persistent storage;

performing, by the network orchestrator, a lookup operation for configuration data corresponding to the specific data node the lookup operation comprising:

satisfying the lookup operation using the configuration in a partially loaded network configuration tree resident in a memory of the network orchestrator when available, and

satisfying the lookup operation by retrieving the configuration data from the persistent storage for use by the partially loaded network configuration tree when the configuration data is not available in the partially loaded network configuration tree; and

enqueuing, by the network orchestrator and in response to initiation of the lookup operation, a background loading operation that loads from the persistent storage configuration data for one or more children nodes associated with the specific data node into the memory of the network orchestrator for use by the partially loaded network configuration tree.

12 . The apparatus of claim 11 , wherein the background loading operation is performed to enqueue only configuration data for immediate children nodes associated with the specific data node.

13 . The apparatus of claim 11 , wherein the partially loaded network configuration tree comprises an in-memory tree that represents a subset of configuration data stored in persistent storage.

14 . The apparatus of claim 11 , further comprising:

performing, by the network orchestrator, a range scan of the persistent storage to retrieve the configuration data for the one or more children nodes associated with the specific data node.

15 . The apparatus of claim 11 , further comprising:

evaluating, by the network orchestrator, a plurality of expressions to perform a validation operation involving a data model corresponding to nodes associated with the request to determine expressions among the plurality of expressions that are relevant to nodes associated with the request;

extracting, by the network orchestrator, sub-paths corresponding to the expressions among the plurality of expressions that are relevant to nodes associated with the request; and

using the sub-paths as hints to prefetch upcoming point lookups on sub-trees associated with the lookup operation by retrieving, from the persistent storage, configuration data for the sub-trees into the memory of the network orchestrator for use by the partially loaded network configuration tree.

16 . The apparatus of claim 15 , wherein:

the plurality of expressions comprise XPath expressions, and

the data model is constructed using a YANG data modeling language.

17 . The apparatus of claim 11 , further comprising:

loading, by the network orchestrator, configuration data corresponding to nodes associated with the request into the memory of the network orchestrator in background threads in parallel from the persistent storage; and

mapping, by the network orchestrator, the configuration data into a helper data structure where locations of nodes in the partially loaded network configuration tree are associated with corresponding nodes associated with the request in the helper data structure.

18 . The apparatus of claim 17 , further comprising:

generating, by the network orchestrator, links in the partially loaded network configuration tree; and

removing, by the network orchestrator, entries in the helper data structure that are associated with corresponding to nodes associated with the request in response to generating the links in the partially loaded network configuration tree.

19 . The apparatus of claim 17 , further comprising:

ensuring, by the network orchestrator, a schema order associated with nodes associated with the request while mapping the configuration data into the helper data structure.

20 . A tangible, non-transitory, computer-readable medium storing program instructions that cause a device to execute a process comprising:

receiving, by a network orchestrator, a request for performance of a data operation, wherein the request identifies a specific data node of a network configuration tree representing configuration data of a computer network, the configuration data stored in a persistent storage;

performing, by the network orchestrator, a lookup operation for configuration data corresponding to the specific data node the lookup operation comprising:

satisfying the lookup operation using the configuration in a partially loaded network configuration tree resident in a memory of the network orchestrator when available, and

satisfying the lookup operation by retrieving the configuration data from the persistent storage for use by the partially loaded network configuration tree when the configuration data is not available in the partially loaded network configuration tree; and

enqueuing, by the network orchestrator and in response to initiation of the lookup operation, a background loading operation that loads from the persistent storage configuration data for one or more children nodes associated with the specific data node into the memory of the network orchestrator for use by the partially loaded network configuration tree.