Apparatus, method, and computer program product for distributed semantic search

1 . An apparatus comprising at least one processor and at least one non-transitory memory comprising program code stored thereon, wherein the at least one non-transitory memory and the program code are configured to, with the at least one processor, cause the apparatus to at least:

partition a main text data set representing a master corpus into a plurality of node text data sets each representing a portion of the master corpus based at least in part on a predefined partitioning schema, wherein each node text data set of the plurality of node text data sets corresponds uniquely to a semantic search node of a plurality of semantic search nodes;

for each semantic search node of the plurality of semantic search nodes:

select, from a plurality of index structures associated with a semantic search process, an optimal index structure based at least in part on hyperparameters associated with the plurality of index structures; and

generate a node index based at least in part on the node text data set corresponding to the semantic search node according to the selected optimal index structure;

store the plurality of node indexes in a node index store as a file in a columnar file format comprising a table in which each of the node indexes is stored as a corresponding column of binary values;

for each semantic search node of the plurality of semantic search nodes, load the corresponding node index from the node index store into a persistent node index cache of the semantic search node, wherein the node index persists independently with respect to distinct executions of a corresponding instance of a semantic search;

receive a query vector representing input for the semantic search over the master corpus;

at each semantic search node of the plurality of semantic search nodes, generate a node results set representing output of the semantic search over the portion of the master corpus represented by the node text data set corresponding to the semantic search node by executing an instance of a semantic search process based at least in part on the query vector and on the node index corresponding to the semantic search node; and

generate a master results set representing output of the semantic search over the master corpus by combining a plurality of node results sets comprising the node results set generated at each semantic search node of the plurality of semantic search nodes.

2 . The apparatus of claim 1 , wherein each node results set of the plurality of node results sets comprises a top-k set of ranked results determined by the instance of the semantic search process corresponding to the semantic search node at which the node results set is generated, and combining the plurality of node results sets comprises determining a global top-k set of the ranked results from all of the node results sets in the plurality of node results sets based at least in part on similarity scores, wherein the global top-k set of the ranked results are determined after the node results sets are retrieved from persistent node index caches.

3 . The apparatus of claim 1 , wherein the predefined partitioning schema is configured such that the portion of the master corpus represented by each node text data set of the plurality of node text data sets is determined based at least in part on time data, region data, and/or categories associated with each discrete object in the master corpus.

4 . The apparatus of claim 1 , wherein generating the node index corresponding to each semantic search node of the plurality of semantic search nodes comprises:

generating, for each semantic search node of the plurality of semantic search nodes, a node vector set corresponding to the semantic search node based at least in part on the node text data set corresponding to the semantic search node by embedding text from the node text data set using a trained language model; and

generating the node index corresponding to each semantic search node of the plurality of semantic search nodes based at least in part on the node vector set.

5 . The apparatus of claim 4 , wherein the at least one non-transitory memory and the program code are configured to, with the at least one processor, further cause the apparatus to at least:

tune the trained language model for optimal compatibility with the semantic search process based at least in part on hyperparameters and sample query data comprising labeled semantic search inputs.

6 . The apparatus of claim 1 , wherein the main text data set representing the master corpus comprises aggregated enterprise data collected by an enterprise management system with respect to one or more systems associated with an enterprise, the query vector is generated based at least in part on semantic search request input received via a semantic search request interface presented via the enterprise management system, and the enterprise management system, in response to receiving the master results set perform one or more enterprise management operations with respect to one or more systems associated with the enterprise based at least in part on the master results set.

7 . A computer-implemented method comprising:

partitioning a main text data set representing a master corpus into a plurality of node text data sets each representing a portion of the master corpus based at least in part on a predefined partitioning schema, wherein each node text data set of the plurality of node text data sets corresponds uniquely to a semantic search node of a plurality of semantic search nodes;

for each semantic search node of the plurality of semantic search nodes:

selecting, from a plurality of index structures associated with a semantic search process, an optimal index structure based at least in part on hyperparameters associated with the plurality of index structures; and

generating a node index based at least in part on the node text data set corresponding to the semantic search node according to selected optimal index structure;

storing the plurality of node indexes in a node index store as a file in a columnar file format comprising a table in which each of the node indexes is stored as a corresponding column of binary values;

for each semantic search node of the plurality of semantic search nodes, loading the corresponding node index from the node index store into a persistent node index cache of the semantic search node, wherein the node index persists independently with respect to distinct executions of a corresponding instance of a semantic search;

receiving a query vector representing input for the semantic search over the master corpus;

at each semantic search node of the plurality of semantic search nodes, generating a node results set representing output of the semantic search over the portion of the master corpus represented by the node text data set corresponding to the semantic search node by executing an instance of a semantic search process based at least in part on the query vector and on the node index corresponding to the semantic search node; and

generating a master results set representing output of the semantic search over the master corpus by combining a plurality of node results sets comprising the node results set generated at each semantic search node of the plurality of semantic search nodes.

8 . The method of claim 7 , wherein each node results set of the plurality of node results sets comprises a top-k set of ranked results determined by the instance of the semantic search process corresponding to the semantic search node at which the node results set is generated, and combining the plurality of node results sets comprises determining a global top-k set of the ranked results from all of the node results sets in the plurality of node results sets based at least in part on similarity scores, wherein the global top-k set of the ranked results are determined after the node results sets are retrieved from persistent node index caches.

9 . The method of claim 7 , wherein the predefined partitioning schema is configured such that the portion of the master corpus represented by each node text data set of the plurality of node text data sets is determined based at least in part on time data, region data, and/or categories associated with each discrete object in the master corpus.

10 . The method of claim 7 , wherein generating the node index corresponding to each semantic search node of the plurality of semantic search nodes comprises:

generating, for each semantic search node of the plurality of semantic search nodes, a node vector set corresponding to the semantic search node based at least in part on the node text data set corresponding to the semantic search node by embedding text from the node text data set using a trained language model; and

generating the node index corresponding to each semantic search node of the plurality of semantic search nodes based at least in part on the node vector set.

11 . The method of claim 10 , further comprising tuning the trained language model for optimal compatibility with the semantic search process based at least in part on hyperparameters and sample query data comprising labeled semantic search inputs.

12 . A computer program product comprising at least one non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising an executable portion configured to:

partition a main text data set representing a master corpus into a plurality of node text data sets each representing a portion of the master corpus based at least in part on a predefined partitioning schema, wherein each node text data set of the plurality of node text data sets corresponds uniquely to a semantic search node of a plurality of semantic search nodes;

for each semantic search node of the plurality of semantic search nodes:

select, from a plurality of index structures associated with a semantic search process, an optimal index structure based at least in part on hyperparameters associated with the plurality of index structures; and

generate a node index based at least in part on the node text data set corresponding to the semantic search node according to the selected optimal index structure;

store the plurality of node indexes in a node index store as a file in a columnar file format comprising a table in which each of the node indexes is stored as a corresponding column of binary values;

for each semantic search node of the plurality of semantic search nodes, load the corresponding node index from the node index store into a persistent node index cache of the semantic search node, wherein the node index persists independently with respect to distinct executions of a corresponding instance of a semantic search;

receive a query vector representing input for the semantic search over the master corpus;

at each semantic search node of the plurality of semantic search nodes, generate a node results set representing output of a semantic search over the portion of the master corpus represented by the node text data set corresponding to the semantic search node by executing an instance of a semantic search process based at least in part on the query vector and on the node index corresponding to the semantic search node; and

generate a master results set representing output of a semantic search over the master corpus by combining a plurality of node results sets comprising the node results set generated at each semantic search node of the plurality of semantic search nodes.