Index machine
In an aspect, provided is a method comprising receiving, at a master node, capability information associated with a plurality of worker nodes, receiving, at the master node, an indexation request, and in response to the indexation request, distributing one or more tasks to the plurality of worker nodes based on the respective capability information, wherein the one or more tasks relate to generating a plurality of indexlets.
1. An apparatus comprising:
at least one processor; and
memory comprising processor-executable instructions that, when executed by the at least one processor, cause the apparatus to:
receive, by a master node, an indexation request associated with a data model;
send, to a first worker node of a plurality of worker nodes, a first task that causes the first worker node to generate a first symbol map, wherein the first symbol map bidirectionally indexes a first portion of the data model;
determine that the first worker node is unavailable for at least one of: a lookup operation or an insert operation; and
send, to a second worker node of the plurality of worker nodes, a second task that causes the second worker node to generate a second symbol map, wherein the second symbol map bidirectionally indexes at least one of: the first portion of the data model or a second portion of the data model.
2. The apparatus of claim 1 , wherein the lookup operation comprises processing at least part of a query associated with the first symbol map, and wherein the insert operation comprises inserting a plurality of symbols into the first symbol map.
3. The apparatus of claim 1 , wherein the first symbol map comprises a first state associated with the first worker node, wherein the first state is indicative of the first worker node being in a stand-by state, and wherein the processor-executable instructions further cause the apparatus to:
cause, based on the first state, the second worker node to be instantiated;
receive, from the second worker node, capability information associated with the second worker node; and
send, to the second worker node, based on the capability information and the determination that the first worker node is unavailable, the second task, wherein the second symbol map bidirectionally indexes the second portion of the data model.
4. The apparatus of claim 1 , wherein the first symbol map comprises a first state associated with the first worker node, wherein the first state is indicative of the first worker node being in either a serving state or a closed state, and wherein the second symbol map bidirectionally indexes the first portion of the data model.
5. The apparatus of claim 1 , wherein the processor-executable instructions that cause the apparatus to determine that the first worker node is unavailable for at least one of: the lookup operation or the insert operation further cause the apparatus to receive, from the first worker node, an indication that the first symbol map has reached an optimal capacity limit.
6. An apparatus comprising:
at least one processor; and
memory comprising processor-executable instructions that, when executed by the at least one processor, cause the apparatus to:
receive, by a first worker node of a plurality of worker nodes, from a master node, a first task associated with a data model;
generate, based on the first task, a first symbol map that bidirectionally indexes a first portion of the data model;
determine that the first symbol map has reached an optimal capacity limit; and
send, to the master node, an indication that the first symbol map has reached the optimal capacity limit, wherein the indication causes the master node to send a second task to a second worker node of the plurality of worker nodes.
7. The apparatus of claim 6 , wherein the processor-executable instructions that cause the apparatus to determine that the first symbol map has reached the optimal capacity limit further cause the apparatus to:
insert a plurality of symbols into the first symbol map; and
determine, based on the plurality of symbols, that the first symbol map has reached the optimal capacity limit.
8. The apparatus of claim 6 , wherein the processor-executable instructions further cause the apparatus to:
receive, by the master node from the first worker node, the indication that the first symbol map has reached the optimal capacity limit; and
send, to the second worker node, the second task.
9. The apparatus of claim 6 , wherein the processor-executable instructions further cause the apparatus to:
receiving, by the second worker node, from the master node, the second task; and
generating, by the second worker node, based on the second task, a second symbol map that bidirectionally indexes at least one of: the first portion of the data model or a second portion of the data model.
10. The apparatus of claim 6 , wherein the first symbol map comprises a first state associated with the first worker node indicative of the first worker node being in a stand-by state, and wherein the processor-executable instructions further cause the apparatus to at least one of:
cause, by the master node, based on the first state, the second worker node to be instantiated;
receive, by the master node from the second worker node, capability information associated with the second worker node;
send, by the master node to the second worker node, based on the capability information and a determination that the first worker node is unavailable, the second task; or
cause, by the master node, based on the first state, the second worker node to generate a second symbol map, wherein the second symbol map bidirectionally indexes the first portion of the data model.
11. One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:
receive, by a master node, an indexation request associated with a data model;
send, to a first worker node of a plurality of worker nodes, a first task that causes the first worker node to generate a first symbol map, wherein the first symbol map bidirectionally indexes a first portion of the data model;
determine that the first worker node is unavailable for at least one of: a lookup operation or an insert operation; and
send, to a second worker node of the plurality of worker nodes, a second task that causes the second worker node to generate a second symbol map, wherein the second symbol map bidirectionally indexes at least one of: the first portion of the data model or a second portion of the data model.
12. The one or more non-transitory computer-readable media of claim 11 , wherein the lookup operation comprises processing at least part of a query associated with the first symbol map, and wherein the insert operation comprises inserting a plurality of symbols into the first symbol map.
13. The one or more non-transitory computer-readable media of claim 11 , wherein the first symbol map comprises a first state associated with the first worker node, wherein the first state is indicative of the first worker node being in a stand-by state, and wherein the processor-executable instructions further cause the computing device to:
cause, based on the first state, the second worker node to be instantiated;
receive, from the second worker node, capability information associated with the second worker node; and
send, to the second worker node, based on the capability information and the determination that the first worker node is unavailable, the second task, wherein the second symbol map bidirectionally indexes the second portion of the data model.
14. The one or more non-transitory computer-readable media of claim 11 , wherein the first symbol map comprises a first state associated with the first worker node, wherein the first state is indicative of the first worker node being in either a serving state or a closed state, and wherein the second symbol map bidirectionally indexes the first portion of the data model.
15. The one or more non-transitory computer-readable media of claim 11 , wherein the processor-executable instructions that cause the computing device to determine that the first worker node is unavailable for at least one of: the lookup operation or the insert operation further cause the computing device to receive, from the first worker node, an indication that the first symbol map has reached an optimal capacity limit.
16. One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:
receive, by a first worker node of a plurality of worker nodes, from a master node, a first task associated with a data model;
generate, based on the first task, a first symbol map that bidirectionally indexes a first portion of the data model;
determine that the first symbol map has reached an optimal capacity limit; and
send, to the master node, an indication that the first symbol map has reached the optimal capacity limit, wherein the indication causes the master node to send a second task to a second worker node of the plurality of worker nodes.
17. The one or more non-transitory computer-readable media of claim 16 , wherein the processor-executable instructions that cause the apparatus to determine that the first symbol map has reached the optimal capacity limit further cause the computing device to:
insert a plurality of symbols into the first symbol map; and
determine, based on the plurality of symbols, that the first symbol map has reached the optimal capacity limit.
18. The one or more non-transitory computer-readable media of claim 16 , wherein the processor-executable instructions further cause the computing device to:
receive, by the master node from the first worker node, the indication that the first symbol map has reached the optimal capacity limit; and
send, to the second worker node, the second task.
19. The one or more non-transitory computer-readable media of claim 16 , wherein the processor-executable instructions further cause the computing device to:
receiving, by the second worker node, from the master node, the second task; and
generating, by the second worker node, based on the second task, a second symbol map that bidirectionally indexes at least one of: the first portion of the data model or a second portion of the data model.
20. The one or more non-transitory computer-readable media of claim 16 , wherein the first symbol map comprises a first state associated with the first worker node indicative of the first worker node being in a stand-by state, and wherein the processor-executable instructions further cause the computing device to at least one of:
cause, by the master node, based on the first state, the second worker node to be instantiated;
receive, by the master node from the second worker node, capability information associated with the second worker node;
send, by the master node to the second worker node, based on the capability information and a determination that the first worker node is unavailable, the second task; or
cause, by the master node, based on the first state, the second worker node to generate a second symbol map, wherein the second symbol map bidirectionally indexes the first portion of the data model.