Optimized dynamic large scale and large document ingestion for search engines using service mesh
In one embodiment, an illustrative method herein comprises: obtaining, by a process, a file having a given size; assigning, by the process, the file to a particular size-range bucket of a plurality of size-range buckets of a data ingestion pipeline, the plurality of size-range buckets having a corresponding size-based configuration; and forwarding, by the process, the file into a particular size-based service mesh ingress gateway of the data ingestion pipeline according to the particular size-range bucket to cause processing of the file within the data ingestion pipeline according to the corresponding size-based configuration of the particular size-range bucket.
1 . A method, comprising:
obtaining, by a process, a file having a given size;
assigning, by the process, the file to a particular size-range bucket of a plurality of size-range buckets of a data ingestion pipeline, the plurality of size-range buckets having a corresponding size-based configuration; and
forwarding, by the process, the file into a particular size-based service mesh ingress gateway of the data ingestion pipeline according to the particular size-range bucket to cause processing of the file within the data ingestion pipeline according to the corresponding size-based configuration of the particular size-range bucket.
2 . The method as in claim 1 , further comprising:
pre-processing the file prior to forwarding the file.
3 . The method as in claim 2 , wherein pre-processing comprises an aggregation of a plurality of files.
4 . The method as in claim 1 , wherein the plurality of size-range buckets each correspond to separate service mesh ingress gateway and a corresponding cloud platform internal load balancer to a particular instance of a distributed search and analytics engine.
5 . The method as in claim 1 , wherein the corresponding size-based configuration of the plurality of size-range buckets are based on smaller-sized buckets managing a larger number of smaller-sized files with lower processing and memory resources than larger-sized buckets that manage a smaller number of larger-sized files with higher processing and memory resources.
6 . The method as in claim 1 , further comprising:
monitoring file traffic of the data ingestion pipeline; and
deriving observability data based on the file traffic of the data ingestion pipeline.
7 . The method as in claim 6 , wherein the observability data comprises one or more of: sizes of files; a percentage of distribution of file sizes; incoming request durations for connections for document ingestion; server responses and their HTTP Codes for document ingestion; server incoming request volume; average response time for requests; number of requests per second across microservices; and nodes used for document ingestion; or errors.
8 . The method as in claim 6 , further comprising:
tuning the corresponding size-based configuration of the plurality of size-range buckets based on applying machine learning to the observability data.
9 . The method as in claim 1 , further comprising:
annotating the file according to the particular size-range bucket.
10 . The method as in claim 9 , wherein annotating adds a cloud storage publish/subscribe message attribute, the method further comprising:
pushing the file to a cloud storage publish/subscribe messaging topic.
11 . The method as in claim 10 , wherein one or more subscription filters of the data ingestion pipeline are configured to filter files based on the cloud storage publish/subscribe message attribute to cause an appropriate size-based service mesh ingress gateway to process the file.
12 . The method as in claim 1 , wherein the corresponding size-based configuration comprises processor and memory specifications.
13 . The method as in claim 1 , wherein the corresponding size-based configuration comprises one or more of: file batch sizes, number of files within a given batch, timeout, or worker threads.
14 . The method as in claim 1 , wherein the corresponding size-based configuration comprises one or more of pod replica configurations or horizontal pod autoscaler configurations.
15 . The method as in claim 1 , further comprising:
implementing an artificial intelligence agent to dynamically configure ingestion of files to increase efficiency of indexing files based on file sizes.
16 . A tangible, non-transitory, computer-readable medium having computer-executable instructions stored thereon that, when executed by a processor on a computer, cause the computer to perform a method comprising:
obtaining a file having a given size;
assigning the file to a particular size-range bucket of a plurality of size-range buckets of a data ingestion pipeline, the plurality of size-range buckets having a corresponding size-based configuration; and
forwarding the file into a particular size-based service mesh ingress gateway of the data ingestion pipeline according to the particular size-range bucket to cause processing of the file within the data ingestion pipeline according to the corresponding size-based configuration of the particular size-range bucket.
17 . The tangible, non-transitory, computer-readable medium as in claim 16 , wherein the plurality of size-range buckets each correspond to separate service mesh ingress gateway and a corresponding cloud platform internal load balancer to a particular instance of a distributed search and analytics engine.
18 . The tangible, non-transitory, computer-readable medium as in claim 16 , wherein the corresponding size-based configuration of the plurality of size-range buckets are based on smaller-sized buckets managing a larger number of smaller-sized files with lower processing and memory resources than larger-sized buckets that manage a smaller number of larger-sized files with higher processing and memory resources.
19 . The tangible, non-transitory, computer-readable medium as in claim 16 , wherein the method further comprises:
monitoring file traffic of the data ingestion pipeline; and
deriving observability data based on the file traffic of the data ingestion pipeline.
20 . An apparatus, comprising:
one or more network interfaces to communicate with a network;
a processor coupled to the one or more network interfaces and configured to execute one or more processes; and
a memory configured to store a process that is executable by the processor, the process, when executed, configured to:
obtain a file having a given size;
assign the file to a particular size-range bucket of a plurality of size-range buckets of a data ingestion pipeline, the plurality of size-range buckets having a corresponding size-based configuration; and
forward the file into a particular size-based service mesh ingress gateway of the data ingestion pipeline according to the particular size-range bucket to cause processing of the file within the data ingestion pipeline according to the corresponding size-based configuration of the particular size-range bucket.