Conditioning of productivity application file content for ingestion by an artificial intelligence model
A data processing system for conditioning productivity application file content for ingestion by an artificial intelligence (AI) language model includes a server system hosting a conversion service, the server system comprising processing and memory resources; an Application Programming Interface (API) for the conversion service to receive productivity application files at any scale; and a format converter of the conversion service, the format converter converting a productivity application file to a format compatible with the AI language model to enable the AI language model to operate on the file content based on a user instruction submitted with the file content as output by the format converter.
1 . A data processing system for conditioning productivity application file content for ingestion by an artificial intelligence (AI) language model, the data processing system comprising:
a server system hosting a first conversion service, an AI service, and a second conversion service, the server system comprising processing and memory resources, and the first conversion service including:
an Application Programming Interface (API) coupled to a productivity application to receive productivity application files having productivity application file content;
a format converter coupled to the API for converting the productivity application file content to conditioned content, the conditioned content having a format compatible with the AI language model to enable the AI language model to operate on the conditioned content based on a user instruction submitted with the conditioned content;
a data subsampler coupled to the format converter to receive the conditioned content, the data subsampler operating on the conditioned content to generate at least one of a new title, a summary, or a file name to assist a user in identifying and organizing the conditioned content according to the user instruction, wherein;
the AI service is coupled to the first conversion service to receive the conditioned content and the user instruction, the AI service executing the user instruction to operate on the conditioned content using the AI language model to produce augmented content; and
the second conversion service is coupled to the AI service to receive an output from the AI language model, the second conversion service operating on the output to produce a new productivity application file that includes the augmented content, wherein the new productivity application file may be opened by the user using the productivity application to access the augmented content according to the user instruction and at least one of the new title, the summary, and the file name.
2 . The data processing system of claim 1 , further comprising a script to submit batches of productivity application files to the first conversion service via the API in a scalable manner.
3 . The data processing system of claim 1 , wherein the format converter comprises a microworker with access to an application code library for the productivity application that produced the productivity application files being converted.
4 . The data processing system of claim 1 , further comprising a Personal Identifiable Information (PII) scrubber to remove PII or other confidential information from the productivity application file content before submission to the AI language model.
5 . The data processing system of claim 4 , wherein the PII scrubber utilizes an AI model trained to identify and remove the PII or the other confidential information from the productivity application file content.
6 . The data processing system of claim 1 , wherein the first conversion service further comprises a language detection/translation module to detect a non-English language in the productivity application file content and translate the productivity application file content to English.
7 . The data processing system of claim 1 , wherein the API is a Representational State Transfer (REST) API.
8 . The data processing system of claim 7 , wherein the API further comprises a gRPC (Remote Procedure Call) service.
9 . A method for conditioning productivity application file content for ingestion by an artificial intelligence (AI) language model, the method comprising:
receiving productivity application files having productivity application file content with a server system hosting a first conversion service via an Application Programming Interface (API) coupled to a productivity application;
converting the content of the productivity application files to conditioned content, the conditioned content having a format compatible with the AI language model to enable the AI language model to operate on the conditioned content based on a user instruction submitted with the conditioned content;
receiving by a data subsampler the conditioned content, the data subsampler operating on the conditioned content to generate at least one of a new title, a summary, or a file name to assist a user in identifying and organizing the conditioned content according to the user instruction;
receiving by the AI service the conditioned content and the user instruction, the AI service executing the user instructions to operate on the conditioned content using the AI language model to produce augmented content; and
receiving by a second conversion service an output from the AI language model, the second conversion service operating on the output to produce a new productivity application file that includes the augmented content, wherein the new productivity application file may be opened by the user using the productivity application to access the augmented content according to the user instruction and at least one of the new title, the summary, and the file name.
10 . The method of claim 9 , further comprising accepting the productivity application files from a script configured to submit batches of productivity application files to the first conversion service via the API in a scalable manner, the script operative to adjust batch size or a number of batches to scale for any size of database of the productivity application files.
11 . The method of claim 9 , wherein converting the content comprises using a microworker with access to an application code library for the productivity application that produced the productivity application files being converted.
12 . The method of claim 9 , further comprising, with the conversion service, removing Personal Identifiable Information (PII) or other confidential information from the productivity application file content before submission to the AI language model.
13 . The method of claim 12 , further comprising using an AI model trained to identify and remove the PII or the other confidential information from the productivity application file content.
14 . The method of claim 9 , further comprising, with the conversion service, detecting a non-English language in the productivity application file content and translating the productivity application file content to English.
15 . The method of claim 9 , wherein the API is a Representational State Transfer (REST) API.
16 . The method of claim 15 , wherein the API further comprises a gRPC (Remote Procedure Call) service.
17 . The method of claim 9 , further comprising performing token and subsampling of the productivity application file content.
18 . A non-transitory computer-readable medium comprising a first conversion service, an AI service, and a second conversion service, for conditioning productivity application file content for ingestion by an artificial intelligence (AI) language model, the medium comprising instructions that, when executed by a host device, cause the host device to:
receive productivity application files having productivity application file content with a server system hosting a conversion service via an Application Programming Interface (API) coupled to a productivity application;
convert the content of the productivity application files to conditioned content, the conditioned content having a format compatible with the AI language model to enable the AI language model to operate on the conditioned content based on a user instruction submitted with the conditioned content;
receive by a data subsampler the conditioned content, the data subsampler operating on the conditioned content to generate at least one of a new title, a summary, or a file name to assist a user in identifying and organizing the conditioned content according to the user instruction;
receive by the AI service the conditioned content and the user instruction, the AI service executing the user instruction to operate on the conditioned content using the AI language model to produce augmented content; and
receive by the second conversion service an output from the AI language model, the second conversion service operating on the output to produce a new productivity application file that includes the augmented content, wherein the new productivity application file may be opened by the user using the productivity application to access the augmented content according to the user instruction and at least one of the new title, the summary, and the file name.