Granted Patent B1

US 12,437,113 · App. 19/204,523 · Granted Oct 7, 2025

Inventors: Harrison Dahme (Stateline, NV); Chris Martin (Seattle, WA); Nicholas Roberts-Huntley (New York, NY); Jason Johnson (Ashland, OR)

Assignee: K2 NETWORK LABS, INC.

G06F21/6254G06F21/602G06N7/01

Abstract

Claims (133)

1. A method for orchestrating automated data processing and transformation, comprising:

receiving, by a centralized orchestrator, a request to process a dataset associated with a client;

initiating, by the orchestrator, a data ingestion process to obtain sample data from the dataset, wherein the data ingestion process is executed on client premises or in a cloud environment;

analyzing, by a semantic analysis module controlled by the orchestrator, the sample data to determine semantic types of data fields within the dataset;

generating, by a transformation module controlled by the orchestrator, data transformation instructions based on the determined semantic types;

deploying, by the orchestrator, a data processing pipeline to a client-controlled environment, wherein the pipeline is executed within the client premises;

configuring, by the orchestrator, privacy preservation parameters for the data processing pipeline to identify and obfuscate potential personally identifiable information (PII) in the dataset;

instructing the data processing pipeline to apply the data transformation instructions and privacy preservation parameters to the dataset;

determining, by a configuration module controlled by the orchestrator, data storage configurations for the transformed dataset;

directing the storage of the transformed dataset according to the determined data storage configurations, wherein the storage occurs in a client-controlled environment or a cloud environment;

generating, by a machine learning module that is executed in the cloud or on client premises, a machine learning model based on the transformed dataset; and

storing the machine learning model in a model repository accessible to the client.

2. The method of claim 1 , further comprising:

implementing access controls for the transformed dataset or machine learning model through role-based permission systems with cryptographic verification;

tracking data lineage throughout the transformation process using a directed acyclic graph (DAG) representation with cryptographic hashes;

optimizing query performance on the transformed dataset using automated indexing based on identified access patterns;

scheduling recurring data processing jobs based on configurable parameters; or

providing programmatic access to the orchestrated functionalities through APIs.

3. The method of claim 1 , wherein analyzing the sample data to determine semantic types comprises:

applying a transformer-based machine learning classifier to predict semantic types;

validating the predicted semantic types using pattern matching or contextual analysis;

generating confidence scores for one or more predicted semantic type;

refining the semantic type predictions based on statistical analysis of the sample data; or

generating human-readable descriptions of identified semantic types.

4. The method of claim 1 , wherein generating data transformation instructions comprises:

identifying one or more required data normalization steps based on the determined semantic types;

generating SQL-based transformation queries to standardize data formats;

creating data quality validation rules to ensure consistency of transformed data;

optimizing the transformation instructions for distributed execution; or

automatically generating documentation of transformation logic.

5. The method of claim 1 , wherein configuring privacy preservation parameters comprises:

implementing k-anonymity algorithms for categorical data fields;

creating encrypted views of sensitive data fields while maintaining referential integrity; or

generating privacy impact assessments that quantify re-identification risk.

6. The method of claim 1 , wherein deploying the data processing pipeline comprises:

generating a directed acyclic graph (DAG) representation of the transformation steps;

optimizing the DAG for parallel execution based on data dependencies;

implementing checkpointing mechanisms to ensure pipeline resilience;

dynamically scaling pipeline resources based on data volume; or

implementing secure communication channels between pipeline components.

7. The method of claim 1 , wherein generating the machine learning model comprises:

selecting model architectures based on the semantic types identified in the transformed dataset;

implementing distributed learning techniques to train models while preserving data privacy;

applying transfer learning to leverage pre-trained models relevant to the identified semantic types; or

continuously refining the model using incremental learning techniques.

8. A system for orchestrating automated data processing and transformation, comprising:

a centralized orchestrator implemented on a server system comprising at least one processor and memory;

a client computing architecture comprising at least one processor and memory;

wherein the centralized orchestrator is configured to:

receive a request to process a dataset associated with a client;

initiate a data ingestion process to obtain sample data from the dataset, wherein the data ingestion process is executed on the client computing architecture or in a cloud environment;

control a semantic analysis module to analyze the sample data to determine semantic types of data fields within the dataset;

control a transformation module to generate data transformation instructions based on the determined semantic types;

deploy a data processing pipeline to the client computing architecture;

configure privacy preservation parameters for the data processing pipeline to identify and obfuscate potential personally identifiable information (PII) in the dataset;

instruct the data processing pipeline to apply the data transformation instructions and privacy preservation parameters to the dataset;

control a configuration module to determine data storage configurations for the transformed dataset;

direct the storage of the transformed dataset according to the determined data storage configurations, wherein the storage occurs in the client computing architecture or the cloud environment;

control a machine learning module executed in the cloud environment or on the client computing architecture to generate a machine learning model based on the transformed dataset; and

store the machine learning model in a model repository accessible to the client.

9. The system of claim 8 , wherein the centralized orchestrator is further configured to:

implement access controls for the transformed dataset or machine learning model through role-based permission systems with cryptographic verification;

track data lineage throughout the transformation process using a directed acyclic graph (DAG) representation with cryptographic hashes;

optimize query performance on the transformed dataset using automated indexing based on identified access patterns;

schedule recurring data processing jobs based on configurable parameters; or

provide programmatic access to the orchestrated functionalities through APIs.

10. The system of claim 8 , wherein the semantic analysis module is configured to:

apply a transformer-based machine learning classifier to predict semantic types;

validate the predicted semantic types using pattern matching or contextual analysis;

generate confidence scores for one or more predicted semantic type;

refine the semantic type predictions based on statistical analysis of the sample data; or

generate human-readable descriptions of identified semantic types.

11. The system of claim 8 , wherein the transformation module is configured to:

identify one or more required data normalization steps based on the determined semantic types;

generate SQL-based transformation queries to standardize data formats;

create data quality validation rules to ensure consistency of transformed data;

optimize the transformation instructions for distributed execution; or

automatically generate documentation of transformation logic.

12. The system of claim 8 , wherein configuring privacy preservation parameters comprises:

implement k-anonymity algorithms for categorical data fields;

create encrypted views of sensitive data fields while maintaining referential integrity; or

generate privacy impact assessments that quantify re-identification risk.

13. The system of claim 8 , wherein deploying the data processing pipeline comprises:

generate a directed acyclic graph (DAG) representation of the transformation steps;

optimize the DAG for parallel execution based on data dependencies;

implement checkpointing mechanisms to ensure pipeline resilience;

dynamically scale pipeline resources based on data volume; or

implement secure communication channels between pipeline components.

14. The system of claim 8 , wherein the machine learning module is configured to:

select model architectures based on the semantic types identified in the transformed dataset;

implement distributed learning techniques to train models while preserving data privacy;

apply transfer learning to leverage pre-trained models relevant to the identified semantic types; or

continuously refine the model using incremental learning techniques.

15. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

receiving a request to process a dataset associated with a client;

initiating a data ingestion process to obtain sample data from the dataset, wherein the data ingestion process is executed on client premises or in a cloud environment;

analyzing, by a semantic analysis module, the sample data to determine semantic types of data fields within the dataset;

generating, by a transformation module, data transformation instructions based on the determined semantic types;

deploying a data processing pipeline to a client-controlled environment, wherein the pipeline is executed within the client premises;

configuring privacy preservation parameters for the data processing pipeline to identify and obfuscate potential personally identifiable information (PII) in the dataset;

instructing the data processing pipeline to apply the data transformation instructions and privacy preservation parameters to the dataset;

determining, by a configuration module, data storage configurations for the transformed dataset;

directing the storage of the transformed dataset according to the determined data storage configurations, wherein the storage occurs in a client-controlled environment or a cloud environment;

generating, by a machine learning module that is executed in the cloud or on client premises, a machine learning model based on the transformed dataset; and

storing the machine learning model in a model repository accessible to the client.

16. The non-transitory computer-readable medium of claim 15 , wherein the operations further comprise:

implementing access controls for the transformed dataset or machine learning model through role-based permission systems with cryptographic verification;

tracking data lineage throughout the transformation process using a directed acyclic graph (DAG) representation with cryptographic hashes;

optimizing query performance on the transformed dataset using automated indexing based on identified access patterns;

scheduling recurring data processing jobs based on configurable parameters; or

providing programmatic access to the orchestrated functionalities through APIs.

17. The non-transitory computer-readable medium of claim 15 , wherein analyzing the sample data to determine semantic types comprises:

applying a transformer-based machine learning classifier to predict semantic types;

validating the predicted semantic types using pattern matching or contextual analysis;

generating confidence scores for at least one predicted semantic type;

refining the semantic type predictions based on statistical analysis of the sample data; or

generating human-readable descriptions of identified semantic types.

18. The non-transitory computer-readable medium of claim 15 , wherein generating data transformation instructions comprises:

identifying one or more required data normalization steps based on the determined semantic types;

generating SQL-based transformation queries to standardize data formats;

creating data quality validation rules to ensure consistency of transformed data;

optimizing the transformation instructions for distributed execution; or

automatically generating documentation of transformation logic.

19. The non-transitory computer-readable medium of claim 15 , wherein configuring privacy preservation parameters comprises:

implementing k-anonymity algorithms for categorical data fields;

creating encrypted views of sensitive data fields while maintaining referential integrity; or

generating privacy impact assessments that quantify re-identification risk.

20. The non-transitory computer-readable medium of claim 15 , wherein deploying the data processing pipeline comprises:

generating a directed acyclic graph (DAG) representation of the transformation steps;

optimizing the DAG for parallel execution based on data dependencies;

implementing checkpointing mechanisms to ensure pipeline resilience;

dynamically scaling pipeline resources based on data volume; or

implementing secure communication channels between pipeline components.

Assignments (1)

References Cited (48)

Cited By (7)

US 12,567,000 US 12,602,504 US 12,632,442 US 12,645,674 US 12,657,294 US 12,700,999 US 12,719,869