IP Library › Granted Patent US 12,699,727
Granted Patent B1
US 12,699,727 · App. 19/184,423 · Granted Aug 4, 2026

Multimedia intelligence agent

Inventors: Ron Kimmel (Haifa, IL); Ron Slossberg (Raanana, IL); Aviad Zabatani (Kfar Vitkin, IL); Sagi Ben Moshe (Ein Ayala, IL); Ofir Mulla (Talme Elazar, IL); Noam Rotstein (Haifa, IL); Tomer Weiss (Kadarim, IL)
Assignee: LUMANA INC.
G06F16/435G06F16/243G06F16/2438
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,699,727
App. No.
19/184,423
Filed
Apr 21, 2025
Granted
Aug 4, 2026
Kind
B1
Art Unit
2165
USPC
707/765
Abstract

A system and method for structured querying. A method includes embedding text of a request to create an embedding. The embedding is compared to query structure examples. Each query structure example includes natural language text and a structured query. The structured query of each query structure example is structured according to a format used by a database. Matching query structure examples are identified among the query structure examples based on the comparison. A first query is generated based on the matching query structure examples and the text of the request. The first query is provided to a language model in order to obtain a set of outputs. At least a portion of the database is queried using a second query. The second query is based on the set of outputs from the language model and is structured according to the format used by the database.

Claims (52)

1 . A method for structured querying, comprising:

embedding text of a request in order to create an embedding of the text of the request;

comparing the embedding of the text of the request to at least a portion of each of a plurality of query structure examples during a comparison, wherein each query structure example of the plurality of query structure examples includes a natural language text portion and a structured query portion, wherein the structured query portion of each query structure example is structured according to a format used by a database, wherein the comparison includes comparing the embedding of the text of the request to a plurality of embeddings of the plurality of query structure examples, wherein the embedding of the at least a portion of each of the plurality of query structure examples is a reduced representation of the at least a portion of the query structure example;

identifying at least one matching query structure example among the plurality of query structure examples based on the comparison;

generating a first query based on the at least one matching query structure example and the text of the request;

providing the first query to a language model in order to obtain a set of outputs from the language model; and

querying at least a portion of the database using a second query, wherein the second query is based on the set of outputs from the language model, wherein the second query is structured according to the format used by the database, wherein the at least a portion of the database is queried by a first system, wherein a plurality of media content items is stored in a plurality of storages accessible to a plurality of second systems;

causing transmission of at least a portion of at least one media content item among the plurality of media content items from the plurality of second systems to a third system, wherein the at least one media content item is determined based on the results of querying the at least a portion of the database using the second query performed by the first system, wherein causing the transmission of the at least a portion of the identified at least one media content item further includes sending a corresponding identifier for each of the at least one media content item to a respective second system of the plurality of second systems which is configured to access a respective storage among the plurality of storages in which the media content item is stored.

2 . The method of claim 1 , further comprising:

providing at least one function to the language model, wherein the at least one function is configured to access the database when called by the language model, wherein the set of outputs from the language model includes at least one call to the at least one function.

3 . The method of claim 2 , wherein the request is from a first entity of a plurality of entities, wherein the database includes a plurality of subsets, wherein each subset of the database corresponds to a respective entity of the plurality of entities, wherein the at least one function is configured to limit access by the language model to only at least one subset of the plurality of subsets corresponding to the first entity.

4 . The method of claim 1 , wherein the request is from a first entity of a plurality of entities, wherein the database includes a plurality of subsets, wherein each subset of the database corresponds to a respective entity of the plurality of entities, further comprising:

identifying at least one first subset among the plurality of subsets which corresponds to the first entity, wherein only the at least one first subset of the database is queried using the second query.

5 . The method of claim 1 , wherein the request indicates at least one attribute of a plurality of attributes in natural language, further comprising:

defining an event with respect to the at least one attribute;

analyzing media content in order to detect the event in at least one portion of the media content; and

causing streaming of the at least one portion of the media content when the event is detected.

6 . The method of claim 1 , wherein the request is received via a first language model session with a first language model, wherein the language model to which the first query is provided is a second language model, wherein the first query is provided to the second language model during a second language model session.

7 . The method of claim 1 , wherein the database stores attribute data indicating a plurality of attributes identified within a plurality of media content items, wherein results of querying the database using the second query include at least one instance of at least one attribute among the plurality of attributes, further comprising:

identifying the at least one media content item among the plurality of media content items as including the at least one attribute based on the results of querying the database using the second query.

8 . The method of claim 1 , wherein the first system is a cloud server, wherein each of the plurality of second systems is an edge system, wherein the first system is remote from each of the plurality of second systems.

9 . A non-transitory computer-readable medium having stored thereon instructions for causing a processing circuitry to execute a process, the process comprising:

embedding text of a request in order to create an embedding of the text of the request;

comparing the embedding of the text of the request to at least a portion of each of a plurality of query structure examples during a comparison, wherein each query structure example of the plurality of query structure examples includes a natural language text portion and a structured query portion, wherein the structured query portion of each query structure example is structured according to a format used by a database, wherein the comparison includes comparing the embedding of the text of the request to a plurality of embeddings of the plurality of query structure examples, wherein the embedding of the at least a portion of each of the plurality of query structure examples is a reduced representation of the at least a portion of the query structure example;

identifying at least one matching query structure example among the plurality of query structure examples based on the comparison;

generating a first query based on the at least one matching query structure example and the text of the request;

providing the first query to a language model in order to obtain a set of outputs from the language model; and

querying at least a portion of the database using a second query, wherein the second query is based on the set of outputs from the language model, wherein the second query is structured according to the format used by the database, wherein the at least a portion of the database is queried by a first system, wherein a plurality of media content items is stored in a plurality of storages accessible to a plurality of second systems;

causing transmission of at least a portion of at least one media content item among the plurality of media content items from the plurality of second systems to a third system, wherein the at least one media content item is determined based on the results of querying the at least a portion of the database using the second query performed by the first system, wherein causing the transmission of the at least a portion of the identified at least one media content item further includes sending a corresponding identifier for each of the at least one media content item to a respective second system of the plurality of second systems which is configured to access a respective storage among the plurality of storages in which the media content item is stored.

10 . A system for structured querying, comprising:

a processing circuitry; and

a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to:

embed text of a request in order to create an embedding of the text of the request;

compare the embedding of the text of the request to at least a portion of each of a plurality of query structure examples during a comparison, wherein each query structure example of the plurality of query structure examples includes a natural language text portion and a structured query portion, wherein the structured query portion of each query structure example is structured according to a format used by a database, wherein the system is further configured to compare the embedding of the text of the request to a plurality of embeddings of the plurality of query structure examples, wherein the embedding of the at least a portion of each of the plurality of query structure examples is a reduced representation of the at least a portion of the query structure example;

identify at least one matching query structure example among the plurality of query structure examples based on the comparison;

generate a first query based on the at least one matching query structure example and the text of the request;

provide the first query to a language model in order to obtain a set of outputs from the language model; and

query at least a portion of the database using a second query, wherein the second query is based on the set of outputs from the language model, wherein the second query is structured according to the format used by the database, wherein the at least a portion of the database is queried by a first system, wherein a plurality of media content items is stored in a plurality of storages accessible to a plurality of second systems;

cause transmission of at least a portion of at least one media content item among the plurality of media content items from the plurality of second systems to a third system, wherein the at least one media content item is determined based on the results of querying the at least a portion of the database using the second query performed by the first system, wherein causing the transmission of the at least a portion of the identified at least one media content item further includes sending a corresponding identifier for each of the at least one media content item to a respective second system of the plurality of second systems which is configured to access a respective storage among the plurality of storages in which the media content item is stored.

11 . The system of claim 10 , wherein the system is further configured to:

provide at least one function to the language model, wherein the at least one function is configured to access the database when called by the language model, wherein the set of outputs from the language model includes at least one call to the at least one function.

12 . The system of claim 11 , wherein the request is from a first entity of a plurality of entities, wherein the database includes a plurality of subsets, wherein each subset of the database corresponds to a respective entity of the plurality of entities, wherein the at least one function is configured to limit access by the language model to only at least one subset of the plurality of subsets corresponding to the first entity.

13 . The system of claim 10 , wherein the request is from a first entity of a plurality of entities, wherein the database includes a plurality of subsets, wherein each subset of the database corresponds to a respective entity of the plurality of entities, wherein the system is further configured to:

identify at least one first subset among the plurality of subsets which corresponds to the first entity, wherein only the at least one first subset of the database is queried using the second query.

14 . The system of claim 10 , wherein the request indicates at least one attribute of a plurality of attributes in natural language, wherein the system is further configured to:

define an event with respect to the at least one attribute;

analyze media content in order to detect the event in at least one portion of the media content; and

cause streaming of the at least one portion of the media content when the event is detected.

15 . The system of claim 10 , wherein the request is received via a first language model session with a first language model, wherein the language model to which the first query is provided is a second language model, wherein the first query is provided to the second language model during a second language model session.

16 . The system of claim 10 , wherein the database stores attribute data indicating a plurality of attributes identified within a plurality of media content items, wherein results of querying the database using the second query include at least one instance of at least one attribute among the plurality of attributes, wherein the system is further configured to:

identify the at least one media content item among the plurality of media content items as including the at least one attribute based on the results of querying the database using the second query.

17 . The system of claim 10 , wherein the first system is a cloud server, wherein each of the plurality of second systems is an edge system, wherein the first system is remote from each of the plurality of second systems.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 22, 2025
From: KIMMEL, RON; SLOSSBERG, RON; ZABATANI, AVIAD; BEN MOSHE, SAGI; MULLA, OFIR; ROTSTEIN, NOAM; WEISS, TOMER
To: LUMANA INC.
Reel/Frame 070911/0837 →
References Cited (62)
US 4604694A · Hough · 1986 [cited by applicant]
US 5339427A · Elko et al. · 1994 [cited by applicant]
US 9152666B2 · Lin et al. · 2015 [cited by applicant]
US 10049308B1 · Dhua et al. · 2018 [cited by applicant]
US 10223604B2 · Bahl et al. · 2019 [cited by applicant]
US 11238093B2 · Ayush et al. · 2022 [cited by applicant]
US 11314985B2 · Clayton et al. · 2022 [cited by applicant]
US 11631162B2 · Price et al. · 2023 [cited by applicant]
US 11676247B2 · Zimmer et al. · 2023 [cited by applicant]
US 11842727B2 · Metallinou et al. · 2023 [cited by applicant]
US 11862171B2 · Li et al. · 2024 [cited by applicant]
US 11886828B1 · Gray et al. · 2024 [cited by applicant]
US 11947923B1 · Jain et al. · 2024 [cited by applicant]
US 11977854B2 · Tunstall-Pedoe et al. · 2024 [cited by applicant]
US 11989527B2 · Tunstall-Pedoe et al. · 2024 [cited by applicant]
US 11995412B1 · Mishra · 2024 [cited by applicant]
US 12008739B2 · Xu et al. · 2024 [cited by applicant]
US 12073180B2 · Tunstall-Pedoe et al. · 2024 [cited by applicant]
US 20150012661A1 · Elmore · 2015 [cited by applicant]
US 20150213056A1 · Maluf et al. · 2015 [cited by applicant]
US 20200242774A1 · Park et al. · 2020 [cited by applicant]
US 20210097691A1 · Liu · 2021 [cited by applicant]
US 20210218819A1 · Lawson et al. · 2021 [cited by applicant]
US 20210256061A1 · Janakiraman · 2021 [cited by examiner]
US 20210326674A1 · Liu et al. · 2021 [cited by applicant]
US 20210365736A1 · Kearney et al. · 2021 [cited by applicant]
US 20220067994A1 · Neuberger et al. · 2022 [cited by applicant]
US 20220138170A1 · Misiewicz · 2022 [cited by examiner]
US 20220180447A1 · Kearney et al. · 2022 [cited by applicant]
US 20220207707A1 · Yang et al. · 2022 [cited by applicant]
US 20220383864A1 · Gruber et al. · 2022 [cited by applicant]
US 20220398247A1 · Wu · 2022 [cited by examiner]
US 20230042224A1 · Patel et al. · 2023 [cited by applicant]
US 20230126177A1 · Xu et al. · 2023 [cited by applicant]
US 20230169074A1 · Kim · 2023 [cited by examiner]
US 20230177683A1 · Lenis et al. · 2023 [cited by applicant]
US 20230326212A1 · Chawda et al. · 2023 [cited by applicant]
US 20230342884A1 · Singh et al. · 2023 [cited by applicant]
US 20230351102A1 · Tran · 2023 [cited by applicant]
US 20240062067A1 · Hu et al. · 2024 [cited by applicant]
US 20240143835A1 · Khodadadeh et al. · 2024 [cited by applicant]
US 20240205405A1 · Tanaka et al. · 2024 [cited by applicant]
US 20240252048A1 · Tran · 2024 [cited by applicant]
US 20240320807A1 · Zhong et al. · 2024 [cited by applicant]
US 20240338802A1 · Yoon et al. · 2024 [cited by applicant]
US 20240347059A1 · Manjunath et al. · 2024 [cited by applicant]
US 20250053732A1 · Itani · 2025 [cited by applicant]
US 20250077509A1 · Schindel · 2025 [cited by examiner]
US 20250086211A1 · Bolcer · 2025 [cited by examiner]
US 20250148657A1 · Hirota et al. · 2025 [cited by applicant]
CN 110868598B · 2021 [cited by applicant]
CN 116541490A · 2023 [cited by applicant]
CN 117725259A · 2024 [cited by applicant]
CN 117743610A · 2024 [cited by applicant]
CN 117972127A · 2024 [cited by applicant]
CN 118051635A · 2024 [cited by applicant]
WO 2023161630A1 · 2023 [cited by applicant]
English translation of CN119597785A using Google Patents. Publication date Mar. 11, 2025. 9 pages. (Year: 2025). [cited by examiner]
English Translation of CN117112590A to Sun et al (Year: 2023). [cited by examiner]
O'Connor, Ryan. “Introduction to Diffusion Models for Machine Learning.” AssemblyAI, May 12, 2022, www.assemblyai.com/blog/diffusion-models-for-machine-learning-introduction/. Date Accessed Dec. 14, 2023. [cited by applicant]
Kirichenko, Polina, Pavel Izmailov, and Andrew Gordon Wilson. “Last layer re-training is sufficient for robustness to spurious correlations.” arXiv preprint arXiv:2204.02937 (2022). (Year: 2022). [cited by applicant]
Liusie, Adian, Potsawee Manakul, and Mark Gales. “Mitigating word bias in zero-shot prompt-based classifiers.” Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings). 2023. (Year: 2023). [cited by applicant]