Method and system for generating data-enriching augmented reality applications from a domain-specific language

1. A system for generating data-enriching augmented reality (AR) applications from a domain-specific language, the system comprising:

a data store; and

an AR application generating module configured to generate an AR application using a domain-specific language (DSL) and a fuzzy join operation, wherein the AR application is configured to augment real-world data with supplementary data from a remote data source, in an AR environment, wherein the real-world data comprises a plurality of modalities and the AR function comprises a plurality of embedding functions where each embedding function of the plurality of embedding functions corresponds to a different modality of the plurality of modalities and , wherein the plurality of modalities corresponding to the plurality of embedding functions comprise a transcription modality, an image modality, a speech modality, and a geolocation modality, and

wherein the AR application is configured to augment real-world data with supplementary data from the remote data source comprises the AR application configured to:

express the real-world data as an embedding vector of a fixed dimension;

determine at least one embedding vector of a plurality of embedding vectors that is similar to the embedding vector of the real-world data using a nearest neighbor search executed over the plurality of embedding vectors, wherein each embedding vector of the plurality of embedding vectors expresses the supplementary data from the remote source and is of the fixed dimension; and

perform the fuzzy join operation of the embedding vector of the real-world data and the at least one embedding vector of the plurality of embedding vectors to augment the real-world data with supplementary data from the remote data source.

2. The system of claim 1 , wherein the AR application generating module is configured to receive a program written in the DSL, and generate the AR application based on the received program.

3. The system of claim 1 , wherein the AR application is defined in a stateful manner and wherein the fuzzy join operation is defined in a declarative manner.

4. The system of claim 1 , wherein the DSL is based on a grammar, wherein the grammar is defined with data sources and with transformations to join remotely retrieved data with the real-world data.

5. The system of claim 1 , wherein the real-world data is live data from a camera feed.

6. The system of claim 1 , wherein the DSL is configured to provide scripts that define a directed acyclic graph (DAG) of nodes that represent event-driven data flow elements.

7. The system of claim 1 , wherein the data store comprises at least one of a remote source, an embedding store, or a real-world context.

8. The system of claim 1 , wherein the fuzzy join operation is configured to perform a left-outer join between real-world context and remote data from the remote sources, which have both been transformed by the plurality of embedding functions.

9. A method for generating data-enriching augmented reality (AR) applications from a domain-specific language, the method comprising:

receiving a program written in a domain-specific language (DSL); and

generating an AR application based on the received program using the DSL and a fuzzy join operation, wherein the AR application is configured to augment real-world data with supplementary data from a remote data source, in an AR environment, wherein the real-world data comprises a plurality of modalities and the AR function comprises a plurality of embedding functions where each embedding function of the plurality of embedding functions corresponds to a different modality of the plurality of modalities, wherein the plurality of modalities corresponding to the plurality of embedding functions comprise a transcription modality, an image modality, a speech modality, and a geolocation modality,

wherein the AR application configured to augment real-world data with supplementary data from the remote data source comprises the AR application configured to:

express the real-world data as an embedding vector of a fixed dimension;

determine at least one embedding vector of a plurality of embedding vectors that is similar to the embedding vector of the real-world data using a nearest neighbor search executed over the plurality of embedding vectors, wherein each embedding vector of the plurality of embedding vectors expresses the supplementary data from the remote source and is of the fixed dimension; and

perform the fuzzy join operation of the embedding vector of the real-world data and the at least one embedding vector of the plurality of embedding vectors to augment the real-world data with supplementary data from the remote data source.

10. The method of claim 9 , further comprising defining the AR application defined in a stateful manner and wherein the fuzzy join operation is defined in a declarative manner.

11. The method of claim 9 , wherein the DSL is based on a grammar, wherein the grammar is defined with data sources and with transformations to join remotely retrieved data with real-world data.

12. The method of claim 9 , wherein the real-world data is live data from a camera feed.

13. The method of claim 9 , wherein the DSL is configured to provide scripts that define a directed acyclic graph (DAG) of nodes that represent event-driven data flow elements.

14. The method of claim 9 , wherein the fuzzy join operation is configured to perform a left-outer join between real-world context and remote data from remote sources, which have both been transformed by embedding functions.

15. The method of claim 9 , wherein the plurality of embedding functions comprise a food and recipes embedding function that aligns images of food in the real-world data with recipes from the supplementary data.