Extensible architecture with multimodal feature fusion for document classification

1 . A system, comprising:

a non-transitory memory; and

one or more hardware processors coupled with the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations comprising:

extracting, from a first document, a first set of features corresponding to a first modality and a second set of features corresponding to a second modality;

encoding, using a first encoder, the first set of features into a first embedding;

encoding, using a second encoder, the second set of features into a second embedding;

generating a first fused embedding, wherein the generating the first fused embedding comprises (i) converting the first embedding from a first latent space to a normalized latent space, (ii) converting the second embedding from a second latent space to the normalized latent space, and (iii) merging the converted first embedding and the converted second embedding; and

determining, using a machine learning model, a similarity score representing a similarity between the first document and a second document based on the first fused embedding.

2 . The system of claim 1 , wherein the encoding the first set of features comprises:

encoding, using the first encoder, the first set of features into a first plurality of embeddings, wherein each embedding in the first plurality of embeddings corresponds to a distinct element within the first document; and

generating the first embedding based on aggregating the first plurality of embeddings.

3 . The system of claim 1 , wherein the operations further comprise:

extracting, from the first document, a third set of features corresponding to a third modality;

encoding, using a third encoder, the third set of features into a third embedding; and

combining the third embedding with the first fused embedding.

4 . The system of claim 1 , wherein the merging the converted first embedding and the converted second embedding comprises:

performing at least one of a summation or an average operation to

the converted first embedding and the converted second embedding.

5 . The system of claim 1 , wherein the converting the first embedding comprises padding one or more additional values to the first embedding.

6 . The system of claim 1 , wherein the operations further comprise:

determining the normalized latent space based on the first latent space and the second latent space.

7 . The system of claim 1 , wherein the operations further comprise:

generating a second fused embedding based on features extracted from the second document, wherein the determining the similarity score is further based on the second fused embedding.

8 . A method, comprising:

extracting, from a first image of a first document, first features corresponding to a first set of modalities;

encoding a first portion of the first features into a first embedding;

encoding a second portion of the first features into a second embedding;

generating a first fused embedding based on fusing the first embedding and the second embedding;

determining a first similarity score representing a first similarity between the first document and a second document based on the first fused embedding and a second fused embedding generated for a second image of the second document;

extracting, from the first image of the first document, second features corresponding to a second set of modalities;

encoding the second features into a third embedding;

determining a second similarity score representing a second similarity between the first document and the second document based on the third embedding; and

classifying the first document based on a combined score generated from the first similarity score and the second similarity score.

9 . The method of claim 8 , further comprising:

generating a comparison embedding based on performing a function on the first fused embedding and the second fused embedding, wherein the first similarity score is determined further based on the comparison embedding.

10 . The method of claim 9 , wherein the function comprises a subtraction function.

11 . The method of claim 9 , wherein the function comprises a dot product function.

12 . The method of claim 8 , wherein the first embedding corresponds to a first latent space, wherein the second embedding corresponds to a second latent space, and wherein the method further comprises:

converting the first embedding from the first latent space to a normalized latent space; and

converting the second embedding from the second latent space to the normalized latent space, wherein the generating the first fused embedding is further based on the converted first embedding and the converted second embedding.

13 . The method of claim 12 , further comprising:

determining the normalized latent space based on the first latent space and the second latent space.

14 . The method of claim 8 , wherein the determining the first similarity score is further based on the third embedding.

15 . A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations comprising:

obtaining, via a user interface of a device, a first image of a first document;

extracting, from the first image, first features corresponding to a plurality of modalities;

encoding a first portion of the first features into a first embedding;

encoding a second portion of the first features into a second embedding;

generating a first fused embedding based on fusing the first embedding and the second embedding;

extracting, from the first image, second features corresponding to a modality that is not included in the plurality of modalities;

encoding the second features into a third embedding;

determining, using a machine learning model, a similarity score representing a similarity between the first document and a second document based on the first fused embedding, the third embedding, and a second fused embedding generated for the second document; and

determining a classification for the first document based on the similarity score.

16 . The non-transitory machine-readable medium of claim 15 , wherein the first image is obtained based on a request for a first type of documents, and wherein the operations further comprise:

prompting, via the user interface, a user of the device for a second image when the classification indicates that the first document corresponds to a second type of documents different from the first type of documents.

17 . The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise:

restricting an account associated with the device access to one or more resources when the classification indicates that the first document is a match with the second document.

18 . The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise:

generating the second fused embedding based on features extracted from the second document.

19 . The non-transitory machine-readable medium of claim 15 , wherein the first fused embedding, the second fused embedding, and the third embedding are provided to the machine learning model as input values.

20 . The non-transitory machine-readable medium of claim 15 , wherein the machine learning model is a first machine learning model, wherein the similarity score is a first similarity score, and wherein the operations further comprising:

determining, using a second machine learning model, a second similarity score representing a second similarity between the first document and the second document based on the third embedding; and

generating a combined score based on the first similarity score and the second similarity score, wherein the determining the classification is further based on the combined score.