Granted Patent B2

US 12,277,190 · App. 17/244,558 · Granted Apr 15, 2025

Inventors: Karan Walia (Brampton, CA); Anton Mamonov (Toronto, CA); Sobi Walia (Brampton, CA)

G06F16/9577G06F16/2237G06F16/289G06F16/951G06F16/986G06F18/22

Abstract

Claims (83)

1. A computer-implemented method of selecting a new web element among a plurality, “n,” of new web elements in a new web page on which a task is to be automatically performed, the selected new web element related to a known web element, where interaction with the known web element has been previously recorded in a first document, the method comprising:

generating a second document based on the first document, the second document including a reference to the known web element;

storing a first plurality of vectors for the known web element;

storing n second pluralities of vectors, one second plurality of vectors for each new web element among the n new web elements, each second plurality of vectors having a plurality, “m,” of vectors;

wherein each vector among the m vectors in each second plurality of vectors among the second pluralities of vectors has a corresponding vector in the first plurality of vectors;

for each second plurality of vectors of the n second pluralities of vectors, generating a similarity score between:

each vector in the first plurality of vectors; and

the corresponding vector in the each second plurality of vectors;

selecting the new web element having the second plurality of vectors with the highest similarity score, thereby identifying the selected new web element that is most related to the known web element; and

updating the second document to replace the reference to the known web element with a reference to the selected new web element.

2. The method of claim 1 , wherein a vector in the first plurality of vectors is representative of a property of the known web element in an object model.

3. The method of claim 2 , wherein the property comprises at least one of:

a tag for the known web element,

a class for the known web element,

a position for the known web element,

a height for the known web element,

a width for the known web element, and

a text used within known web element.

4. The method of claim 2 , wherein the object model comprises a Document Object Model (DOM).

5. The method of claim 1 , wherein a vector in the second plurality of vectors is representative of a property of the new web element in an object model.

6. The method of claim 5 , wherein the property comprises at least one of:

a tag for the new web element,

a class for the new web element,

a position for the new web element,

a height for the new web element,

a width for the new web element, and

a text used within new web element.

7. The method of claim 5 , wherein the object model comprises a Document Object Model (DOM).

8. The method of claim 1 , wherein the generating the similarity score comprises determining an aggregated cosine distance.

9. A computer-implemented method of selecting a new web element among a plurality, “n,” of new web elements in a new web page on which a task is to be automatically performed, the selected new web element related to a known web element, where interaction with the known web element has been previously recorded in a first document, the method comprising:

generating a second document based on the first document, the second document including a reference to the known web element;

receiving a position and dimensions for the known web element;

receiving a position and dimensions for each new web element among the n new web elements;

for each new web element among the n new web elements, generating a similarity score between:

the position and dimensions of the known web element; and

the position and dimensions of the each n new web element;

selecting the new web element with the highest similarity score, thereby identifying the selected new web element most related to the known web element; and

updating the second document to replace the reference to the known web element with a reference to the selected new web element.

10. The method of claim 9 , wherein the generating the similarity score comprises using an intersection-over-union analysis.

11. The method of claim 9 , wherein the plurality n of new web elements is a candidate set of web elements.

12. The method of claim 11 , comprising determining the candidate set of web elements.

13. The method of claim 12 , wherein the determining comprises:

storing a first plurality of vectors for the known web element;

storing n second pluralities of vectors, one second plurality of vectors for each new web element among the n new web elements, each second plurality of vectors having a plurality, “m,” of vectors;

wherein each vector among the m vectors in each second plurality of vectors among the second pluralities of vectors has a corresponding vector in the first plurality of vectors;

for each second plurality of vectors of the n second pluralities of vectors, generating a vector similarity score between:

each vector in the first plurality of vectors; and

the corresponding vector in the each second plurality of vectors; and

selecting the candidate plurality of web elements being the second pluralities of vectors with the vector similarity score above a threshold.

14. The method of claim 13 , wherein the generating the vector similarity score comprises determining an aggregate cosine distance.

15. The method of claim 13 , wherein a vector in the first plurality of vectors is representative of a property of the known web element in an object model.

16. The method of claim 15 , wherein the property of the known web element comprises at least one of:

a tag for the known web element,

a class for the known web element,

a position for the known web element,

a height for the known web element,

a width for the known web element, and

a text used within the known web element.

17. The method of claim 15 , wherein the object model comprises a Document Object Model (DOM).

18. The method of claim 13 , wherein a vector in the second plurality of vectors is representative of a property of the new web element in an object model.

19. The method of claim 18 , wherein the property of the new web element comprises at least one of:

a tag for the new web element,

a class for the new web element,

a position for the new web element,

a height for the new web element,

a width for the new web element, and

a text used within the new web element.

20. The method of claim 18 , wherein the object model comprises a Document Object Model (DOM).

21. A computer-implemented method of selecting a new web element among a plurality, “n,” of new web elements in a new web page on which a task is to be automatically performed, the selected new web element related to a known web element, where interaction with the known web element has been previously recorded in a first document, the method comprising:

generating a second document based on the first document, the second document including a reference to the known web element;

storing a first plurality of vectors for the known web element;

storing n second pluralities of vectors, one second plurality of vectors for each new web element among the n new web elements, each second plurality of vectors having a plurality, “m,” of vectors;

wherein each vector among the m vectors in each second plurality of vectors among the second pluralities of vectors has a corresponding vector in the first plurality of vectors;

for each second plurality of vectors of the n second pluralities of vectors, generating a similarity score between:

each vector in the first plurality of vectors; and

the corresponding vector in the each second plurality of vectors;

determining that no second plurality of vectors have a similarity score above a threshold;

responsive to the determining that no second plurality of vectors have a similarity score above a threshold:

establishing a virtual network computing (VNC) connection to an electronic device;

transmitting a visual representation of the new web page;

receiving an indication of a pixel location of a particular web element in the visual representation; and

selecting, as the new web element, the particular web element at the pixel location; and

updating the second document to replace the reference to the known web element with a reference to the new web element.

Assignments (2)

Continuity (1)

References Cited (151)