Automatic summarization of remotely-guided augmented reality sessions

1. A method comprising operating at least one hardware processor to:

receive a recording of a remotely-guided augmented reality (AR) session, the recording comprising:

images of a scene, captured by a camera of a local user,

position and orientation data of the camera,

annotations generated by a remote user at points-of-interest (POIs) in a three-dimensional (3D) representation of the scene; and

automatically generate an electronic document which comprises a summary of the remotely-guided AR session, by:

selecting prominent images out of the images of the scene, by:

(a) defining the images of the scene as a set of candidate images (G),

(b) adding, to an empty set (S), those of the images of G on which the annotations were indicated during the remotely-guided AR session,

(c) removing from G images whose pairwise viewed region distance from any image in S is below a predefined threshold, and

(d) adding to S additional images from G, using a non-maximal suppression algorithm in which images of G are sorted according to a focus score (F) and are rejected according to viewed region distance,

such that S includes the prominent images,

projecting the annotations to matching locations in some of the prominent images, based on the POIs of the annotations and on the position and orientation data of the camera, and

including, in the summary: the prominent images, including those of the prominent images having the projected annotations.

2. The method of claim 1 , wherein the pairwise viewed region distance is determined, for any pair of images I i and I j , by:

defining the 3D representation of the scene as a sparse point cloud {P k |k∈[1, 2, . . . n]} having n points;

defining π i,k as a projection of P k to I i , based on the position and orientation data of the camera associated with I i ; and

defining the pairwise viewed region distance between

I

i

⁢

⁢

and

⁢

⁢

I

j

⁢

⁢

as

⁢

⁢

D

i

,

j

=

Σ

k

⁢



π

i

,

k

-

π

j

,

k



2

∴

2

wherein i, j, a, k, and n are integers.

3. The method of claim 2 , wherein the non-maximal suppression algorithm comprises:

defining the focus score F of each image in G as an aggregate of overall point focus scores of all points P k whose projections are within the respective image:

F

i

=

Σ

k

⁢

F

i

,

k

,

wherein each of the point focus scores is determined by:

for each point P k , and for each image I i in G:

(a) if π i,k is outside boundaries of I i , the point focus score for point P k is set to zero,

(b) if π i,k is inside boundaries of I i :

(i) defining d i,k =∥π i,k −C i ∥ 2 2 as a distance of point P k from a center (C i ) of I i ,

(ii) defining

F

i

,

k

=

1

a

+

d

i

,

k

2

 as a point focus score of point P k in I i , and

(iii) defining

F

k

=

Σ

i

⁢

F

i

,

k

 as the overall point focus score of point P k across all of G.

4. The method of claim 3 , wherein the non-maximal suppression algorithm further comprises:

sorting the images in G in descending order according to their focus scores; and

iteratively, until G is empty:

(a) adding the first image in G to S, and

(b) removing from G any other image of G whose viewed region distance from the first image is below a predefined threshold,

such that, when G is empty, S includes the prominent images.

5. The method of claim 1 , wherein:

the recording of the remotely-guided AR session further comprises conversation text between the local user and the remote user;

the automatic generation of the electronic document further comprises temporally aligning the conversation text with the projected annotations; and

the inclusion in the summary is also of a dissemination of the conversation text amongst those of the prominent images having the projected annotations, wherein the dissemination is according to the temporal alignment.

6. A system comprising:

(i) at least one hardware processor; and

(ii) a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by said at least one hardware processor to:

receive a recording of a remotely-guided augmented reality (AR) session, the recording comprising:

images of a scene, captured by a camera of a local user,

position and orientation data of the camera,

annotations generated by a remote user at points-of-interest (POIs) in a three-dimensional (3D) representation of the scene; and

automatically generate an electronic document which comprises a summary of the remotely-guided AR session, by:

selecting prominent images out of the images of the scene, by:

(a) defining the images of the scene as a set of candidate images (G),

(b) adding, to an empty set (S), those of the images of G on which the annotations were indicated during the remotely-guided AR session,

(c) removing from G images whose pairwise viewed region distance from any image in S is below a predefined threshold, and

(d) adding to S additional images from G, using a non-maximal suppression algorithm in which images of G are sorted according to a focus score (F) and are rejected according to viewed region distance,

such that S includes the prominent images,

projecting the annotations to matching locations in some of the prominent images, based on the POIs of the annotations and on the position and orientation data of the camera, and

including, in the summary: the prominent images, including those of the prominent images having the projected annotations.

7. The system of claim 6 , wherein the pairwise viewed region distance is determined, for any pair of images I i and I j , by:

defining the 3D representation of the scene as a sparse point cloud {P k |k∈[1, 2, . . . n]} having n points;

defining π i,k as a projection of P k to I i , based on the position and orientation data of the camera associated with I i ; and

defining the pairwise viewed region distance between I i and I j as

D

i

,

j

=

Σ

k

⁢



π

i

,

k

-

π

j

,

k



2

∴

2

wherein i, j, a, k, and n are integers.

8. The system of claim 7 , wherein the non-maximal suppression algorithm comprises:

defining the focus score F of each image in G as an aggregate of overall point focus scores of all points P k whose projections are within the respective image:

F

i

=

Σ

k

⁢

F

i

,

k

,

wherein each of the point focus scores is determined by:

for each point P k , and for each image I i in G:

(a) if π i,k is outside boundaries of I i , the point focus score for point P k is set to zero,

(b) if π i,k is inside boundaries of I i :

(i) defining d i,k =∥π i,k −C i ∥ 2 2 as a distance of point P k from a center (C i ) of I i ,

(ii) defining

F

i

,

k

=

1

a

+

d

i

,

k

2

 as a point focus score of point P k in I i , and

(iii) defining

F

k

=

Σ

i

⁢

F

i

,

k

 as the overall point focus score of point P k across all of G.

9. The system of claim 8 , wherein the non-maximal suppression algorithm further comprises:

sorting the images in G in descending order according to their focus scores; and

iteratively, until G is empty:

(a) adding the first image in G to S, and

(b) removing from G any other image of G whose viewed region distance from the first image is below a predefined threshold,

such that, when G is empty, S includes the prominent images.

10. The system of claim 6 , wherein:

the recording of the remotely-guided AR session further comprises conversation text between the local user and the remote user;

the automatic generation of the electronic document further comprises temporally aligning the conversation text with the projected annotations; and

the inclusion in the summary is also of a dissemination of the conversation text amongst those of the prominent images having the projected annotations, wherein the dissemination is according to the temporal alignment.

11. A computer program product comprising a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by at least one hardware processor to:

receive a recording of a remotely-guided augmented reality (AR) session, the recording comprising:

images of a scene, captured by a camera of a local user,

position and orientation data of the camera,

annotations generated by a remote user at points-of-interest (POIs) in a three-dimensional (3D) representation of the scene; and

automatically generate an electronic document which comprises a summary of the remotely-guided AR session, by:

selecting prominent images out of the images of the scene, by:

(a) defining the images of the scene as a set of candidate images (G),

(b) adding, to an empty set (S), those of the images of G on which the annotations were indicated during the remotely-guided AR session,

(c) removing from G images whose pairwise viewed region distance from any image in S is below a predefined threshold, and

(d) adding to S additional images from G, using a non-maximal suppression algorithm in which images of G are sorted according to a focus score (F) and are rejected according to viewed region distance,

such that S includes the prominent images,

projecting the annotations to matching locations in some of the prominent images, based on the POIs of the annotations and on the position and orientation data of the camera, and

including, in the summary: the prominent images, including those of the prominent images having the projected annotations.

12. The computer program product of claim 11 , wherein the pairwise viewed region distance is determined, for any pair of images I i and I j , by:

defining the 3D representation of the scene as a sparse point cloud {P k |k∈[1, 2, . . . n]} having n points;

defining π i,k as a projection of P k to I i , based on the position and orientation data of the camera associated with I i ; and

defining the pairwise viewed region distance between I i and I j as

D

i

,

j

=

Σ

k

⁢



π

i

,

k

-

π

j

,

k



2

∴

2

wherein i, j, a, k, and n are integers.

13. The computer program product of claim 12 , wherein the non-maximal suppression algorithm comprises:

defining the focus score F of each image in G as an aggregate of overall point focus scores of all points P k whose projections are within the respective image:

F

i

=

Σ

k

⁢

F

i

,

k

,

wherein each of the point focus scores is determined by:

for each point P k , and for each image I i in G:

(a) if π i,k is outside boundaries of I i , the point focus score for point P k is set to zero,

(b) if I i,k is inside boundaries of I i :

(i) defining d i,k =∥π i,k −C i ∥ 2 2 as a distance of point P k from a center (C i ) of I i ,

(ii) defining

F

i

,

k

=

1

a

+

d

i

,

k

2

 as a point focus score of point P k in I i , and

(iii) defining

F

k

=

Σ

i

⁢

F

i

,

k

 as the overall point focus score of point P k across all of G.

14. The computer program product of claim 13 , wherein the non-maximal suppression algorithm further comprises:

sorting the images in G in descending order according to their focus scores; and

iteratively, until G is empty:

(a) adding the first image in G to S, and

(b) removing from G any other image of G whose viewed region distance from the first image is below a predefined threshold,

such that, when G is empty, S includes the prominent images.

15. The computer program product of claim 11 , wherein:

the recording of the remotely-guided AR session further comprises conversation text between the local user and the remote user;

the automatic generation of the electronic document further comprises temporally aligning the conversation text with the projected annotations; and

the inclusion in the summary is also of a dissemination of the conversation text amongst those of the prominent images having the projected annotations, wherein the dissemination is according to the temporal alignment.