Bar code detection and authentication for contactless transactions using wearable multimedia device
In an embodiment, a wearable multimedia device, comprises: a housing configured to attach to clothing of a user; a wide field-of-view camera embedded in the housing; one or more processors; memory storing instructions that when executed by the one or more processors, causes the one or more processors to perform operations comprising: capturing, using the camera, an image of a scene that includes a bar code displayed by a terminal device; determining an intent of the user to read the bar code; in accordance with the user's intent to read the barcode: cropping the barcode from the image of the scene; establishing, using the bar code, a communication channel with the terminal device; receiving, using the communication channel, transaction data from the terminal device; and responsive to the transaction data, sending, using the communication channel, authentication credentials to the terminal device.
1. An apparatus, comprising:
a wearable multimedia device worn by a user comprising:
a housing;
a wide field-of-view camera embedded in the housing;
a depth sensor;
one or more processors; and
a memory storing instructions that when executed by the one or more processors, causes the one or more processors to perform operations comprising:
capturing, using the camera, an image of a scene that includes a plurality of unlabeled objects and a user gesture;
capturing, using the depth sensor, depth data of the scene;
performing semantic segmentation on the image of the scene to predict an object mask for each unlabeled object;
performing instance segmentation on pixel data within the each object mask;
labeling, based on the instance segmentation, that one of the plurality of unlabeled objects is a contactless terminal device;
associating, with sensor fusion, an intent of the user to engage with the contactless terminal device based at least in part on the labeled contactless terminal device, the depth data, and the user gesture, where the user gesture is the user pointing in a direction of the contactless terminal device;
establishing a communication channel between the wearable multimedia device and the contactless terminal device;
receiving, using the communication channel, data from the contactless terminal device;
responsive to the received data, sending, using the communication channel, authentication credentials to the contactless terminal device;
receiving, using the communication channel, access from the contactless terminal device based at least in part on the authentication credentials; and
responsive to the received access to the contactless terminal device, performing an interaction with the contactless terminal device.
2. The apparatus of claim 1 , further comprising:
searching for a bar code in the image of the scene; and automatically cropping the bar code from the image of the scene; and
establishing, using the bar code, the communication channel between the wearable multimedia device and the terminal device.
3. The apparatus of claim 2 , wherein the bar code is a quick response (QR) code.
4. The apparatus of claim 2 , wherein the bar code is displayed on a monitor of the contactless terminal device.
5. The apparatus of claim 1 , wherein the image of the scene captured by the camera and the depth data of the scene captured by a depth sensor of the wearable multimedia device are used to determine that the user is gesturing.
6. The apparatus of claim 1 , wherein determining the intent of the user to engage with the contactless terminal device further comprises determining that the gesture is the user pointing their hand or finger in the direction of the contactless terminal device.
7. The apparatus of claim 1 , wherein the image of the scene captured by the camera and user speech captured by a microphone of the wearable multimedia device at the scene are used to determine the intent of the user to engage with the contactless terminal device.
8. The apparatus of claim 1 , wherein the image of the scene captured by the camera, user speech captured by a microphone of the wearable multimedia device, and depth data captured by the depth sensor of the wearable multimedia device at the scene are used to determine the intent of the user to engage with the contactless terminal device.
9. The apparatus of claim 1 , wherein at least one of location, speed or direction of the user captured by at least one sensor of the wearable multimedia device is used to determine the intent of the user to engage with the contactless terminal device.
10. The apparatus of claim 1 , wherein the image of the scene captured by the camera or depth data captured by a depth sensor of the wearable multimedia device is used to determine a type of the contactless terminal device.
11. The apparatus of claim 1 , wherein a past history of communications with the contactless terminal device is used to determine the intent of the user to engage with the contactless terminal device.
12. The apparatus of claim 1 , wherein the authentication credentials include biometric sensor data.
13. The apparatus of claim 1 , further comprising:
processing, using a machine learning model, the image of the scene to detect and label the contactless terminal device in the scene.
14. The apparatus of claim 1 , further comprising:
after authentication of the wearable multimedia device by the contactless terminal device using the authentication credentials, capturing a control gesture of the user; and
sending control data to the contactless terminal device for controlling the contactless terminal device based on the control gesture.
15. The apparatus of claim 14 , wherein the contactless terminal device is a media playback device and the control gesture is a swipe gesture for changing content played on the media playback device.
16. The apparatus of claim 15 , wherein a speed of the swipe gesture controls a speed of the content changing.
17. A method comprising:
capturing, using a wide field-of-view camera of a wearable multimedia device worn by a user, an image of a scene that includes a plurality of unlabeled objects and a user gesture, wherein the wearable multimedia device comprises a housing, the wide field-of-view camera embedded in the housing, a depth sensor, one or more processors, and a memory storing instructions that are executed by the one or more processors;
capturing, by the one or more processors using the depth sensor, depth data of the scene;
performing, by the one or more processors, semantic segmentation on the image of the scene to predict an object mask for each unlabeled object;
performing, by the one or more processors, instance segmentation on pixel data within the each object mask;
labeling, by the one or more processors based on the instance segmentation, that one of the plurality of unlabeled objects is a contactless terminal device;
associating, by the one or more processors with sensor fusion, an intent of the user to engage with the contactless terminal device based at least in part on the user gesture, the depth data and the labeled contactless terminal device, where the user gesture is the user pointing in a direction of the contactless terminal device;
establishing, by the one or more processors, a communication channel between the wearable multimedia device and the contactless terminal device;
receiving, by the one or more processors using the communication channel, data from the contactless terminal device;
responsive to the received data, sending, using the communication channel, authentication credentials to the contactless terminal device;
receiving, by the one or more processors using the communication channel, access from the contactless terminal device based at least in part on the authentication credentials; and
responsive to the received access to the contactless terminal device, performing, by the one or more processors, an interaction with the contactless terminal device.
18. The method of claim 17 , further comprising:
searching, by the one or more processors, for a bar code in the image of the scene;
automatically cropping, by the one or more processors, the bar code from the image of the scene; and
establishing, by the one or more processors using the bar code, the communication channel between the wearable multimedia device and the terminal device.
19. The method of claim 17 , wherein determining the intent of the user to engage with the contactless terminal device further comprises determining, by the one or more processors, that the gesture is the user pointing their hand or finger in the direction of the contactless terminal device.
20. The method of claim 17 , wherein the image of the scene captured the camera and user speech captured by a microphone of the wearable multimedia device at the scene are used to determine, by the one or more processors, the intent of the user to engage with the contactless terminal device.