IP Library Granted Patent US 12,555,332
Granted Patent B2
US 12,555,332 · App. 18/355,600 · Granted Feb 17, 2026

Augmented reality with non-line-of-sight perception

Inventors: Fadel Adib (Cambridge, MA); Tara Boroushaki (Cambridge, MA); Laura Noelle Dodds (Cambridge, MA); Aline Eid (Ann Arbor, MI); Maisy Lilian Lam (Cambridge, MA)
Assignee: Massachusetts Institute of Technology
G06T19/006G01S5/02G01S13/06G02B27/017G01S13/90
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,555,332
App. No.
18/355,600
Granted
Feb 17, 2026
Kind
B2
Abstract

According to one aspect of the disclosure, a mobile augmented reality (AR) system can include: a receiver configured to receive radio frequency (RF) signals from one or more items located within an environment; a tracking module configured to generate tracking data responsive to a location of the system within the environment over time; a display device; and one or more processors configured to determine a location of at least one of the one or more items within the environment using the received RF signals and the tracking data, and generate a visual representation of the location of the at least one item for display on the display device.

Claims (47)

1 . A mobile augmented reality (AR) system comprising:

a receiver configured to receive radio frequency (RF) signals from one or more items located within an environment;

a tracking module configured to generate tracking data responsive to a location of the system within the environment over time;

a display device; and

one or more processors configured to:

determine a location of at least one of the one or more items within the environment using the received RF signals and the tracking data, the determining of the location including:

performing reverse synthetic aperture radar (R-SAR) on the RF signals to track locations of a radio frequency identification (RFID) tag of the at least one of the one or more items as it is being picked up,

determining an identifier of the at least one of the one or more items based at least in part on the RFID tag locations, and

verifying the at least one of the one or more items is a correct item to be picked up based on comparing the determined item identifier to an expected item identifier; and

generate a visual representation of the location of the at least one of the one or more items for display on the display device.

2 . The mobile AR system of claim 1 wherein the display device is a visor of an AR headset and the receiver is integrated into the AR headset.

3 . The mobile AR system of claim 2 wherein the tracking module is external to the AR headset.

4 . The mobile AR system of claim 2 wherein the receiver comprises a conformal antenna located about the visor of the AR headset.

5 . The mobile AR system of claim 1 wherein the display device is a screen of a mobile handheld device further comprising the receiver, tracking module, and one or more processors.

6 . The mobile AR system of claim 1 wherein the receiver forms a part of a transceiver further configured to transmit RF signals within the environment, and wherein the received RF signals are in response to the transmitted RF signals.

7 . The mobile AR system of claim 6 wherein the received RF signals are received from one or more RFID tags collocated with respective ones of the one or more items.

8 . The mobile AR system of claim 6 wherein the one or more processors are configured to determine the location of the at least one of the one or more items using time of flight (ToF) between the transmitted RF signals and the received RF signals.

9 . The mobile AR system of claim 1 wherein the one or more processors are configured to determine the location of the at least one of the one or more items using synthetic aperture radar (SAR) with the tracking data used to obtain SAR antenna locations.

10 . The mobile AR system of claim 1 wherein the visual representation of the location of the at least one of the one or more items comprises at least one of:

an arrow indicating the location of the at least one of the one or more items; or

a holographic sphere or ellipse indicating the location of the at least one of the one or more items.

11 . A conformal antenna for use with an augmented reality (AR) headset, the conformal antenna comprising:

one or more conductive surfaces configured to conform about a front surface of the AR headset and to receive radio frequency (RF) signals from one or more items located within an environment,

wherein the AR headset is configured to:

perform reverse synthetic aperture radar (R-SAR) on the RF signals to track locations of a radio frequency identification (RFID) tag of an item being picked up from the one or more items located within an environment;

determine an identifier of the item being picked up based at least in part on the RFID tag locations; and

verify the item being picked up is a correct item to be picked up based on comparing the determined item identifier to an expected item identifier.

12 . The conformal antenna of claim 11 wherein the one or more conductive surfaces are configured transmit RF signals within the environment, and wherein the received RF signals are in response to the transmitted RF signals.

13 . The conformal antenna of claim 12 wherein the received RF signals corresponds to reflections of the transmitted RF signals off the one or more items.

14 . The conformal antenna of claim 11 wherein the AR headset includes one or more sensors positioned about the front surface, wherein the one or more conductive surfaces are configured to conform about the front surface without substantially masking the one or more sensors.

15 . The conformal antenna of claim 11 wherein the one or more conductive surfaces comprises one or more tapers and one or more slots.

16 . The conformal antenna of claim 11 wherein the received RF signals comprise at least one of:

ultra-high frequency (UHF) signals;

WiFi signals;

millimeter wave signals; or

terahertz (THz) signals.

17 . A method for verifying an item being picked up, the method comprising:

receiving radio frequency (RF) signals from a plurality of items within an environment;

receiving tracking data responsive to a location of an item being picked up over time;

performing reverse synthetic aperture radar (R-SAR) on the RF signals to track locations of a radio frequency identification (RFID) tag of the item being picked up;

determining an identifier of the item being picked up using the received RF signals and the tracking data and by comparing the tracking data to the RFID tag locations; and

verifying the item is a correct item to be picked up based on comparing the determined item identifier to an expected item identifier.

18 . The method of claim 17 further comprising generating user feedback in response to the verifying of the item.

19 . The method of claim 17 wherein the item is being picked up by one of:

a human hand;

a robot gripper; or

a grabber tool.

Assignments (2)
CONFIRMATORY LICENSE Recorded Jun 12, 2025
From: MASSACHUSETTS INSTITUTE OF TECHNOLOGY
To: NATIONAL SCIENCE FOUNDATION
Reel/Frame 071548/0325 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 16, 2023
From: ADIB, FADEL; BOROUSHAKI, TARA; DODDS, LAURA NOELLE; EID, ALINE; LAM, MAISY LILIAN
To: MASSACHUSETTS INSTITUTE OF TECHNOLOGY
Reel/Frame 064608/0305 →
Continuity (2)
Provisional Application 63408240 · Sep 20, 2022
Related Publication 20240096034A1 · Mar 21, 2024
References Cited (78)
US 9948413B1 · Zhou · 2018 [cited by examiner]
US 20090215536A1 · Yee · 2009 [cited by examiner]
US 20160005233A1 · Fraccaroli · 2016 [cited by examiner]
US 20190113609A1 · Baheti · 2019 [cited by examiner]
US 20200119780A1 · Klemmer · 2020 [cited by examiner]
US 20240037907A1 · Koch · 2024 [cited by examiner]
CN 112085134A · 2020 [cited by examiner]
WO WO2023183659A1 · 2023 [cited by examiner]
Author: Liang et al.; Title: E3DinSAR: 3-D Localization of RFID-Tagged Objects Based on Interference Synthetic Apertures; Source: IEEE Internet of Things Journal, vol. 7; Date: Dec. 12, 2020 (Year: 2020). [cited by examiner]
Boroushaki et al., “FuseBot: RF-Visual Mechanical Search”; Massachusetts Institute of Technology, Robotics: Science and Systems Jun. 2022-Jul. 2022; 12 pages. [cited by applicant]
Chatzistefanou et al., “Tag Localization by Handheld UHF RFID Reader with Optical and RFID Landmarks”; IEEE Journal of Radio Frequency Identification, Nov. 2022; 12 pages. [cited by applicant]
Doong, “A closed-form formula for GPS GDOP computation”; GPS Solut (2009), Nov. 2008; pp. 183-190 (8 pages). [cited by applicant]
Rashid et al., “Bringing Online Shopping Experience to Offline Retail through Augmented Reality and RFID”; ResearchGate, Conference Paper, Oct. 2015; 8 pages. [cited by applicant]
Kunkel et al., “A Concept for Infrastructure Independent Localization and Augmented Reality Visualization of RFID Tags”, IEEE MTT-S International Microwave Workshop on Wireless Sensing, Local Positioning and RFID (IMWS … [cited by applicant]
Yamashita et al., “Pedestrian Navigation System for Visually Impaired People Using HoloLens and RFID”; 2017 Conference on Technologies and Applications of Artificial Intelligence (TAAI), Dec. 2017; 6 pages. [cited by applicant]
Sain, “A new era for RFID in retail”; Accenture, 2021; 14 pages. [cited by applicant]
Adib et al., “See Through Walls with Wi-Fi!”; Massachusetts Institute of Technology, Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM, Aug. 2013; 12 pages. [cited by applicant]
Azuma, “A Survey of Augmented Reality”; Presence, vol. 6, No. 4, Aug. 1997; pp. 355-385 (31 pages). [cited by applicant]
Azzouzi et al., “New Measurement Results for the Localization of UHF RFID Transponders Using an Angle of Arrival (AoA) Approach”; 2011 IEEE Conference on RFID, Apr. 2011; pp. 91-97 (7 pages). [cited by applicant]
Xie et al., “TaggedAR: An RFID-Based Approach for Recognition of Multiple Tagged Objects in Augmented Reality Systems”; IEEE Transactions on Mobile Computing, vol. 18, No. 5, May 2019; pp. 1188-1202 (15 pages). [cited by applicant]
Xie et al., “Tell Me What I See: Recognize RFID Tagged Objects in Augmented Reality Systems”; UBICOMP '16, Sep. 2016; pp. 916-927 (12 pages). [cited by applicant]
Boroushaki et al., “FuseBot: Mechanical Search of Rigid and Deformable Objects via Multi-Modal Perception”; Massachusetts Institute of Technology, Research Square, Feb. 2023; 29 pages. [cited by applicant]
Boroushaki et al., “Robotic Grasping of Fully-Occluded Objects RF Perception”; Massachusetts Institute of Technology, 2021 IEEE International Conference on Robotics and Automation, May 30-Jun. 5, 2021; 7 pages. [cited by applicant]
Boroushaki et al., “RFusion: Robotic Grasping via RF-Visual Sensing and Learning”; Massachusetts Institute of Technology, Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, Nov. 2021; 14 pages. [cited by applicant]
Cai et al., “Study of Human Head Effects on UWB Antenna”; IEEE International Workshop on Antenna Technology (IWAT 2005), Mar. 2005; pp. 310-313 (4 pages). [cited by applicant]
Chawla et al., “Real-Time RFID Localization Using RSS”; 2014 International Conference on Localization and GNSS, Jun. 2013; 6 pages. [cited by applicant]
Cheng et al., “In-Air Gesture Interaction: Real Time Hand Posture Recognition Using Passive RFID Tags”; Special Section on Artificial Intelligence in Cyberseurity, Jul. 2019; 13 pages. [cited by applicant]
Cihangir et al., “Dual-Band 4G Eyewear Antenna and SAR Implications”; IEEE Transactions on Antennas and Propagation, vol. 65, No. 4, Apr. 2017; 5 pages. [cited by applicant]
Cihangir et al., “Feasibility Study of 4G Cellular Antennas for Eyewear Communicating Devices”; IEEE Antennas and Wireless Propagation Letters, vol. 12, 2013; 4 pages. [cited by applicant]
Crespo-Bardera et al., “Design and Analysis of Conformal Antenna for Future Public Safety Communications”; IEEE Antennas & Propagation Magazine, Jul. 2020; pp. 94-102 (9 pages). [cited by applicant]
Gareis et al., “A MIMO UHF-RFID SAR 3D Locating System for Autonomous Inventory Robots”; 2020 IEEE MTT-S International Conference on Microwaves for Intelligent Mobility (ICMIM), Nov. 2020; 4 pages. [cited by applicant]
Haydar et al., “Design and Analysis of Conformal Antennas for Smart Glasses”; 12th European Conference on Antennas and Propagation (EuCAP 2018), Apr. 2018; 5 pages. [cited by applicant]
Jidhage et al., “Hooked Loop Antenna Concept for Bluetooth Headset Applications”; IEEE Antennas and Propagation Society Symposium, Jun. 2004; pp. 3521-3524 (4 pages). [cited by applicant]
Kronberger et al., “UHF RFID Localization System Based on a Phased Array Antenna”; 2011 IEEE International Symposium on Antennas and Propagation (APSURSI), Jul. 2011; 4 pages. [cited by applicant]
Liu et al., “Technical Evaluation of HoloLens for Multimedia: A First Look”; IEEE MultiMedia, vol. 25, Issue 4, Oct.-Dec. 2018; pp. 8-18 (11 pages). [cited by applicant]
Luo et al., “3D Backscatter Localization for Fine-Grained Robotics”; Proceedings of the 16 [cited by applicant]
Ma et al., “Minding the Billions: Ultra-wideband Localization for Deployed RFID Tags”; Proceedings of the 23 [cited by applicant]
Mahmood et al., “Recent Advances in Wearable Antenna Technologies: A Review”; Progress in Electromagnetics Research B, vol. 89, 1-27, 2020; 27 pages. [cited by applicant]
Miesen et al., “UHF RFID Localization Based on Synthetic Apertures”; IEEE Transactions on Automation Science and Engineering, vol. 10, No. 3, Jul. 2013; pp. 807-815 (9 pages). [cited by applicant]
Motroni et al., “A Multi-Antenna SAR-based method for UHF RFID Tag Localization via UGV”; 2018 IEEE International Conference on RFID Technology & Application (RFID-TA), Sep. 2018; 6 pages. [cited by applicant]
Ni et al., “LANDMARC: Indoor Location Sensing Using Active RFID”; Wireless Networks 10, 2004; pp. 701-710 (10 pages). [cited by applicant]
Plicanic et al., “Actual Diversity Performance of a Multiband Diversity Antenna With Hand and Head Effects”; IEEE Transactions on Antennas and Propagation, vol. 57, No. 5, May 2009; pp. 1547-1556 (10 pages). [cited by applicant]
Shangguan et al., “The Design and Implementation of a Mobile RFID Tag Sorting Robot”; Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services, 2016; pp. 31-42 (12 pages). [cited by applicant]
Wang et al., “RF-Compass: Robot Object Manipulation Using RFIDs”; Proceedings of the 19th Annual International Conference on Mobile Computing & Networking, 2013; pp. 3-14 (12 pages). [cited by applicant]
Wang et al., “Dude, Where's My Card? RFID Positioning That Works with Multipath and Non-Line of Sight”; Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM, Aug. 2013; pp. 51-62 (12 pages). [cited by applicant]
Wang et al., “RF-IDraw: Virtual Touch Screen in the Air Using RF Signals”; ACM SIGCOMM Computer Communication Review, vol. 44, Issue 4, Oct. 2014; pp. 235-246 (12 pages). [cited by applicant]
Wang et al., “Sub-6GHZ 4G/5G Conformal Glasses Antennas”; IEEE Access, vol. 7, Dec. 2019; pp. 182027-182036 (10 pages). [cited by applicant]
Wei et al., “Design of a Wideband Horizontally Polarized Omnidirectional Printed Loop Antenna”; IEEE Antennas and Wireless Propagation Letters, vol. 11, 2012; pp. 49-52 (4 pages). [cited by applicant]
Yang et al., “Tagoram: Real-Time Tracking of Mobile RFID Tags to High Precision Using COTS Devices”; Proceedings of the 20th Annual International Conference on Mobile Computing and Networking, Sep. 2014; pp. 237-248 (12… [cited by applicant]
Yazdandoost et al., “Ultra Wideband L-Loop Antenna”; 2005 IEEE International Conference on Ultra-Wideband, Sep. 2005; pp. 201-205 (5 pages). [cited by applicant]
Yeo et al., “Miniaturized Wideband Loop Antenna Using a Multiple Half-Circular-Ring-Based Loop Structure and Horizontal Slits for Terrestrial DTV and UDH TV Applications”; Sensors 2021, 21, 2916, Apr. 2021; 14 pages. [cited by applicant]
Zhang et al., “RFID-assisted assembly guidance system in an augmented reality environment”; International Journal of Production Research, Vo. 49, No. 13, Jul. 2011; pp. 3919-3938 (21 pages). [cited by applicant]
Zhao et al., “PEC: Synthetic Aperture RFID Localization with Aperture Position Error Compensation”; 16 [cited by applicant]
Zhou et al., “Two-dimension Localization of Passive RFID Tags Using AOA Estimation”; 2011 IEEE International Instrumentation and Measurement Technology Conference, May 2011; 5 pages. [cited by applicant]
“ALN-9640 Squiggle Inlay”, Alien Technology Corporation, Feb. 2014; 4 pages. [cited by applicant]
“About HoloLens 2”, article dated Mar. 12, 2023; Microsoft; [retrieved on Jul. 13, 2023]. Retrieved from the Internet <URL: https://learn.microsoft.com/en-us/hololens/hololens2-hardware>; 10 pages. [cited by applicant]
“Hand tracking—MRTK2”, article dated Aug. 1, 2022; Microsoft; [retrieved on Jul. 14, 2023]. Retrieved from the Internet <URL: https://learn.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk2/features/input/hand-… [cited by applicant]
“Camera setup in Unity”, article dated Mar. 2, 2023; Microsoft; [retrieved on Jul. 14, 2023]. Retrieved from the Internet <URL: https://learn.microsoft.com/en-us/windows/mixed-reality/develop/unity/camera-in-unity?tabs=… [cited by applicant]
Microsoft webpage, “Microsoft Visual Studio”; Microsoft; [retrieved on Jul. 14, 2023]. Retrieved from the Internet <URL: https://visualstudio.microsoft.com/>; 11 pages. [cited by applicant]
Nuand webpage, “bladeRF 2.0 micro”; Nuand Microchip Specifications, Nuand LLC; [retrieved on Jul. 14, 2023]. Retrieved from the Internet <URL: https://www.nuand.com/bladerf-2-0-micro/>; 10 pages. [cited by applicant]
TeamViewer webpage, “TeamViewer Frontline”; TeamViewer Frontline Specifications, TeamViewer Germany GmbH; [retrieved on Jul. 14, 2023]. Retrieved from the Internet <URL: https://www.teamviewer.com/en-us/products/frontli… [cited by applicant]
Microsoft webpage, “HoloLens 2”, HoloLens2 Specifications, Microsoft; [retrieved on Jul. 13, 2023]. Retrieved from the Internet <URL: https://www.microsoft.com/en-us/hololens/hardware#document-experiences>; 8 pages. [cited by applicant]
Culliford et al., “Facebook invests billions in metaverse efforts as ad business slows”, Reuters, Oct. 2021; 4 pages. [cited by applicant]
T. Dengel, “Disruption Denied: How Next-Generation Logistics Create A Resilient Supply Chain”, Forbes, SAP Brandvoice, Jun. 2022; 5 pages. [cited by applicant]
Fillmore et al., “AR and VR in the Workplace”, IBM, IBM Institute for Business Value, Sep. 2020; 12 pages. [cited by applicant]
S. Galer, “Virtual Reality Emerges As Powerful Employee Training Tool”, Forbes, SAP Brandvoice, May 2022; 5 pages. [cited by applicant]
L. Gates, “Augmented Reality (AR) to Improve Employee Performance”, Insight, Aug. 2018; 6 pages. [cited by applicant]
“EPC Radio-Frequency Identity Protocols Generation-2 UHF RFID”, EPCglobal Gen2 Specification, Version 2.0.1, Apr. 2015; 152 pages. [cited by applicant]
Hao et al., “Antennas and Propagation for Body Centric Wireless Communication”, IEEE/ACES International Conference on Wireless Communications and Applied Computational Electromagnetics, 2005; 4 pages. [cited by applicant]
Kumar et al., “Accurate indoor localization with zero start-up cost”, Proceedings of the 20th Annual International Conference on Mobile Computing and Networking, Sep. 2014; pp. 483-494 (12 pages). [cited by applicant]
Lin et al., “Beyond one-dollar mouse: A battery-free device for 3D human-computer interaction via RFID tags”, 2015 IEEE Conference on Computer Communications (INFOCOM), Apr.-May 2015; pp. 1661-1669 (9 pages). [cited by applicant]
A. Sag, “Why Microsoft Won The $22 Billion Army Hololens 2 AR Deal”, Forbes, Apr. 2021; 6 pages. [cited by applicant]
D. Kaye, “Apple teases metaverse AR plans, stock jumps”, Reuters, Jan. 2022; 2 pages. [cited by applicant]
Smith et al., “Efficient 3-D Near-Field MIMO-SAR Imaging for Irregular Scanning Geometries”, IEEE Access, vol. 10, Jan. 2022; pp. 10283-10294 (12 pages). [cited by applicant]
R. Tedrake, “Robotic Manipulation—Perception, Planning, and Control”, Course Notes for MIT 6.4210, 2022; 162 pages. [cited by applicant]
Alvarez et al., “System Based on Compact mmWave Radar and Natural Body Movement for Assisting Visually Impaired People”, IEEE Access, vol. 9, Sep. 2021; pp. 125042-125051 (10 pages). [cited by applicant]
Alvarez-Narciandi et al., “Towards Turning Smartphones Into mmWave Scanners”, IEEE Access, vol. 9, Mar. 2021; pp. 45147-45154 (8 pages). [cited by applicant]
Alvarez-Narciandi et al., “Freehand, Agile, and High-Resolution Imaging With Compact mm-Wave Radar”, IEEE Access, vol. 7, Jul. 2019; pp. 95516-95526 (11 pages). [cited by applicant]