IP Library › Granted Patent US 12,657,933
Granted Patent B2
US 12,657,933 · App. 17/662,165 · Granted Jun 16, 2026

Method, apparatus, and computer program product for map data generation from probe data imagery

Inventors: Fei Tang (Aarau, CH); Ole Henry Dorum (Chicago, IL); Soojung Hong (Zurich, CH); Arash Ostadzadeh (Eindhoven, NL)
Assignee: HERE GLOBAL B.V.
G06V20/588G06V10/82G06V20/58
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,657,933
App. No.
17/662,165
Granted
Jun 16, 2026
Kind
B2
Abstract

A method is provided to using a generative adversarial network to generate map geometry from images representing probe data. Methods may include: receiving a rasterized image representative of probe data within a geographic area, where each pixel of the rasterized image includes a property representing at least one component of the probe data; generating a prediction image of road features within the geographic area using a generative adversarial network based on trained model parameters and the rasterized image; determining one or more map elements based, at least in part, on the prediction image and georeferenced locations of the road features within the prediction image; and updating a map of the geographic area with one or more map elements.

Claims (33)

1 . An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and computer program code configured to, with the processor, cause the apparatus to at least:

receive a rasterized image representative of probe data within a geographic area, wherein each pixel of the rasterized image is encoded with at least one property representing at least one component of the probe data associated with map geometry, wherein the at least one component comprises one or more of a travel speed, an average heading angle, a lane marking observation, or a probe data point count;

deblur the rasterized image to obtain a prediction image of road features within the geographic area using a generative adversarial network, wherein the generative adversarial network is trained using direct image translation techniques applied to pairs of training images comprising probe data density histogram images and corresponding ground truth label images;

process the prediction image using a deep neural network trained on ground truth labeled map elements to determine one or more map elements based, at least in part, on the prediction image and georeferenced locations of the road features within the prediction image; and

update a map of the geographic area with the one or more map elements.

2 . The apparatus of claim 1 , wherein the at least one property representing at least one component of the probe data comprises a pixel value, and wherein the at least one component of the probe data comprises a probe density represented by the pixel value.

3 . The apparatus of claim 1 , wherein the at least one property representing at least one component of the probe data comprises a pixel value, and wherein the at least one component of the probe data comprises an average probe speed represented by the pixel value.

4 . The apparatus of claim 1 , wherein the at least one property representing at least one component of the probe data comprises a pixel value, and wherein the at least one component of the probe data comprises a predominant probe heading represented by the pixel value.

5 . The apparatus of claim 1 , wherein causing the apparatus to receive the rasterized image representative of probe data within the geographic area comprises causing the apparatus to receive a rasterized image of normalized probe data within the geographic area, wherein the normalized probe data is normalized based, at least in part, on at least one of a functional class of a road segment along which the probe data was captured or a total volume of received probe data along the road segment along which the received probe data was captured.

6 . The apparatus of claim 1 , wherein the generative adversarial network is a conditional generative adversarial network, wherein each pixel of the probe data density histogram images represents normalized probe count for a location corresponding to a respective pixel.

7 . The apparatus of claim 6 , wherein the ground truth label images comprise images depicting labeled map elements.

8 . The apparatus of claim 1 , wherein the probe data comprises at least one of a vehicle location or a location of an object detected by a vehicle.

9 . A method comprising:

receiving a rasterized image representative of probe data within a geographic area, wherein each pixel of the rasterized image is encoded with at least one property representing at least one component of the probe data associated with map geometry, wherein the at least one component comprises one or more of a travel speed, an average heading angle, a lane marking observation, or a probe data point count;

deblurring the rasterized image to obtain a prediction image of road features within the geographic area using a generative adversarial network, wherein the generative adversarial network is trained using direct image translation techniques applied to pairs of training images comprising probe data density histogram images and corresponding ground truth label images;

processing the prediction image using a deep neural network trained on ground truth labeled map elements to determine one or more map elements based, at least in part, on the prediction image and georeferenced locations of the road features within the prediction image; and

updating a map of the geographic area with the one or more map elements.

10 . The method of claim 9 , wherein the at least one property representing at least one component of the probe data comprises a pixel value, and wherein the at least one component of the probe data comprises a probe density represented by the pixel value.

11 . The method of claim 9 , wherein the at least one property representing at least one component of the probe data comprises a pixel value, and wherein the at least one component of the probe data comprises an average probe speed represented by the pixel value.

12 . The method of claim 10 , wherein the probe data comprises at least one of a vehicle location or a location of an object detected by a vehicle.

13 . The method of claim 9 , wherein receiving the rasterized image representative of probe data within the geographic area comprises receiving a rasterized image of normalized probe data within the geographic area, wherein the normalized probe data is normalized based, at least in part, on at least one of a functional class of a road segment along which the probe data was captured or a total volume of received probe data along the road segment along which the received probe data was captured.

14 . The method of claim 9 , wherein the generative adversarial network is a conditional generative adversarial network, wherein each pixel of the probe data density histogram images represents normalized probe count for a location corresponding to a respective pixel.

15 . The method of claim 14 , wherein the ground truth label images comprise images depicting labeled map elements.

16 . A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising program code instructions to:

receive a rasterized image representative of probe data within a geographic area, wherein each pixel of the rasterized image is encoded with at least one property representing at least one component of the probe data associated with map geometry, wherein the at least one component comprises one or more of a travel speed, an average heading angle, a lane marking observation, or a probe data point count;

deblur the rasterized image to obtain a prediction image of road features within the geographic area using a generative adversarial network, wherein the generative adversarial network is trained using direct image translation techniques applied to pairs of training images comprising probe data density histogram images and corresponding ground truth label images;

process the prediction image using a deep neural network trained on ground truth labeled map elements to determine one or more map elements based, at least in part, on the prediction image and georeferenced locations of the road features within the prediction image; and

update a map of the geographic area with the one or more map elements.

17 . The computer program product of claim 16 , wherein the program code instructions to receive the rasterized image of probe data within the geographic area comprise program code instructions to receive a rasterized image of normalized probe data within the geographic area, wherein the normalized probe data is normalized based, at least in part, on at least one of a functional class of a road segment along which the probe data was captured or a total volume of received probe data along the road segment along which the received probe data was captured.

18 . The computer program product of claim 16 , wherein the generative adversarial network is a conditional generative adversarial network, wherein each pixel of the probe data density histogram images represents normalized probe count for a location corresponding to a respective pixel.

19 . The apparatus of claim 1 , wherein each pixel of the rasterized image is encoded with at least one property representing the probe data point count associated with a respective pixel and at least one additional property representing a travel speed, an average heading angle, or a lane marking observation associated with the respective pixel.

20 . The apparatus of claim 19 , wherein the least one property representing the probe data point count associated with the respective pixel is encoded in a first channel, wherein the at least one additional property is encoded in a second channel.

21 . The apparatus of claim 20 , wherein the first channel comprises an intensity value, and wherein the second channel comprises a color value.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 5, 2022
From: TANG, FEI; DORUM, OLE HENRY; HONG, SOOJUNG; OSTADZADEH, ARASH
To: HERE GLOBAL B.V.
Reel/Frame 059831/0534 →
Continuity (1)
Related Publication 20230360407A1 · Nov 9, 2023
References Cited (123)
US 8949021B2 · Witmer · 2015 [cited by applicant]
US 9171485B2 · Gautama et al. · 2015 [cited by applicant]
US 9177404B2 · Ramachandran et al. · 2015 [cited by applicant]
US 9658074B2 · Dorum · 2017 [cited by applicant]
US 10266280B2 · Derenick et al. · 2019 [cited by applicant]
US 10444020B2 · Dorum · 2019 [cited by applicant]
US 10546400B2 · Dorum · 2020 [cited by applicant]
US 10580292B2 · Dorum · 2020 [cited by applicant]
US 10760920B2 · Sekiyama · 2020 [cited by applicant]
US 11030476B2 · Xu et al. · 2021 [cited by applicant]
US 11068515B2 · Bukowski · 2021 [cited by applicant]
US 11093760B2 · Savla et al. · 2021 [cited by applicant]
US 11192558B2 · Thompson · 2021 [cited by applicant]
US 11209548B2 · Yang et al. · 2021 [cited by applicant]
US 11210537B2 · Koivisto et al. · 2021 [cited by applicant]
US 11227500B2 · Wang · 2022 [cited by applicant]
US 11244500B2 · Marschner et al. · 2022 [cited by applicant]
US 20030009287A1 · Howard et al. · 2003 [cited by applicant]
US 20090210388A1 · Elson et al. · 2009 [cited by applicant]
US 20130033591A1 · Takahashi et al. · 2013 [cited by applicant]
US 20140219558A1 · Teng et al. · 2014 [cited by applicant]
US 20160239983A1 · Dorum et al. · 2016 [cited by applicant]
US 20160358477A1 · Ansari · 2016 [cited by examiner]
US 20170169313A1 · Choi et al. · 2017 [cited by applicant]
US 20170177933A1 · Mittal et al. · 2017 [cited by applicant]
US 20180003512A1 · Lynch · 2018 [cited by applicant]
US 20180107190A1 · Marshall et al. · 2018 [cited by applicant]
US 20190147320A1 · Mattyus et al. · 2019 [cited by applicant]
US 20190147331A1 · Arditi · 2019 [cited by applicant]
US 20190170519A1 · Anwar et al. · 2019 [cited by applicant]
US 20190221033A1 · Messerlie et al. · 2019 [cited by applicant]
US 20190258878A1 · Koivisto et al. · 2019 [cited by applicant]
US 20190272434A1 · Dorum · 2019 [cited by examiner]
US 20190287393A1 · Fowe et al. · 2019 [cited by applicant]
US 20190325738A1 · Dorum · 2019 [cited by examiner]
US 20190355103A1 · Baek et al. · 2019 [cited by applicant]
US 20200302223A1 · Dutta et al. · 2020 [cited by applicant]
US 20200410274A1 · Satoh et al. · 2020 [cited by applicant]
US 20210012166A1 · Braley et al. · 2021 [cited by applicant]
US 20210019516A1 · Mittal · 2021 [cited by examiner]
US 20210056847A1 · Saxena et al. · 2021 [cited by applicant]
US 20210113130A1 · Tran · 2021 [cited by applicant]
US 20210150278A1 · Dudzik et al. · 2021 [cited by applicant]
US 20210164787A1 · Soni et al. · 2021 [cited by applicant]
US 20210209368A1 · Hao et al. · 2021 [cited by applicant]
US 20210224466A1 · Nehrenberg et al. · 2021 [cited by applicant]
US 20210302170A1 · Xie et al. · 2021 [cited by applicant]
US 20210325898A1 · Golov · 2021 [cited by applicant]
US 20210333124A1 · Heo et al. · 2021 [cited by applicant]
US 20210342585A1 · Fleisig et al. · 2021 [cited by applicant]
US 20220113162A1 · Nomura · 2022 [cited by applicant]
US 20220156612A1 · Ren et al. · 2022 [cited by applicant]
US 20220198339A1 · Zhao et al. · 2022 [cited by applicant]
US 20220277647A1 · Guo et al. · 2022 [cited by applicant]
US 20220366259A1 · Wang et al. · 2022 [cited by applicant]
US 20230213945A1 · Sajjan et al. · 2023 [cited by applicant]
US 20230221136A1 · Rodrigues · 2023 [cited by applicant]
US 20230252795A1 · Tong et al. · 2023 [cited by applicant]
CN 101924647A · 2010 [cited by applicant]
CN 107743431A · 2018 [cited by applicant]
EP 3280974A1 · 2018 [cited by applicant]
WO WO2011023247A1 · 2011 [cited by applicant]
WO WO2016162665A1 · 2016 [cited by applicant]
WO WO2021002190A1 · 2021 [cited by applicant]
Zhang, Xiangrong, et al. “Aerial image road extraction based on an improved generative adversarial network.” Remote Sensing 11.8 (2019): 930. (Year: 2019). [cited by examiner]
Xiao, Xuerong, Swetava Ganguli, and Vipul Pandey. “VAE-Info-cGAN: generating synthetic images by combining pixel-level and feature-level geospatial conditional inputs.” Proceedings of the 13th ACM SIGSPATIAL Internation… [cited by examiner]
Zhang, Ying, et al. “An enhanced GAN model for automatic satellite-to-map image conversion.” IEEE 2020 (Year: 2020). [cited by examiner]
Xiao, Xuerong et al. “VAE-Info-cGAN: Generating synthetic images by combining pixel-level and feature-level geospatial conditional inputs.” Proceedings of the 13th ACM SIGSPATIAL International Workshop on Computational … [cited by examiner]
Notice of Allowance for U.S. Appl. No. 17/662,129 dated Dec. 9, 2024. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 17/662,151 dated Dec. 20, 2024. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 17/662,136 dated Dec. 23, 2024. [cited by applicant]
Agmon et al., “An algorithm for finding the distribution of maximal entropy”, Journal of Computational Physics, vol. 30, No. 2 (Feb. 1979), 9 pages. [cited by applicant]
Alotaibi A., “Deep Generative Adversarial Networks for Image-to-Image Translation: A Review”, Symmetry, vol. 12, No. 10, (Oct. 16, 2020), 26 pages. [cited by applicant]
Arman et al., “Lane-level routable digital map reconstruction for motorway networks using low-precision GPS data”, Transportation Research Part C: Emerging Technologies, (Jun. 3, 2021), 21 pages. [cited by applicant]
Batra, A., “Road Topology Extraction from Satellite Images by Knowledge Sharing”, International Institute of Information Technology, Deemed University, (Jul. 2019), 75 pages. [cited by applicant]
Biagioni et al., “Inferring Road Maps from Global Positioning System Traces: Survey and Comparative Evaluation”, Department of Computer Science, University of Illinois at Chicago, (2012), 11 pages. [cited by applicant]
Chen et al., “Probabilistic Modeling of Traffic Lanes from GPS Traces”, 18th ACM SIGSPATIAL International Symposium on Advances in Geographic Information Systems, ACM-GIS 2010, (Nov. 3-5, 2010), 8 pages. [cited by applicant]
Dorum, O., “Deriving Double-Digitized Road Network Geometry from Probe Data”, SIGSPATIAL '17: Proceedings of the 25th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, (Nov. 2017), 1… [cited by applicant]
Hartmann et al., “Night time road curvature estimation based on convolutional neural networks”, 2013 IEEE Intelligent Vehicles Symposium (IV), (Jun. 23-26, 2013), 6 pages. [cited by applicant]
He et al., “Sat2Graph: Road Graph Extraction through Graph-Tensor Encoding”, arXiv:2007.09547v1, (Jul. 19, 2020), 17 pages. [cited by applicant]
Horritt et al., “Developing a Prototype Tool for Mapping Flooding From All Sources Phase 1: Scoping and Conceptual Method Development”, Department for Environment Food and Rural Affairs, Flood and Coastal Erosion Risk M… [cited by applicant]
Kaji et al., “Overview of Image-to-Image Translation by Use of Deep Neural Networks: Denoising, Super-Resolution, Modality Conversion, and Reconstruction in Medical Imaging”, Radiological Physics and Technology 12(4), (… [cited by applicant]
Kupyn et al., “DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better”, 2019 IEEE/CVF International Conference on Computer Vision (ICCV), (2019), 10 pages. [cited by applicant]
Kupyn et al., “DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks”, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, (Nov. 19, 2017), pp. 8183-8192. [cited by applicant]
Mi et al., “HDMapGen: A Hierarchical Graph Generative Model of High Definition Maps”, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (Jun. 1, 2021), 10 pages. [cited by applicant]
Narayan et al., “Maximum Entropy Image Restoration in Astronomy”, Annual Review of Astronomy and Astrophysics, vol. 24, No. 1, (Nov. 2003), 44 pages. [cited by applicant]
Redmon et al., “YOLO: Real-Time Object Detection”, Retrieved on Oct. 18, 2022, Retrieved from the Internet: URL<https://pjreddie.com/darknet/yolo>, (2018), 9 pages. [cited by applicant]
Spolti et al., “Application of U-Net and Auto-Encoder to the Road/Non-Road Classification of Aerial Imagery in Urban Environments”, 15th International Conference on Computer Vision Theory and Applications, (Jan. 2020), … [cited by applicant]
Vu, “Vehicle Perception: Localization, Mapping with Detection, Classification and Tracking of Moving Objects”, Computer Science, Institut National Polytechnique de Grenoble—INPG, (2009), 127 pages. [cited by applicant]
Xiao et al., “VAE-info-cGAN: generating synthetic images by combining pixel-level and feature-level geospatial conditional inputs”, arXiv:2012.04196v1, (Dec. 8, 2020), 10 pages. [cited by applicant]
Zhang et al., “A Fast Learning Method for Accurate and Robust Lane Detection Using Two-Stage Feature Extraction with YOLO v3”, Sensors 2018 (Dec. 6, 2018), 20 pages. [cited by applicant]
U.S. Appl. No. 17/662, 129, filed May 5, 2022, entitled, “Method, Apparatus, and Computer Program Product for Map Geometry Generation Based on Object Detection”, 40 pages. [cited by applicant]
U.S. Appl. No. 17/662,158, filed May 5, 2022, entitled, “Method, Apparatus, and Computer Program Product for Probe Data-Based Geometry Generation”, 35 pages. [cited by applicant]
U.S. Appl. No. 17/662,151, filed May 5, 2022, entitled, “Method, Apparatus, and Computer Program Product for Map Geometry Generation Based on Data Aggregation And Conflation With Statistical Analysis”, 35 pages. [cited by applicant]
U.S. Appl. No. 17/662,136, filed May 5, 2022, entitled, “Method, Apparatus, and Computer Program Product for Lane Geometry Generation Based on Graph Estimation”, 40 pages. [cited by applicant]
U.S. Appl. No. 17/662,144, filed May 5, 2022, entitled, “Method, Apparatus, and Computer Program Product for Map Geometry Generation Based on Data Aggregation and Conflation”, 43 pages. [cited by applicant]
Final Office Action for U.S. Appl. No. 17/662,129 dated Feb. 15, 2024. [cited by applicant]
Final Office Action for U.S. Appl. No. 17/662,136 dated Mar. 27, 2024. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/662,158 dated Mar. 27, 2024. [cited by applicant]
Advisory Action for U.S. Appl. No. 17/662,129 dated Apr. 25, 2024. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/662,129 dated Aug. 24, 2023. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/662,158 dated Sep. 13, 2023. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/662,136 dated Sep. 14, 2023. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/662,144 dated Jan. 5, 2024. [cited by applicant]
Extended European Search Report for European Application No. 23171578.0 dated Oct. 10, 2023, 8 pages. [cited by applicant]
Extended European Search Report for European Application No. 23171576.4 dated Oct. 6, 2023, 9 pages. [cited by applicant]
Extended European Search Report for European Application No. 23171637.4 dated Oct. 19, 2023, 7 pages. [cited by applicant]
Extended European Search Report for European Application No. 23171632.5 dated Oct. 10, 2023, 8 pages. [cited by applicant]
Final Office Action for U.S. Appl. No. 17/662,144 dated Jun. 18, 2024. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/662,129 dated Jun. 5, 2024. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/662,136 dated Sep. 10, 2024. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/662,144 dated Oct. 23, 2024. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/662,151 dated Sep. 27, 2024. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/662,158 dated Sep. 25, 2024. [cited by applicant]
Final Office Action for U.S. Appl. No. 17/662,158 dated Mar. 18, 2025. [cited by applicant]
Advisory Action for U.S. Appl. No. 17/662,158 dated May 15, 2025. [cited by applicant]
Final Office Action for U.S. Appl. No. 17/662,144 dated Mar. 11, 2025. [cited by applicant]
Advisory Action for U.S. Appl. No. 17/662,144 dated May 19, 2025. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/662,144 dated Jul. 1, 2025. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 17/662,144 dated Oct. 7, 2025. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 17/662,158 dated Oct. 30, 2025. [cited by applicant]
Office Action for European Application No. 23171578.0 dated Feb. 3, 2026, 7 pages. [cited by applicant]
Niroshan et al., “Post-analysis of OSM-GAN Spatial Change Detection”, Lecture Notes in Computer Science, vol. 13238, (May 19, 2022), 15 pages. [cited by applicant]