IP Library › Granted Patent US 12,326,996
Granted Patent B2
US 12,326,996 · App. 18/629,255 · Granted Jun 10, 2025

System and machine learning method for separating noise and signal in multitouch sensors

Inventors: Lichen Wang (San Jose, CA); Behrooz Shahsavari (Los Altos Hills, CA); Hojjat Seyed Mousavi (San Jose, CA); Nima Ferdosi (San Jose, CA); Baboo V. Gowreesunker (Mountain View, CA)
Assignee: Apple Inc.
G06F3/04182G06F3/044G06N3/044G06N3/08G06F2203/04104
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,326,996
App. No.
18/629,255
Granted
Jun 10, 2025
Kind
B2
Abstract

In some examples, touch data can include noise. Machine learning techniques, such as gated recurrent units and convolutional neural networks can be used to mitigate noise present in touch data. In some examples, a gated recurrent unit stage and a convolutional neural network stage can be arranged in series, such as by providing the output of the gated recurrent unit as input to the convolutional neural network. The gated recurrent unit can remove noise caused by a first component of the electronic device and the convolutional neural network can remove noise caused by a second component of the electronic device, for example. Thus, together, the gated recurrent unit and the convolutional neural network can remove or substantially reduce the noise in the touch data.

Claims (68)

1. An electronic device comprising:

one or more processors configured to execute software that causes the electronic device to:

obtain first touch data representing output, sensed using one or more touch sensing electrodes of a touch sensitive surface, indicating one or more touches of the touch sensitive surface by one or more touch objects, wherein the first touch data includes data representing a first noise;

input the first touch data into a gated recurrent unit implemented by the one or more processors executing the software;

remove, with the gated recurrent unit, the first noise from the first touch data to obtain second touch data that indicates the one or more touches of the touch sensitive surface by the one or more touch objects; and

output, from the gated recurrent unit, the second touch data.

2. The electronic device of claim 1 , wherein:

the first touch data includes a number of one-dimensional arrays of touch data;

the gated recurrent unit includes a number of channels equal to the number of one-dimensional arrays of the first touch data; and

each respective one-dimensional array of the first touch data is input to each respective channel of the gated recurrent unit.

3. The electronic device of claim 2 , wherein:

the gated recurrent unit includes a plurality of layers, each layer including one node in each channel;

each respective node outputs a state to one or more nodes of the same layer in channels adjacent to a channel of a respective node; and

each respective node outputs an output to a node in a same channel as the respective node and in a layer that is adjacent to a layer of the respective node.

4. The electronic device of claim 1 , wherein:

the first noise is noise caused by operation of a display.

5. The electronic device of claim 1 , wherein the software further causes the electronic device to, prior to inputting the first touch data that includes first noise into the gated recurrent unit, train the gated recurrent unit, wherein training the gated recurrent unit comprises:

sampling a plurality of touch sensitive surfaces while display components of the plurality of touch sensitive surfaces are powered off and objects are proximate to or touching the touch sensitive surfaces to obtain training touch data and testing touch data;

sampling the plurality of touch sensitive surfaces while the display components of the plurality of touch sensitive surfaces are powered on and no objects are proximate to or touching the plurality of touch sensitive surfaces to obtain training noise data and testing noise data;

combining the training touch data and the training noise data to create training data; and

combining the testing touch data and the testing noise data to create testing data.

6. The electronic device of claim 1 , wherein the software further causes the electronic device to, prior to inputting the first touch data that includes first noise into the gated recurrent unit, perform a calibration procedure, the calibration procedure including:

at a touch screen of the electronic device, while no objects are proximate to or touching the touch screen, displaying a plurality of images while sensing touch data to collect noise data; and

calculating a plurality of characteristic vectors of the noise data for which the plurality of characteristic vectors are linearly combined to model the noise data.

7. The electronic device of claim 6 , wherein the gated recurrent unit includes a plurality of nodes arranged in a plurality of channels and layers, and each node retains a state, and the software further causes the electronic device to, prior to inputting the first touch data that includes the first noise into the gated recurrent unit:

providing nodes within a number of the layers with a characteristic vector as an initial state of the nodes within the number of the layers.

8. The electronic device of claim 6 , wherein the software further causes the electronic device to:

concatenate the plurality of characteristic vectors with the first touch data that includes the first noise, wherein inputting the first touch data that includes the first noise into the gated recurrent unit includes inputting the concatenation of the characteristic vectors and the first touch data that includes the first noise into the gated recurrent unit.

9. The electronic device of claim 6 , wherein the gated recurrent unit includes a plurality of nodes arranged in a plurality of channels and layers, and each node retains a state, and the software further causes the electronic device to, prior to inputting the first touch data that includes first noise into the gated recurrent unit:

concatenate the characteristic vectors; and

provide the concatenation of the characteristic vectors to a first layer of the gated recurrent unit as an initial state of the first layer.

10. The electronic device of claim 6 , wherein the software further causes the electronic device to, prior to inputting the touch data that includes the first noise into the gated recurrent unit:

concatenate the characteristic vectors;

input the concatenation of the characteristic vectors into a calibration gated recurrent unit; and

provide an output of the calibration gated recurrent unit to an attention layer of the gated recurrent unit.

11. The electronic device of claim 1 , wherein the software further causes the electronic device to, prior to inputting the first touch data that includes the first noise into the gated recurrent unit:

sense the first touch data using a respective sampling frequency of a plurality of sampling frequencies; and

concatenate the respective sampling frequency with the first touch data that includes the first noise, wherein inputting the first touch data that includes the first noise into the gated recurrent unit includes inputting the concatenation of the respective sampling frequency and the first touch data that includes the first noise into the gated recurrent unit.

12. The electronic device of claim 1 , wherein the gated recurrent unit is a minimal gated recurrent unit.

13. The electronic device of claim 1 , wherein the gated recurrent unit is spatially recurrent.

14. The electronic device of claim 1 , wherein removal of the first noise from the first touch data to obtain the second touch data with the gated recurrent unit further comprises:

estimate, using the gated recurrent unit, a plurality of coefficients;

scale a plurality of predetermined vectors with the plurality of coefficients to obtain scaled vectors; and

sum the scaled vectors to obtain an estimation of the first noise.

15. A method, implemented by one or more processors executing software, comprising:

obtaining first touch data representing output, sensed using one or more touch sensing electrodes of a touch sensitive surface, indicating one or more touches of the touch sensitive surface by one or more touch objects, wherein the first touch data includes data representing a first noise;

inputting the first touch data that includes the first noise into a gated recurrent unit implemented by the one or more processors executing the software;

removing, with the gated recurrent unit, the first noise from the first touch data to obtain second touch data that indicates the one or more touches of the touch sensitive surface by the one or more touch objects; and

outputting, from the gated recurrent unit, the second touch data.

16. The method of claim 15 , wherein:

the first touch data includes a number of one-dimensional arrays of touch data;

the gated recurrent unit includes a number of channels equal to the number of one-dimensional arrays of the first touch data; and

each respective one-dimensional array of the first touch data is input to each respective channel of the gated recurrent unit.

17. A non-transitory computer-readable medium comprising software instructions that, when executed by one or more processors of an electronic device comprising a touch screen, cause the electronic device to:

obtain first touch data representing output, sensed using one or more touch sensing electrodes of a touch sensitive surface, indicating one or more touches of the touch sensitive surface by one or more touch objects, wherein the first touch data includes data representing a first noise;

input the first touch data into a gated recurrent unit implemented by the one or more processors executing the software;

remove, with the gated recurrent unit, the first noise from the first touch data to obtain second touch data that indicates the one or more touches of the touch sensitive surface by the one or more touch objects; and

output, from the gated recurrent unit, the second touch data.

18. The non-transitory computer-readable medium of claim 17 , wherein:

the first touch data includes a number of one-dimensional arrays of touch data;

the gated recurrent unit includes a number of channels equal to the number of one-dimensional arrays of the first touch data; and

each respective one-dimensional array of the first touch data is input to each respective channel of the gated recurrent unit.

19. The non-transitory computer-readable medium of claim 18 , wherein:

the gated recurrent unit includes a plurality of layers, each layer including one node in each channel;

each respective node outputs a state to one or more nodes of the same layer in channels adjacent to a channel of a respective node; and

each respective node outputs an output to a node in a same channel as the respective node and in a layer that is adjacent to a layer of the respective node.

20. The non-transitory computer-readable medium of claim 17 , wherein:

the first noise is noise caused by operation of a display.

Continuity (3)
Continuation 17406990 · Aug 19, 2021
Provisional Application 63070779 · Aug 26, 2020
Related Publication 20240345683A1 · Oct 17, 2024
References Cited (98)
US 5483261A · Yasutake · 1996 [cited by applicant]
US 5488204A · Mead et al. · 1996 [cited by applicant]
US 5825352A · Bisset et al. · 1998 [cited by applicant]
US 5835079A · Shieh · 1998 [cited by applicant]
US 5880411A · Gillespie et al. · 1999 [cited by applicant]
US 6188391B1 · Seely et al. · 2001 [cited by applicant]
US 6310610B1 · Beaton et al. · 2001 [cited by applicant]
US 6323846B1 · Westerman et al. · 2001 [cited by applicant]
US 6690387B2 · Zimmerman et al. · 2004 [cited by applicant]
US 7015894B2 · Morohoshi · 2006 [cited by applicant]
US 7184064B2 · Zimmerman et al. · 2007 [cited by applicant]
US 7663607B2 · Hotelling et al. · 2010 [cited by applicant]
US 8479122B2 · Hotelling et al. · 2013 [cited by applicant]
US 8520913B2 · Dean et al. · 2013 [cited by applicant]
US 9098148B2 · Sharma · 2015 [cited by applicant]
US 9146630B2 · Choi et al. · 2015 [cited by applicant]
US 9164618B2 · Perski et al. · 2015 [cited by applicant]
US 9244545B2 · Hinckley et al. · 2016 [cited by applicant]
US 9256321B2 · Wang et al. · 2016 [cited by applicant]
US 9558455B2 · Johnson et al. · 2017 [cited by applicant]
US 9904412B2 · Stevenson et al. · 2018 [cited by applicant]
US 10261685B2 · Deselaers et al. · 2019 [cited by applicant]
US 10345970B2 · Qiao et al. · 2019 [cited by applicant]
US 10928955B1 · Reed et al. · 2021 [cited by applicant]
US 11481070B1 · Ganjali et al. · 2022 [cited by applicant]
US 11599223B1 · Gowreesunker et al. · 2023 [cited by applicant]
US 11954288B1 · Lichen et al. · 2024 [cited by applicant]
US 20060197753A1 · Hotelling · 2006 [cited by applicant]
US 20060214921A1 · Takahashi et al. · 2006 [cited by applicant]
US 20070257890A1 · Hotelling et al. · 2007 [cited by applicant]
US 20120056846A1 · Zaliva · 2012 [cited by applicant]
US 20120113064A1 · White et al. · 2012 [cited by applicant]
US 20120194452A1 · Cho et al. · 2012 [cited by applicant]
US 20120249433A1 · Deng et al. · 2012 [cited by applicant]
US 20130222290A1 · Choi et al. · 2013 [cited by applicant]
US 20140028621A1 · Hotelling et al. · 2014 [cited by applicant]
US 20140062949A1 · Chang et al. · 2014 [cited by applicant]
US 20140071360A1 · Chang et al. · 2014 [cited by applicant]
US 20140118297A1 · Huang et al. · 2014 [cited by applicant]
US 20140204058A1 · Huang et al. · 2014 [cited by applicant]
US 20140267132A1 · Rabii et al. · 2014 [cited by applicant]
US 20150091850A1 · Morein et al. · 2015 [cited by applicant]
US 20150242009A1 · Xiao et al. · 2015 [cited by applicant]
US 20150293621A1 · Singh et al. · 2015 [cited by applicant]
US 20160196003A1 · Yoon et al. · 2016 [cited by applicant]
US 20160291792A1 · Sawahata et al. · 2016 [cited by applicant]
US 20160309307A1 · Agarwal et al. · 2016 [cited by applicant]
US 20160357315A1 · Huang et al. · 2016 [cited by applicant]
US 20170003785A1 · Berget et al. · 2017 [cited by applicant]
US 20170090668A1 · Agarwal et al. · 2017 [cited by applicant]
US 20170090670A1 · Stevenson et al. · 2017 [cited by applicant]
US 20170139527A1 · Nathan et al. · 2017 [cited by applicant]
US 20170192604A1 · Stevenson et al. · 2017 [cited by applicant]
US 20170285804A1 · Li et al. · 2017 [cited by applicant]
US 20180059852A1 · Sterling et al. · 2018 [cited by applicant]
US 20180129311A1 · Westhues et al. · 2018 [cited by applicant]
US 20180188938A1 · Deselaers et al. · 2018 [cited by applicant]
US 20180329542A1 · Forlines et al. · 2018 [cited by applicant]
US 20180348949A1 · Kim et al. · 2018 [cited by applicant]
US 20190079634A1 · Kravets et al. · 2019 [cited by applicant]
US 20190122076A1 · Sen et al. · 2019 [cited by applicant]
US 20200142532A1 · Kravets et al. · 2020 [cited by applicant]
US 20200210045A1 · Mohamed et al. · 2020 [cited by applicant]
US 20200241691A1 · Kobayashi · 2020 [cited by applicant]
US 20210191563A1 · Reed et al. · 2021 [cited by applicant]
US 20210397327A1 · Lee · 2021 [cited by applicant]
US 20220019311A1 · Seyed Mousavi et al. · 2022 [cited by applicant]
US 20230016448A1 · Ganjali et al. · 2023 [cited by applicant]
JP 2000163031A · 2000 [cited by applicant]
JP 2002342033A · 2002 [cited by applicant]
WO 2017151757A1 · 2017 [cited by applicant]
Corrected Notice of Allowability received for U.S. Appl. No. 17/123,015, mailed on Oct. 19, 2023, 2 pages. [cited by applicant]
Ex Parte Quayle Action received for U.S. Appl. No. 17/249,791, mailed on Jan. 5, 2022, 8 pages. [cited by applicant]
Final Office Action received for U.S. Appl. No. 16/107,037, mailed on Jan. 13, 2020, 17 pages. [cited by applicant]
Final Office Action received for U.S. Appl. No. 17/123,015, mailed on Apr. 13, 2023, 14 pages. [cited by applicant]
Final Office Action received for U.S. Appl. No. 17/406,990, mailed on Apr. 4, 2023, 13 pages. [cited by applicant]
Non-Final Office Action received for U.S. Appl. No. 16/107,037, mailed on Jul. 30, 2019, 15 pages. [cited by applicant]
Non-Final Office Action received for U.S. Appl. No. 17/123,015, mailed on Oct. 27, 2022, 12 pages. [cited by applicant]
Non-Final Office Action received for U.S. Appl. No. 17/249,791, mailed on Apr. 12, 2022, 12 pages. [cited by applicant]
Non-Final Office Action received for U.S. Appl. No. 17/406,990, mailed on Sep. 28, 2022, 9 pages. [cited by applicant]
Non-Final Office Action received for U.S. Appl. No. 17/933,974, mailed on Mar. 2, 2023, 19 pages. [cited by applicant]
Non-Final Office Action received for U.S. Appl. No. 16/107,037, mailed on May 6, 2020, 20 pages. [cited by applicant]
Notice of Allowance received for U.S. Appl. No. 16/107,037, mailed on Oct. 22, 2020, 9 pages. [cited by applicant]
Notice of Allowance received for U.S. Appl. No. 17/123,015, mailed on Oct. 10, 2023, 8 pages. [cited by applicant]
Notice of Allowance received for U.S. Appl. No. 17/177,977, mailed on Nov. 4, 2021, 9 pages. [cited by applicant]
Notice of Allowance received for U.S. Appl. No. 17/249,791, mailed on Nov. 7, 2022, 11 pages. [cited by applicant]
Notice of Allowance received for U.S. Appl. No. 17/406,990, mailed on Aug. 15, 2023, 11 pages. [cited by applicant]
Notice of Allowance received for U.S. Appl. No. 17/406,990, mailed on Dec. 7, 2023, 8 pages. [cited by applicant]
Notice of Allowance received for U.S. Appl. No. 17/463,195, mailed on Jun. 10, 2022, 11 pages. [cited by applicant]
Notice of Allowance received for U.S. Appl. No. 17/933,974, mailed on Aug. 21, 2023, 12 pages. [cited by applicant]
Aggarwal, C. C., “An Introduction to Neural Networks”, Neural Networks and Deep Learning. Springer, Cham, Available Online at: <https://doi.org/10.1007/978-3-319-94463-0_1>, 2018, pp. 1-52. [cited by applicant]
Aggarwal, C. C., “Convolutional Neural Networks”, Neural Networks and Deep Learning. Springer, Cham, Available online at: <https://doi.org/10.1007/978-3-319-94463-0_8>, 2018, pp. 315-371. [cited by applicant]
Aggarwal, C. C., “Recurrent Neural Networks”, Neural Networks and Deep Learning. Springer, Cham, Available online at: <https://doi.org/10.1007/978-3-319-94463-0_7>, 2018, pp. 271-313. [cited by applicant]
Le, Huy V., “Hand-and-Finger-Awareness for Mobile Touch Interaction using Deep Learning”, University of Suttgart, Germany, 2019, 258 pages. [cited by applicant]
Lee et al., “A Multi-Touch Three Dimensional Touch-Sensitive Tablet”, CHI'85 Proceedings, Apr. 1985, pp. 21-25. [cited by applicant]
Rubine, Dean, “Combining Gestures and Direct Manipulation”, CHI'92, May 3-7, 1992, pp. 659-660. [cited by applicant]
Rubine, Dean H., “The Automatic Recognition of Gestures”, CMU-CS-91-202, Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Computer Science at Carnegie Mellon University, Dec… [cited by applicant]
Westerman, Wayne, “Hand Tracking, Finger Identification, and Chordic Manipulation on a Multi-Touch Surface”, A Dissertation Submitted to the Faculty of the University of Delaware in Partial Fulfillment of the Requiremen… [cited by applicant]