IP Library Granted Patent US 12,681,638
Granted Patent B2
US 12,681,638 · App. 18/476,724 · Granted Jul 14, 2026

Hardware accessible memory fabric

Inventors: Timothy A. Stabrawa (Lombard, IL); Zachary A. Cornelius (Buffalo Grove, IL); John Overton (Chicago, IL); Andrew S. Poling (Lombard, IL); Jesse Taylor (Glenside, PA)
Assignee: Kove IP, LLC
G06F3/0607G06F3/0604G06F3/0631G06F3/0647G06F3/067G06F3/0683G06F12/08G06F12/10H04L67/1097G06F2212/1048Y02D10/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,681,638
App. No.
18/476,724
Filed
Sep 28, 2023
Granted
Jul 14, 2026
Kind
B2
Art Unit
2137
USPC
711/209
Abstract

Dynamically provisionable and allocatable memory external to a requesting apparatus may be provided. A method includes: receiving a memory allocation request for allocating primary memory from a component of the client device; in response to the memory allocation request, selecting a subset of a region of memory in a memory device to be a portion of primary memory allocated for the client device, wherein the region of memory in the memory device is allocated for the client device before the memory allocation request is received; mapping at least the portion of primary memory to an address space; and accessing, by a hardware-accessible interface of the client device, data in the portion of primary memory via a client-side memory access, wherein a communication interface of the memory device is configured to access the subset of the region of memory in the memory device as part of the client-side memory access.

Claims (32)

1 . A method performed by a client device, the method comprising:

receiving a memory allocation request for allocating primary memory from a component of the client device;

in response to the memory allocation request, selecting a subset of a region of memory in a memory device to be a portion of primary memory allocated for the client device, wherein the region of memory in the memory device is allocated for the client device before the memory allocation request is received;

mapping at least the portion of primary memory to an address space; and

accessing, by a hardware-accessible interface of the client device, data in the portion of primary memory via a client-side memory access, wherein a communication interface of the memory device is configured to access the subset of the region of memory in the memory device as part of the client-side memory access.

2 . The method of claim 1 , wherein the client device communicates with the memory device based on a Peripheral Component Interconnect Express (PCIe) protocol.

3 . The method of claim 1 , wherein the client-side memory access is performed by using a protocol for accessing memory.

4 . The method of claim 3 , wherein the protocol comprises a PCIe protocol.

5 . The method of claim 1 , wherein the client device and the memory device are in communication over at least one of: a network; or a switched fabric.

6 . The method of claim 1 , wherein another client device accesses the data in the portion of primary memory as shared memory with the client device.

7 . The method of claim 1 , wherein the client-side memory access comprises one or more atomic operations.

8 . The method of claim 7 , wherein the one or more atomic operations comprise at least one of: a fetch-and-add operation or a compare-and-swap operation.

9 . The method of claim 1 , wherein the client device comprises a chipset or a processor, and wherein the chipset or the processor comprises a component of the hardware-accessible interface.

10 . The method of claim 9 , wherein, in response to a cache-invalidate indication of a memory access protocol, the component of the hardware-accessible interface causes one or more cache lines of the client device to be invalidated.

11 . The method of claim 1 , wherein at least a portion of the region of memory in the memory device is accessed by an application component via a second client-side memory access.

12 . The method of claim 1 , wherein at least a portion of the region of memory is registered with the communication interface of the memory device.

13 . The method of claim 1 , wherein the client-side memory access indicates a starting memory location corresponding to the portion of primary memory.

14 . The method of claim 1 , wherein the portion of primary memory is accessed in response to the client-side memory access, and wherein the client-side memory access comprises a corresponding address indicative of the portion of primary memory.

15 . The method of claim 1 , wherein another device is prevented from accessing the data in the portion of primary memory based on one or more access controls.

16 . The method of claim 1 , wherein the region of memory in the memory device is allocated by an allocation logic.

17 . The method of claim 1 , wherein a region access logic enforces one or more access controls for the region of memory in the memory device.

18 . A client device comprising a memory for storing computer instructions and a processor in communication with the memory, wherein, when the processor executes the computer instructions, the processor is configured to cause the client device to:

receive a memory allocation request for allocating primary memory from a component of the client device;

in response to the memory allocation request, select a subset of a region of memory in a memory device to be a portion of primary memory allocated for the client device, wherein the region of memory in the memory device is allocated for the client device before the memory allocation request is received;

map at least the portion of primary memory to an address space; and

access, by a hardware-accessible interface of the client device, data in the portion of primary memory via a client-side memory access, wherein a communication interface of the memory device is configured to access the subset of the region of memory in the memory device as part of the client-side memory access.

19 . The client device of claim 18 , wherein the client device communicates with the memory device based on a Peripheral Component Interconnect Express (PCIe) protocol.

20 . A non-transitory storage medium for storing computer readable instructions, the computer readable instructions, when executed by a processor in a client device, causing the processor to:

receive a memory allocation request for allocating primary memory from a component of the client device;

in response to the memory allocation request, select a subset of a region of memory in a memory device to be a portion of primary memory allocated for the client device, wherein the region of memory in the memory device is allocated for the client device before the memory allocation request is received;

map at least the portion of primary memory to an address space; and

access, by a hardware-accessible interface of the client device, data in the portion of primary memory via a client-side memory access, wherein a communication interface of the memory device is configured to access the subset of the region of memory in the memory device as part of the client-side memory access.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 28, 2023
From: STABRAWA, TIMOTHY A.; CORNELIUS, ZACHARY A.; OVERTON, JOHN; POLING, ANDREW S.; TAYLOR, JESSE I.
To: KOVE CORPORATION
Reel/Frame 065064/0021 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 28, 2023
From: KOVE IO, INC.
To: KOVE IP, LLC
Reel/Frame 065064/0093 →
CHANGE OF NAME Recorded Sep 28, 2023
From: KOVE CORPORATION
To: KOVE IO, INC.
Reel/Frame 065082/0217 →
Continuity (7)
Continuation 17170280 · Feb 8, 2021
Continuation 16439053 · Jun 12, 2019
Continuation 15800891 · Nov 1, 2017
Continuation 14554655 · Nov 26, 2014
Continuation 14530908 · Nov 3, 2014
Provisional Application 62051144 · Sep 16, 2014
Related Publication 20240020003A1 · Jan 18, 2024
References Cited (105)
US 6049853A · Kingsbury et al. · 2000 [cited by applicant]
US 7925711B1 · Gopalan et al. · 2011 [cited by applicant]
US 8307154B2 · Stabrawa et al. · 2012 [cited by applicant]
US 8510524B1 · Corbett et al. · 2013 [cited by applicant]
US 8745276B2 · Bloch et al. · 2014 [cited by applicant]
US 8775755B2 · Magenheimer et al. · 2014 [cited by applicant]
US 8799914B1 · Metcalf · 2014 [cited by applicant]
US 8914458B2 · Raindel et al. · 2014 [cited by applicant]
US 9015426B2 · Stabrawa et al. · 2015 [cited by applicant]
US 9575889B2 · Chang et al. · 2017 [cited by applicant]
US 10235064B1 · Natanzon et al. · 2019 [cited by applicant]
US 20030188114A1 · Lubbers et al. · 2003 [cited by applicant]
US 20030188218A1 · Lubbers et al. · 2003 [cited by applicant]
US 20040093389A1 · Mohammed et al. · 2004 [cited by applicant]
US 20040143562A1 · Chen et al. · 2004 [cited by applicant]
US 20040143718A1 · Chen et al. · 2004 [cited by applicant]
US 20040221102A1 · Watanabe · 2004 [cited by applicant]
US 20050044301A1 · Vasilevsky et al. · 2005 [cited by applicant]
US 20050204045A1 · Belkin et al. · 2005 [cited by applicant]
US 20050216552A1 · Fineberg et al. · 2005 [cited by applicant]
US 20060236063A1 · Hausauer et al. · 2006 [cited by applicant]
US 20080082593A1 · Komarov et al. · 2008 [cited by applicant]
US 20080104337A1 · VelurEunni · 2008 [cited by applicant]
US 20080235409A1 · Ryzhykh · 2008 [cited by applicant]
US 20090113425A1 · Ports et al. · 2009 [cited by applicant]
US 20100250867A1 · Bettger et al. · 2010 [cited by applicant]
US 20110264949A1 · Ikeuchi et al. · 2011 [cited by applicant]
US 20120079318A1 · Colgrove et al. · 2012 [cited by applicant]
US 20120221803A1 · Stabrawa et al. · 2012 [cited by applicant]
US 20120222052A1 · Magenheimer et al. · 2012 [cited by applicant]
US 20130185720A1 · Tuch et al. · 2013 [cited by applicant]
US 20130290546A1 · Samih et al. · 2013 [cited by applicant]
US 20140089451A1 · Eran et al. · 2014 [cited by applicant]
US 20140164545A1 · Davis et al. · 2014 [cited by applicant]
US 20150020071A1 · Phelan et al. · 2015 [cited by applicant]
US 20150293881A1 · Raikin et al. · 2015 [cited by applicant]
US 20150370494A1 · Wang · 2015 [cited by applicant]
US 20160019117A1 · Kumarasamy et al. · 2016 [cited by applicant]
US 20160077761A1 · Stabrawa et al. · 2016 [cited by applicant]
US 20160077966A1 · Stabrawa et al. · 2016 [cited by applicant]
US 20160077975A1 · Stabrawa et al. · 2016 [cited by applicant]
US 20170147227A1 · Stabrawa et al. · 2017 [cited by applicant]
US 20170147437A1 · Borlick et al. · 2017 [cited by applicant]
US 20170160979A1 · Golander et al. · 2017 [cited by applicant]
US 20170192865A1 · Pan · 2017 [cited by applicant]
US 20170285997A1 · Stabrawa et al. · 2017 [cited by applicant]
US 20170286702A1 · Cheung · 2017 [cited by applicant]
US 20180081559A1 · Stabrawa et al. · 2018 [cited by applicant]
WO WO2008132760A2 · 2008 [cited by applicant]
Liran Liss, “On Demand Paging for User-level Networking,” dated Sep. 30, 2013, pp. 1-25, Mellanox Technologies, Sunnyvale, CA. [cited by applicant]
Liran Liss, “On-Demand-Paging (ODP) Update,” dated Apr. 28, 2014, available at https://www.youtube.com/watch?v=KbrlsXQbHCw. [cited by applicant]
Brian Van Essen et al., “DI-MMAP—a scalable memory-map runtime for out-of-core data-intensive applications,” Cluster Computing, dated Oct. 4, 2013, pp. 15-28, vol. 18, Issue 1, published by Springer, New York City, NY. [cited by applicant]
Hannes Mühleisen et al., “Peak Performance—Remote Memory Revisited,” dated Jun. 24, 2013, pp. 1-7, DaMoN' 13, New York, NY. [cited by applicant]
Kevin Te-Ming Lim, “Disaggregated Memory Architectures for Blade Servers,” dated 2010, pp. 1-148, published by The University of Michigan, Ann Arbor, MI. [cited by applicant]
Shuang Liang et al. “Swapping to Remote Memory over Infiniband: An Approach using a High Performance Network Block Device,” dated Sep. 2005, pp. 1-10, published by The Ohio State University, Columbus OH. [cited by applicant]
Shuang Liang et al. “Swapping to Remote Memory over Infiniband: An Approach using a High Performance Network Block Device,” dated Sep. 2005, pp. 1-30, Cluster Computing, 2005, IEEE International, Conference located in B… [cited by applicant]
Stephen Mathew Rumble, “Memory and Object Management in RAMCloud,” dated Mar. 2014, pp. 1-161, published by Stanford University, Stanford, CA. [cited by applicant]
Katnegermis, “rdma/rdma_common.c at master katnegermis/rdma,” GitHub, dated Apr. 27, 2014, pp. 1-4, available at http://github.com/katnegermis/rdma/blob/master/rdma_common.c. [cited by applicant]
Katnegermis, rdma/rdma_server.c at master katnegermis/rdma, GitHub, dated May 6, 2014, pp. 1-7, available at http://github.com/katnegermis/rdma/blob/master/rdma_server_c. [cited by applicant]
Robert Russell, “Introduction to RDMA Programming”, dated April 17, 2014, pp. 1-76, available at http://web.archive.org/web/20140417205540/http://www.cs.unh.edu/~rdr/rdma-intro-module.ppt. [cited by applicant]
Anonymous, “Implementing calloc(),” dated Apr. 12, 2010, pp. 1-3, published online by Lockless, available at URL http://web.archive.org/web/20100412183814/http://locklessinc.com/articles/calloc. [cited by applicant]
Konstantinos Karampogias, “An In-Memory RDMA-Based Architecture for the Hadoop Distributed Filesystem Master Thesis,” dated Aug. 21, 2012, pp. 1-49, published by IBM Research—Zurich, Rüschlikon, Switzerland, available a… [cited by applicant]
Anonymous, “Remote direct memory access,” dated May 12, 2016, pp. 1-2, published online by Wikipedia at URL https://en.wikipedia.org/wiki/Remote_direct_memory_access. [cited by applicant]
Anonymous, “Copy-on-write,” dated Nov. 26, 2014, pp. 1-3, Wikipedia, published by Wikipedia at URL: https://en.wikipedia.org/w/index.php?title=Copy-on-write&oldid=635508783. [cited by applicant]
Chao Wang et al., “NVMalloc: Exposing an Aggregate SSD as a Memory Partition in Extreme-Scale Machines”, dated 2012, pp. 957-968, Parallel & Distributed Processing Symposium (IPDPS), 2012 IEEE 26th International, Shangh… [cited by applicant]
“hugetlbpage”, dated Sep. 13, 2015, pp. 1-6, The Linux Kernel Archives, available at URL https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt. [cited by applicant]
Iulian Moraru, “nvmalloc”, dated Feb. 27, 2013, p. 1, published online by GitHub at URL https://github.com/efficient/nvram/tree/master/nvmalloc. [cited by applicant]
Christoffer Dall, et al., “KVM/ARM: The Design and Implementation of the Linux ARM Hypervisor,” dated Mar. 1, 2014, pp. 333-347, ASPLOS '14, Mar. 1-4, 2014, Salt Lake City, UT, Copyright is held by the owner/author(s). … [cited by applicant]
Avi Kivity et al., “kvm: the Linux Virtual Machine Monitor,” dated Jun. 27, 2007, pp. 225-232, Proceedings of the Linux Symposium, vol. 1, published online by Kernel, Ottawa, Ontario, Canada. [cited by applicant]
Josh Triplett, “Using the KVM API,” dated Sep. 29, 2015, pp. 1-12, published by LWN.net at URL https://lwn.net/Articles/658511. [cited by applicant]
Jes Sorensen, “KVM Live Snapshot support,” dated Jun. 1, 2011, pp. 1-22, published by RedHat, LinuxCon, Japan. [cited by applicant]
“Warning: The xenpaging code is new and not fully debugged . . . ”, p. 1, Retrieved from the Internet on Jan. 27, 2016, at URL: https://xenbits.xon.org/docs/4.3-testing/,misc/xenpacing.txt. [cited by applicant]
Brian Van Essen, “tagged_page.c” (source code file is part of DI-MMAP, Version 1.0), dated 2012, pp. 1-13, Lawrence Livermore National Security, LLC, retrieved from URL https://bitbucket.org/vanessen/di-mmap/src. [cited by applicant]
Brian Van Essen, “Snippet of ‘tagged_page.c’” (source code file is part of DI-MMAP, Version 1.0), dated 2012, p. 1, Lawrence Livermore National Security, LLC, retrieved from URL https:/bitbucket.org/vanessen/di-mmap/src. [cited by applicant]
Brian Van Essen, “helpers.c” (source code file is part of DI-MMAP, Version 1.0), dated 2012, pp. 1-19, Lawrence Livermore National Security, LLC, retrieved from URL https:/bitbucket.org/vanessen/di-mmap/src. [cited by applicant]
Brian Van Essen, “Snippet of ‘helpers.c’” (source code file is part of DI-MMAP, Version 1.0), dated 2012, p. 1, Lawrence Livermore National Security, LLC, retrieved from URL https://bitbucket.org/vanessen/di-mmap/src. [cited by applicant]
Anonymous, “Memory Resource Controller” (memory cgroup documentation—Kernel.org), dated Jan. 25, 2016, pp. 1-23, published by Kernel.org at URL https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt. [cited by applicant]
Ahmed Al-Moayed et al., “Quality of Service Attributes in Web Services,” dated 2010, pp. 367-372, 2010 Fifth International Conference on Software Engineering Advances, © 2010 IEEE. [cited by applicant]
Aleksandar Dragojević et al., FaRM: Fast Remote Memory, Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI '14), ISBN 978-1-931971-09-6, dated Apr. 2, 2014, pp. 1-15, published… [cited by applicant]
International Search Report, issued in International Application No. PCT/US2015/050170, dated Dec. 8, 2015, pp. 1-14, European Patent Office, Rijswijk, Netherlands. [cited by applicant]
Partial International Search Report, issued in International Application No. PCT/US2015/050177, dated Dec. 17, 2015, pp. 1-8, European Patent Office, Rijswijk, Netherlands. [cited by applicant]
International Search Report, issued in International Application No. PCT/US2015/050177, dated May 4, 2016, pp. 1-28, European Patent Office, Rijswijk, Netherlands. [cited by applicant]
Non-Final Office Action, issued in U.S. Appl. No. 14/530,908, dated Apr. 8, 2016, pp. 1-21, United States Patent and Trademark Office, Alexandria, VA. [cited by applicant]
Non-Final Office Action, issued in U.S. Appl. No. 14/554,655, dated Apr. 6, 2016, pp. 1-22, United States Patent and Trademark Office, Alexandria, VA. [cited by applicant]
Non-Final Office Action, issued in U.S. Appl. No. 14/854,657, dated Oct. 23, 2017, pp. 1-12, United States Patent and Trademark Office, Alexandria, VA. [cited by applicant]
Final Office Action, issued in U.S. Appl. No. 14/530,908, dated Jul. 5, 2016, pp. 1-22, United States Patent and Trademark Office, Alexandria, VA. [cited by applicant]
Extended European Search Report, issued in European Patent Application No. 16162354.1, dated Jul. 28, 2016, pp. 1-11, European Patent Office, Munich, Germany. [cited by applicant]
Non-Final Office Action, issued in U.S. Appl. No. 15/621,537, dated Jun. 29, 2017, pp. 1-14, United States Patent and Trademark Office, Alexandria, VA. [cited by applicant]
Non-Final Office Action, issued in U.S. Appl. No. 14/554,655, dated Jan. 11, 2017, pp. 1-8, United States Patent and Trademark Office, Alexandria, VA. [cited by applicant]
Non-Final Office Action, issued in U.S. Appl. No. 15/800,891, dated Apr. 2, 2018, pp. 1-14, United States Patent and Trademark Office, Alexandria, VA. [cited by applicant]
Final Office Action, issued in U.S. Appl. No. 14/854,657, dated Apr. 5, 2018, pp. 1-13, United States Patent and Trademark Office, Alexandria, VA. [cited by applicant]
Non-Final Office Action, issued in U.S. Appl. No. 15/424,395, dated Apr. 2, 2018, pp. 1-17, United States Patent and Trademark Office, Alexandria, VA. [cited by applicant]
Final Office Action, issued in U.S. Appl. No. 15/424,395, dated Dec. 18, 2018, pp. 1-11, United States Patent and Trademark Office, Alexandria, VA. [cited by applicant]
Final Office Action, issued in U.S. Appl. No. 14/854,657, dated Aug. 24, 2018, pp. 1-6, United States Patent and Trademark Office, Alexandria, VA. [cited by applicant]
European Office Action, issued in European Application No. 15782105.9, dated Nov. 29, 2018, pp. 1-7, European Patent Office Rijswijk, Netherlands. [cited by applicant]
European Office Action, issued in European Application No. 15 770 761.3, dated Nov. 29, 2018, pp. 1-7, European Patent Office, Rijswijk, Netherlands. [cited by applicant]
Kevin Lim et al., “Disaggregated Memory for Expansion and Sharing in Blade Servers,” International Symposium on Computer Architecture, dated Jun. 20, 2009, pp. 1-2, published by ACM, Austin, TX. [cited by applicant]
Final Office Action, issued in U.S. Appl. No. 15/800,891, dated Jul. 9, 2018, pp. 1-16, U.S. Patent and Trademark Office, Alexandria, VA. [cited by applicant]
Non-Final Office Action, issued in U.S. Appl. No. 16/050,974, dated Dec. 10, 2019, pp. 1-17, U.S. Patent and Trademark Office, Alexandria, VA. [cited by applicant]
Non-Final Office Action, issued in U.S. Appl. No. 16/050,974, dated Oct. 27, 2020, pp. 1-11, U.S. Patent and Trademark Office, Alexandria, VA. [cited by applicant]
Extended European Search Report, issued in European Patent Application No. 20207722.8, dated Dec. 4, 2020, European Patent Office, Munich, Germany. [cited by applicant]
Andreas Hollmann et al., “Invasive Computing: An Application Assisted Resource Management Approach,” Multicore Software Engineering, Performance, and Tools, dated May 31, 2012, pp. 82-85, published by Springer-Verlag, B… [cited by applicant]
Emmanuel Cecchet, “Memory Mapped Networks: a new deal for Distributed Shared Memories? The SciFS experience,” ISBN 978-0-7695-1745-2, Cluster Computing, 2002, dated Sep. 23, 2002, pp. 231-238, 2002 IEEE International Co… [cited by applicant]
Non-Final Office Action, issued in U.S. Appl. No. 16/395,329, dated Dec. 22, 2020, pp. 1-19, U.S. Patent and Trademark Office, Alexandria, VA. [cited by applicant]
Yukun Liu et al., “The Development Tutorial via UNIX Kernel Services,” UNIX Operating System, online SBN 978-3-642-20432-6, dated 2011, pp. 1-368, published online by Springer Berlin Heidelberg at URL https://doi.org/10… [cited by applicant]