IP Library Granted Patent US 12,321,597
Granted Patent B2
US 12,321,597 · App. 17/862,145 · Granted Jun 3, 2025

Method and apparatus to enable individual non volatile memory express (NVME) input/output (IO) queues on differing network addresses of an NVME controller

Inventors: James P. Freyensee (Hillsboro, OR); Phil C. Cayton (Warren, OR); Dave B. Minturn (Hillsboro, OR); Jay E. Sternberg (North Plains, OR)
G06F3/061G06F3/0655G06F3/0656G06F3/0688
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,321,597
App. No.
17/862,145
Granted
Jun 3, 2025
Kind
B2
Abstract

Methods and apparatus related to enabling individual NVMe (Non-Volatile Memory express) IO (Input Output or I/O) queues on differing network addresses of an NVMe controller are described. In one embodiment, a plurality of backend controller logic is coupled to a plurality of non-volatile memory devices. One or more virtual controller target logic (coupled to the plurality of backend controller logic) transmit data from a first portion of a plurality of IO queues to a first backend controller logic of the plurality of the backend controller logic. The one or more virtual controller target logic transmit data from a second portion of the plurality of IO queues to a second backend controller logic of the plurality of backend controller logic. Other embodiments are also disclosed and claimed.

Claims (31)

1. An apparatus comprising:

circuitry to provide a requester with access to at least one Non-Volatile Memory express (NVMe) compatible target device, wherein the at least one NVMe compatible target device is to operate consistent at least with a Non-Volatile Memory Express Specification;

circuitry to process a request from the requester to communicate with the at least one NVMe compatible target device; and

an interface to transmit the request in at least one Ethernet packet to at least one target device associated with the at least one NVMe compatible target device, wherein the request comprises an NVMe administrative command and the NVMe administrative command comprises one or more of: Create Submission/Completion IO Queues, or Identify Controller.

2. The apparatus of claim 1 , wherein the at least one NVMe compatible target device is associated with one or more physical NVMe compatible controllers associated with one or more volatile or non-volatile memory devices.

3. The apparatus of claim 1 , wherein the at least one NVMe compatible target device is identified to the requester as a single NVMe device and wherein the transmit the request in the at least one Ethernet packet to the target device associated with the at least one NVMe compatible target device comprises transmit the request to multiple NVMe devices.

4. The apparatus of claim 1 , wherein the interface is to transmit the request to the at least one NVMe compatible target device over a fabric.

5. The apparatus of claim 4 , wherein the transmit the request is compatible with one or more of: Remote Direct Memory Access (RDMA), Fibre Channel, and/or Ethernet.

6. The apparatus of claim 1 , wherein the at least one target device comprises multiple NVMe compatible target devices and wherein the interface is to transmit the request to the multiple NVMe compatible target devices.

7. The apparatus of claim 1 , wherein the at least one NVMe compatible target device is to provide access to at least one admin queue and at least one IO queue.

8. A method comprising:

providing access to at least one Non-Volatile Memory express (NVMe) compatible target device, wherein the at least one NVMe compatible target device is to operate consistent at least with a Non-Volatile Memory Express Specification;

processing a request from a requester to communicate with the at least one NVMe compatible target device; and

transmitting the request in at least one Ethernet packet to at least one target device associated with the at least one NVMe compatible target device, wherein the request comprises an NVMe administrative command and the NVMe administrative command comprises one or more of: Create Submission/Completion IO Queues, or Identify Controller.

9. The method of claim 8 , wherein the at least one NVMe compatible target device is associated with one or more physical NVMe compatible controllers associated with one or more volatile or non-volatile memory devices.

10. The method of claim 8 , wherein the transmitting the request in at least one Ethernet packet to at least one target device associated with the at least one NVMe compatible target device comprises transmitting the request in the at least one Ethernet packet to at least one target device associated with the at least one NVMe compatible target device over a fabric.

11. The method of claim 10 , wherein the transmitting the request is compatible with one or more of: Remote Direct Memory Access (RDMA), Fibre Channel, or Ethernet.

12. The method of claim 8 , wherein the at least one target device comprises multiple NVMe compatible target devices and wherein the transmitting the request comprises transmitting the request to the multiple NVMe compatible target devices.

13. The method of claim 8 , wherein the at least one NVMe compatible target device provides access to at least one admin queue and at least one IO queues.

14. A system comprising:

a host computing system comprising:

at least one memory;

at least one processor to:

execute a requester software; and

circuitry to:

provide access, to the requester software, to at least one Non-Volatile Memory express (NVMe) compatible target device, wherein the at least one NVMe compatible target device is to operate consistent at least with a Non-Volatile Memory Express Specification,

process a request from the requester software to communicate with the at least one NVMe compatible target device; and

an interface to transmit the request in at least one Ethernet packet to at least one target device associated with the at least one NVMe compatible target device, wherein the request comprises an NVMe administrative command and the NVMe administrative command comprises one or more of: Create Submission/Completion IO Queues, or Identify Controller.

15. The system of claim 14 , wherein the at least one NVMe compatible target device is associated with one or more physical NVMe compatible controllers associated with one or more volatile or non-volatile memory devices.

16. The system of claim 15 , wherein the non-volatile memory is to comprise one or more of: nanowire memory, Ferro-electric Transistor Random Access Memory (FeTRAM), Magnetoresistive Random Access Memory (MRAM), flash memory, Spin Torque Transfer Random Access Memory (STTRAM), Resistive Random Access Memory, byte addressable 3-Dimensional Cross Point Memory, PCM (Phase Change Memory), and volatile memory backed by a power reserve to retain data during power failure or power disruption.

17. The system of claim 14 , wherein the interface is to transmit the request to the at least one NVMe compatible target device over a fabric and the transmit the request is compatible with one or more of: Remote Direct Memory Access (RDMA), Fibre Channel, or Ethernet.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 11, 2025
From: INTEL CORPORATION
To: SK HYNIX NAND PRODUCT SOLUTIONS CORP. (DBA SOLIDIGM)
Reel/Frame 072549/0289 →
Continuity (3)
Continuation 16381994 · Apr 11, 2019
Continuation 14976949 · Dec 21, 2015
Related Publication 20220350483A1 · Nov 3, 2022
References Cited (136)
US 5396637A · Harwell et al. · 1995 [cited by applicant]
US 5603038A · Crump et al. · 1997 [cited by applicant]
US 5809526A · Patel · 1998 [cited by applicant]
US 5860083A · Sukegawa · 1999 [cited by applicant]
US 5978924A · Ahn · 1999 [cited by applicant]
US 6393518B2 · Koivuniemi · 2002 [cited by applicant]
US 6523102B1 · Dye et al. · 2003 [cited by applicant]
US 6535996B1 · Brewer et al. · 2003 [cited by applicant]
US 6968424B1 · Danilak · 2005 [cited by applicant]
US 7146455B2 · Cheng · 2006 [cited by applicant]
US 7260695B2 · Batchelor et al. · 2007 [cited by applicant]
US 7437597B1 · Kruckemyer et al. · 2008 [cited by applicant]
US 7653699B1 · Colgrove et al. · 2010 [cited by applicant]
US 7716525B1 · Buchko et al. · 2010 [cited by applicant]
US 7802145B1 · Bainbridge et al. · 2010 [cited by applicant]
US 7954006B1 · Mangipudi · 2011 [cited by applicant]
US 8069309B1 · Long · 2011 [cited by applicant]
US 8325554B2 · Sweere et al. · 2012 [cited by applicant]
US 8510598B2 · Vedder et al. · 2013 [cited by applicant]
US 8612676B2 · Dahlen et al. · 2013 [cited by applicant]
US 8966164B1 · Asnaashari et al. · 2015 [cited by applicant]
US 10013168B2 · Cayton et al. · 2018 [cited by applicant]
US 10176116B2 · Sternberg et al. · 2019 [cited by applicant]
US 10200376B2 · Cayton et al. · 2019 [cited by applicant]
US 10620840B2 · Freyensee et al. · 2020 [cited by applicant]
US 10893050B2 · Cayton et al. · 2021 [cited by applicant]
US 10970231B2 · Sternberg et al. · 2021 [cited by applicant]
US 20020095602A1 · Pherson et al. · 2002 [cited by applicant]
US 20030191734A1 · Voigt · 2003 [cited by applicant]
US 20060015683A1 · Ashmore et al. · 2006 [cited by applicant]
US 20060126833A1 · O'Leary et al. · 2006 [cited by applicant]
US 20060138231A1 · Elberbaum · 2006 [cited by applicant]
US 20060274566A1 · Takashima et al. · 2006 [cited by applicant]
US 20070124604A1 · Feldstein et al. · 2007 [cited by applicant]
US 20070229521A1 · Li et al. · 2007 [cited by applicant]
US 20070294543A1 · Chiang · 2007 [cited by applicant]
US 20090172372A1 · Kumar et al. · 2009 [cited by applicant]
US 20090182962A1 · Khmelnitsky et al. · 2009 [cited by applicant]
US 20090254705A1 · Abali et al. · 2009 [cited by applicant]
US 20090271563A1 · Gopalan et al. · 2009 [cited by applicant]
US 20090313416A1 · Nation · 2009 [cited by applicant]
US 20100011155A1 · Kitagawa · 2010 [cited by applicant]
US 20100011261A1 · Cagno et al. · 2010 [cited by applicant]
US 20100080057A1 · Reuter et al. · 2010 [cited by applicant]
US 20100095053A1 · Bruce et al. · 2010 [cited by applicant]
US 20100114889A1 · Rabii et al. · 2010 [cited by applicant]
US 20100198972A1 · Umbehocker · 2010 [cited by applicant]
US 20100202240A1 · Moshayedi et al. · 2010 [cited by applicant]
US 20110161748A1 · Casper et al. · 2011 [cited by applicant]
US 20110231594A1 · Sugimoto et al. · 2011 [cited by applicant]
US 20110321123A1 · Ishikawa · 2011 [cited by applicant]
US 20120047314A1 · Chen et al. · 2012 [cited by applicant]
US 20120096332A1 · Furno et al. · 2012 [cited by applicant]
US 20120166891A1 · Dahlen et al. · 2012 [cited by applicant]
US 20120198136A1 · Moshayedi et al. · 2012 [cited by applicant]
US 20120271990A1 · Chen et al. · 2012 [cited by applicant]
US 20130019048A1 · Bland et al. · 2013 [cited by applicant]
US 20130019076A1 · Amidi et al. · 2013 [cited by applicant]
US 20140068200A1 · Schnapp et al. · 2014 [cited by applicant]
US 20140189203A1 · Suzuki et al. · 2014 [cited by applicant]
US 20140201562A1 · Breakstone et al. · 2014 [cited by applicant]
US 20140281040A1 · Liu · 2014 [cited by applicant]
US 20140304461A1 · Kitahara et al. · 2014 [cited by applicant]
US 20150089287A1 · Jayakumar et al. · 2015 [cited by applicant]
US 20150095554A1 · Asnaashari et al. · 2015 [cited by applicant]
US 20150134880A1 · Danilak et al. · 2015 [cited by applicant]
US 20150143134A1 · Hashimoto · 2015 [cited by applicant]
US 20150212734A1 · Asnaashari · 2015 [cited by examiner]
US 20150248366A1 · Bergsten et al. · 2015 [cited by applicant]
US 20150319237A1 · Hussain et al. · 2015 [cited by applicant]
US 20150319243A1 · Hussain et al. · 2015 [cited by applicant]
US 20160085718A1 · Huang · 2016 [cited by applicant]
US 20160188414A1 · Jayakumar et al. · 2016 [cited by applicant]
US 20170123656A1 · Benisty et al. · 2017 [cited by applicant]
US 20170177216A1 · Freyensee · 2017 [cited by examiner]
US 20170185310A1 · Cayton et al. · 2017 [cited by applicant]
US 20180004559A1 · Geml et al. · 2018 [cited by applicant]
US 20180088978A1 · Li et al. · 2018 [cited by applicant]
US 20180188974A1 · Cayton et al. · 2018 [cited by applicant]
US 20180260148A1 · Klein · 2018 [cited by applicant]
US 20210311882A1 · Sternberg et al. · 2021 [cited by applicant]
CN 104572516A · 2015 [cited by applicant]
CN 102959525A · 2016 [cited by applicant]
KR 20120015313A · 2012 [cited by applicant]
KR 20140142530A · 2014 [cited by applicant]
TW 225589B · 2004 [cited by applicant]
TW 201118546A · 2011 [cited by applicant]
WO 2003060716A1 · 2003 [cited by applicant]
WO 2006039711A1 · 2006 [cited by applicant]
WO 2015041698A1 · 2015 [cited by applicant]
WO 2015080690A1 · 2015 [cited by applicant]
WO 2016105814A1 · 2016 [cited by applicant]
WO 2017112021A1 · 2017 [cited by applicant]
WO 2017112285A1 · 2017 [cited by applicant]
WO 2018038840A2 · 2018 [cited by applicant]
Korean and English Translation of Korean First Office Action for Patent Application No. 10-2018-7014348, Mailed Mar. 2, 2023, 7 pages. [cited by applicant]
“Intel Xeon Processor C550/C3500 Series-based Platform for Storage Applications” 2010, 6 pages, Intel Corporation, USA. [cited by applicant]
“NVM Express” retrieved from http://www.nvmexpress.org on Nov. 8, 2015, 2 pages. [cited by applicant]
“PCI Express Base Specification”, Specification, Nov. 10, 2010, 860 pages, Revision 3.0, PCI-SIG, Beaverton, OR. [cited by applicant]
Advisory Action for U.S. Appl. No. 14/976,949, Mailed May 3, 2018, 4 pages. [cited by applicant]
Christie, M., et al., “Red Hat Enterprise Linux 5 Online Storage Reconfiguration Guide”, Red Hat Enterprise Linux 5, Edition 1, © 2008, 43 pp. [cited by applicant]
Final Office Action for U.S. Appl. No. 14/976,949, Mailed Dec. 13, 2017, 28 pages. [cited by applicant]
First Chinese Office Action of P.R. China State Intellectual Property Office for Patent Application No. 201680067899.6, Mailed May 19, 2021, 11 pages. [cited by applicant]
First Office Action for U.S. Appl. No. 14/976,949, Mailed Jun. 16, 2017, 25 pages. [cited by applicant]
First Office Action for U.S. Appl. No. 16/381,994, Mailed Oct. 28, 2021, 26 pages. [cited by applicant]
International Preliminary Report on Patentability for International Application No. PCT/US2017/029982, dated Dec. 6, 2018, 11 pp (77.346PCT (IPRP). [cited by applicant]
International Preliminary Report on Patentability received for International Application No. PCT/US2011/061466, dated Jun. 25, 2013, 5 pages. [cited by applicant]
International Preliminary Report on Patentability received for International Application No. PCT/US2015/062555, dated Jul. 6, 2017, 8 pages. [cited by applicant]
International Preliminary Report on Patentability, received for International Patent Application No. PCT/US2013/061188, dated Mar. 29, 2016, 6 pages. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2017/029982, dated Jul. 28, 2017, 14 pp. (77.346PCT (ISR & WO). [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2017/043568, dated Oct. 20, 2017, 15 pp. [cited by applicant]
International Search Report and Written Opinion received for International Application No. PCT/US2011/061466, dated Jul. 31, 2012, 7 pages. [cited by applicant]
International Search Report and Written Opinion received for International Application No. PCT/US2013/061188, dated Jun. 20, 2014, 10 pages. [cited by applicant]
International Search Report and Written Opinion received for International Application No. PCT/US2015/062555, dated Mar. 15, 2016, 11 pages. [cited by applicant]
International Search Report and Written Opinion received for International Application No. PCT/US2016/055226, dated Jan. 19, 2017, 13 pages. [cited by applicant]
International Search Report and Written Opinion received for International Application No. PCT/US2016/063551, dated Mar. 13, 2017, 11 pages. [cited by applicant]
Marks, K., “An NVM Express Tutorial”, Flash Memory Summit 2013, 92 pp. [cited by applicant]
Minturn, D., “NVM Express Over Fabrics”, Mar. 15-18, 2015, NVM Express, Total 13 pp. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 16/381,994, Mailed Mar. 9, 2022, 10 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 16/240,705, Mailed Dec. 1, 2020, 10 pages. [cited by applicant]
Notice of Allowance received for U.S. Appl. No. 12/976,545, dated Aug. 15, 2013, 7 pages. [cited by applicant]
NVM Express, Inc., “NVM Express Over Fabrics”, Revision 1.0, Jun. 5, 2106, © 2007-2016, Total 49 pp. [cited by applicant]
NVM Express, Inc., “NVM Express”, Revision 1.2a, Oct. 23, 2015, © 2007-2015, Total 209 pp. [cited by applicant]
Office Action and Search Report received for Taiwanese Patent Application No. 104138741, dated Oct. 19, 2016, 8 bages including 1 page of English translation. [cited by applicant]
Office Action received for Taiwanese Patent Application No. 104138741, dated Mar. 16, 2017, 5 pages. [cited by applicant]
Office Action received for U.S. Appl. No. 12/976,545, dated Apr. 12, 2013, 8 pages. [cited by applicant]
Office Action received for U.S. Appl. No. 12/976,545, dated Dec. 5, 2012, 9 pages. [cited by applicant]
Office Action received for U.S. Appl. No. 14/127,548, dated Jul. 17, 2015, 9 pages. [cited by applicant]
Office Action received for U.S. Appl. No. 14/583,037, dated Feb. 24, 2017, 7 pages. [cited by applicant]
Office Action received for U.S. Appl. No. 14/583,037, dated May 19, 2016, 6 pages. [cited by applicant]
Office Action received for U.S. Appl. No. 14/583,037, dated Nov. 4, 2016, 6 pages. [cited by applicant]
Onufryk, P., “What's New in NVMe 1.1 and Future Directions”, NVM Express, Flash Memory Summit 2013, Santa Clara, CA, Total 26 pp. [cited by applicant]
Sardella, S., “NVMe Over Fabrics Agenda”, Aug. 11, 2015, NVM Express, Total 14 pp. [cited by applicant]
Second Chinese Office Action of P.R. China State Intellectual Property Office for Patent Application No. 201680067899.6, Mailed Oct. 15, 2021, 3 pages. [cited by applicant]
Second Office Action for U.S. Appl. No. 14/976,949, Mailed Jun. 4, 2018, 29 pages. [cited by applicant]
Wikipedia, “Internet Storage Name Service”, [online], last modified on Aug. 12, 2016, [Retrieved on Sep. 18, 2016], retrieved from the Internet at , 2 pp. [cited by applicant]