nvme_driver: split the mesh channel to the queue handler between commands and requests (#2708)#2723
Merged
mattkur merged 1 commit intomicrosoft:release/1.7.2511from Feb 2, 2026
Conversation
…ands and requests (microsoft#2708) Currently, in QueueHandler::poll_fn, we register a waker with the receiving channel only when there is space available in the submission queue (SQ). This channel multiplexes two kinds of traffic: requests that write to the submission queue, and control‑plane requests for the handler. When the submission queue is full, poll_fn does not register a waker with recv. Instead, it relies entirely on incoming completions and interrupts to wake the task. As a result, if completions are slow, not only is command submission affected, but control‑plane operations such as Save, Inspect, and NextAen are also delayed—even though they are unrelated to SQ writes. With this change, the run loop now listens on three separate channels: 1. A command channel for requests that write to the submission queue 2. A control‑plane channel for operations like Save and Inspect 3. A completion channel This separation ensures that control‑plane operations can continue to make progress even when the submission queue is full and completions are slow. In particular, Save() should no longer be blocked by SQ backpressure. Note: I am still working on a VMM test to validate behavior when the submission queue is full. (cherry picked from commit 86a1cba)
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors the NVMe queue pair’s internal messaging so that control‑plane operations (like Save and Inspect) are no longer backpressured by a full submission queue or slow completions, improving responsiveness under load.
Changes:
- Splits the previous single
meshchannel betweenQueuePairandQueueHandlerinto two channels: one for control‑plane requests (Req) and one for data‑plane commands (Cmd), updatingQueueHandlerLoopandIssueraccordingly. - Refactors
QueueHandler::runto poll the command channel only when there is SQ and pending‑command capacity, while always polling the control‑plane channel and giving it priority over completions. - Updates inspection wiring (
#[inspect(flatten, with = ...)]) and save/restore paths to use the new control‑plane channel (send_req/recv_req) without changing external APIs.
| let event = if !self.drain_after_restore { | ||
| // Normal processing of the requests and completions. | ||
| poll_fn(|cx| { | ||
| // Look for NVME commands |
There was a problem hiding this comment.
Comment uses the acronym "NVME" while the rest of this module and NVMe spec consistently use "NVMe"; please fix the casing here for consistency (e.g., "NVMe commands").
Suggested change
| // Look for NVME commands | |
| // Look for NVMe commands |
mattkur
approved these changes
Feb 2, 2026
Contributor
mattkur
left a comment
There was a problem hiding this comment.
Has been tested in production and in local vmm_tests (that change is still iterating in main).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Clean cherry pick of PR #2708
Currently, in QueueHandler::poll_fn, we register a waker with the receiving channel only when there is space available in the submission queue (SQ). This channel multiplexes two kinds of traffic: requests that write to the submission queue, and control‑plane requests for the handler.
When the submission queue is full, poll_fn does not register a waker with recv. Instead, it relies entirely on incoming completions and interrupts to wake the task. As a result, if completions are slow, not only is command submission affected, but control‑plane operations such as Save, Inspect, and NextAen are also delayed—even though they are unrelated to SQ writes.
With this change, the run loop now listens on three separate channels:
This separation ensures that control‑plane operations can continue to make progress even when the submission queue is full and completions are slow. In particular, Save() should no longer be blocked by SQ backpressure.
Note: I am still working on a VMM test to validate behavior when the submission queue is full.