Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the shutdown order reported in #426.
Summary
admin_clideclared its clients first and registered theIBManager::stop()guard after them(
storageClienton line 100, the guard on line 111), so at scope exit the guard ran first:IBManager::stop()executed while the clients - and their RDMA queue pairs - were still alive, andthe clients were destroyed afterwards. For any admin command that touches the storage client
(
create-target,remove-target, ...) the storage-facing RC QP was therefore destroyed withoutcloseGracefully(): the storage side kept an orphan QP and only noticed it when its periodicWRType::CHECKexhausted the RC retry budget.This moves the guard above the client declarations, so it is registered first and therefore runs
last: clients first,
IBManager::stop()last. That is the ordersrc/fuse/hf3fs_fuse.cppalreadyuses (the
IBManagerguard is registered right afterIBManager::start(), theSCOPE_EXIT { d.stop(); }for the clients later), and the orderFuseClients::stop()implements explicitly (meta -> storage -> mgmtd -> net client).
Root cause
Before:
Automatic objects are destroyed in reverse order of declaration, so the guard registered at line
111 was destroyed before
storageClientat line 100.IBManager::stop()->IBManager::reset()stops and resetssocketManager_. When the storage client is finallydestroyed,
StorageClientImpl::~StorageClientImpl()->stop()->messenger_.stopAndJoin()->IOWorker::stopAndJoin()->dropConnections()destroys the transports, andTransport::~Transport()callsIBManager::close(). WithsocketManager_ == nullptr,IBManager::closeImpl()just lets theIBSocketdie, andIBSocket::~IBSocket()only moves theQP to
IBV_QPS_ERR: theImmData::close()message is never posted, so the peer never learns thatthe connection is gone.
The other clients were already fine:
mgmtdClient->stop()andclient->stopAndJoin()run whileIBManager is still alive, which is why the mgmtd QP closed gracefully in the report while the
storage-facing QP did not.
Changes
src/client/bin/admin_cli.cc: moveSCOPE_EXIT { hf3fs::net::IBManager::stop(); };above theclient declarations, with a comment explaining why the position matters. One line moved, no API
change, no other binary touched.
Why this is safe
IBManagerwas never started;IBManager::stop()->reset()is thena no-op on default-constructed members.
--release_versionearly return still happens before the guard is registered, so that pathis unchanged.
Monitor::stop()still runs before the clients are destroyed, exactly as before.IBSocketcould outliveIBManager::reset()(socket manager and eventloop already gone) is closed. The storage QPs are now dropped while the IB event loop is still
running, so
IBSocketManager::stopAndJoin()- called fromIBManager::stop()- still has itsdrain window to push the close message out.
What was not validated
This is a static analysis. I could not build or run 3FS (no RDMA hardware), so the ordering is
confirmed from the source and from the runtime log in #426, but the effect on a live fabric was
not measured.
The retry horizon quoted in #426 (~1764 s) is not reproducible from the shipped configuration:
retry_cnt = 7andtimeout = 14inconfigs/admin_cli.tomlandconfigs/storage_main.tomlimply sub-second RC retry exhaustion, and
check_connections_intervaldefaults to 60 s, whichwould surface the orphan QP within about a minute. That deployment must override these values.
The defect and this fix do not depend on the exact horizon.
Possible follow-up (not in this patch):
IBManager::closeImpl()currently drops a socket silentlywhen
socketManager_ == nullptr; logging or rejecting a not-closed/READY socket there would makethis class of shutdown-order bug self-reporting.
Prepared with AI assistance; reviewed before submission.