Skip to content

Fix admin_cli shutdown order: stop IBManager after the clients - #429

Open
PerryLink wants to merge 1 commit into
deepseek-ai:mainfrom
PerryLink:fix/admin-cli-ibmanager-shutdown-order
Open

PerryLink wants to merge 1 commit into
deepseek-ai:mainfrom
PerryLink:fix/admin-cli-ibmanager-shutdown-order

Conversation

@PerryLink

Copy link
Copy Markdown

Fixes the shutdown order reported in #426.

Summary

admin_cli declared its clients first and registered the IBManager::stop() guard after them
(storageClient on line 100, the guard on line 111), so at scope exit the guard ran first:
IBManager::stop() executed while the clients - and their RDMA queue pairs - were still alive, and
the clients were destroyed afterwards. For any admin command that touches the storage client
(create-target, remove-target, ...) the storage-facing RC QP was therefore destroyed without
closeGracefully(): the storage side kept an orphan QP and only noticed it when its periodic
WRType::CHECK exhausted the RC retry budget.

This moves the guard above the client declarations, so it is registered first and therefore runs
last: clients first, IBManager::stop() last. That is the order src/fuse/hf3fs_fuse.cpp already
uses (the IBManager guard is registered right after IBManager::start(), the
SCOPE_EXIT { d.stop(); } for the clients later), and the order FuseClients::stop()
implements explicitly (meta -> storage -> mgmtd -> net client).

Root cause

Before:

std::shared_ptr<StorageClient> storageClient;    // line 100
...
SCOPE_EXIT { hf3fs::net::IBManager::stop(); };   // line 111

Automatic objects are destroyed in reverse order of declaration, so the guard registered at line
111 was destroyed before storageClient at line 100. IBManager::stop() ->
IBManager::reset() stops and resets socketManager_. When the storage client is finally
destroyed, StorageClientImpl::~StorageClientImpl() -> stop() -> messenger_.stopAndJoin() ->
IOWorker::stopAndJoin() -> dropConnections() destroys the transports, and
Transport::~Transport() calls IBManager::close(). With socketManager_ == nullptr,
IBManager::closeImpl() just lets the IBSocket die, and IBSocket::~IBSocket() only moves the
QP to IBV_QPS_ERR: the ImmData::close() message is never posted, so the peer never learns that
the connection is gone.

The other clients were already fine: mgmtdClient->stop() and client->stopAndJoin() run while
IBManager is still alive, which is why the mgmtd QP closed gracefully in the report while the
storage-facing QP did not.

Changes

src/client/bin/admin_cli.cc: move SCOPE_EXIT { hf3fs::net::IBManager::stop(); }; above the
client declarations, with a comment explaining why the position matters. One line moved, no API
change, no other binary touched.

Why this is safe

  • If no command needed IB, IBManager was never started; IBManager::stop() -> reset() is then
    a no-op on default-constructed members.
  • The --release_version early return still happens before the guard is registered, so that path
    is unchanged.
  • Monitor::stop() still runs before the clients are destroyed, exactly as before.
  • The window in which an IBSocket could outlive IBManager::reset() (socket manager and event
    loop already gone) is closed. The storage QPs are now dropped while the IB event loop is still
    running, so IBSocketManager::stopAndJoin() - called from IBManager::stop() - still has its
    drain window to push the close message out.
  • Nothing else in this file depends on IBManager being stopped early.

What was not validated

This is a static analysis. I could not build or run 3FS (no RDMA hardware), so the ordering is
confirmed from the source and from the runtime log in #426, but the effect on a live fabric was
not measured.

The retry horizon quoted in #426 (~1764 s) is not reproducible from the shipped configuration:
retry_cnt = 7 and timeout = 14 in configs/admin_cli.toml and configs/storage_main.toml
imply sub-second RC retry exhaustion, and check_connections_interval defaults to 60 s, which
would surface the orphan QP within about a minute. That deployment must override these values.
The defect and this fix do not depend on the exact horizon.

Possible follow-up (not in this patch): IBManager::closeImpl() currently drops a socket silently
when socketManager_ == nullptr; logging or rejecting a not-closed/READY socket there would make
this class of shutdown-order bug self-reporting.

Prepared with AI assistance; reviewed before submission.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant