Repository navigation
KeyError not caught when popping a connection from pool.idle #827
Description
Hello,
With the latest version of the client, I have connection issues due to the aboved mentioned bug:
The try / except statement here bugs if pool.active is also empty.
I have trouble tracing the issue, but I have the impression that there is a circular call somewhere. I gather from looking at the code that it should be possible to access the module version without an authentication, but the authenticated method to get the code version from the auth module is called within the except statement if the idle pool is empty, as shown in this traceback after a connection error:
Details
Error connecting to irods for object ndc80gfp cdc11cfp_CFP.tif
Traceback (most recent call last):
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/pool.py", line 86, in get_connection
conn = self.idle.pop()
^^^^^^^^^^^^^^^
KeyError: 'pop from an empty set'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/omero-quay/src/omero_quay/irods/data.py", line 212, in _exists
return super()._exists(isaobject)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/omero-quay/src/omero_quay/core/interface.py", line 582, in _exists
if localobject := self._find_by_id(isaobject):
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/omero-quay/src/omero_quay/irods/data.py", line 520, in _find_by_id
.all()
^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/query.py", line 227, in all
result_set = self.execute()
^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/query.py", line 200, in execute
with self.sess.pool.get_connection() as conn:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/pool.py", line 18, in method_
ret = method(self, *s, **kw)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/pool.py", line 109, in get_connection
conn = Connection(self, self.account)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/connection.py", line 109, in __init__
auth_module.login(self, **self.auth_options)
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/auth/native.py", line 24, in login
_authenticate_native(conn, req=opt)
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/auth/native.py", line 40, in _authenticate_native
_native_ClientAuthState(conn, scheme=_scheme).authenticate_client(
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/auth/__init__.py", line 189, in authenticate_client
resp = self.call(next_operation, to_send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/auth/__init__.py", line 177, in call
resp = func(request)
^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/auth/native.py", line 112, in native_auth_client_response
resp = _auth_api_request(self.conn, server_req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/auth/__init__.py", line 149, in _auth_api_request
response = conn.recv()
^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/connection.py", line 188, in recv
raise exc
irods.exception.SYS_LIBRARY_ERROR: None
CRITICAL irods.connection:connection.py:174 [Errno 104] Connection reset by peer
ERROR irods.connection:connection.py:175 Could not receive server response
ERROR omero_quay.iRODSData.man_e88f1a22:data.py:214 Error connecting to irods for object license.txt
Traceback (most recent call last):
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/pool.py", line 86, in get_connection
conn = self.idle.pop()
^^^^^^^^^^^^^^^
KeyError: 'pop from an empty set'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/connection.py", line 166, in recv
msg = iRODSMessage.recv(self.socket)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/message/__init__.py", line 346, in recv
rsp_header_size = _recv_message_in_len(sock, 4)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/message/__init__.py", line 225, in _recv_message_in_len
buf = sock.recv(size_left, socket.MSG_WAITALL)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/omero-quay/src/omero_quay/irods/data.py", line 212, in _exists
return super()._exists(isaobject)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/omero-quay/src/omero_quay/core/interface.py", line 582, in _exists
if localobject := self._find_by_id(isaobject):
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/omero-quay/src/omero_quay/irods/data.py", line 520, in _find_by_id
.all()
^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/query.py", line 227, in all
result_set = self.execute()
^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/query.py", line 200, in execute
with self.sess.pool.get_connection() as conn:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/pool.py", line 18, in method_
ret = method(self, *s, **kw)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/pool.py", line 109, in get_connection
conn = Connection(self, self.account)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/connection.py", line 109, in __init__
auth_module.login(self, **self.auth_options)
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/auth/native.py", line 24, in login
_authenticate_native(conn, req=opt)
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/auth/native.py", line 40, in _authenticate_native
_native_ClientAuthState(conn, scheme=_scheme).authenticate_client(
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/auth/__init__.py", line 189, in authenticate_client
resp = self.call(next_operation, to_send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/auth/__init__.py", line 177, in call
resp = func(request)
^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/auth/native.py", line 71, in native_auth_client_request
resp = _auth_api_request(self.conn, server_req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/auth/__init__.py", line 149, in _auth_api_request
response = conn.recv()
^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/connection.py", line 177, in recv
raise NetworkException("Could not receive server response")
irods.exception.NetworkException: Could not receive server response
CRITICAL irods.connection:connection.py:174 [Errno 104] Connection reset by peer
ERROR irods.connection:connection.py:175 Could not receive server response
ERROR omero_quay.iRODSData.man_e88f1a22:data.py:214 Error connecting to irods for object ndc80gfp cdc11cfp_GFP.tif
Traceback (most recent call last):
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/pool.py", line 86, in get_connection
conn = self.idle.pop()
^^^^^^^^^^^^^^^
KeyError: 'pop from an empty set'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/connection.py", line 166, in recv
msg = iRODSMessage.recv(self.socket)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/message/__init__.py", line 346, in recv
rsp_header_size = _recv_message_in_len(sock, 4)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/message/__init__.py", line 225, in _recv_message_in_len
buf = sock.recv(size_left, socket.MSG_WAITALL)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/omero-quay/src/omero_quay/irods/data.py", line 212, in _exists
return super()._exists(isaobject)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/omero-quay/src/omero_quay/core/interface.py", line 582, in _exists
if localobject := self._find_by_id(isaobject):
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/omero-quay/src/omero_quay/irods/data.py", line 520, in _find_by_id
.all()
^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/query.py", line 227, in all
result_set = self.execute()
^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/query.py", line 200, in execute
with self.sess.pool.get_connection() as conn:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/pool.py", line 18, in method_
ret = method(self, *s, **kw)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/pool.py", line 109, in get_connection
conn = Connection(self, self.account)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/connection.py", line 109, in __init__
auth_module.login(self, **self.auth_options)
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/auth/native.py", line 24, in login
_authenticate_native(conn, req=opt)
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/auth/native.py", line 40, in _authenticate_native
_native_ClientAuthState(conn, scheme=_scheme).authenticate_client(
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/auth/__init__.py", line 189, in authenticate_client
resp = self.call(next_operation, to_send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/iroods/auth/__init__.py", line 177, in call
resp = func(request)
^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/auth/native.py", line 71, in native_auth_client_request
resp = _auth_api_request(self.conn, server_req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/auth/__init__.py", line 149, in _auth_api_request
response = conn.recv()
^^^^^^^^^^^
File "/home/omero/.conda/envs/quay/lib/python3.12/site-packages/irods/connection.py", line 177, in recv
raise NetworkException("Could not receive server response")
irods.exception.NetworkException: Could not receive server response
...I'm opening a PR with a proposed fix :)
Best,
Guillaume
Activity
d-w-moore commentedon Sep 2, 2026
@glyg - In the original problem description, I see what appears to be three separate error dumps concatenated into the same file. I could be wrong, but that is what it looks like. If true, then there probably is no looping behavior happening....
If this is a misapprehension on my part and all lines in that excerpt are actually meant to be part of a whole, cohesive error dump, let me know.
korydraughn commentedon Sep 2, 2026
From #828 (comment):
trel commentedon Sep 14, 2026
New/Correct related issue is at #831
korydraughn commentedon Sep 14, 2026
We'll use this issue for the documentation effort.
Reopening.
korydraughn commentedon Sep 14, 2026
The PR for the documentation effort is #832.