SDA database reports that ingestion was completed succesfully, but the submission service did not report ingestion as completed.

[KarlG-nbis]

Describe the bug

Using version: 2026.6.0 of the metadata-submitter

There is a dataset which was published successfully, and ingestion, dataset creation, etc went all good on the sda side, it was added to rems and can be downloaded succesfully. But in the metadata-submitter database:

SELECT * FROM submissions WHERE submission_id = 'aa-dataset-nzrd4e-jycyhc';

=>

aa-dataset-nzrd4e-jycyhc,Random MnEgFoYQjv,878edf4064341dc5297ef4c127a373241a8bbfc3@lifescience-ri.eu,,Bigpicture,Random data dataset,A random dataset,2026-07-14 11:19:08.630407 +00:00,2026-07-14 11:19:46.972237 +00:00,true,false,2026-07-14 11:19:46.967555 +00:00,,"{""name"": ""Random MnEgFoYQjv"", ""rems"": {""workflowId"": 36, ""organizationId"": ""Synthetic Org""}, ""title"": ""Random data dataset"", ""workflow"": ""Bigpicture"", ""projectId"": ""878edf4064341dc5297ef4c127a373241a8bbfc3@lifescience-ri.eu"", ""description"": ""A random dataset""}"

Main concern:

and in the files table:

SELECT * FROM files WHERE submission_id = 'aa-dataset-nzrd4e-jycyhc';

=>

aa-file-xfskn9-v5pa2h,aa-dataset-nzrd4e-jycyhc,,DATASET_aa-dataset-nzrd4e-jycyhc/METADATA/dataset.xml.c4gh,1053,,,,error,External service error: admin,transient_error,1,2026-07-14 11:19:40.504892 +00:00,2026-07-14 11:24:15.386590 +00:00
aa-file-jvfb8m-9b2zgm,aa-dataset-nzrd4e-jycyhc,,DATASET_aa-dataset-nzrd4e-jycyhc/METADATA/policy.xml.c4gh,2857,,,,error,External service error: admin,transient_error,1,2026-07-14 11:19:40.981948 +00:00,2026-07-14 11:24:18.608311 +00:00
aa-file-ypcqkb-p4vzfy,aa-dataset-nzrd4e-jycyhc,,DATASET_aa-dataset-nzrd4e-jycyhc/METADATA/image.xml.c4gh,15456,,,,error,External service error: admin,transient_error,1,2026-07-14 11:19:41.929227 +00:00,2026-07-14 11:24:21.732372 +00:00
aa-file-n6jmtt-shvhq8,aa-dataset-nzrd4e-jycyhc,,DATASET_aa-dataset-nzrd4e-jycyhc/METADATA/annotation.xml.c4gh,3140,,,,error,External service error: admin,transient_error,1,2026-07-14 11:19:42.916615 +00:00,2026-07-14 11:24:24.863434 +00:00
aa-file-cgc58g-5pw9zr,aa-dataset-nzrd4e-jycyhc,,DATASET_aa-dataset-nzrd4e-jycyhc/METADATA/observation.xml.c4gh,1246,,,,error,External service error: admin,transient_error,1,2026-07-14 11:19:43.485468 +00:00,2026-07-14 11:24:27.996934 +00:00
aa-file-p3ryfj-2msnnv,aa-dataset-nzrd4e-jycyhc,,DATASET_aa-dataset-nzrd4e-jycyhc/METADATA/observer.xml.c4gh,293,,,,error,External service error: admin,transient_error,1,2026-07-14 11:19:43.984085 +00:00,2026-07-14 11:24:31.118755 +00:00
aa-file-nupznp-4yrgxf,aa-dataset-nzrd4e-jycyhc,aa-annotation-ktdp5g-bsrf33,DATASET_aa-dataset-nzrd4e-jycyhc/ANNOTATIONS/annotation_bIiLxpPgLj.geojson.c4gh,,SHA256,eeb41e4f84a01c354d96a7d697b13618497058e890c455b12b97ad35420746ba,e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,error,External service error: admin,transient_error,1,2026-07-14 11:19:09.000678 +00:00,2026-07-14 11:24:34.241882 +00:00
aa-file-fxttuw-p8q2g6,aa-dataset-nzrd4e-jycyhc,aa-annotation-jefwa9-ggwnw3,DATASET_aa-dataset-nzrd4e-jycyhc/ANNOTATIONS/annotation_yaACOLRMJG.geojson.c4gh,,SHA256,eeb41e4f84a01c354d96a7d697b13618497058e890c455b12b97ad35420746ba,e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,error,External service error: admin,transient_error,1,2026-07-14 11:19:09.006226 +00:00,2026-07-14 11:24:37.356274 +00:00
aa-file-cfa6kd-wxukav,aa-dataset-nzrd4e-jycyhc,,DATASET_aa-dataset-nzrd4e-jycyhc/METADATA/sample.xml.c4gh,4835,,,,error,External service error: admin,transient_error,1,2026-07-14 11:19:44.613833 +00:00,2026-07-14 11:24:53.025450 +00:00
aa-file-5qyrfm-954r3z,aa-dataset-nzrd4e-jycyhc,,DATASET_aa-dataset-nzrd4e-jycyhc/METADATA/staining.xml.c4gh,778,,,,error,External service error: admin,transient_error,1,2026-07-14 11:19:45.341789 +00:00,2026-07-14 11:24:56.319040 +00:00
aa-file-44sukk-9d4ca6,aa-dataset-nzrd4e-jycyhc,,DATASET_aa-dataset-nzrd4e-jycyhc/LANDING_PAGE/landing_page.xml.c4gh,7735,,,,error,External service error: admin,transient_error,1,2026-07-14 11:19:46.120125 +00:00,2026-07-14 11:24:59.394001 +00:00
aa-file-ew9sgj-yuen6g,aa-dataset-nzrd4e-jycyhc,,DATASET_aa-dataset-nzrd4e-jycyhc/LANDING_PAGE/THUMBNAILS/1.jpg.c4gh,,,,,error,External service error: admin,transient_error,1,2026-07-14 11:19:46.340589 +00:00,2026-07-14 11:25:02.887080 +00:00
aa-file-e44exa-8yk68g,aa-dataset-nzrd4e-jycyhc,aa-image-uhwm4d-hq76fc,DATASET_aa-dataset-nzrd4e-jycyhc/IMAGES/IMAGE_bIiLxpPgLj/bIiLxpPgLj_1.dcm.c4gh,,SHA256,d63ccaa66fc4d565d5bfd846f82191af4204e9b162aea6dc311d353614f3210c,e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,error,External service error: admin,transient_error,1,2026-07-14 11:19:08.882999 +00:00,2026-07-14 11:25:05.964619 +00:00
aa-file-2tmnva-9wn8zf,aa-dataset-nzrd4e-jycyhc,aa-image-uhwm4d-hq76fc,DATASET_aa-dataset-nzrd4e-jycyhc/IMAGES/IMAGE_bIiLxpPgLj/bIiLxpPgLj_2.dcm.c4gh,,SHA256,42184f289044c0050bd3331f074bf3a5690ad0158e98c437b72cdbf968549219,e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,error,External service error: admin,transient_error,1,2026-07-14 11:19:08.909382 +00:00,2026-07-14 11:25:09.064881 +00:00
aa-file-yh2e85-f2y8vw,aa-dataset-nzrd4e-jycyhc,aa-image-uhwm4d-hq76fc,DATASET_aa-dataset-nzrd4e-jycyhc/IMAGES/IMAGE_bIiLxpPgLj/bIiLxpPgLj_3.dcm.c4gh,,SHA256,5544f8e90ec2362d0f047a9a966d023219be87b27115ddc4fec62f12750e61d6,e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,error,External service error: admin,transient_error,1,2026-07-14 11:19:08.928069 +00:00,2026-07-14 11:25:12.160140 +00:00
aa-file-zc6aeu-ta7jw3,aa-dataset-nzrd4e-jycyhc,aa-image-uhwm4d-hq76fc,DATASET_aa-dataset-nzrd4e-jycyhc/IMAGES/IMAGE_bIiLxpPgLj/bIiLxpPgLj_4.dcm.c4gh,,SHA256,b95acc8ee1e330e21a01a1666a88f408b1d91aea3c4c20ce865e5de4ecb17668,e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,error,External service error: admin,transient_error,1,2026-07-14 11:19:08.935259 +00:00,2026-07-14 11:25:15.331003 +00:00
aa-file-q2ssqz-kf6tfq,aa-dataset-nzrd4e-jycyhc,aa-image-uhwm4d-hq76fc,DATASET_aa-dataset-nzrd4e-jycyhc/IMAGES/IMAGE_bIiLxpPgLj/bIiLxpPgLj_5.dcm.c4gh,,SHA256,a4d93cc8b2425444730d9745582c57763794048478690b0c55ab7c3a36ccede0,e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,error,External service error: admin,transient_error,1,2026-07-14 11:19:08.945060 +00:00,2026-07-14 11:25:18.514071 +00:00
aa-file-h8j7dk-n5sc4p,aa-dataset-nzrd4e-jycyhc,aa-image-uhwm4d-hq76fc,DATASET_aa-dataset-nzrd4e-jycyhc/IMAGES/IMAGE_bIiLxpPgLj/bIiLxpPgLj_6.dcm.c4gh,,SHA256,d33960d767bb6258be0d1d885acceb511818a355ebe4b538098dba79aff722e0,e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,error,External service error: admin,transient_error,1,2026-07-14 11:19:08.952352 +00:00,2026-07-14 11:25:21.703767 +00:00
aa-file-gnhh26-txpqv9,aa-dataset-nzrd4e-jycyhc,aa-image-8r6342-vh5e7g,DATASET_aa-dataset-nzrd4e-jycyhc/IMAGES/IMAGE_yaACOLRMJG/yaACOLRMJG_4.dcm.c4gh,,SHA256,aee44c12b51d76ca5e63de681dd86490c6ccc572507fcf32e04241de105f57ec,e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,error,External service error: admin,transient_error,1,2026-07-14 11:19:08.974091 +00:00,2026-07-14 11:25:24.985936 +00:00
aa-file-kr3vt7-bkwvjm,aa-dataset-nzrd4e-jycyhc,aa-image-8r6342-vh5e7g,DATASET_aa-dataset-nzrd4e-jycyhc/IMAGES/IMAGE_yaACOLRMJG/yaACOLRMJG_5.dcm.c4gh,,SHA256,d88f2d7dac3cd21122f71acaf8efcb25dcd22d940745c3543e56833f0d687702,e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,error,External service error: admin,transient_error,1,2026-07-14 11:19:08.983880 +00:00,2026-07-14 11:25:28.224231 +00:00
aa-file-5vf49z-fpy4n8,aa-dataset-nzrd4e-jycyhc,,DATASET_aa-dataset-nzrd4e-jycyhc/LANDING_PAGE/THUMBNAILS/0.jpg.c4gh,,,,,error,External service error: admin,transient_error,1,2026-07-14 11:19:46.353668 +00:00,2026-07-14 11:24:40.510589 +00:00
aa-file-5spa2g-cfx75n,aa-dataset-nzrd4e-jycyhc,aa-image-8r6342-vh5e7g,DATASET_aa-dataset-nzrd4e-jycyhc/IMAGES/IMAGE_yaACOLRMJG/yaACOLRMJG_6.dcm.c4gh,,SHA256,334407908d16d629d3f0b2c7e782fdf223bcc3761ab23ec8d810922d50e205fc,e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,error,External service error: admin,transient_error,1,2026-07-14 11:19:08.993568 +00:00,2026-07-14 11:25:28.244225 +00:00
aa-file-7zthjk-pwfucj,aa-dataset-nzrd4e-jycyhc,aa-image-8r6342-vh5e7g,DATASET_aa-dataset-nzrd4e-jycyhc/IMAGES/IMAGE_yaACOLRMJG/yaACOLRMJG_1.dcm.c4gh,,SHA256,8686256901b3403cb185a5c6253e1561b17cf1d13342a6bf8345c412d8c31c30,e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,error,External service error: admin,transient_error,1,2026-07-14 11:19:08.957521 +00:00,2026-07-14 11:24:43.644679 +00:00
aa-file-yqhm4s-b2m3xr,aa-dataset-nzrd4e-jycyhc,aa-image-8r6342-vh5e7g,DATASET_aa-dataset-nzrd4e-jycyhc/IMAGES/IMAGE_yaACOLRMJG/yaACOLRMJG_2.dcm.c4gh,,SHA256,953927614b4e7698f5fb8ff8a9da099010af205ed091e11c9ca3733db65f08c0,e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,error,External service error: admin,transient_error,1,2026-07-14 11:19:08.962711 +00:00,2026-07-14 11:24:46.777867 +00:00
aa-file-rxdwvs-zn5svp,aa-dataset-nzrd4e-jycyhc,aa-image-8r6342-vh5e7g,DATASET_aa-dataset-nzrd4e-jycyhc/IMAGES/IMAGE_yaACOLRMJG/yaACOLRMJG_3.dcm.c4gh,,SHA256,25c901e3556d2f78a88127b536ed9e7c8909e8eb6358db0e0891b7fc3ee112d0,e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,error,External service error: admin,transient_error,1,2026-07-14 11:19:08.968299 +00:00,2026-07-14 11:24:49.888567 +00:00

Note all files has ingest_status: error, ingest_error: External service error: admin, and ingest_error_type: transient_error

And metadata submitter is on each Ingest scan cycle logging _sync_file_ingest_states: No Admin API status for submission xyz, file_id: abc, path=123 yet for each file

[rasko-leinonen-csc]

Joonatan> Submission is marked as published prior ingestion starts. The ingestion runs in the background until it is completed. The ingestion can stay in an unfinished state forever if there are repeated file ingestions errors.

Karl: the SDA database reports that ingestion was completed succesfully, but the submission service did not report ingestion as completed. Possibly, when a file has been processed it does not show in a list call yet.

[KarlG-nbis]

Looked into the logs abit more when this happend

aa-dataset-nzrd4e-jycyhc_issue_logs.csv

and what appears to happen is that everything went fine with the files, but then there was an issue releasing the dataset, and the await self._admin_handler.release_dataset(submission_id) ingest.py#L181 failed, which reset the the ingest cycle, but since the

await self._admin_handler.create_dataset(
    CreateDatasetRequest(user=user_id, accession_ids=file_ids, dataset_id=submission_id)
)

had succeeded the files are no longer exposed on the `await self._admin_handler.get_user_files(user_id, submission_id).

The reason for the release_dataset failure is most likely a race condition since, the 200 response from the create_dataset request does not mean that the dataset has been created, just that the request to create the dataset has been added to the queue, so on the subsequent release_dataset call the dataset might not yet have been created if there are other messages before the create request in the queue, etc.

[KarlG-nbis]

Also on the note of the dataset creation, it would be good for the metadata-submitter to support some type of batching of the /dataset/create requests, eg 1000 files per request, to avoid potential timeouts, etc, during the request handling

But related to knowing when to call the dataset/release, I will discuss with the team, but most likely we will add a new API to get the status of the dataset, and either all files(but that would probably require pagination), or just the number of files in a datasets, so you can check when all /dataset/create requests have been processed.

[KarlG-nbis]

We now have a new API /dataset/{datasetID} to get the state of a dataset, including status, and NrOfFiles added, so would suggest to keep pooling that until it the expected nr of files have been added to be sure that all dataset create requests have been actioned