running tmt on Ubuntu 26.10 ends in 'Unable to find tree root ...'

[tflink]

When I try to run tmt on a Ubuntu 26.10 machine, the run goes like:

# tmt run plans -n rocm-inbox-rocrand provision -h local discover -h fmf execute -h tmt
/var/tmp/tmt/run-001

/plans/rocm-inbox-rocrand
    discover
        directory: /opt/rocm-inbox-testing
        how: fmf
        tests: rocrand/selftest

plan failed.

The exception was caused by 1 earlier exceptions

Cause number 1:

    No metadata found in the '/var/tmp/tmt/run-001/plans/rocm-inbox-rocrand/discover/default-0/tests' directory. Use 'tmt init' to get started.

    The exception was caused by 1 earlier exceptions

    Cause number 1:

        Unable to find tree root for '/var/tmp/tmt/run-001/plans/rocm-inbox-rocrand/discover/default-0/tests'.

I don't understand why, but this only seems to happen on 26.10 and I've been able to reproduce with both the 1.76.0 that's packaged in my ppa and with 1.78.0 from git. I have not been able to reproduce this on 26.04.

ppa with tmt package: https://launchpad.net/~tflink/+archive/ubuntu/rocm-inbox-testing-tools
test suite used: https://github.com/AMD-Linux-Open-Source-Graphics/rocm-inbox-testing
docker images for 26.04 (resolute) and 26.10 (stonking): https://hub.docker.com/r/tflink/rocm-inbox-testing-ubuntu-base/tags

[LecrisUT]

Can you run it with some TMT_SHOW_TRACEBACK=1 to get some more tracebacks? I will try to reproduce it, can you guide me in how to set this up on a fresh ubuntu container/vm?

[tflink]

# TMT_SHOW_TRACEBACK=1 tmt run plans -n rocm-inbox-rocrand provision -h local discover -h fmf execute -h tmt
/var/tmp/tmt/run-001

/plans/rocm-inbox-rocrand
    discover
        directory: /opt/rocm-inbox-testing
        how: fmf
        tests: rocrand/selftest

plan failed.

    Traceback (most recent call last):

    File /usr/lib/python3/dist-packages/tmt/__main__.py, line 39, in run_cli
      tmt.cli._root.main()


    File /usr/lib/python3/dist-packages/click/core.py, line 1514, in __call__
      return self.main(*args, **kwargs)


    File /usr/lib/python3/dist-packages/click/core.py, line 1435, in main
      rv = self.invoke(ctx)


    File /usr/lib/python3/dist-packages/click/core.py, line 1902, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))


    File /usr/lib/python3/dist-packages/click/core.py, line 1934, in invoke
      return _process_result(rv)


    File /usr/lib/python3/dist-packages/click/core.py, line 1871, in _process_result
      value = ctx.invoke(self._result_callback, value, **ctx.params)


    File /usr/lib/python3/dist-packages/click/core.py, line 853, in invoke
      return callback(*args, **kwargs)


    File /usr/lib/python3/dist-packages/click/decorators.py, line 34, in new_func
      return f(get_current_context(), *args, **kwargs)


    File /usr/lib/python3/dist-packages/tmt/cli/_root.py, line 512, in finito
      click_context.obj.run.go()


    File /usr/lib/python3/dist-packages/tmt/base/run.py, line 750, in go
      raise tmt.utils.GeneralError('plan failed.') from error



The exception was caused by 1 earlier exceptions

Cause number 1:

    No metadata found in the '/var/tmp/tmt/run-001/plans/rocm-inbox-rocrand/discover/default-0/tests' directory. Use 'tmt init' to get started.

        Traceback (most recent call last):

        File /usr/lib/python3/dist-packages/tmt/base/run.py, line 746, in go
          plan.go()


        File /usr/lib/python3/dist-packages/tmt/base/plan.py, line 1287, in go
          step.go()


        File /usr/lib/python3/dist-packages/tmt/steps/discover/__init__.py, line 918, in go
          self.discover_tests(phase)


        File /usr/lib/python3/dist-packages/tmt/steps/discover/__init__.py, line 676, in discover_tests
          phase.go(path=path, logger=logger)


        File /usr/lib/python3/dist-packages/tmt/steps/discover/fmf.py, line 664, in go
          self._tests = self.do_the_discovery(path)


        File /usr/lib/python3/dist-packages/tmt/steps/discover/fmf.py, line 797, in do_the_discovery
          tests += tree.tests(


        File /usr/lib/python3/dist-packages/tmt/base/core.py, line 2462, in tests
          for test in name_filter(self.tree.prune(keys=keys, names=[name], sort=sort))


        File /usr/lib/python3/dist-packages/tmt/base/core.py, line 2348, in tree
          raise tmt.utils.MetadataError(



    The exception was caused by 1 earlier exceptions

    Cause number 1:

        Unable to find tree root for '/var/tmp/tmt/run-001/plans/rocm-inbox-rocrand/discover/default-0/tests'.

            Traceback (most recent call last):

            File /usr/lib/python3/dist-packages/tmt/base/core.py, line 2346, in tree
              self._tree = fmf.Tree(str(self._path))


            File /usr/lib/python3/dist-packages/fmf/base.py, line 114, in __init__
              self._initialize(path=data)


            File /usr/lib/python3/dist-packages/fmf/base.py, line 174, in _initialize
              raise utils.RootError(

[tflink]

The tests in the repo I'm using require AMD GPUs but that is only during execution and thus, I assume isn't relevant for this issue and will omit those parts but as a heads up, the actual tests will fail if they execute. This image uses tmt 1.76.0 from my ppa because I haven't gotten around to updating the packages but I've seen the same issue on 1.78.0 when I install from git.

clone https://github.com/AMD-Linux-Open-Source-Graphics/rocm-inbox-testing.git somewhere, I will use /var/tmp/rocm-inbox-testing for this example

start the container: podman run --rm -it --security-opts label=disable -v /var/tmp/rocm-inbox-testing:/opt/rocm-inbox-testing docker.io/tflink/rocm-inbox-testing-ubuntu-base:stonking

Navigate to the tests cd /opt/rocm-inbox-testing

Run tmt run plans -n rocm-inbox-rocrand provision -h local --feeling-safe discover -h fmf execute -h tmt to see the issue

If you repeat with tflink/rocm-inbox-testing-ubuntu-base:resolute instead of the stonking tag, the issue does not happen and the test will fail, as expected when there is no AMD GPU access in the container.

[LecrisUT]

Ok, I've found the issue and it is a rsync a b/ != rsync a b kind of issue. I am afraid if this is an issue with the rsync version used. There was a recent rsync vibe-coding fiasco recently, but 3.4.4 should be a safe one 🤔

[LecrisUT]

Aha, found the issue

$ cp -a --reflink=auto /var/tmp/tmt-test/./ /var/tmp/dest2

copies tmt-test folder instead of the contents. This is a rather big regression, @tflink can you follow up on this in Ubuntu land, it seems related to the cp (uutils coreutils) 0.10.0 rust coreutils transition and it is a major regression.

[tflink]

https://bugs.launchpad.net/ubuntu/+source/rust-coreutils/+bug/2167447