Running/Initializing in subdirectory

[LecrisUT]

I am new to this, so sorry if I wasn't able to read the documentation about this, but is it possible to initialize/run tmt in a subdirectory? Upstream is already setup that test folder contains pytest, and I don't want to pollute that folder with these files. Can I instead put those folders in .fmf or a .tmt subfolder so that it is automatically detected as the root, or pass a config option pointing to such a root?

[lukaszachy]

Directories starting with dot are not processed when tmt looks for *fmf files. So you cannot store plan or test metadata there.

You can have fmf root (directory which contains .fmf/version) anywhere in the git repo, however it behaves as boundary so files outside of fmf root (but in the git repo) might not be copied where you need them.

So I'd say that if I understand your problem you might want to create fmf root in the git root directly. Depending if you expect to have multiple fmf files you might to store them in some subdir or have them directly in the fmf root.

Eg.

.
├── .fmf
│   └── version
├── test
│   └── pytest
└── tmt.fmf

Where tmt.fmf contains (it is the Plan):

execute:
    how: tmt

discover:
    how: shell
    tests:
    - name: Run the pytest
      require:
      - pytest
      path: /test
      test: ./pytest

[lukaszachy]

Test has path defined so tmt will change the directory when executing the test.

[LecrisUT]

First a clarification, I want to define and add new smoke tests that do not run pytest. Specifically this is in regards to packit where I want to test that after dnf install my_package, all of the installs work without having to manually specify LD_LIBRARY_PATH etc.

If I understand correctly it is possible to just have one monolithic tmt.fmf file with all the tests defined in it? Ideally I would want to have

.
├── .fmf
│   └── version
├── test
│   └── pytest
├── tmt
│   └── test
│       └── check_version_with_pacakge1
│       └── check_version_with_pacakge2
└── tmt.fmf

[lukaszachy]

Yes, that is possible. However if you have more test scripts and you want to have them as separate tests it could be more future proof to this:

.
├── .fmf
│   └── version
├── test
│   └── pytest
└── tmt
    └── test
        ├── check_version_with_pacakge1
        ├── check_version_with_pacakge2
        └── main.fmf

Where the tmt/test/main.fmf will contains something as

# common attributes for both tests....
require:
- OTHER_PACKAGE_TO_INSTALL

/name-for-check_version_with_pacakge1:
   test: ./check_version_with_pacakge1
/name-for-check_version_with_pacakge2:
   test: ./check_version_with_pacakge2

This will create 2 tests:

$ tmt t ls
/tmt/test/name-for-check_version_with_pacakge1
/tmt/test/name-for-check_version_with_pacakge2

[LecrisUT]

Ok. So the only restriction is that the folder cannot be hidden?

[psss]

Exploring hidding directories has been already implemented in fmf:

• Explore metadata in hidden files and directories fmf#264

@LecrisUT, anything still missing here?

[psss]

Testing Farm handles path of the fmf identifier in some different way.
@LecrisUT, could you please provide a specific / tangible example so that we can discuss that?

[LecrisUT]

So there are a few aspects to this issue and #2184. I will try to summarize and reformulate them, and recreate the issues as needed.

---

Accessing files outside of the TMT_TREE

Consider a test like

discover:
  how: fmf
  url: example.com
  path: /path/to/tmt/tree

In the test above, the test can only access files from the tree upwards, i.e. nothing from the original git repo root. But contract that with running with dist-git-extract. In that case the full source tree is available under TMT_SOURCE_DIR.

Ideally we should allow the user to access the full source dir as needed with TMT_SOURCE_DIR regardless if it's from dist-git, git repo or extracted artifact.

This is relevant to projects which do not want to add the .fmf folder int the top-level path.

---

Discover fmf behaves unexpectedly when run with -r path/to/subdir

Let's take an example from on our own project

tmt/packaging/plans/upstream.fmf

Lines 3 to 9 in 25045c9

	discover:
	how: fmf
	# Using a hack for when running in upstream where it is able to
	# discover tests from outside the current fmf tree.
	# https://github.com/teemtee/tmt/issues/2568
	path: .
	filter: "tier:1"

This behaves differently depending on:

• Running with -r packaging: the . there resolves to the git root (what we want)
• Running by first cd packaging: the . there resolves to the tmt tree in packaging which has no tests

This is clearly a bug since the -r should not affect this resolution. But unfortunately some projects have started to use this hack (mostly from my design) in order to discover tests from outside of the current tree. We should have a way to preserve some backwards compatibility, but the only way I can think of from the user perspective is to redefine the plan and support

discover:
    how: fmf
    path: ${TMT_SOURCE_DIR}
    filter: "tier:1"

However we resolve this bug, it introduces a behavior change, so we might need to make it dependent on #3802.

---

Documenting the various path variables and path interactions

Based on the issue above, the path handling can be very obscure. In a similar manner to #4364 plans to clarify the environment variables, these paths should also be clarified in a more central location:

• What are all of the path variables defined, e.g. TMT_TREE, TMT_SOURCE_DIR, TMT_TEST_DATA. When are they defined, what do they point to, etc.
• What special paths does tmt define? Similar to the above but including non-exposed paths. E.g. the workdir of run, plan, step, phase, or other path structures that we rely on like the WORKDIR_ROOT
• What is the general intuition of how path is supposed to work across all plugins. E.g. where does / and . point to, are they dependent on the current fmf object or fmf file?

---

These are the main issues that I can remember, if there are other issues that I have opened on a similar topic let me know to try and see if it's a duplicate of these or needs some reclarification.