Enhance the test control options available on a per-test basis

[sterlingalexander]

Hi,

Let me start off by saying I am making an assumption, that a test should handle it's setup and cleanup as completely as it can without the plan writer having to consider what's required for the test to run and what the environment will be when the next test in the plan runs. This would mean that tests 'just run' when included in plans and the plan writer could make any adjusts they want (or not) to modify them.

Currently, this extra setup can be handled in sidecar test scripts or can be coupled into the test plan itself. To me, it seems like there may be a set of common tasks that would benefit from having a dedicated way to describe and execute using the test metadata. As an example:

When testing the linux kernel, if one wants to run specific tests, like kunit, then the kernel command line needs be booted with a specific flag set. The kunit tests will produce a passing result without the flag, but this is a false positive, they don't run. The kunit tests are the only kernel tests that need this flag, it's self-contained to this specific kernel internal test harness.

Getting this option on the command line can be accomplished at request/plan level using --kickstart kernel-options-post='kunit.enable=1' or other plan definitions, but this option persists for any other test run in that plan. The current, test-scoped solution is to have a script 'do the test setup and clean up', but it would be nice if this were an option for the test metadata as well. Maybe something like:

setup:
  how: <shell|plugin|tmt-builtin interface>
  <...actions....>

With the contract of 'any action taken during the setup phase will persist only for the duration of this test and then be reversed'. If this contract guarantee is feasible, then shell is probably not a realistic option, but I included it there to represent the current reality.

Another thought would be to use this metadata to group discovered tests into sub-plans of tests that have similar requirement, with worst-case being N plans for N tests.

This would not completely obviate the need for scripts/side-car setup of tests, but it could push a lot of common configuration cases into the test metadata, and decouple them from the plan. If tests themselves provided all the necessary configuration/metadata so they were guaranteed to run in arbitrary test plans it would remove the need for plan writers to have knowledge of all the test requirements. This would create an 'off the shelf' testing environment, where adjusts would be based on specific knowledge and requirements, but the base overall state of the tests would hopefully make plan writing and testing a bit easier.

Finally, if I'm missing something obvious feel free to close this...and provide a pointer in the right direction.

[psss]

Hi, thanks for sharing the ideas. Seems we've already touched some of the topics you mention in existing issues:

Currently, this extra setup can be handled in sidecar test scripts or can be coupled into the test plan itself. To me, it seems like there may be a set of common tasks that would benefit from having a dedicated way to describe and execute using the test metadata.

I believe this one is related to the topic of how to efficiently handle test setup and cleanup. Several ways of how to resolve this have been brainstormed and are listed under the following umbrella issue:

• Shared test setup and cleanup #4524

With the contract of 'any action taken during the setup phase will persist only for the duration of this test and then be reversed'.

This seems to be related to the idea of the proposed isolate key which would ensure that an identical / clean test environment would be provided for each test, either by allocating a separate (sub)plan (i.e. a dedicated guest instance) or having some form of snapshot or rollback functionality:

• Implement the isolate key for isolating tests #4143

Another thought would be to use this metadata to group discovered tests into sub-plans of tests that have similar requirement, with worst-case being N plans for N tests.

Yes, this is something very close to the idea of having the hardware key on the test level, which allow individual tests to specify what they need and the implementation logic would take care of grouping tests according to specific hardware requirements. In your case it would be something more generic, but in essence I'd say a very similar implementation:

• Introduce a new test attribute hardware #765

This would not completely obviate the need for scripts/side-car setup of tests, but it could push a lot of common configuration cases into the test metadata, and decouple them from the plan. If tests themselves provided all the necessary configuration/metadata so they were guaranteed to run in arbitrary test plans it would remove the need for plan writers to have knowledge of all the test requirements. This would create an 'off the shelf' testing environment, where adjusts would be based on specific knowledge and requirements, but the base overall state of the tests would hopefully make plan writing and testing a bit easier.

Yeah, this approach definitely makes sense. And could simplify writing/maintaining tests. Hopefully by implementing some/all of these ideas we could make some tangible steps towards this vision. @sterlingalexander, could you please check the above-mentioned issues and get involved there if interested? Please, let us know whether they would cover your use cases or if there's still anything missing. Thanks.

[sterlingalexander]

Added a comment here that probably should have landed here instead.