CFE-4708: Added support for a path-augment to specify files to ship beyond the extension-matching

[SimonThalvorsen]

Ticket: CFE-4708
Changelog: Added input_paths_extra to allow specified files to ship independent of what is specified in input_name_patterns[_extra]

[cf-bottom]

Thank you for submitting a pull request! Maybe @nickanderson can review this?

[nickanderson]

No ticket in the commit subject, that's for the PR.

I am not sure about this.

I understand that you are adding a file path to the files distributed in update.

• This is a fully qualified file path? Can you give an example of what values a user would be expected to use here?
• You say these files would be updated each time, regardless of validated_updates_ready, so these files could update but the rest of the policy could be skipped?

[SimonThalvorsen]

No ticket in the commit subject, that's for the PR.

I am not sure about this.

I understand that you are adding a file path to the files distributed in update.

* This is a fully qualified file path? Can you give an example of what values a user would be expected to use here?

* You say these files would be updated each time, regardless of validated_updates_ready, so these files could update but the rest of the policy could be skipped?

I am not certain that the hashing in GeneratePolicyRelease() is what messes it up, but the issue
is that originally I just added a variable input_paths_extra to contain the extra filepaths that may exists outside of what is defined in input_name_patterns (since these gets copied from before). In the first iteration i just expandend the file_select u_input_files-body to use the path_name-parameter and then use it like this:

leaf_name => {
  @(update_def.input_name_patterns), @(update_def.input_name_patterns_extra)
};
path_name => { @(update_def.input_paths_extra) };
file_result => "leaf_name|path_name";

but that only worked for the initial copy, so if the project got rebuilt with the same files(but different content) only the .sh (and assumes .txt, .cf and other accepted extensions) would get marked as changed and the client got the new content, but keeping the old, unchanged version of the other files.

The usage is intended for files added by cfbs input-type-file, where the supplied files may not match what is set inside ìnput_name_patterns and adding the extension (or no extension at all) to input_name_patterns_extra may cause more files that what one would want to get shipped.

[nickanderson]

No ticket in the commit subject, that's for the PR.
I am not sure about this.
I understand that you are adding a file path to the files distributed in update.

* This is a fully qualified file path? Can you give an example of what values a user would be expected to use here?

* You say these files would be updated each time, regardless of validated_updates_ready, so these files could update but the rest of the policy could be skipped?

I am not certain that the hashing in GeneratePolicyRelease() is what messes it up, but the issue is that originally I just added a variable input_paths_extra to contain the extra filepaths that may exists outside of what is defined in input_name_patterns (since these gets copied from before). In the first iteration i just expandend the file_select u_input_files-body to use the path_name-parameter and then use it like this:

leaf_name => {
  @(update_def.input_name_patterns), @(update_def.input_name_patterns_extra)
};
path_name => { @(update_def.input_paths_extra) };
file_result => "leaf_name|path_name";

but that only worked for the initial copy, so if the project got rebuilt with the same files(but different content) only the .sh (and assumes .txt, .cf and other accepted extensions) would get marked as changed and the client got the new content, but keeping the old, unchanged version of the other files.

The usage is intended for files added by cfbs input-type-file, where the supplied files may not match what is set inside ìnput_name_patterns and adding the extension (or no extension at all) to input_name_patterns_extra may cause more files that what one would want to get shipped.

I would remove git from the equation initially. Edit files in /var/cfengine/masterfiles on a hub and understand that basic flow first.
from memory ...
validated_updates_ready is defined when the client repaires it's copy of cf_promsies_validated. cf_promises_validated on the hubs /var/cfengine/masterfiles is updated when the hub sees that it's running and validates new policy in inputs. So, you change masterfiles, update.cf runs from inputs, promises.cf runs from inputs, which causes cf_promises_validated in /var/cfengine/masterfiles to be updated, then clients see that file changed, which lifts the gate for the client to scan all the rest of the files for change.

[SimonThalvorsen]

So this is still a work in progress, but the current issue is mainly this

Files that get added by the def.json only gets copied over on the 2nd run of update.cf, I assume this is the staleness as described in this comment, but not certain. If the file already exists, then update/update_policy fetches and copies without problems

[nickanderson]

I don't think it should behave this way (copying the files each time).

If you move the class guard to if => directly on the promise, the promise will only be in the context of a validated policy update which also has these extra files enabled.

There are two separate flows ... one where masterfiles is from a git repo and when thats the case cf_promises_release_id is the git commit hash. When its not git backed that id is computed from a tree hash in core. That tree hash I think only coveres specific files but I think that is probably wrong and should probably consider all files, just like git would.

[nickanderson]

I will push a change to your pr here lmk how that works for you

[nickanderson]

concat() line broken comment => values.

[nickanderson]

I don't think it should behave this way (copying the files each time).

If you move the class guard to if => directly on the promise, the promise will only be in the context of a validated policy update which also has these extra files enabled.

There are two separate flows ... one where masterfiles is from a git repo and when thats the case cf_promises_release_id is the git commit hash. When its not git backed that id is computed from a tree hash in core. That tree hash I think only coveres specific files but I think that is probably wrong and should probably consider all files, just like git would.