Skip to content

Add support for excluding files based on .gitignore to copy_tree - #5112

Open
happz wants to merge 7 commits into
copy-tree-rsync-strategyfrom
copy-tree-rsync-exclude
Open

happz wants to merge 7 commits into
copy-tree-rsync-strategyfrom
copy-tree-rsync-exclude

Conversation

@happz

@happz happz commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

When copying a tree from a git repository, it is useful to not copy
files ignored by git, i.e. "honoring" the .gitignore files. This has
been implemented at one place - plan populating its worktree - but there
are other places that would benefit from this feature.

Therefore copy_tree gains new parameters, existing strategies are
extended where possible.

Pull Request Checklist

  • implement the feature
  • write the documentation
  • extend the test coverage

@happz happz added the ci | full test Pull request is ready for the full test execution label Aug 7, 2026
@happz
happz force-pushed the copy-tree-rsync-exclude branch from 6960fb7 to cfd4820 Compare August 7, 2026 19:07
Comment thread tmt/utils/filesystem.py Outdated
Comment thread tmt/utils/filesystem.py
@happz

happz commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

@LecrisUT would you have any idea why https://artifacts.dev.testing-farm.io/923c8df6-c4cb-41fa-899f-5fafc4068b8d/work-stream10y1vl1yxx/plans/provision/container/install/centos/stream10/execute/data/guest/default-0/tests/prepare/artifact/install-cases-3/output.txt would be sensitive to copy_tree() changes? It seems to be related to the addition of .gitignore-based exclusion, previous patches seem to be fine.

@LecrisUT

Copy link
Copy Markdown
Member

@LecrisUT would you have any idea why https://artifacts.dev.testing-farm.io/923c8df6-c4cb-41fa-899f-5fafc4068b8d/work-stream10y1vl1yxx/plans/provision/container/install/centos/stream10/execute/data/guest/default-0/tests/prepare/artifact/install-cases-3/output.txt would be sensitive to copy_tree() changes? It seems to be related to the addition of .gitignore-based exclusion, previous patches seem to be fine.

The rpms in that test are in .gitignore, but they are being "force included" in the tests. The rpms are built on the host and shared with a few other tests as well, made much more sense before the container plans split.

But that does raise the question if there are users who would do something similar outside of a clean testing-farm execution

@happz

happz commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

@LecrisUT would you have any idea why https://artifacts.dev.testing-farm.io/923c8df6-c4cb-41fa-899f-5fafc4068b8d/work-stream10y1vl1yxx/plans/provision/container/install/centos/stream10/execute/data/guest/default-0/tests/prepare/artifact/install-cases-3/output.txt would be sensitive to copy_tree() changes? It seems to be related to the addition of .gitignore-based exclusion, previous patches seem to be fine.

The rpms in that test are in .gitignore, but they are being "force included" in the tests. The rpms are built on the host and shared with a few other tests as well, made much more sense before the container plans split.

Any idea how to change to test to amend it with the .gitignore filter?

@LecrisUT

Copy link
Copy Markdown
Member

Any idea how to change to test to amend it with the .gitignore filter?

The quick fix is to drop the tests/prepare/artifact/rpms/.gitignore copy in

rlRun "cp -a rpms $testdir/data/" 0 "Copy rpms data"

but also I think we should make this configurable

@happz
happz force-pushed the copy-tree-rsync-exclude branch from 67cf7f0 to 0725140 Compare August 10, 2026 10:26
Comment thread tmt/utils/filesystem.py
@happz
happz force-pushed the copy-tree-rsync-exclude branch from 0725140 to 98739f5 Compare August 10, 2026 10:37
@thrix thrix added this to planning Aug 12, 2026
@thrix thrix moved this to implement in planning Aug 12, 2026
@happz
happz force-pushed the copy-tree-rsync-exclude branch from 98739f5 to b47083a Compare August 16, 2026 19:13
@happz
happz force-pushed the copy-tree-rsync-exclude branch from b47083a to e3ad657 Compare August 16, 2026 20:44
@LecrisUT

Copy link
Copy Markdown
Member

Any idea how to change to test to amend it with the .gitignore filter?

The quick fix is to drop the tests/prepare/artifact/rpms/.gitignore copy in

rlRun "cp -a rpms $testdir/data/" 0 "Copy rpms data"

but also I think we should make this configurable

/tests/prepare/artifact/providers/repository-file on centos-7 is of the same nature as ☝️ so similar fix

@happz
happz force-pushed the copy-tree-rsync-exclude branch from 2c761f2 to 24c9963 Compare August 19, 2026 18:16
@happz
happz force-pushed the copy-tree-rsync-exclude branch from 24c9963 to 0b70854 Compare August 20, 2026 13:25
@happz happz added status | blocked The merging of PR is blocked on some other issue status | blocking other work An important pull request, blocking other pull requests or issues labels Aug 20, 2026
Comment thread tests/unit/test_filesystem.py Outdated

subprocess.check_call(['git', 'init'], cwd=source_dir)

(source_dir / 'this-file-is-apparently-ignored.ignore-me').touch()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? The .gitignore in subdir should ignore only from that path down. Implementation wise also it doesn't look to deal with nested git repos. There are at least 4 cases to consider:

  • top-level .gitignore
  • subdir .gitignore
  • submodule ignores
  • .git/info/exclude

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? The .gitignore in subdir should ignore only from that path down.

Yes, hence adding an apparently ignorable file above the .gitignore. If this works, the file should land in the destination as it is not in fact ignored by git.

Implementation wise also it doesn't look to deal with nested git repos.

Hmm, probably a new can of worms. I wouldn't mind adding a "nested git repos/submodules not supported yet" note for now. rsync strategy could support it well, shutil most likely not.

There are at least 4 cases to consider:

  • top-level .gitignore
  • subdir .gitignore

These should be supported and covered by the implementation and tests.

  • submodule ignores

Inclined to ignore this for now.

  • .git/info/exclude

Inclined to "ignore" this for now, although it should be possible to support this with what we already have here. I will take a look at this one.

@happz
happz force-pushed the copy-tree-rsync-exclude branch 2 times, most recently from 3def217 to fab0ef2 Compare August 26, 2026 21:43
Comment thread tmt/utils/filesystem.py
@happz
happz force-pushed the copy-tree-rsync-exclude branch from fab0ef2 to ea3ed3b Compare August 26, 2026 21:56
Comment thread tmt/utils/filesystem.py
gitignore = current / '.gitignore'

if gitignore.is_file():
filters += ['--filter', f'dir-merge,- {gitignore}']

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dir-merge filter option in rsync expects a relative path pattern, not an absolute path to a specific file. This line attempts to pass an absolute path like /path/to/parent/.gitignore which will not work correctly.

The dir-merge filter is designed to find and apply rules from files matching a pattern within the transfer tree, not to reference specific files outside the source directory. For parent .gitignore files above src but within git_root, rsync cannot directly process them with dir-merge.

To fix this, parent .gitignore files should either be:

  1. Read and their rules converted to explicit --exclude patterns, or
  2. Handled differently, potentially by using --exclude-from with a temporary merged file
# Option 1: Read parent .gitignore and convert to exclude patterns
with open(gitignore) as f:
    for line in f:
        line = line.strip()
        if line and not line.startswith('#'):
            filters += ['--exclude', line]
Suggested change
filters += ['--filter', f'dir-merge,- {gitignore}']
with open(gitignore) as f:
for line in f:
line = line.strip()
if line and not line.startswith('#'):
filters += ['--exclude', line]

Spotted by Graphite

Fix in Graphite


Is this helpful? React 👍 or 👎 to let us know.

@happz
happz force-pushed the copy-tree-rsync-exclude branch from d8306d9 to 0f7a812 Compare September 16, 2026 07:30
@therazix
therazix force-pushed the copy-tree-rsync-exclude branch from 0f7a812 to 06fd410 Compare September 16, 2026 13:36
When copying a tree from a git repository, it is useful to not copy
files ignored by git, i.e. "honoring" the `.gitignore` files. This has
been implemented at one place - plan populating its worktree - but there
are other places that would benefit from this feature.

Therefore `copy_tree` gains new parameters, existing strategies are
extended where possible.
@happz
happz force-pushed the copy-tree-rsync-exclude branch from 06fd410 to 388586e Compare September 16, 2026 20:56

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci | full test Pull request is ready for the full test execution status | blocked The merging of PR is blocked on some other issue status | blocking other work An important pull request, blocking other pull requests or issues

Projects

Status: implement

Development

Successfully merging this pull request may close these issues.

3 participants