I'm seeing numerous performance problems with etckeeper on my servers. It generally happens when there is a git repository underneath `/etc` or some other thing that generates lots of small files. Typically, what I try next is to make git ignore those files, which works for the git side of things, but not for the etckeeper side of things, as all those files still get listed (and parsed) by etckeeper's `.etckeeper` metadata file.

The problem here is the metadata "ignore" filter doesn't really work. We can see this in multiple different bug reports here, like [[todo/etkeeper_warning_about_hardlinks_doesn__39__t_exclude_ignored_files]], [[todo/Do_not_recreate_ignored_empty_directory]], or [[todo/what_if_there_is_a_Git_repo_somewhere_underneath___47__etc__63__/]]. I think the approach taken to solve this problem is incorrect, which is why I am opening this issue to regroup all of them in one single place.

From what I can tell, the `30store-metadata` script tries to ignore files ignored by git. But it tries to do that using `grep`, and git ignore files are not `grep` patterns at all. In fact, they're not even regular expressions, they are like glob patterns, but with many different exceptions (like `!` to negate a pattern, for example). I do not think there is a command that can faithfully reproduce this outside of git itself.

Right now, we basically do this:

 1. list all empty directories, and add them to the metadata, regardless of gitignore (which is [[todo/Do_not_recreate_ignored_empty_directory]])
 2. then list all files and directories (empty or not), try to filter out ignored files, and try to ignore normal modes, which means:
    1. `find $NOVCS \( -type f -or -type d \)` (basically skips `.git` and friends)
    2. `sed 's/^\.\///' | grep -xFvf $(git ls-files -oi --exclude-standard; git ls-files -oi --exclude-standard --directory | sort -u)`
    3. some inline perl script which actually systematically chmods all files, and optionally changes the owner and group if they are not the EUID/EGID

Now, the problem that concerns us here is mostly in step 2.2 above. That `grep` command is bad for many reasons, but the first of which is `-x`: that will do a full match on the entire line so if you have, say, `puppet` in your `.gitignore` that will match the `puppet` directory, but nothing *underneath*, which is not how gitignore files work *at all*.

I think the proper way to do this is to actually start from files git actually tracks, since in that step, we're trying to keep track of modes and owners of actual files managed by git. It seems to me much better to make git list the files, then process *that*, than try to reimplement git-ignore outside of git.

I think the pseudocode then would be something like, for the git case:

 1. `git ls-files | sort -u | maybe_chmod_chown`
 2. `git ls-files --others --exclude-standard --directory | sed -e "s/^/mkdir -p /"`

... and that's it! The trick here is we separate the normal file tracking (first step) from the empty directory listing (second step). I haven't actually tested this because I'm out of battery in that third yak razor, but I wanted to brainstorm this here first to see if we could somehow make sense of this.

(Also, I don't think that the `maybe_chmod_chown` script should systematically change the mode on files, but that's a different story (and optimization)...)

-- [[anarcat]]

Update: and I think the patch is [something like this](https://salsa.debian.org/debian/etckeeper/-/commit/0a4305886c6e9501eec223880fa9ae8776b4947e) (see [branch](https://salsa.debian.org/debian/etckeeper/-/commits/new-git-ignores) for latest. it doesn't *quite* work the way I expected, unfortunately. a few examples:

[[!format diff """
diff --git a/.etckeeper b/.etckeeper
index 71fb3188..8b34a087 100755
--- a/.etckeeper
+++ b/.etckeeper
@@ -3,34 +3,28 @@
 mkdir -p './ImageMagick'
 mkdir -p './X11/xkb'
 mkdir -p './X11/xorg.conf.d'
-mkdir -p './apm/event.d'
-mkdir -p './apm/scripts.d'
+mkdir -p './apm'
[...]
"""]]

in the above example, you see that `apm` is correctly added but not the underlying `events.d` and `scripts.d`...

it *does* correctly follow ignores for most stuff however, which is an improvement. I did have to bend over backwards to remove symlinks from the listing, with that ungodly `sed`. and, in general, we have quoting problems here because we pipe filenames that might have newlines in them... thankfully, we might consider `/etc` trusted, but that's something that makes me uncomfortable about this whole thing in the first place...

here at least, a bunch of stuff is cleaned up:

[[!format txt """
root@marcos:/etc# git diff --cached --stat
 .etckeeper | 636 ++++------------------------------------------------------------------------------------------------------------------------------------------------
 1 file changed, 17 insertions(+), 619 deletions(-)
root@marcos:/etc# wc -l .etckeeper
7419 .etckeeper
"""]]

Anyways, let me know what you think. The main tradeoff here is that we lose empty subdirectories, which maybe is a big deal, but for my use cases, I don't expect etckeeper to recover everything like this, that's what backups are for. ;)
