Sparse Checkout #6169

YuKitsune · 2022-01-12T01:20:13Z

This PR aims to implement basic sparse checkout functionality into LibGit2 building on @jochenhz's original PR.

What does it do?

Status / Diff takes GIT_INDEX_ENTRY_SKIP_WORKTREE into account
sparse-checkout file is treated as an attr_file, hence following gitignore rules
Skips sparse files on checkout
Sparse checkout can be enabled with core.sparseCheckout
support init, set, add, reapply and disable commands all supported
Expose an interface function git_sparse_check_path to query whether or not a file is excluded from checkout
Set GIT_INDEX_ENTRY_SKIP_WORKTREE flag on git_index_read_tree based on the sparse checkout ruleset.

What does it not do?

Smart trickery to determine whether or not a full tree is sparse (would be a very welcome optimization)
Support core.sparseCheckoutCone
No sparse checkout option on clone (but this can be achieved with GIT_CHECKOUT_NONE and git_index_read_tree)

Note

I've got some tests around worktrees, they seem to work
Given how complex sparse checkout can be, I'm almost certain I haven't gotten this implementation 100% correct, though feel free to prove me wrong 👍

YuKitsune · 2022-01-12T01:22:37Z

src/sparse.c

+	if ((error = git_str_joinpath(&infopath, repo->gitdir, "info")) < 0) {
+		if (error != GIT_ENOTFOUND)
+			goto done;
+		error = 0;
+	}


Would like to have used something like git_repository__item_path here, but it would always get the info path for the root repository, not the worktrees. Keen to know if there's a nicer way to do this.

boretrk · 2022-01-21T23:45:58Z

include/git2/diff.h

+	/** Skip files in the diff that are excluded by the `sparse-checkout` file.
+	 * Set to 1 to skip sparse files, 0 otherwise
+	 */
+	int skip_sparse_files;


This one might require some discussion on how flags are handled.
There is a flags field, but it is almost full. Only (1u << 27) is available.

Most of the flags are in enums, I think the reason for this is that it provides some type safety while still being passable as arguments to functions.
In this particular case the flags are never used as a parameter, the user passes the git_diff_options struct instead.
This means that it would be possible to specify the flags in bitfields as unsigned int skip_sparse_files:1;
Unfortunately we can't split out the existing enum to a regular bitfield without being incompatible but IMO that would be a good thing to do for flags passed in a larger struct like this.

Pinging in @ethomson since he probably have given some more thought on what to do when we run out of flags.

boretrk · 2022-01-21T23:53:32Z

src/sparse.h

+	GIT_SPARSE_NOTFOUND = -1,
+	GIT_SPARSE_NOCHECKOUT = 0,
+	GIT_SPARSE_CHECKOUT = 1,
+};


Could we give this enum a typedef so that the proper type can be used instead of int?

I've called it git_sparse_status for now, though I'm open to some better name suggestions.

src/util.c

.gitignore

boretrk · 2022-01-22T00:12:07Z

src/checkout.c

+				const git_index_entry *e = git_index_get_byindex(data->index, pos);
+				if (e == NULL ||
+						(e->flags_extended & GIT_INDEX_ENTRY_SKIP_WORKTREE) == 0 ||
+						(data->strategy & GIT_CHECKOUT_REMOVE_SPARSE_FILES) != 0) {


Do you think it would be possible to add some comment on what is going on here?

boretrk · 2022-01-22T11:06:00Z

src/sparse.c

+    scan = (char *)git_str_cstr(&data);
+    while (!error && *scan) {
+
+		buf = git__strtok_(&scan, "\r\n", "\n", "\0");


This isn't how strtok works. strtok matches any character in the string.
git__strtok(&scan, "\r\n") is what you want here.
If either '\r' or '\n' is encountered it would split on them and empty segments are skipped over.
So "\r\n" would first match on '\r' and then skip over the '\n'.
It would also take "\n\n\r" or any combination of them.

But over all it seems a bit strange to have this splitting and parsing here.
https://www.git-scm.com/docs/git-sparse-checkout says:

By default, the sparse-checkout file uses the same syntax as .gitignore files.

Shouldn't we aim to have the same function for parsing them?

I haven't really thought it through but I have this feeling that it could be possible to merge the spare checkout list and the ignore list for some checks but maybe there is something that prevents that.

I've replaced git__strtok_ with git__strtok, I couldn't find any "git ignore list"-like functions around, so I quickly threw this together. I'm open to some ideas on how we can make this list function a bit nicer.

I've also consolidated the ignore parsing functions, so that parse_sparse_file now uses parse_ignore_file instead. I'd like to keep the parse_sparse_file function around since cone mode should change the parsing behaviour a bit.

YuKitsune · 2022-06-21T05:36:13Z

Bump.
Is there anything I can do to help progress this further?

VinnyOG · 2022-08-05T18:08:12Z

@YuKitsune Is this PR in a state that it's working for you? I would like to patch it into a project that I'm working on and wondering if there are any rough edges I should know about.

Thanks for your work in continuing on this!

take skip worktree flag into account when handling unmatched items while diffing Signed-off-by: Jochen Hunz <j.hunz@anchorpoint.app> move does_negate_rule from ignore.c to attr_file.h so that it can be used by sparse checkout code as well Signed-off-by: Jochen Hunz <j.hunz@anchorpoint.app> add core.sparseCheckout to configmap cache Signed-off-by: Jochen Hunz <j.hunz@anchorpoint.app> implement very basic handling of .git/info/spare-checkout file. The sparse-checkout file is treated as a git_attr_file. A public API git_sparse_check_path reads the sparse-checkout file from the attribute file cache and checks the path against this attribute file using the gitignore ruleset Signed-off-by: Jochen Hunz <j.hunz@anchorpoint.app> implement git_sparse__lookup function that can be called on initialised git_sparse object Signed-off-by: Jochen Hunz <j.hunz@anchorpoint.app> when doing git_index_read_tree set skip-worktree flag for index entries that are excluded by the sparse checkout ruleset - if sparse checkout is enabled Signed-off-by: Jochen Hunz <j.hunz@anchorpoint.app> fix cleanup of git_sparse structure when core.sparseCheckout is disabled Signed-off-by: Jochen Hunz <j.hunz@anchorpoint.app> set skip-worktree flag when index entry is initialized and path is sparse Signed-off-by: Jochen Hunz <j.hunz@anchorpoint.app> cache sparse-checkout values when iterating a tree so that it can be accessed by checkout and diff functionalities quickly. Signed-off-by: Jochen Hunz <j.hunz@anchorpoint.app> skip sparse files for unmatched new and old items if requested Signed-off-by: Jochen Hunz <j.hunz@anchorpoint.app> skip sparse files on checkout, add option to remove sparse files on checkout Signed-off-by: Jochen Hunz <j.hunz@anchorpoint.app> add test repository for sparse unit tests Signed-off-by: Jochen Hunz <j.hunz@anchorpoint.app> add tests for git_sparse_check_path and git status on sparse repositories Signed-off-by: Jochen Hunz <j.hunz@anchorpoint.app> test that index operations set skip-worktree flag if needed Signed-off-by: Jochen Hunz <j.hunz@anchorpoint.app> implement git sparse checkout tests Signed-off-by: Jochen Hunz <j.hunz@anchorpoint.app> fix: parsing sparse file does not require context for subdirs Fix build errors form rebase Fix build errors form rebase Implement init and set Fix `git_sparse_checkout_set` not enabling sparse checkout WIP: Implementing other sparse-checkout commands - Temporarily remove cone references, will need to revisit - Updating references in tests fix sparse_checkout_list impl update tests re-sync `git_attr__does_negate_rule` update test coverage make `git_sparse_checkout_init` with default patterns fix init overwriting sparse-checkout file docs tidy-up Tidy-up Update sparse checkout tests update set.c implement git sparse-checkout apply hack: restore working directory on disable Updating tests Tidy up Fix missing methods from rebase move sparse diff flag to separate field seems we've run out of space on that enum... add version to sparse checkout init opts fix more missing methods from rebase Move diff ignore sparse flag fix sparse checkout file being written to the root Fix up assertions add worktree tests tidy up args and internals Further tidy up fix sparse-checkout file not writing to worktree gitdir remove todo add git sparse status typedef remove `git__strtok_` tidy `parse_sparse_file` consolidate ignore parsing add comment addressing skip_worktree and sparse checkout update gitignore Fix build errors from rebase Move sparse tests

YuKitsune · 2022-08-10T23:21:48Z

@VinnyOG We haven't gotten around to making use of this. Our use-case requires a new build of LibGit2Sharp, which proved to be too much of a pain.

extrawurst · 2022-08-17T12:42:30Z

I wish this would be getting attention

YuKitsune · 2022-08-30T23:08:29Z

I've mucked up the rebase and the PR auto-closed... 🤦
Doesn't seem to let me re-open it, so I've created a new one over here #6394
Sorry for the mess 😅

YuKitsune commented Jan 12, 2022

View reviewed changes

YuKitsune marked this pull request as ready for review January 12, 2022 01:35

boretrk reviewed Jan 21, 2022

View reviewed changes

boretrk reviewed Jan 22, 2022

View reviewed changes

src/util.c Outdated Show resolved Hide resolved

boretrk reviewed Jan 22, 2022

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

boretrk reviewed Jan 22, 2022

View reviewed changes

YuKitsune force-pushed the features/sparse-checkout branch from 79e51b2 to ec21633 Compare February 4, 2022 05:17

YuKitsune force-pushed the features/sparse-checkout branch from ec21633 to 38f99fc Compare March 14, 2022 04:56

zentron mentioned this pull request May 19, 2022

Sparse checkout support #2263

Open

YuKitsune force-pushed the features/sparse-checkout branch from 572a8f3 to 8c06663 Compare May 29, 2022 23:04

VinnyOG pushed a commit to 8thwall/libgit2 that referenced this pull request Aug 5, 2022

[sparse] squash and rebase libgit2#6169 sparse checkout support

a5c563a

VinnyOG mentioned this pull request Aug 5, 2022

[sparse] squash and rebase #6169 sparse checkout support 8thwall/libgit2#26

Draft

7 tasks

YuKitsune force-pushed the features/sparse-checkout branch from 8c06663 to 1882cdf Compare August 30, 2022 22:50

YuKitsune closed this Aug 30, 2022

YuKitsune force-pushed the features/sparse-checkout branch from 1882cdf to bdab223 Compare August 30, 2022 22:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sparse Checkout #6169

Sparse Checkout #6169

YuKitsune commented Jan 12, 2022 •

edited

Loading

YuKitsune Jan 12, 2022

boretrk Jan 21, 2022

boretrk Jan 22, 2022

boretrk Jan 21, 2022

YuKitsune Feb 4, 2022

boretrk Jan 22, 2022

boretrk Jan 22, 2022 •

edited

Loading

YuKitsune Feb 4, 2022

YuKitsune commented Jun 21, 2022

VinnyOG commented Aug 5, 2022

YuKitsune commented Aug 10, 2022

extrawurst commented Aug 17, 2022

YuKitsune commented Aug 30, 2022 •

edited

Loading

Sparse Checkout #6169

Sparse Checkout #6169

Conversation

YuKitsune commented Jan 12, 2022 • edited Loading

What does it do?

What does it not do?

Note

YuKitsune Jan 12, 2022

Choose a reason for hiding this comment

boretrk Jan 21, 2022

Choose a reason for hiding this comment

boretrk Jan 22, 2022

Choose a reason for hiding this comment

boretrk Jan 21, 2022

Choose a reason for hiding this comment

YuKitsune Feb 4, 2022

Choose a reason for hiding this comment

boretrk Jan 22, 2022

Choose a reason for hiding this comment

boretrk Jan 22, 2022 • edited Loading

Choose a reason for hiding this comment

YuKitsune Feb 4, 2022

Choose a reason for hiding this comment

YuKitsune commented Jun 21, 2022

VinnyOG commented Aug 5, 2022

YuKitsune commented Aug 10, 2022

extrawurst commented Aug 17, 2022

YuKitsune commented Aug 30, 2022 • edited Loading

YuKitsune commented Jan 12, 2022 •

edited

Loading

boretrk Jan 22, 2022 •

edited

Loading

YuKitsune commented Aug 30, 2022 •

edited

Loading