Skip to content

WIP candidate merge for partial clone (2017/10/31) #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 30 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
8db82bc
dir: allow exclusions from blob in addition to file
jeffhostetler Jun 29, 2017
d9ae44a
oidmap: add oidmap iterator methods
jeffhostetler Oct 27, 2017
1ce5756
oidset: add iterator methods to oidset
jeffhostetler Oct 27, 2017
5d03320
list-objects: filter objects in traverse_commit_list
jeffhostetler Jun 30, 2017
04b52a0
rev-list: add list-objects filtering support
jeffhostetler Oct 27, 2017
f747a3d
pack-objects: add list-objects filtering
jeffhostetler Oct 27, 2017
f5b3d08
upload-pack: add object filtering for partial clone
jeffhostetler Oct 27, 2017
fa9dd86
clone, fetch-pack, index-pack, transport: partial clone
jeffhostetler Oct 30, 2017
2be7708
fetch: refactor calculation of remote list
jonathantanmy Sep 29, 2017
8c76466
fetch: add object filtering for partial fetch
jeffhostetler Oct 30, 2017
4696eb4
remote-curl: add object filtering for partial clone
jeffhostetler Oct 30, 2017
afdaced
extension.partialclone: introduce partial clone extension
jeffhostetler Oct 30, 2017
89b80a6
fsck: introduce partialclone extension
jonathantanmy Oct 30, 2017
47cd09c
fsck: support refs pointing to promisor objects
jonathantanmy Oct 30, 2017
067a054
fsck: support referenced promisor objects
jonathantanmy Oct 24, 2017
6c45842
fsck: support promisor objects as CLI argument
jonathantanmy Oct 24, 2017
d205b91
index-pack: refactor writing of .keep files
jonathantanmy Oct 24, 2017
3a7785f
introduce fetch-object: fetch one promisor object
jonathantanmy Oct 24, 2017
5d832f8
sha1_file: support lazily fetching missing objects
jonathantanmy Oct 30, 2017
6ca6e7d
rev-list: support termination at promisor objects
jonathantanmy Oct 30, 2017
3195942
gc: do not repack promisor packfiles
jonathantanmy Oct 30, 2017
a14cbbc
pack-objects: test support for blob filtering
jonathantanmy Oct 30, 2017
f562efd
fetch-pack: test support excluding large blobs
jonathantanmy Oct 30, 2017
0ea4b50
fetch: add from_promisor and exclude-promisor-objects parameters
jonathantanmy Oct 30, 2017
ac98b8b
t5500: add fetch-pack tests for partial clone
jonathantanmy Oct 30, 2017
97f9eb9
t5601: test for partial clone
jonathantanmy Oct 30, 2017
c2ca0eb
t5500: more tests for partial clone and fetch
jonathantanmy Oct 30, 2017
238fea9
unpack-trees: batch fetching of missing blobs
jonathantanmy Oct 30, 2017
ed27342
fetch-pack: restore save_commit_buffer after use
jonathantanmy Oct 30, 2017
d2da45f
index-pack: silently assume missing objects are promisor
jeffhostetler Oct 30, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Documentation/config.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3268,6 +3268,10 @@ uploadpack.packObjectsHook::
was run. I.e., `upload-pack` will feed input intended for
`pack-objects` to the hook, and expects a completed packfile on
stdout.

uploadpack.allowFilter::
If this option is set, `upload-pack` will advertise partial
clone and partial fetch object filtering.
+
Note that this configuration variable is ignored if it is seen in the
repository-level config (this is a safety measure against fetching from
Expand Down
12 changes: 11 additions & 1 deletion Documentation/git-pack-objects.txt
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ SYNOPSIS
'git pack-objects' [-q | --progress | --all-progress] [--all-progress-implied]
[--no-reuse-delta] [--delta-base-offset] [--non-empty]
[--local] [--incremental] [--window=<n>] [--depth=<n>]
[--revs [--unpacked | --all]] [--stdout | base-name]
[--revs [--unpacked | --all]]
[--stdout [--filter=<filter-spec>] | base-name]
[--shallow] [--keep-true-parents] < object-list


Expand Down Expand Up @@ -236,6 +237,15 @@ So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle.
With this option, parents that are hidden by grafts are packed
nevertheless.

--filter=<filter-spec>::
Requires `--stdout`. Omits certain objects (usually blobs) from
the resulting packfile. See linkgit:git-rev-list[1] for valid
`<filter-spec>` forms.

--exclude-promisor-objects::
Silently omit referenced but missing objects from the packfile.
This is used with partial clone.

SEE ALSO
--------
linkgit:git-rev-list[1]
Expand Down
5 changes: 4 additions & 1 deletion Documentation/git-rev-list.txt
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,10 @@ SYNOPSIS
[ --fixed-strings | -F ]
[ --date=<format>]
[ [ --objects | --objects-edge | --objects-edge-aggressive ]
[ --unpacked ] ]
[ --unpacked ]
[ --filter=<filter-spec> ] ]
[ --filter-print-missing ]
[ --filter-print-omitted ]
[ --pretty | --header ]
[ --bisect ]
[ --bisect-vars ]
Expand Down
6 changes: 6 additions & 0 deletions Documentation/gitremote-helpers.txt
Original file line number Diff line number Diff line change
Expand Up @@ -466,6 +466,12 @@ set by Git if the remote helper has the 'option' capability.
Transmit <string> as a push option. As the push option
must not contain LF or NUL characters, the string is not encoded.

'option filter <filter-spec>'::
An object filter specification for partial clone or fetch
as described in rev-list.

TODO document 'option from-promisor' and 'option no-haves' ?

SEE ALSO
--------
linkgit:git-remote[1]
Expand Down
30 changes: 30 additions & 0 deletions Documentation/rev-list-options.txt
Original file line number Diff line number Diff line change
Expand Up @@ -706,6 +706,36 @@ ifdef::git-rev-list[]
--unpacked::
Only useful with `--objects`; print the object IDs that are not
in packs.

--filter=<filter-spec>::
Only useful with one of the `--objects*`; omits objects (usually
blobs) from the list of printed objects. The '<filter-spec>'
may be one of the following:
+
The form '--filter=blob:none' omits all blobs.
+
The form '--filter=blob:limit=<n>[kmg]' omits blobs larger than n bytes
or units. The value may be zero. Special files matching '.git*' are
alwayse included, regardless of size.
+
The form '--filter=sparse:oid=<oid-ish>' uses a sparse-checkout
specification contained in the object (or the object that the expression
evaluates to) to omit blobs not required by the corresponding sparse
checkout.
+
The form '--filter=sparse:path=<path>' similarly uses a sparse-checkout
specification contained in <path>.

--filter-print-missing::
Prints a list of the missing objects for the requested traversal.
Object IDs are prefixed with a ``?'' character. The object type
is printed after the ID. This may be used with or without any of
the above filtering options.

--filter-print-omitted::
Only useful with one of the above `--filter*`; prints a list
of the omitted objects. Object IDs are prefixed with a ``~''
character.
endif::git-rev-list[]

--no-walk[=(sorted|unsorted)]::
Expand Down
8 changes: 8 additions & 0 deletions Documentation/technical/pack-protocol.txt
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,7 @@ out of what the server said it could do with the first 'want' line.
upload-request = want-list
*shallow-line
*1depth-request
[filter-request]
flush-pkt

want-list = first-want
Expand All @@ -227,6 +228,8 @@ out of what the server said it could do with the first 'want' line.
additional-want = PKT-LINE("want" SP obj-id)

depth = 1*DIGIT

filter-request = PKT-LINE("filter" SP filter-spec)
----

Clients MUST send all the obj-ids it wants from the reference
Expand All @@ -249,6 +252,11 @@ complete those commits. Commits whose parents are not received as a
result are defined as shallow and marked as such in the server. This
information is sent back to the client in the next step.

The client can optionally request that pack-objects omit various
objects from the packfile using one of several filtering techniques.
These are intended for use with partial clone and partial fetch
operations. See `rev-list` for possible "filter-spec" values.

Once all the 'want's and 'shallow's (and optional 'deepen') are
transferred, clients MUST send a flush-pkt, to tell the server side
that it is done sending the list.
Expand Down
8 changes: 8 additions & 0 deletions Documentation/technical/protocol-capabilities.txt
Original file line number Diff line number Diff line change
Expand Up @@ -309,3 +309,11 @@ to accept a signed push certificate, and asks the <nonce> to be
included in the push certificate. A send-pack client MUST NOT
send a push-cert packet unless the receive-pack server advertises
this capability.

filter
------

If the upload-pack server advertises the 'filter' capability,
fetch-pack may send "filter" commands to request a partial clone
or partial fetch and request that the server omit various objects
from the packfile.
22 changes: 22 additions & 0 deletions Documentation/technical/repository-version.txt
Original file line number Diff line number Diff line change
Expand Up @@ -86,3 +86,25 @@ for testing format-1 compatibility.
When the config key `extensions.preciousObjects` is set to `true`,
objects in the repository MUST NOT be deleted (e.g., by `git-prune` or
`git repack -d`).

`partialcloneremote`
~~~~~~~~~~~~~~~~~~~~

When the config key `extensions.partialcloneremote` is set, it indicates
that the repo was created with a partial clone (or later performed
a partial fetch) and that the remote may have omitted sending
certain unwanted objects. Such a remote is called a "promisor remote"
and it promises that all such omitted objects can be fetched from it
in the future.

The value of this key is the name of the promisor remote.

`partialclonefilter`
~~~~~~~~~~~~~~~~~~~~

When the config key `extensions.partialclonefilter` is set, it gives
the initial filter expression used to create the partial clone.
This value becomed the default filter expression for subsequent
fetches (called "partial fetches") from the promisor remote. This
value may also be set by the first explicit partial fetch following a
normal clone.
4 changes: 4 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -792,6 +792,7 @@ LIB_OBJS += ewah/ewah_bitmap.o
LIB_OBJS += ewah/ewah_io.o
LIB_OBJS += ewah/ewah_rlw.o
LIB_OBJS += exec_cmd.o
LIB_OBJS += fetch-object.o
LIB_OBJS += fetch-pack.o
LIB_OBJS += fsck.o
LIB_OBJS += gettext.o
Expand All @@ -807,6 +808,8 @@ LIB_OBJS += levenshtein.o
LIB_OBJS += line-log.o
LIB_OBJS += line-range.o
LIB_OBJS += list-objects.o
LIB_OBJS += list-objects-filter.o
LIB_OBJS += list-objects-filter-options.o
LIB_OBJS += ll-merge.o
LIB_OBJS += lockfile.o
LIB_OBJS += log-tree.o
Expand Down Expand Up @@ -836,6 +839,7 @@ LIB_OBJS += pack-write.o
LIB_OBJS += pager.o
LIB_OBJS += parse-options.o
LIB_OBJS += parse-options-cb.o
LIB_OBJS += partial-clone-utils.o
LIB_OBJS += patch-delta.o
LIB_OBJS += patch-ids.o
LIB_OBJS += path.o
Expand Down
3 changes: 3 additions & 0 deletions builtin/cat-file.c
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
#include "tree-walk.h"
#include "sha1-array.h"
#include "packfile.h"
#include "partial-clone-utils.h"

struct batch_options {
int enabled;
Expand Down Expand Up @@ -475,6 +476,8 @@ static int batch_objects(struct batch_options *opt)

for_each_loose_object(batch_loose_object, &sa, 0);
for_each_packed_object(batch_packed_object, &sa, 0);
if (is_partial_clone_registered())
warning("This repository has partial clone enabled. Some objects may not be loaded.");

cb.opt = opt;
cb.expand = &data;
Expand Down
24 changes: 22 additions & 2 deletions builtin/clone.c
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@
#include "run-command.h"
#include "connected.h"
#include "packfile.h"
#include "list-objects-filter-options.h"
#include "partial-clone-utils.h"

/*
* Overall FIXMEs:
Expand Down Expand Up @@ -60,6 +62,7 @@ static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;
static int option_dissociate;
static int max_jobs = -1;
static struct string_list option_recurse_submodules = STRING_LIST_INIT_NODUP;
static struct list_objects_filter_options filter_options;

static int recurse_submodules_cb(const struct option *opt,
const char *arg, int unset)
Expand Down Expand Up @@ -135,6 +138,7 @@ static struct option builtin_clone_options[] = {
TRANSPORT_FAMILY_IPV4),
OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
TRANSPORT_FAMILY_IPV6),
OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
OPT_END()
};

Expand Down Expand Up @@ -886,6 +890,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
struct refspec *refspec;
const char *fetch_pattern;

fetch_if_missing = 0;

packet_trace_identity("clone");
argc = parse_options(argc, argv, prefix, builtin_clone_options,
builtin_clone_usage, 0);
Expand Down Expand Up @@ -1073,6 +1079,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
warning(_("--shallow-since is ignored in local clones; use file:// instead."));
if (option_not.nr)
warning(_("--shallow-exclude is ignored in local clones; use file:// instead."));
if (filter_options.choice)
warning(_("--filter is ignored in local clones; use file:// instead."));
if (!access(mkpath("%s/shallow", path), F_OK)) {
if (option_local > 0)
warning(_("source repository is shallow, ignoring --local"));
Expand Down Expand Up @@ -1104,7 +1112,13 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
transport_set_option(transport, TRANS_OPT_UPLOADPACK,
option_upload_pack);

if (transport->smart_options && !deepen)
if (filter_options.choice) {
transport_set_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,
filter_options.raw_value);
transport_set_option(transport, TRANS_OPT_FROM_PROMISOR, "1");
}

if (transport->smart_options && !deepen && !filter_options.choice)
transport->smart_options->check_self_contained_and_connected = 1;

refs = transport_get_remote_refs(transport);
Expand Down Expand Up @@ -1164,13 +1178,18 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
write_refspec_config(src_ref_prefix, our_head_points_at,
remote_head_points_at, &branch_top);

if (filter_options.choice)
partial_clone_utils_register(&filter_options, "origin",
"clone");

if (is_local)
clone_local(path, git_dir);
else if (refs && complete_refs_before_fetch)
transport_fetch_refs(transport, mapped_refs);

update_remote_refs(refs, mapped_refs, remote_head_points_at,
branch_top.buf, reflog_msg.buf, transport, !is_local);
branch_top.buf, reflog_msg.buf, transport,
!is_local && !filter_options.choice);

update_head(our_head_points_at, remote_head, reflog_msg.buf);

Expand All @@ -1191,6 +1210,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
}

junk_mode = JUNK_LEAVE_REPO;
fetch_if_missing = 1;
err = checkout(submodule_progress);

strbuf_release(&reflog_msg);
Expand Down
14 changes: 14 additions & 0 deletions builtin/fetch-pack.c
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
struct oid_array shallow = OID_ARRAY_INIT;
struct string_list deepen_not = STRING_LIST_INIT_DUP;

fetch_if_missing = 0;

packet_trace_identity("fetch-pack");

memset(&args, 0, sizeof(args));
Expand Down Expand Up @@ -143,6 +145,18 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
args.update_shallow = 1;
continue;
}
if (skip_prefix(arg, ("--" CL_ARG__FILTER "="), &arg)) {
parse_list_objects_filter(&args.filter_options, arg);
continue;
}
if (!strcmp("--from-promisor", arg)) {
args.from_promisor = 1;
continue;
}
if (!strcmp("--no-haves", arg)) {
args.no_haves = 1;
continue;
}
usage(fetch_pack_usage);
}
if (deepen_not.nr)
Expand Down
Loading