Provide tarballs for offline-builds

[the-kenny]

Type: feature
Area: build

Many distributions try to switch to fully-reproducible builds for their packages. Fetching stuff from external sources (ipfs, http, git, etc.) is a no-go in such situations. Some package managers like Nix even go as far as blocking all access to resources outside the specified bounds (including network access).

It would be helpful if ipfs-go could provide tarballs which include everything necessary to build ipfs, including all dependencies that would be fetched over the network in a normal build.

Hey, thanks for bringing this up! We generally agree that reproducible builds are something that go-ipfs should be capable of.

Many distributions try to switch to fully-reproducible builds for their packages. Fetching stuff from external sources (ipfs, http, git, etc.) is a no-go in such situations. Some package managers like Nix even go as far as blocking all access to resources outside the specified bounds (including network access).

It's usually just fine to include hashes with every external source you fetch. I'm wondering what the reasoning is to forbid fetching things from the network? Is it to allow fully offline builds? I think that's a different matter from reproducibility.

It would be helpful if ipfs-go could provide tarballs which include everything necessary to build ipfs, including all dependencies that would be fetched over the network in a normal build.

I disagree -- instead of bundling/vendoring everything, we should aim for complete verification of all dependencies. Using Gx is already a great step forward, but we should aim to have all our dependencies stored in IPFS, so that we can address them with a single multihash. Note that IPFS provides self-authenticating data structures, which solve a big part of the problems on the way to reproducible and verifiable builds.

[the-kenny]

I disagree -- instead of bundling/vendoring everything, we should aim for complete verification of all dependencies. Using Gx is already a great step forward, but we should aim to have all our dependencies stored in IPFS, so that we can address them with a single multihash. Note that IPFS provides self-authenticating data structures, which solve a big part of the problems on the way to reproducible and verifiable builds.

That's fine for us, iff ipfs guarantees that the output will be byte-to-byte identical over time, even over releases of gx and ipfs (which it does I suppose).

Is it possible to use gx to fetch all dependencies prior starting the build, or do some of them depend for example on the version of go used?

That's fine for us, iff ipfs guarantees that the output will be byte-to-byte identical over time, even over releases of gx and ipfs (which it does I suppose).

I'm not sure it is right now, and what the state of reproducible builds with Golang generally is at the moment, but yes it should. Only changes of package.json or Golang should result in different binaries.

Is it possible to use gx to fetch all dependencies prior starting the build, or do some of them depend for example on the version of go used?

The dependencies are specified in package.json and gx install fetches them. I'm not sure Gx currently has a command for verifying dependencies that have already been fetched and stored in the filesystem. (cc @whyrusleeping)

Anyhow, please let us know however we can help! :) To my knowledge you'd be the first to roll reproducible builds of go-ipfs and there's probably a lot we can learn from and improve.

[the-kenny]

Lars Gierth notifications@github.com writes:

I'm not sure it is right now, and what the state of reproducible
builds with Golang generally is at the moment, but yes it should. Only
changes of package.json or Golang should result in different binaries.

That's nice to hear! We'll see if go causes problems. As far as I know
it hasn't caused any non-reproducible builds yet.

The dependencies are specified in package.json and gx install
fetches them. I'm not sure Gx currently has a command for verifying
dependencies that have already been fetched and stored in the
filesystem. (cc @whyrusleeping

That's very good! The last time I tried to package a recent version of
ipfs (two months ago or so) I was afraid to touch gx. I'll give it
another try soon(ish).

Anyhow, please let us know however we can help! :) To my knowledge
you'd be the first to roll reproducible builds of go-ipfs and there's
probably a lot we can learn from and improve.

Getting rid of downloads (or other network access) in the build goes a
great length to that, and from what I hear it should be possible. I
don't have enough time at my hand right now but maybe in the next days.

Thanks!

[whyrusleeping]

@the-kenny If you want, you can do a gx install --local and it will put all the source for all the dependencies inside the vendor folder. You could then tar up the whole directory and have your entire batch of source ready for reproducible offline builds.

In the future we want to add in the ability to select a specific compiler through gx, meaning that everyone who builds ipfs at a certain version wouldnt just get the same source, but the same compiler and all dependencies, which if i'm not mistaken (i.e. go doesnt have weird timestamps in the build process) should result in a fully reproducible build.

[Kubuxu]

I recall issue with Golang where as the compilation and linking process are performed concurrently the order or symbols in binary wasn't stable but it might have changed.

Also there is problem with build paths: golang/go#16860 but this could be solved by building in chroot, I think.

[jbenet]

I'm confident golang will get to fully reproducible builds, if it's not
there yet. May be matter of time.

Would be great to start packaging go compiler itself with gx so that gx install --local --dev-tools or something could fetch all the things and
then they could be bundled.

@the-kenny thanks for work on this. Bonus points if you use ipfs and
ipfs get instead of tar to move stuff around ;D -- they don't have to be
connected globally.

We probably should have a tool like ipfs-rcp <from> <to> that uses
ephemeral ipfs nodes with persistent local repos to move stuff across only
two machines, without touching any other nodes.
On Tue, Sep 6, 2016 at 8:51 PM Jakub Sztandera notifications@github.com
wrote:

I recall issue with Golang where as the compilation and linking process
are performed concurrently the order or symbols in binary wasn't stable but
it might have changed.

Also there is problem with build paths: golang/go#16860
golang/go#16860 but this could be solved by
building in chroot, I think.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#3195 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAIcoSBoAm552CNQOVXDNetBQAI0zSJUks5qnbZBgaJpZM4J134Z
.

[Kubuxu]

Looks like golang builds themselves are reproducible:

sha256sum *
d15b90d23c5249d443f243713017f15a26503ae3514f1d20660df96ebf4ac1b6  build1
d15b90d23c5249d443f243713017f15a26503ae3514f1d20660df96ebf4ac1b6  build2
d15b90d23c5249d443f243713017f15a26503ae3514f1d20660df96ebf4ac1b6  build3
d15b90d23c5249d443f243713017f15a26503ae3514f1d20660df96ebf4ac1b6  build4

I did remove the pkg cache after the 2nd one.

[Kubuxu]

Resolved by #4920