npm on IPFS #2
Comments
|
just idle thought? Or discussion? |
|
@whyrusleeping notes from discussion in person :) |
|
As mentioned in the first checkbox, one blocker for this is that someone needs to make a |
|
@jbenet Hmm... After you explained the ipfs docker mount hack yesterday, what if we mount ipfs on node_modules and |
|
@ehd how did that work out? |
|
@whyrusleeping It works :) $ cat add.js
'use strict';
module.exports = function(a, b) {
return a + b;
};
$ ipfs add add.js
added QmXctmnj6tEMSJsb433jKwcdGwuBy4uGMnqva5Nkvp5E7x add.js
$ cat prog.js
'use strict';
var add = require('QmXctmnj6tEMSJsb433jKwcdGwuBy4uGMnqva5Nkvp5E7x');
console.log(add(1, 2));
$ node prog.js
3Two things:
|
|
@ehd thats great! what would the ideal interface be for you to want to use ipfs for js modules? |
|
Ideally we'd have a simple bootstrapping js/node program in ipfs that can Another related idea we've bounced: storing computational results in ipfs. Side note: Ironically I'm in a place without wifi and just looked up |
|
Update on the list above: @krl made https://github.com/ipfs/node-ipfsd-ctl @ehd all of that sounds awesome!
Yeah! like a special Btw, what @bengl and I want to do above with
that sounds awesome
@krl's https://github.com/ipfs/node-ipfsd-ctl can make ephemeral ipfs nodes using the go-ipfs package, so you dont even have to install ipfs beforehand. it's huge (cause go-ipfs is huge) but once we have node-ipfs this will be a drop-in replacement.
Absolutely! And-- i want to make a way to do browserify with partial results separated out. THOUGH, rabin fingerprinting will get us 80% of the way there without the worries anyway.
Yeahhh i really want this. wonder what the overhead + pains of doing this might be... hard to get right because people dont hash things or use cache headers correctly :/ |
You'd have to patch node. All the There are also many lifecycle scripts and compilation steps and all kinds of other things that are done to modules during the install process. In some modules, the results of these scripts are different depending on where they are in the I think it's much more practical to keep a strict separation between the source of packages (ipfs/ipns, in an ideal world) and the current install directory. Requiring them directly might work for JS-only modules with no lifecycle scripts. But then you're not supporting a huge chunk of available modules. I'm only really talking about node here, but for people using frontend build pipelines that depend on npm, the same problems can arise. So yeah, I'm not sure grabbing node dependencies directly from ipfs at run-time is realistic. |
|
@bengl ah thanks! makes a lot of sense. i guess we could do it to specific packages people decide to do this with (i.e. people explicitly publish them + require them with hashes, so they take on the responsibility of not making all the lifecycle stuff happen?) not sure. |
|
Yeah for sure. ipfs-pm/ipfspm/ippm. You'd pretty much have to make a clear separation from npm at that point, I'd think. I still would like to see the ipfs leveraged as the package source, as we discussed before. Should be easy once |
|
@bengl maybe we can hack that ( |
|
Definitely, I took a look at the spec today. @bengl do you have any of the work you've made towards it published anywhere? |
|
So really we're looking at potentially two different use-cases for Use Case 1: Raw Blob StoreIn this case, we're really just plopping blobs onto IPFS and getting their IPFS paths back in the metadata for later retrieval. So in Use Case 2: IPNS Mutable File System ThingIn this case (which could use the previous case under the hood), in the constructor for the blob store, we'd provide all the necessary data to read and write to an IPNS namespace (keypair? just the hash? Sorry, my IPNS knowledge is rather limited). Here, Unfortunately I don't spend as much time on this as I'd like, so I'm unfamiliar with how things actually work in IPNS-land right now. Sorry! Here's the abstract-blob-store API reference. |
|
@bengl yep, you've got it right :) We'd want a third module: |
|
Pardon my late replies, I’m on a vacation until Thursday.
That’s true. But you can change the meaning of
That’s unfortunately true, but I think we can do lots of useful stuff without modules that rely on native compilation. Most compilation I see when running |
|
First stab at https://github.com/ipfs/ipfs-blob-store Currently uses raw blocks, and is only content addressable. It does not use ipns at the moment, passes 45 of the tests (out of a bunch more, I don't fully understand tape). |
|
i think it's time to resurrect this effort. |
|
npm uses tar right? |
|
👍 |
|
👍 💥 |
|
Looking for some clarifications:
this is so awesome! |
Just
No.
Yes, that's the first thing |
|
Hi. Trying to make sure I grok the big picture here, in terms of what an ipfs-npm request looks like. Please correct me as I go. ;)
How does |
|
@whyrusleeping helped me better understand the intent here. Looks like I was getting ahead of myself; it sounds like the first goal is to opt for decentralized before distributed: let users run their own npm registry that uses IPFS as the blob store rather than the local fs. Lingering Q: how will our reginabox map module names to IPFS addresses? |
|
(decentralized first, then distributed: https://ipfs.io/ipfs/QmZpfhN5rucQC4kx7Gu5udS8FXxsxMddiDCvG8WFjG8SMv) |
|
Just PR'ed ipfs-blob-store using mfs (mutable file system) ipfs-shipyard/ipfs-blob-store#5 @bengl I'm being unable to attach a custom blob-store to registry-static, looking at the --help menu, the --blobstore option doesn't appear (for ref posted the help menu below) and when I run something like Although I have the module exporting the blob store: $ ls -al
total 464
# ...
-rw-r--r-- 1 david staff 84 Oct 12 19:36 registry-ipfs-blob-store.js
# ...Could you give me a hint on how to do it? Thank you :) note:
|
|
(The root ipns name would point to another ipns key, the current signing key from a rotation, etc. that way can revoke it by moving ipns -> ipns ptr at a given time, bounded only by record lifetimes) — On Tue, Nov 3, 2015 at 9:55 AM, Stephan Seidt [email protected]
|
|
@ehd that is similar to what we are doing, where we have one PEER that has the entire npm by a similar folder structure (the one used in registry-static) and then the others, by copying that root path |
Sounds great. Can't wait.
What do you mean by "the others"? Here's an interesting project by @mjackson: http://npmcdn.com – It's a heroku app with a CDN in front, serving files out of npm packages. Idea: Use trusted computation instead of trusted names for this problem (Distributed request network?)
Someone else executes it for me, caches the result. I trust them to provide If I paid them to do this for me, and they provided a binding legal contract: Very nice and trustworthy. You know who's got at providing a useful and stable environment for running code? CI services. There probably is a nicer blockchainy/ethereum smart contract way, but I'm not well versed here. Something like npmcdn would be trivial with this: /ipcompute/funhash/inputhash -> result? There are many things we could use that work like npmcdn Benefit:
Q:
|
|
@ehd like Amazon lambda/WebTask for the network? That is a field that is very very exciting and something we keep going back, it is definitely a valid idea. Now when it comes to npm on IPFS, what we want is to be able to install the modules from the network, which is far less ambitious that having a universal way to do computation in a P2P network (btw I did work towards that goal in my MSc thing that I got to share at DTN last May, we can talk more or open a issue about it :):)). The main challenge was, without forcing users to adopt a new way to install their modules, how could we offer the same Quick update on state of things
|
Yeah, it's so cool! I'll try to visualize my idea on the weekend.
And that's awesome. It'll be really cool and useful to have the that for npm end users and also the idea above. It relies on some system serving all the archives and metadata, and using npm on ipfs instead of the registry brings those benefits with it. |
So, right now, using IPFS 0.4.0, with
This is to download the entire npm into an IPFS Node, once our fat Node has finished that, you will be able to do:
@bengl, I wanted to make this setup more nice, without all of the installing steps, any suggestions? Would it be good to have all of this setting up be part of reginabox, so that devs only had to do |
|
@diasdavid how about adding another module |
|
So it would more like this: $ npm i -g reginabox-cli
$ ipfs daemon
$ reginabox mirror npm-registry
# internet connection lighting up because of all the traffic |
|
@dignifiedquire, regianbox is already a CLI :) so I guess we just need the changes on https://github.com/diasdavid/reginabox to go to https://github.com/yahoo/reginabox |
|
hmmmmmmmm now that you say it like that cough ignore my comment, but it would still be nice to put all that manual work you mentioned in there or somewhere else to automate it more |
|
It has begun /Volumes/npm/custom-blob-store
❯ ls npm-registry |wc -l
185 |
|
👍 |
|
|
I'd suggest that a module that just implements the registry API by talking to IPFS would be sufficient, rather than having everyone use That way users could do something like: and then start that and use it as a registry in their
So, in summary, I'd say: a small module implementing only the http server that serves from Also, the fat server that's doing the following of the public registry doesn't need |
|
@bengl ok, sounds good, starting https://github.com/diasdavid/registry-mirror :) |
|
@diasdavid @bengl registry-static is pretty slow :( not really using the full connection or cpu available. I'm only at 10% after 20h even though my connection could have downloaded 800gb in this time according to my calcs |
|
@dignifiedquire No excuses, but here's an explanation of what's going on: That being said, PRs to have more efficient pipelining of HTTP requests in |
|
@bengl created a PR to follow-registry to improve performance :) |
|
Published the first version https://github.com/diasdavid/registry-mirror Wanna give it a try? Now it is only registry-mirror --ipfs --clone` and good to go :) |
|
I think replicating the entire registry into ipfs is the wrong approach here, as the popularity of npm will make the replication into another system a constant catchup. The use case really seems to speed up npm in places of poor internet connectivity to the npm registry — australia, conferences, islands, etc. The usage can be opted into. Essentially one only needs to replicate the files into ipfs that they are using. Essentially one could alias "npm install" to be "install from ipfs if there, otherwise install from npm then publish to ipfs" — one would need a ipns space to alias package names and versions to directory hashes, however I imagine the registry also has the hashes so would be easy enough to verify (is this npm install alias publishing the same version to hash link as to what the npm registry indicates). An alias could also be creates for "npm publish" to publish to ipfs after publishing to the registry. While not comprehensive would allow moderare opting in abilities. |
|
It's really time for NPM on IPFS: How one developer just broke Node, Babel and thousands of projects... |
|
Perhaps @rstacruz from http://github.com/rstacruz/pnpm may be interested in this? pnpm's architecture works really well for my ideal here, as pnpm I believe downloads each package to user directory, then symlinks the package directories to the relevant Combining this with IPFS, this user This works for hosting files, but doesn't work for hosting meta data and publishing. How is that planned to be addressed? Specifically publishing. |
ipfs-blob-storeipfs-daemon-ctrlcontrol whatever daemon is installedipfs-blob-storetoreginaboxcc @bengl
The text was updated successfully, but these errors were encountered: