Plans to add ls, object.stat, swarm.connect?

[bmann]

We’re planning to use a shared IPFS worker on our webnative apps, so your data is available instantly across all those apps. We’ve added a HTML file and worker JS file to our auth lobby, so you can load that in using an iframe.

Staging version of that HTML file:
https://round-aquamarine-dinosaur.fission.app/ipfs.html

Ideally you won’t need to think about iframes at all, and the webnative sdk will handle that for you.

For this to work we would need a few extra methods on ipfs-message-port-client:

• ls, listing directory contents.
• object.stat, getting the size of directories and files.
• swarm.connect, connecting to peers.

Are you planning on adding these methods?

Also posted to our forum https://talk.fission.codes/t/shared-ipfs-worker/996 cc @icidasset

[welcome]

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

• "Priority" labels will show how urgent this is for the team.
• "Status" labels will show if this is ready to be worked on, blocked, or in progress.
• "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

[achingbrain]

Yes, we're planning on fleshing out the API so thanks for mentioning the methods you use, it helps us prioritise accordingly.

We're trying to retire the object API, have you considered using ipfs.files.stat to get the sizes of directories & files instead?

E.g.

const stats = await ipfs.files.stat('/ipfs/Qmfoo')

[icidasset]

@achingbrain Thanks! We'll try out ipfs.files.stat. I didn't know you could use the MFS functions with a /ipfs/ prefix? Or is that only applicable to ipfs.files.stat?

[achingbrain]

The docs could definitely be improved here (and also js-IPFS supports IPFS paths in a few more places than go-IPFS) but the IPFS namespace is overlaid on top of the MFS commands so you can use /ipfs/Qmfoo style strings as paths where you're not trying to mutate an IPFS path.

Eg:

✅ ipfs.files.cp('/ipfs/Qmfoo', '/path/in/local/mfs') - copy a file from the wider IPFS namespace into your MFS
⛔ ipfs.files.cp('/ipfs/Qmfoo', '/ipfs/Qmbar') // ipfs is immutable, this makes no sense
⛔ ipfs.files.rm('/ipfs/Qmfoo') // ipfs is immutable, this makes no sense
✅ ipfs.files.stat('/ipfs/Qmfoo') - get stats on an IPFS path
✅ ipfs.files.ls('/ipfs/Qmfoo') - list the contents of an IPFS path
etc

[Gozala]

We’re planning to use a shared IPFS worker on our webnative apps, so your data is available instantly across all those apps.

Hi @bmann thanks for reaching out! I'm happy that fission team is looking into adopting shared worker and would love to help make that effort succeed. I am also happy to jump to a call so we can sync up on higher bandwidth channel.

We’ve added a HTML file and worker JS file to our auth lobby, so you can load that in using an iframe.
Staging version of that HTML file:
https://round-aquamarine-dinosaur.fission.app/ipfs.html

I am afraid I don't have enough context about auth lobby or what that iframe is meant for, but I am a guessing that different apps run on different origins and iframe is meant to provide shared origin for the shared worker. If my guess is accurate that is pretty much the plan for shared ipfs node across origins. There are some details about it here https://hackmd.io/@gozala/H1LqrvvcI#Cross-Origin

Ideally you won’t need to think about iframes at all, and the webnative sdk will handle that for you.

Absolutely! That is a plan, but there are quite a few things that need to be hammered out (which linked document hints on), like:

• upholding origin isolation
• storage quota management
• finding a good way to share iframe without introducing centralization.

For this to work we would need a few extra methods on ipfs-message-port-client:

• ls, listing directory contents.
• object.stat, getting the size of directories and files.
• swarm.connect, connecting to peers.

Are you planning on adding these methods?

We have arrived to the current API subset based on the feedback from teams that were interested in this effort. It would really help to get a better understand what are concrete APIs used for so we can meet those needs instead of aiming for full API compatibility with JS IPFS, mostly because doing so in an efficient way is going to be challenging and so far we have been finding that what devs need can be achieved in much simpler and efficient ways.

[bmann]

@Gozala we’re in Discord chat at https://fission.codes/discord or send me an email at boris @ fission.codes — would love to get on a call.

[icidasset]

Hey @Gozala

I am a guessing that different apps run on different origins and iframe is meant to provide shared origin for the shared worker.

You guessed correctly ☺️
Let me know if I need to expand on this.

It would really help to get a better understand what are concrete APIs used for so we can meet those needs instead of aiming for full API compatibility with JS IPFS.

Makes total sense.
(1) We can probably remove the need for object.stat, by replacing it with files.stat as suggested previously.

(2) We want ipfs.ls, because:

• We need to tell if a link is a directory or not
• Modification time of files & directories
• Size of files & directories

We could keep track of all that metadata ourselves, but I guess that would defeat the purpose of ipfs.ls? Correct me if I'm wrong.

(3) And lastly, we want swarm.connect, to well, connect to peers. We're currently using this to ensure our connection to bootstrapped nodes. We sometimes lose the connection to those nodes (page could be open for days), so we ensure that connection using swarm.connect.

[Gozala]

Let me know if I need to expand on this.

Thanks for clarifications, they do help. I do however still would like to understand more generally the use case. The reason is that all the teams I have talked to mostly used Dag API with IPFS network for storing and retrieving data, and tiny subset of file API mostly to refer to it from the data points. There is also yet unmet need for pubsub API. Given that insight we started thinking about https://github.com/ipfs/js-dag-service

If fission's use cases and requirements are different it would really help to understand in which way, which can be little tricky to gather from pure API requirements.

Makes total sense.
(1) We can probably remove the need for object.stat, by replacing it with files.stat as suggested previously.

(2) We want ipfs.ls, because:

• We need to tell if a link is a directory or not
• Modification time of files & directories
• Size of files & directories

All that information can be obtained from ipfs.files.stat, if you do want to list directory entries that obviously would require ls.

It would be also good to understand if MFS fits into fission in some way or if it is irrelevant from fission's standpoint.

We could keep track of all that metadata ourselves, but I guess that would defeat the purpose of ipfs.ls? Correct me if I'm wrong.

You definitely should not have to do that. If you do still need ls please let me know, and I'll write a patch to add it.

(3) And lastly, we want swarm.connect, to well, connect to peers. We're currently using this to ensure our connection to bootstrapped nodes. We sometimes lose the connection to those nodes (page could be open for days), so we ensure that connection using swarm.connect.

Having to dial just to reestablish lost connections seems like a workaround to the problem that should not be there in first place. Meaning that connection manager should just do it (and I know where is major overhaul is in progress there).

It is also kind of an API I'm bit hesitant towards, because with node sharing across origins (or native support in browsers) it's going to require some extra caution. That said I would really like to support your effort in adopting shared worker. I think it would be best to move the piece of logic that ensures connections to the bootstrap nodes to the worker side, where you have full IPFS node available. If for some reason app logic needs to guide this process (meaning you need input from the app state to decide what to connect to) I would really like to understand that better.

[icidasset]

I do however still would like to understand more generally the use case.

Definitely! The use case is a decentralised file system, with a public part and private/encrypted part. That file system is shared between applications which can store data on there. So like other file systems, you have files and directories, we need to know the creation/modification timestamps and the sizes of the files and directories. Apps have access to specific parts of the file system, and permission is given by the user (on that auth lobby we talked about earlier). If you'd like to know more, we have a whitepaper that explains it in full detail: https://whitepaper.fission.codes/file-system/file-system-basics

Here's our app called Drive which you can use to browse through your entire file system.

All that information can be obtained from ipfs.files.stat

Oh cool, didn't know that! Also saw in the code that the mtime is provided, which is not mentioned in the documentation.

if you do want to list directory entries that obviously would require ls

Yeah, we need to list directory entries.
Although, with our file system implementation, we might be able to do it without, but I'm not entirely sure.
That said, we would need to call ipfs.files.stat for each directory entry, I'd think ls is probably more performant?

So yeah, if you could add ls that'd be awesome.

Having to dial just to reestablish lost connections seems like a workaround to the problem that should not be there in first place.

For sure 💯 Good to know it's being worked on. In that case we won't need swarm.connect.

I think it would be best to move the piece of logic that ensures connections to the bootstrap nodes to the worker side, where you have full IPFS node available

Makes sense 👍