Hosting Web Apps on IPFS

[cwaring]

Special Edition: IPFS Docs v2 featuring VuePress

Thanks to @hugomrdias and @lidel for chatting through all various scenarios yesterday, below are a few thoughts and next steps balanced against time/effort and overall impact.

The first (and perhaps most important) thing to note is that this problem is not isolated to VuePress, and is currently causing many common web applications with client-side routing and absolute paths to appear broken when loaded via a gateway URL, such as http://gateway.ipfs.io/ipfs/<cidv0>.

In order to make IPFS a first-class citizen for hosting web apps, the consensus is that we should move to subdomain model as soon as possible (e.g. https://<cidv1b32>.ipfs.dweb.link).

• This provides origin isolation for app security and session storage, while also fixing client-side routing and asset issues that people are facing with some of the most popular webpack application bundlers that currently rely on absolute paths (a few examples are Next, Gatsby, create-react-app, Nuxt, and VuePress).

• Additionally, this change will provide familiarity to new users attempting to host these type of web apps on IPFS — thus lowering the barrier to adoption and increasing our reach.

The subdomain pattern is a familiar one for developers. Most hosting services use a https://<subdomain>.<service>.<ext> pattern, and with the recent efforts/refactors across our projects we are very close to making this a reality for IPFS too. To that end, we believe we should invest our effort towards that direction instead of patching frameworks on a case-by-case basis.

With that said, here is a summary of our current options:

Option 1: Make IPFS Companion "redirect to gateway" opt-in and default to dweb links

---

[certainty: high, effort: medium, impact: high]

This option ensures that IPFS Companion will not automatically push users to load a resource via their local gateway node by default, leaving people with a broken experience when an app fails to load correctly. @lidel also suggests that we should move to an opt-in model, so that users have the option to explore sites via their local gateway instead of forcing this as default behaviour. Ultimately, an in-browser proxy would allow us to use cidv1 urls locally alongside dweb links; however, this won't be the case until further work has landed in the Chrome/Firefox vendor space.

For this to work successfully, the following tasks must be completed:

• Make ipfs-dns-deploy use cidv1b32 by default with a dweb preview link.
• Change default behaviour of IPFS Companion to disable auto redirects on DNSLink websites. PR in progress..
• Design a bootloader plugin for VuePress that detects the latest deployed CID and redirects to a cidv1b32 dweb link.

This does mean that a VuePress site will not run locally until we have a subdomain gateway in go-ipfs that we can use as a proxy in ipfs-desktop. However, we believe this is an acceptable tradeoff in the meantime, as this is a niche use case — particularly considering that the main app will provide PWA functionality with offline caching via service workers (enhancing the experience on all platforms, including mobile).

Related issues/efforts:

• CID in subdomain: CID as a Subdomain ipfs/in-web-browsers#89
• Hostnames for localhost HTTP gateway: Hostnames for localhost HTTP gateway ipfs/in-web-browsers#109
• Act as HTTP PROXY for http://.ipfs.localhost: Act as HTTP PROXY for http://<cidv1b32>.ipfs.localhost ipfs/kubo#5982
• Subdomain gateway: Subdomain Gateway ipfs/kubo#6498
• Disable default redirect of DNSLink websites if it breaks Origin: Disable default redirect of DNSLink websites if it breaks Origin ipfs/ipfs-companion#701

Option 2: Create a plugin to rewrite VuePress compiled output

---

[certainty: medium, effort: medium, impact: low]

As sketched out in this gist, we could build a post-process plugin that rewrites all the absolute paths to relative ones, making an app work regardless of path depth. However, this could have a number of unknown and potentially experience-breaking side effects such as interfering with interactivity, routing, failure of offline caching via PWA, and uncertainty around how plugins may respond.

This is approach would be similar to gatsby-plugin-ipfs and may be acceptable for simple sites — but post-processing compiled assets with regex is a fragile and messy solution. We believe this should only be done as an absolute last resort or a stopgap option.

Option 3: Fix the problem upstream in VuePress

---

[certainty: low, effort: high, impact: medium]

We could apply development effort to fix this issue upstream in VuePress. Such work would help resolve additional absolute-path-related issues for other users; however, it's strongly unlikely that we could solve the key issues, patch affected plugin dependencies, and land a merged PR in a timely fashion without unnecessarily delaying the launch of IPFS docs and losing valuable beta user testing time.

The effort here is complex, involves consensus with multiple third parties, and only adds value to a niche use-case. If absolutely necessary, we could investigate pursuing this in the new year, but our recommendation is to avoid postponing beta launch by pursuing one of the other options in the meantime.

Related issues:

• Build with Relative URLs: Build with Relative URLs vuejs/vuepress#796
• Publish source code for vuepress-html-webpack-plugin or unfork it: Publish source code for vuepress-html-webpack-plugin or unfork it vuejs/vuepress#1303

Option 4: Use GitHub Pages, Netlify or similar (stopgap)

---

[certainty: max, effort: zero, impact: 🤷‍♂️]

We are setup this way at the moment and it has enabled us to focus on delivering end user functionality in the docs beta until there was a solution in place to fully migrate to cidv1b32.

Note that Filecoin and IPLD docs are currently configured like this too.

Summary

We are keen to pursue option 1 for maximum overall impact, however I'd like to hear your thoughts. 👂✨

ref: #271

[terichadbourne]

I love the idea of pursuing the goal that helps the most real-world developers host sites on IPFS without having to alter the rest of their stack, including in the very common pattern of client-side routing.

Quick clarifying question on this aspect of Option 1, and apologies in advance because I know my vocabulary isn't up to asking it properly:

This option ensures that IPFS Companion will not automatically push users to load a resource via their local gateway node by default, leaving people with a broken experience when an app fails to load correctly. @lidel also suggests that we should move to an opt-in model, so that users have the option to explore sites via their local gateway instead of forcing this as default behaviour.

Is this about...

(A) letting people choose between accessing the site on "THE gateway" - the thing we run ourselves that is sometimes down - versus "THEIR gateway node" - which I assume is the equivalent of a local/offline cache, only available if they've visited and pinned the site before?

(B) giving people the option to view the site on the "normal internet" as a fallback if "the gateway" (access to IPFS) is down?

Kudos in advance to whoever can parse my question. :)

[olizilla]

how do you like these apples ipfs-shipyard/ipfs-dns-deploy#7

[jessicaschilling]

@olizilla I like these apples very much, they are delicious

[olizilla]

@terichadbourne Yes! it's about always loading the "normal internet" (aka http) version first, and informing the user that an ipfs flavour version of the site is available if they would rather fetch it that way. I assume it would also mean "fallback to ipfs" if http is unavailable.

[lidel]

Starting with IPFS Companion v2.9.0.872 user will load DNSLink website in "normal" way by default (over original HTTP server), but still get context actions including new one "View on Gateway" for opening via Gateway (public or local) in a new tab.

If docs.ipfs.io (or any other DNSLink website) is down, IPFS Companion will recover from Gateway in a new tab (public or local, if user runs their own), but by default we utilize original HTTP server (a public gateway) for the best UX.

To borrow illustration from ipfs/ipfs-companion#826 (comment):

[momack2]

I’m not sure I’ve fully processed option #1 - but my understanding is the suggestion means the vuepress site will still be broken when loaded over ipfs, but to effectively turn off loading over ipfs without explicit user interaction for most people browsing with companion/desktop so they’re less likely to experience this issue in our docs site. Is that an accurate summary? In that model, what would happen if I tried to reload the site from my local gateway (via the button you describe), cohost it using ipfs-cohost, share it with someone else by ipns or ipfs cid, or otherwise fetch the site over the ipfs network? Is there a different answer if I’m using an embedded js-ipfs node vs ipfs-desktop?

More broadly, I don’t think we want to default ipfs-powered sites to use http on first load. Can you explain more why you think that is necessary (@lidel)? Of note, I don’t think a performance reason should apply here - this feedback loop is actually extremely critical for users of companion (frequently developers dogfooding) to understand how long it takes to load from ipfs and we don’t want to remove that dogfooding option from our network use. The solution to that problem is making ipfs more performant, not relying on http for first load. No one is going to regularly use a drop down menu in an extension, even if we make some sort of alert that”this site is ipfs-powered!” - clicking that button would just reload the site slowly, making ipfs look even worse in comparison - so I don’t see that as a mollifying replacement. Most concerningly, this transfers a large number of requests from the actual ipfs network (right now resolved with js-ipfs in companion or go-ipfs in ipfs-desktop) to the ipfs http gateway - which is the opposite direction than we want to be heading and significantly reducing the frequency with which users would interface directly with the ipfs network.

I plan to run the “default-redirect in companion” decision by @jbenet for clear BDFL direction-setting on how companion, embedded nodes, and the gateway should fit together because this would be a large, noticeable, and and high-impact change that has other longer-term strategy implications. Will try to get an answer by Monday so we can fully decide between option 1 and option 2 (agree option 3 is nice, but not achievable on this timeline). Please add context to help clarify options for his review!

Edited: let's have a meeting on Monday CET to discuss the pros/cons of this change before surfacing to @jbenet (IPFS BDFL) for approval/ack.

[olizilla]

I've wanted this for a long time ipfs/ipfs-companion#710

[terichadbourne]

I too look forward to hearing a little more about the order of operations for Option 1. In my experience with standard Offline First patterns and PWAs, I've found that the best user experience is to load first from local cache if you have it (fastest possible load time and avoiding getting stuck in LieFi state but possibly outdated information), then immediately try to pull an updated version from the network. What counts as "the network" is the challenge here. 😂 (And I'm super excited to see the goal of making our docs accessible offline via a PWA! 🎉)

I'd also love clarity on what the Option 1 experience would be like if I visited the new docs site having never installed IPFS or IPFS Companion. We've pushed a lot of content that would normally be on a marketing site into the docs site, so I don't think we should assume that the only folks referencing it are IPFS aficionados.

Just a few thoughts from the vantage of someone who's a self-appointed advocate for newer users, among whom I see "(How) can I host my web site / web app on IPFS?" as one of the most frequently occurring questions...

By pure numbers, there are far more traditional web developers out there than there are DWeb/IPFS enthusiasts who already build on IPFS and have all of our tools installed, and I happen to be on the "more familiar with traditional web dev, new to IPFS" end of that spectrum. The audience of traditional but DWeb-curious web devs -- including those rallying around Offline First and PWAs -- is a huge potential market for adoption in my opinion, and one we should keep in mind as we make decisions. What would they do if faced with our current decision-making process, and how can we best serve them moving forward?

If a traditional web dev were already familiar with tools that met her needs in web app development (VuePress for a documentation site, tools that easily support client-side routing, traditional PWA stacks, etc.), and she bumped into a poor fit between IPFS and those trusted tools, which would win out... her curiosity to try out hosting on IPFS or her reliance on trusted tools of the trade? Recognizing the use cases for which IPFS isn't currently the ideal solution, and responding by making IPFS compatible with a range of existing tools to resolve that (but only when we consider that use case to be a key one for us) is key to increasing adoption and improving our user experience, and I really appreciate the careful thought being put into long-term solutions that will meet the needs of diverse users. ❤️

My concern with creating a plugin for Vuepress (Option 2) or solving things upstream in Vuepress (Option 3) is that both approaches solve a compatibility issue within a very limited scope (as I understand, in a way that won't also apply to the many other stacks with this client-side routing challenge), leaving the problems of hosting web apps on IPFS unaddressed overall. I would definitely lean towards finding solutions within IPFS itself where can have a greater and broader long-term impact. Unless we expect future users to individually create plugins that make IPFS compatible with specific widely-used tools for building web apps, the process of doing so ourselves strikes me as a bit contrary to the dogfooding concept of putting ourselves in our users' shoes. ¯_(ツ)_/¯

As a newer IPFS user from the traditional web dev community, I'm really excited about the possibilities offered by Option 1 as I understand them, including:

• resolving long-term issues with web app routing and paths to increase adoption
• offering great offline access to docs immediately
• collecting user feedback on the new site design as quickly as possible

Thanks to all of the less-new users who are taking the time to dig into all of these options so thoughtfully. You rock! 🎉