Tracking issue for Ember FastBoot

[jtgeibel]

Recently I've been experimenting with the FastBoot PR, and below is my attempt at capturing the work that I think remains to enable FastBoot across the site.

Enable FastBoot for some static pages

As we previously discussed, the existing PR allows us to land support on a single static page to try this out in production:

• /policies - added in Enable Fastboot with USE_FASTBOOT environment variable set to 1 #1900

Groundwork for enabling FastBoot on dynamic pages

Some tasks that I think will be necessary support more dynamic cases:

• Find a way to show the <noscript> message only on pages that haven't been converted yet. - Tweak <noscript> text and placement #2101 tweaks the message and placement, but leaves the message present on all pages.
• Ensure a complete set of headers is sent, based on what we do in the backend middleware. - Move security-related HTTP response headers from Rust to nginx.conf #2100
• For a logged in user, their cookie should be included in the request to the backend.
• When JS is enabled, after the pre-rendered HTML is loaded and Ember boots, a new set of requests are sent from the client to the backend. - ember-data-storefront added in Don't make duplicated requests through Ember Data #2087
• Requests to routes (/does-not-exist) or resources (/crates/does-not-exist) that don't exist, should respond with status 404 instead of 200.
• Update FastBoot logs to align with our typical log format, as described in Tracking issue for Ember FastBoot #1811 (comment) and Enable Fastboot with USE_FASTBOOT environment variable set to 1 #1900 (comment)
• Pass User-Agent header to backend - fixed in adapters/application: Pass User-Agent header to backend in fastboot mode #2048

Page specific bugs

Following are bugs I observed while testing with JS off:

• / - Data doesn't load - fixed in Make / work under FastBoot #1937
• /login - TypeError: window.open is not a function
• /logout - completely blank page
• github_login, /authorize/github - Unclear how the login workflow should work without JS, or if we should even support that
• /crates/test-crate, /crates/test-crate/0.1.5 - "Authors" and "Owners" sections are not populated
• /crates/*/download - I think this endpoint can be removed, it doesn't have an associated route file in the frontend
• `/crates/*/{docs,repo} - Doesn't actually redirect when the metadata is set - added in Make /install work under FastBoot #1912
• /me/*, /dashboard - redirects to /
• /{categories,keywords,users,teams}/does-not-exist - results in a 500 server error
• /search - doesn't load any search results
• /install - a redirect which doesn't work with JS disabled, would be nice to do an HTTP redirect - added in Make /install work under FastBoot #1912

[smarnach]

We also need to get an estimate of the additional load we impose on our servers for rendering the pages – we may need to switch to bigger dynos.

[sivakumar-kailasam]

@locks directed me here to chime in. I'm from the ember learning team & I manage the API docs which runs on fastboot. We use one 2x dyno & we tunnel all our traffic to thar app via fastly.

If you'd like more detailed stats reach out to me on discord & I'll be able to share them.

[kzys]

Regarding the additional load, it would be helpful to share the stats about the endpoints. I assume that /policies is low-traffic but don't know much about other endpoints.

[sgrif]

We also need to get an estimate of the additional load we impose on our servers for rendering the pages – we may need to switch to bigger dynos.

Less than 1% of our traffic is from browsers. I'd be shocked if we needed to increase server capacity as a result of this.

[smarnach]

Less than 1% of our traffic is from browsers.

That's definitely less than I expected. The additional server process we will be running will still consume some memory, regardless of whether it receives any requests, so we should at least take a look at that.

[sgrif]

Do we have control over the logs fastboot produces? It'd be nice if it could match the format of our previous logs. Before fastboot it would look like:

Sept 05 14:59:58 crates-io app/web.1 at=info method=GET path="/policies" request_id=a-uuid-here fwd="1.2.3.4" service=0ms status=200 user_agent="Web Browser"

Now we have:

Sept 05 15:11:46 crates-io app/web.2 2019-09-05T15:11:46.678Z 200 OK /policies

We should at least have the same amount of information that was present before. It'd be nice to put some identifier in the log I can filter on to only see requests handled by fastboot as well

[sgrif]

fastboot appears to be using a ton of memory. Right now the only page being served by fastboot is /policies.

[kzys]

#1827

This caused our memory usage to increase from 50MB to 350MB at boot, and I was able to trivially cause the server to run out of memory by hitting /policies repeatedly.

@sgrif - How much increase have you observed in the production environment? My change adds 2 node processes (foreman and the actual ember server), and it is super helpful to know which one is contributing how much.

[sgrif]

I observed roughly another 300MB from hitting /policies repeatedly before rolling back the deploy, so a total of 600MB difference from the previous deploy. I don't have per-process statistics.

[kzys]

300MB for keeping the processes + 300MB per request? Wow, that's a lot.

Do we have some limits regarding the memory consumption? There is a Heroku wiki about Node's memory usage and I will try to tune what we have.

Regarding memory consumption, do you have some numbers we need to hit?

Adding Node and loading Ember there definitely consume more memory, compared to having just a cargo server. If the minimal Node + Ember setups hits the limits, we probably need to give up Fastboot and try something else.

[sgrif]

Not per-request, that was the total increase in usage I saw after a few hundred requests. Our limit is 512MB, but honestly if this is using more than 100-150MB I will be extremely uncomfortable

[kzys]

I think that 512MB is coming from Heroku's Standard 1X Dyno. Would it be able to upgrade our Dyno to more powerful one, since the server is doing more work for more features?

If we cannot change our dyno and cannot hit the 150MB goal, we need to give up Fastboot.

[sgrif]

Let's see where we're at after someone investigates the memory usage in more depth

[kzys]

Reducing the number of the child processes fastboot-app-server starts seems the way to go.

From Node's perspective, Heroku's dyno has 8 cores and fastboot-app-server starts 8 child processes,

https://github.com/ember-fastboot/fastboot-app-server/blob/bc36d74bea137ca0d3fce35555c67e373d8e6290/src/fastboot-app-server.js#L55

which Heroku specifically says "a common mistake".

https://devcenter.heroku.com/articles/node-memory-use#running-multiple-processes

A common mistake is to determine the number of Node processes to run based on the number of CPUs, but the number of physical CPUs is not the best mechanism for scaling within a virtualized Linux container. The total amount of memory used by all node processes should remain within dyno memory limits.

[kzys]

Let me submit a few small pull requests (starting #1829) to reduce the size of an upcoming FastBoot pull request.

@jtgeibel Can you uncheck /policies since we have reverted the change?