app/v1/auth and api/v1/graph/ql 504 errors

[irhughes]

Do you want to request a feature or report a bug?

Bug?

Intended outcome:

Upgrade to a newer version of Talk

Actual outcome:

Using any version of Talk from 4.6.7 and above the embed js returns 504 errors calling:
https://my-server/api/v1/auth and then https://my-server/api/v1/graph/ql. As a result of this the box that would encompass the comments does not show up and I have a loading spinner instead.

Using Postman the Auth call returns a 204 and the graph/ql call returns a 200 with data.

Please provide as much detail as possible, including a screenshot or copy-paste of any related error messages, logs, or other output that might be related. Places to look for information include your browser console, server console, and network logs. The more information you can give the better.

How to reproduce the issue:

I am using the docker images directly from coralproject/talk:4.6.7 without any changes. I make no changes to the docker-compose-yml other than the version number. I am using default configs with a local redis docker and connecting to MongoDB Atlas for the DB connection. This setup works fine with 4.6.6.

Version and environment

version 4.6.7 and above, docker, EC2 - linux AMI
Chrome and Firefox

[wyattjoh]

Did you set TALK_ROOT_URL to the correct parameter?

If I leave TALK_ROOT_URL=http://127.0.0.1:3000 as it is in the docs, you must only access it from that URL.

If you are accessing it from https://my-server/, you should set TALK_ROOT_URL=https://my-server.

[irhughes]

It is based on an existing config. With the docker-compose.yml pointing to 4.6.6 it works.

[wyattjoh]

Talk itself never throws a 504 error, that would be thrown by a proxy you have inbetween (such as Nginx, Caddy). Check the server logs from both your web proxy and Talk to see if there's any information as we were unable to reproduce that specific condition using the docker-compose.yml file from the docs with the default config.

[irhughes]

The 504 is thrown by the load balancer instantly, not after the expected timeout has passed. This implies that the server running Talk immediately closes the connection and the load balancer then treats it as a timeout.

I did further testing with Postman sending all of the headers and cookies that the website sends and I received the same error as the website. We are sending several large cookies in our request. When I remove any of the larger cookies the correct response is returned. When I checked the node versions between 4.6.6 and 4.6.7 it has been upgraded from v8.12.0 to v8.14.0. Node v8.14.0 introduced a breaking change around header size: nodejs/node#24692

We ran into a similar problem on our Elastic Beanstalk that supports our UI. I believe that our fix then was a configuration change for the nginx server to allow larger headers. Not sure what options are possible with the Talk configuration.

[kgardnr]

@irhughes For more advanced troubleshooting help and consulting on how best to set up and configure Talk, we can provide you with paid support. Just email support at coralproject dot net and someone from our team will get back to you!

In the meantime, I'll close this issue. Thank you!

[irhughes]

We know how to configure Talk. We have had it running in production for several months now. The problem is that you have introduced a change that will break us when we try and upgrade.

[kgardnr]

Where is Talk running in production?

[irhughes]

I am with Brunswick News. I believe that we spoke back in the fall. It is running on our tj.news website.

[kgardnr]

Hey @irhughes I spoke with the team today, we have another suggestion you can try which we'll be posting here shortly, but since this is also tied to your setup, beyond that we would need to explore other options to offer support. Stay tuned!

[wyattjoh]

You can experiment with adding the environment variable of:

NODE_OPTIONS=--max-http-header-size=16000

To see if you can get any better progress with it. That may allow a mitigation.

[irhughes]

So with 4.6.7 it won't work because the Node version doesn't support the --max-http-header-size option.

I went straight to 4.8.7 which has a higher node version that supports the --max-http-header-size option and things are working now.

It might be useful to rename this issue to something about "Requests with large header sizes may fail in versions 4.6.7 and above" so other people can find the issue.

Thanks for the help.