async-http block by Cloudflare but Net::HTTP and Faraday doesn't #190

elct9620 · 2024-11-23T01:42:18Z

I am trying to build a Discord application, the document explains the user agent is necessary to prevent Cloudflare from returning 400 errors.

To make a PoC I write below Ruby code.

API_VERSION = 10
CHANNEL_ID = "xxxx"

Sync do
  Async::HTTP::Internet.post(
    Async::HTTP::Endpoint.parse("https://discord.com/api/v#{API_VERSION}/channels/#{CHANNEL_ID}/messages"),
    {
      'Content-Type' => 'application/json',
      'Authorization' => "Bot #{ENV['DISCORD_BOT_TOKEN']}",
      'User-Agent' => 'DiscordBot (https://example.com 0.1.0)'
    },
    { content: "Hello World" }.to_json
  )
end

The response is 400 bad request from Cloudflare.

However, if using Net::HTTP (ref: discorb) or Faraday is works correctly.

Faraday.post(
  "https://discord.com/api/v#{API_VERSION}/channels/#{CHANNEL_ID}/messages",
  { content: 'Hello World' }.to_json,
  {
      'Content-Type' => 'application/json',
      'Authorization' => "Bot #{ENV['DISCORD_BOT_TOKEN']}",
      'User-Agent' => 'DiscordBot (https://example.com 0.1.0)' # Optional, the "Faraday 2.12.7" as user agent is work
  }
)

On my website, if the bot filter is enabled the Net::HTTP isn't work. But Discord still accepts Net::HTTP requests, I guess there are some behaviors detected by Cloudflare and blocked it.

The text was updated successfully, but these errors were encountered:

ioquatix · 2024-11-23T03:53:01Z

Can you try again using lower case headers, e.g. authorization, content-type and user-agent?

elct9620 · 2024-11-23T04:58:25Z

I had tried user-agent and it did not work, too.

I also changed the user agent, which is unrelated to the user agent that is sent to Discord.

Sync do
  Async::HTTP::Internet.post(
    Async::HTTP::Endpoint.parse("https://discord.com/api/v#{API_VERSION}/channels/#{CHANNEL_ID}/messages"),
    {
      'Content-Type' => 'application/json',
      'Authorization' => "Bot #{ENV['DISCORD_BOT_TOKEN']}",
      'User-Agent' => 'Faraday 2.17.2' # use Faraday user-agent is not work
    },
    { content: "Hello World" }.to_json
  )
end

ioquatix · 2024-11-23T05:15:48Z

Faraday and Net::HTTP both only support HTTP/1 so that may be a difference.

What response are you getting back?

elct9620 · 2024-11-23T05:21:13Z

Just got 400 bad request message from Cloudflare

=> #<Async::HTTP::Protocol::HTTP2::Response:0x63d8 status=400>
irb(main):002> res.body.read
=> "<html>\r\n<head><title>400 Bad Request</title></head>\r\n<body>\r\n<center><h1>400 Bad Request</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n"

ioquatix · 2024-11-23T07:16:12Z

Okay, I'll investigate.

ioquatix · 2024-11-24T04:55:33Z

Okay, I set up my own discord bot, and while I couldn't get it to work fully, I did seem to make progress and found some issues with the original code.

API_VERSION = 10
CHANNEL_ID = "..."
DISCORD_BOT_TOKEN = "... your token here ..."

Sync do
	url = "https://discord.com/api/v#{API_VERSION}/channels/#{CHANNEL_ID}/messages"
	
	response = Async::HTTP::Internet.post(url,
		headers: {
			'authorization' => "Bot #{DISCORD_BOT_TOKEN}",
			'user-agent' => 'AsyncDiscordBot (https://socketry.io/ v0.1.0)',
			'content-type' => 'application/json',
		},
		body: {content: "Hello World"}.to_json
	)
	
	pp response, response.read
end

Async::HTTP::Internet is an extremely high level interface and it takes a string for the URL, not an endpoint instance.

Are you able to try using a string URL and let me know if you can make progress?

elct9620 · 2024-11-24T05:16:16Z

I still got 400 response

irb(main):058> Sync { res = Discord.new(token: ENV['DISCORD_BOT_TOKEN']).post('/channels/#{CHANNEL_ID}/messages', nil, { content: 'Hello, world' }.to_json); pp res, res.read }
"https://discord.com/api/v10/channels/1309482531325870156/messages"
#<Async::HTTP::Protocol::HTTP2::Response:0x77f88 status=400>
"<html>\r\n" +
"<head><title>400 Bad Request</title></head>\r\n" +
"<body>\r\n" +
"<center><h1>400 Bad Request</h1></center>\r\n" +
"<hr><center>cloudflare</center>\r\n" +
"</body>\r\n" +
"</html>\r\n"
=>
[#<Async::HTTP::Protocol::HTTP2::Response:0x77f88 status=400>,
 "<html>\r\n<head><title>400 Bad Request</title></head>\r\n<body>\r\n<center><h1>400 Bad Request</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n"]

The Discord client is a simple Ruby wrapper with Async::HTTP::Internet

module Discord
  class Client
    CLIENT_VERSION = '0.1.0'
    API_VERSION = 10

    USER_AGENT = "DiscordBot (https://example.com, 0.1.0) Ruby/#{RUBY_VERSION}".freeze

    def initialize(token:)
      @token = token
    end

    def default_headers
      {
        'Authorization' => "Bot #{@token}",
        'Content-Type' => 'application/json',
        'User-Agent' => USER_AGENT
      }
    end

    ::Protocol::HTTP::Methods.each do |verb|
      define_method(verb.downcase) do |path, headers = nil, body = nil, &block|
        url = "https://discord.com/api/v#{API_VERSION}#{path}"
        Async::HTTP::Internet.instance.call(verb, url, default_headers.merge(headers || {}), body, &block)
      end
    end
  end
end

ioquatix · 2024-11-24T05:51:05Z

I was getting a 401 response - I think the bot was not authorised to access the channel. I will try your wrapper. If you are on discord, maybe we can chat about it?

elct9620 · 2024-11-24T06:03:00Z

The Send Message permission is required to grant access to the bot.

Yeah, I am on discord.

elct9620 · 2024-11-24T06:09:00Z

Updated:

The Async::HTTP::Internet.instance.call cannot work correctly

module Discord
  class Client
    CLIENT_VERSION = '0.1.0'
    API_VERSION = 10

    USER_AGENT = "DiscordBot (https://example.com, 0.1.0) Ruby/#{RUBY_VERSION}".freeze

    def initialize(token:)
      @token = token
    end

    def default_headers
      {
        'Authorization' => "Bot #{@token}",
        'Content-Type' => 'application/json',
        'User-Agent' => USER_AGENT
      }
    end

    ::Protocol::HTTP::Methods.each do |verb|
      define_method(verb.downcase) do |path, headers = nil, body = nil, &block|
        url = "https://discord.com/api/v#{API_VERSION}#{path}"
-        Async::HTTP::Internet.instance.call(verb, url, default_headers.merge(headers || {}), body, &block)
+        Async::HTTP::Internet.send(verb, url, default_headers.merge(headers || {}), body, &block)
      end
    end
  end
end

I find I can use the below code to send a message correctly 🤔

Sync {
  Async::HTTP::Internet.post("https://discord.com/api/v10/channels/#{CHANNEL_ID}/messages", {
    'Content-Type' => 'application/json',
    'Authorization': "Bot #{BOT_TOKEN}"
  }, {
    content: "Hello, World!"
  }.to_json)
}

Maybe I have something wrong with my wrapper.

ioquatix · 2024-11-24T06:11:10Z

Thanks your screenshot was extremely helpful.

ioquatix · 2024-11-24T06:11:33Z

Please give me a moment to check your code.

ioquatix · 2024-11-24T07:17:21Z

I was equally confused by the old interface of Async::HTTP::Internet.post. I fixed it with the above PR it was released in v0.84.0 and I tested it, my code is now working as expected.

elct9620 · 2024-11-24T07:35:29Z

Finally, the wrapper uses an incorrect method name

-   ::Protocol::HTTP::Methods.each do |verb|
+   ::Protocol::HTTP::Methods.each do |name, verb|
      define_method(verb.downcase) do |path, headers = nil, body = nil, &block|
      define_method(verb.downcase) do |path, headers = nil, body = nil, &block|
        url = "https://discord.com/api/v#{API_VERSION}#{path}"
        Async::HTTP::Internet.instance.call(name, url, headers: default_headers.merge(headers || {}), body:, &block)
      end
    end

ioquatix mentioned this issue Nov 24, 2024

Pass through *arguments and **options for consistency. #191

Merged

3 tasks

elct9620 closed this as completed Nov 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

async-http block by Cloudflare but Net::HTTP and Faraday doesn't #190

async-http block by Cloudflare but Net::HTTP and Faraday doesn't #190

elct9620 commented Nov 23, 2024 •

edited

Loading

ioquatix commented Nov 23, 2024

elct9620 commented Nov 23, 2024

ioquatix commented Nov 23, 2024

elct9620 commented Nov 23, 2024

ioquatix commented Nov 23, 2024

ioquatix commented Nov 24, 2024 •

edited

Loading

elct9620 commented Nov 24, 2024

ioquatix commented Nov 24, 2024 •

edited

Loading

elct9620 commented Nov 24, 2024

elct9620 commented Nov 24, 2024 •

edited

Loading

ioquatix commented Nov 24, 2024

ioquatix commented Nov 24, 2024

ioquatix commented Nov 24, 2024

elct9620 commented Nov 24, 2024

async-http block by Cloudflare but Net::HTTP and Faraday doesn't #190

async-http block by Cloudflare but Net::HTTP and Faraday doesn't #190

Comments

elct9620 commented Nov 23, 2024 • edited Loading

ioquatix commented Nov 23, 2024

elct9620 commented Nov 23, 2024

ioquatix commented Nov 23, 2024

elct9620 commented Nov 23, 2024

ioquatix commented Nov 23, 2024

ioquatix commented Nov 24, 2024 • edited Loading

elct9620 commented Nov 24, 2024

ioquatix commented Nov 24, 2024 • edited Loading

elct9620 commented Nov 24, 2024

elct9620 commented Nov 24, 2024 • edited Loading

ioquatix commented Nov 24, 2024

ioquatix commented Nov 24, 2024

ioquatix commented Nov 24, 2024

elct9620 commented Nov 24, 2024

elct9620 commented Nov 23, 2024 •

edited

Loading

ioquatix commented Nov 24, 2024 •

edited

Loading

ioquatix commented Nov 24, 2024 •

edited

Loading

elct9620 commented Nov 24, 2024 •

edited

Loading