Add support for browser.on(:response).

[postmodern]

I wanted to be able to have a callback similar to on(:request) but triggered once a response was received. This adds an additional callback for the Network.responseReceived event when a response is received. It will look up the Ferrum::Network::Exchange object with the matching requestId and passes it back to the block, once a response has been received for a request.

I didn't know where to add specs for this, since there does not appear to be specs for Ferrum::Page or a spec/page_spec.rb file? I also didn't know where I should mention browser.on(:response) in the README, since browser.on(...) is only mentioned in conjunction to using browser.network.intercept.

[postmodern]

@route would appreciate some guidance on this new feature and where I should add the specs and documentation?

[route]

@postmodern I was reorganizing specs so currently page specs are in the browser_spec. I think we can create page_spec and put it there. Though it overlaps with network_spec a bit, because page.on(:request) only works with interception, but page.on(:response) works without it. I'm now thinking that maybe it's better to use something like page.network.on(:request | :response | :auth). Need to experiment with that...

[ttilberg]

I can't say for certain, but I believe the act of enabling network intercept requires you to authorize the requests to proceed. For example, some of the specs show:

https://github.com/rubycdp/ferrum/blob/main/spec/network/exchange_spec.rb#L28

network.intercept
page.on(:request) { |r, _, _| r.continue }

[postmodern]

@ttilberg hmm that solved the exception, except now it does not appear that my on(:response) callback is being called back.

[ttilberg]

@postmodern I took a look but didn’t find a CDP event translation helper for on(:response). However, in an earlier issue I noted and used network.responseReceived, might this get you on your way?

#197 (comment)

Edit: haha… I just looked at the rest of the context of this issue… lol. I’ll see myself out!

[postmodern]

@ttilberg yeah it's added in this PR. Feel free to pull my branch and try to mess with it. I swear it was working last year.

[postmodern]

OK I added some echo debugging and the @client.on('Network.responseReceived') callback is being called, but it's not finding the exchange object for the given params['requestId'], so the passed in block is never called.

[postmodern]

Got it, I needed to do network.select from the context of Ferrum::Page.

[ilyacherevkov]

This is useful, I use it as a monkey patch while it's not merged.

[ilyacherevkov]

Turned out it doesn't work for really heavy and chunked responses (I m testing particular response which returns 20MB), so I needed to use Network.requestWillBeSent to save mapping table of "requestId" and 'url', Network.loadingFinished to ensure full response got loaded.

Then I used mapping table to match requestId from loadingFinished with url from requestWillBeSen, and thus being 100% sure request got fully loaded.