Send Big POST to HTTP Library, sometimes return empty reply from server

[alexwbaule]

Basic Infos

Hardware

Hardware: ESP-12F
Core Version: 2.3.0

Description

Using the ESP8266WebServer from library, making a big POST in the body, sometimes the reply from server is Empty.

Settings in IDE

Module: Generic ESP8266 Module
Flash Size: 4MB/3MB
CPU Frequency: 80Mhz
Flash Mode: dio
Flash Frequency: 40Mhz
Upload Using: SERIAL
Reset Method: ck

Sketch

I think is not necessary, because my function is not called, the library stops it before. My BODY length is 2277.

Debug Messages

FROM CURL:

curl -s -v -X POST -L  http://192.168.250.141/schedule -d@schedule | jq .
*   Trying 192.168.250.141...
* TCP_NODELAY set
* Connected to 192.168.250.141 (192.168.250.141) port 80 (#0)
> POST /schedule HTTP/1.1
> Host: 192.168.250.141
> User-Agent: curl/7.56.0
> Accept: */*
> Content-Length: 2277
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
> 
* Done waiting for 100-continue
} [2277 bytes data]
* We are completely uploaded and fine
* Empty reply from server
* Connection #0 to host 192.168.250.141 left intact

FROM Arduino IDE with ALL Debugs Enabled:

New client
:ref 2
:ur 2
method: POST url: /schedule search: 
headerName: Host
headerValue: 192.168.250.141
headerName: User-Agent
headerValue: curl/7.56.0
headerName: Accept
headerValue: */*
headerName: Content-Length
headerValue: 2277
headerName: Content-Type
headerValue: application/x-www-form-urlencoded
headerName: Expect
headerValue: 100-continue
:c0 1, 181
:ur 1
:close
WS:dis
:del

Sometimes the 100-continue, works OK, sometimes gives a empty reply.

[devyte]

@alexwbaule please try:

• latest git
• PR Update to SDK 2.1.0 #3215 (SDK 2.1.0)
• PR lwip2: lighter implementation #3362 (lwip2)

[alexwbaule]

Tks @devyte , i will try later and post the result here !

[alexwbaule]

I did the test, and still happens. Using the GIT version.

New client
:ref 2
:ur 2
method: POST url: /schedule search: 
headerName: Host
headerValue: 192.168.250.141
headerName: User-Agent
headerValue: curl/7.56.0
headerName: Accept
headerValue: */*
headerName: Content-Length
headerValue: 2302
headerName: Content-Type
headerValue: application/x-www-form-urlencoded
headerName: Expect
headerValue: 100-continue
:c0 1, 181
:ur 1
:close
WS:dis
:del

alex@pczao:~/PET-POST$ curl -s -v -X POST -L  http://192.168.250.141/schedule -d@schedule | jq .
*   Trying 192.168.250.141...
* TCP_NODELAY set
* Connected to 192.168.250.141 (192.168.250.141) port 80 (#0)
> POST /schedule HTTP/1.1
> Host: 192.168.250.141
> User-Agent: curl/7.56.0
> Accept: */*
> Content-Length: 2302
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
> 
* Done waiting for 100-continue
} [2302 bytes data]
* We are completely uploaded and fine
* Empty reply from server
* Connection #0 to host 192.168.250.141 left intact

[devyte]

@alexwbaule did you try the PR?

[alexwbaule]

Im using now the 2.4.0-rc2, still happens, but it's less. In the 2.4.0-rc2, has this 2 PR's that you refers ?

[devyte]

No, the lwip2 PR is something different that won't get merged until after 2.4 release.

[alexwbaule]

Ok, tonight i will get the lwip2, and use with the 2.4.0-rc2, and i give you a anser.

[alexwbaule]

Hi @devyte , how can i update the lwip2 to this PR version that you talk ?

Looking to the lwip, its part of the sdk... i have to clone the esp8266, and get all the commits from the PR ?

[devyte]

@alexwbaule git allows to pull the PR directly over your local git install. Just google "github how to test pr locally" or something. It's not hard. I'd give you a direct link to a tutorial, but I'm travelling and only have phone access atm.

[alexwbaule]

Hi @devyte , sorry about the delay.

Can you help me get these PRs ?

I'm trying to catch the PRs adding this "fetch = +refs/pull//head:refs/remotes/origin/pr/" to config, and getting the origin again.
If this is right, how can i merge 2 PRs ?

[alexwbaule]

The first one, that you said is a Issue, that generate this PR #3278 Correct ?

[devyte]

Things have advanced somewhat. Latest git already has both PRs, so please install that, select lwip2 from the Arduino menu, rebuild, and retest.

[alexwbaule]

Ok, today i will test this.

[alexwbaule]

Ok, i made the Test:

With LWIP v2 : Fail with the same result at the second POST

> Content-Type: application/json
> Content-Length: 1846
> Expect: 100-continue
> 
* Done waiting for 100-continue
} [1846 bytes data]
* We are completely uploaded and fine
* Empty reply from server
* Connection #0 to host 192.168.250.100 left intact
alex@pczao:~/PET-POST$ 

With Prebuild Sources (gcc): Fail with the same result at the fifth POST

> Content-Type: application/json
> Content-Length: 1846
> Expect: 100-continue
> 
* Done waiting for 100-continue
} [1846 bytes data]
* We are completely uploaded and fine
* Empty reply from server
* Connection #0 to host 192.168.250.100 left intact

With Expressif(xcc): Failt with same result at the first POST
Trying many times and got a reboot

 ets Jan  8 2013,rst cause:4, boot mode:(1,6)

wdt reset

> Content-Type: application/json
> Content-Length: 1846
> Expect: 100-continue
> 
* Done waiting for 100-continue
} [1846 bytes data]
* We are completely uploaded and fine
* Empty reply from server
* Connection #0 to host 192.168.250.100 left intact

With OpenSource (gcc): Fail at the second POST, no reboots.

> Content-Type: application/json
> Content-Length: 1846
> Expect: 100-continue
> 
* Done waiting for 100-continue
} [1846 bytes data]
* We are completely uploaded and fine
* Empty reply from server
* Connection #0 to host 192.168.250.100 left intact

[d-a-v]

@alexwbaule Please confirm the fix, or reopen the issue

[alexwbaule]

Ok, I will test in a couple of hours, just when i arrived at my house.

[alexwbaule]

Hi @d-a-v , still happens, using the lwIP v2.

> Content-Type: application/json
> Content-Length: 1846
> Expect: 100-continue
> 
* Done waiting for 100-continue
} [1846 bytes data]
* We are completely uploaded and fine
* Empty reply from server
* Connection #0 to host 192.168.250.101 left intact
R=0m1,116s U=0m0,008s S=0m0,004s

How to reopen this issue ?

[alexwbaule]

I made more outputs for DEBUG, to see where is the problem.

New client
method: POST url: /schedule search: 
headerName: Host
headerValue: 192.168.250.101
headerName: User-Agent
headerValue: curl/7.57.0
headerName: Accept
headerValue: */*
headerName: Content-Type
headerValue: application/json
headerName: Content-Length
headerValue: 1846
headerName: Expect
headerValue: 100-continue
if !isForm
readBytesWithTimeout: 0
1846
Avaliable: 0
BREAAAAK
plainBuf: 
plainLength < contentLength: 0
1846

With this prints , i see that "client.available()" is returning zero, looking to this function, its point to "getSize()" in libraries/ESP8266WiFi/src/include/ClientContext.h correct ?

[alexwbaule]

Ok !! Let's go again.

Increasing the "HTTP_MAX_POST_WAIT" from 1000 to 5000, helps a lot, sending 20 consecutives POSTS, none of them get the error and the reply was Ok.
Maybe, in the ESP8266WebServer.h if there is a "100-continue" header, the HTTP_POST_WAIT , can be increased based in some algorithm considering the Content-Length.
What you think @devyte and @d-a-v ??

[d-a-v]

I'm more a crash guy than HTTP guy :)

about 100-continue:

To have a server check the request's headers, a client must send Expect: 100-continue as a header in its initial request and receive a 100 Continue status code in response before sending the body

@devyte @igrr should WebServer handle 100-continue or should we unconditionally increase HTTP_POST_WAIT ?

[d-a-v]

@alexwbaule can you test with master then close this issue ?

[alexwbaule]

Hi @d-a-v , you can close this issue, you do exactly what i do in the #3699 (comment) , and using this (5000 msec) works.

TKs.

[d-a-v]

I can't, you can :)

[alexwbaule]

OK ! Closing...