curl is not following 301's fetching https://dot.net/v1/dotnet-install.ps1

[RobertHenry6bev]

curl --output junk.ps1 https://dot.net/v1/dotnet-install.ps1
fails. Curl is not following a 301 presented from the server.
A browser does fine. wget does fine.

@mairaw

[baronfel]

Good spot, this is a behavior difference between the powershell version and the bash version of the scripts.

The bash version doesn't have a centralized place where all http calls are routed through, so it will be more of a whack-a-mole to make sure the -L option is applied everywhere.

[RobertHenry6bev]

I can't use curl to get either the .ps1 or the .sh scripts, never mind trying to execute them and discover that the contents now follow some broken link(s).

[mairaw]

@danzhu54 can we check if this is caused by the WAF policies we put in place?

[davhdavh]

it broke a few days ago, (Thu, 05 Oct 2023 10:25:06 GMT) Invoke-WebRequest 'https://dot.net/v1/dotnet-install.ps1' -OutFile 'dotnet-install.ps1' completed in builds with no issues. The next build on (Sat, 07 Oct 2023 07:16:59 GMT) failed.
Problem seems to be a server down:

* Connection #0 to host dot.net left intact
* Issue another request to this URL: 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.ps1'
*   Trying 13.107.213.73:443...
*   Trying [2620:1ec:46::73]:443...
* connect to 13.107.213.73 port 443 failed: Timed out
*   Trying 13.107.246.73:443...
* ```

[baronfel]

Hey I think this is actually a bit of miscommunication. @RobertHenry6bev you need to pass -L to get curl to follow redirects. Just using curl alone won't follow them. Can you try that and let us know?

[baronfel]

@davhdavh can you provide some more details on how you are trying to download the scripts? I've had success using curl with that link directly today.

[RobertHenry6bev]

In 25 years using curl, I've never had to use -L.

I ran curl only as a quick proxy for my real problem, so perhaps I'm being unfair.

My real problem was: on 10/6 I started using again a previously stable script that installs crank https://github.com/dotnet/crank on a new windows machine, and then uses crank to run some workloads. My scripts no longer worked; they hung when crank itself hung for 20sec; crank complained that "the response from the server was too slow" or some such verbiage, without telling me which server and which URL was to blame. Doing divide and conquer on the commands I threw at crank let me to believe it was crank trying to fetch https://dot.net/v1/dotnet-install.ps1 That's when I tried curl.... I have not written a small program calling the same HttpClient.GetAsync API that crank uses.

[danzhu54]

@danzhu54 can we check if this is caused by the WAF policies we put in place?

Our WAF has been flagging "Unknown bots" and blocking requests associated with different user agents including curl, axios, and PS. We haven't made any changes to the WAF policies in the past month.

The timeout errors could be related to the server issue we were facing on 10/6.

[RobertHenry6bev]

Leaving aside for now the issues with curl, the problem I was trying to reproduce by a simple invocation is coming from either crank or my application which crank runs. The problem I'm really trying to solve only seems to happen on windows running on ARM64. I do not see it on amd64 machines. I think I'll move the discussion over to crank:
dotnet/crank#642

[lewing]

I opened dotnet/arcade#14215 just to move the discussion along

[mairaw]

Is this still happening? Just wanted to make sure whether the new WAF rules we added helped with this or not.

[RobertHenry6bev]

I have not noticed this problem recently.