Add E2E tests using CLI from storage account

[pragnagopa]

CLI builds hat include the latest functions runtime are available on

https://functionsclibuilds.blob.core.windows.net/builds/2/latest/Azure.Functions.Cli.{platform}.zip

where platform in linux-x64, osx-x64, win-x64, win-x86

You can also get the version from

GET https://functionsclibuilds.blob.core.windows.net/builds/2/latest/version.txt

Update CI to use latest CLI.

[pragnagopa]

cc: @soninaren
@tylerl0706 / @daxian-dbw - Let me know if you have any questions

[daxian-dbw]

@pragnagopa We don't use func CLI in our CI.
But we should write some E2E tests for PowerShell functions with the CLI (not sure how nodejs worker does E2E testing) -- is that what you are asking for in this issue?

[pragnagopa]

How do you run E2E tests?

[daxian-dbw]

I don't know 😄 that's why I was asking if that's what you want for this issue. We are not using func CLI in our CI today.

[pragnagopa]

Yes. Please see https://github.com/Azure/azure-functions-java-worker/blob/dev/appveyor.yml for reference

[pragnagopa]

Also, please see https://github.com/Azure/azure-functions-java-worker/tree/dev/endtoendtests/Azure.Functions.Java.Tests.E2E if you can just reuse that project. Ideally, I would maintain one E2E test project that can be reused across all the language workers.

[daxian-dbw]

Great, I will take a look at the project. A quick glance tells me it's using dotnet xunit runner, which has been deprecated recently (https://xunit.github.io/releases/2.4). You possibly will need to update it to use dotnet test.

[pragnagopa]

cc: @paulbatum

Thanks I will look into it. Here is the flow E2E tests

• Add E2E tests that cover supported triggers and bindings
• Build powershell worker
• Download CLI from the URL above. CLI will be updated before every functions runtime release
• Copy powershell worker to CLI\workers directory
• Use CLI to install extensions
• Start functions host via CLI func host start
• Run tests against the localhost.

you can add these either as a nightly runs or run them as part of every commit/PR. Next steps would be

• Deploy to Azure
• Run E2E tests using URL for the deployed APP

As long as CI follows this pattern we are OK. If maintaining a single E2E test project makes sense, I will refactor the tests later.

[TylerLeonhardt]

Something weird is happening with the downloaded Azure.Functions.Cli.zip on macOS... I can't quite figure it out.

Invoke-RestMethod "https://functionsclibuilds.blob.core.windows.net/builds/2/latest/Azure.Functions.Cli.osx-x64.zip" -OutFile Azure.Functions.Cli.zip

Expand-Archive ./Azure.Functions.Cli.zip

./Azure.Functions.Cli/func

This opens the binary as a text file for me in PowerShell:

but in bash, it works once... but now I'm just getting a bunch of:

bash: ./func: Permission denied

It.. shouldn't require sudo...?

[TylerLeonhardt]

chmod -x ./func was needed 🤦‍♂️

[joeyaiello]

What's the proper scoping of this work item? What do we need to achieve for preview?

[TylerLeonhardt]

We have these now for HttpTrigger. We can probably close this. Although I don't know if the expectation is having tests for every trigger... And how we'd do that (how do we trigger a blob trigger without Azure?)

@pragnagopa any thoughts?

[joeyaiello]

Let's sync with them on this next week.

[pragnagopa]

Apologies I missed last weeks ping on this issue.

Yes. We do need tests for the supported triggers. Please see Azure.Functions.Java.Tests.E2E for a sample E2E test project.

This is a requirement for any release. Without test coverage, there is a huge risk of regressions both ways

• Host breaking powershell functions
• Changes in powershell worker not compatible with host changes.

@TylerLeonhardt already added coverage for HttpTriggers. When you add support for rest of the triggers, this work item is part of development

• Add Unit tests
• Add E2E tests

[TylerLeonhardt]

Talked to @paulbatum about this in the sync this week - he said that we can probably get some help setting these E2E tests up (creating the resources in Azure, supplying the subscription).

Ideally, we should be able to grab the E2E tests from Java or Node and run them with a PowerShell Function App.

cc @fabiocav