Description
Unsure how to title this. It's been a week of debugging and isolating with no results.
Is this issue in the correct repo? Also unsure. Help me out here.
I'll walk you through my journey.
I have a very large code base, it runs flawlessly locally. I've deployed it just fine for a long time until recently.
I made a change, including google-cloud-texttospeech in requirements.txt.
It stopped deploying after this (maybe relevant, maybe not).
Removing this change, to the exact same code base as before, still fails to deploy.
Some errors I get at random, and I've tried incredibly hard to isolate.
Both development environments can deploy other things successfully, and push to a new app fine as well.
I have not made any local environment changes at all when it began to fail.
Details:
Azure Functions runtime version 4.24.4.4, Linux premium plan with Elastic Premium EP3.
I deploy directly from visual studio code azure function extension.
My python code follows the folder structure of the v1 coding model (no decorators, lots of folders), however I am on v2 (host.json etc.), this has always worked, and runs locally of course.
Deploying to python 3.9.7.
First development environment:
Manjaro linux, vscode 1.81.1, azure extension 1.12.3
Errors I've encountered seemingly at random, when trying to deploy:
2:50:41 PM debugApp123: **Deployment successful**. deployer = ms-azuretools-vscode deploymentPath = Functions App ZipDeploy. Extract zip. Remote build.
2:51:07 PM debugApp123: Syncing triggers...
2:51:12 PM debugApp123: Querying triggers...
2:51:18 PM debugApp123: **No HTTP triggers found.**
3:34:11 PM debugApp123: Deployment Failed. deployer = ms-azuretools-vscode deploymentPath = Functions App ZipDeploy. Extract zip. Remote build.
3:34:25 PM debugApp123: Deployment failed.
4:00:49 PM: Error: The operation was aborted.
10:52:10 AM ae-API-compute: Deployment successful. deployer = ms-azuretools-vscode deploymentPath = Functions App ZipDeploy. Extract zip. Remote build.
10:52:42 AM ae-API-compute: Syncing triggers...
10:52:49 AM ae-API-compute: Querying triggers...
10:52:52 AM ae-API-compute: WARNING: Some http trigger urls cannot be displayed in the output window because they require an authentication token. Instead, you may copy them from the Azure Functions explorer.
Except it was not successful and functions are empty / don't run (the zip file in webjobstorage contains them thought).
11:55:04 AM debugApp123: Syncing triggers...
11:55:45 AM debugApp123: Syncing triggers (Attempt 2/6)...
11:55:56 AM debugApp123: Syncing triggers (Attempt 3/6)...
11:56:17 AM debugApp123: Syncing triggers (Attempt 4/6)...
11:56:59 AM debugApp123: Syncing triggers (Attempt 5/6)...
11:58:20 AM debugApp123: Syncing triggers (Attempt 6/6)...
11:59:12 AM: Error: Encountered an error (ServiceUnavailable) from host runtime.
And a very exciting shiny rare one:
Offset to Central Directory cannot be held in an Int64.
Second development environment:
Windows, visual studio code, azure extension v1.12.4.
I've also used this environment before, and it has worked, same as the above one.
Errors:
One successful deploy, and randomly most of the above (with no changes), as well as a new one:
10:35:34 AM ae-API-compute: Starting deployment...
10:35:34 AM ae-API-compute: Creating zip package...
10:45:11 AM: Error: socket hang up
Some other random things I've tried:
- different development environments with different OS's, cloning the repo fresh
- rolling back azure function extension in visual studio code to various versions
- protobuf==3.20.*
- setting AzureWebJobsFeatureFlags:EnableWorkerIndexing
- setting SCM_DO_BUILD_DURING_DEPLOYMENT:1
- removing type hints
- restarting
What now?
The errors are not helpful. Logs are missing, I can drill down into events, insights, many different logs, they are all over the place and they are all useless or empty. I've mentioned that I've had multiple 2hr+ calls with the technical support team of various ever-increasing escalations, and they are just as stumped as me (and I am grateful for their efforts).
Any thoughts?
What do I try next? Any information I can provide?
See also (maybe relevant, I don't know at this point):
microsoft/vscode-azurefunctions#3805
microsoft/vscode-azurefunctions#2529
microsoft/Oryx#1774
https://stackoverflow.com/questions/76478668/adding-python-module-google-cloud-storage-is-causing-a-working-azure-function-ap
https://stackoverflow.com/questions/72441758/typeerror-descriptors-cannot-not-be-created-directly
projectkudu/kudu#3348
microsoft/azure-pipelines-tasks#14201
microsoft/vscode-azurefunctions#2529
microsoft/Oryx#1774