Getting only 404 after runtime update to 2.0.11776-alpha

[forki]

The beta runtime was updated to 11776 and now I only get 404 if I try to run my functions in the portal. All functions used to work and now the are broken.

Investigative information

Please provide the following:

• Function App version (1.0 or 2.0-beta): beta
• Function App name: ecarbeta
• Function name: e.g. CheckTransactions
• Region: Europe West

Repro steps

go to portal an run function CheckTransactions
--> 404

Expected behavior

functions executes and returns without error

Actual behavior

getting 404 and function is not executed at all

Known workarounds

None. It's completely broken for me

/cc @fabiocav

[forki]

Created support request: 118052218231390

[forki]

I deleted that functionapp and created a new one called "ecarjobs". but same thing. only 404

[forki]

From logs:
Warning: Only got partial types from assembly: CheckTransactions, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
Exception message: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
at System.Reflection.RuntimeModule.GetTypes()
at System.Reflection.Assembly.GetTypes()
at Microsoft.Azure.WebJobs.Host.Indexers.DefaultTypeLocator.FindTypes(Assembly assembly, IEnumerable`1 extensionAssemblies) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Indexers\DefaultTypeLocator.cs:line 120

[cyky]

Version 11776 seems to have some breaking changes. See details: Azure Functions Runtime 2.0 Preview Breaking Change Notice

You can rollback to earlier version and that might fix the problem for now.

[forki]

@cyky tbh I actually want to have the new version resolution changes. I'd appreciate someone from the team or support takes a deeper look. At least the error reporting needs to be improve here.

[cyky]

@forki I understand. Just wanted to let you know that there is "workaround" as I stumbled here due to a similar problem.

[pragnagopa]

Added this to breaking changes investigations issue.

[fabiocav]

@forki we'll be looking at the details to see what is going on with your app. Support might be unable to help since this is with the preview bits.

[forki]

Thanks @fabiocav.

[paulbatum]

@fabiocav I think the underlying assembly load failure is being hidden because that exception handler does not inspect the LoaderExceptions

Its also interesting that this code path has now become a hotspot for assembly load failures. Do you know why this is the case? All sorts of assembly load failures were possible before the loadcontext changes but they never manifested in this code path as best as I can recall...

[fabiocav]

@paulbatum yes, that code has to be enhanced to properly report the error.

We've seen failures with that code path before the changes (we looked at a few last week), but it all depends on when those assemblies are requested. In the functions scenarios, we'd typically see that early as we probe for extensions.

[paulbatum]

Last week doesn't count - LoadContext changes were already in at that point! I'm talking about the many months previous.

[fabiocav]

Those issues weren't new, though. I pretty sure some of them were quite old, and that was my point; we've seen this before the changes were in place.

[fabiocav]

There's definitely an unrelated issue here being uncovered, as the DefaultTypeLocator shouldn't be in play at all. Looking at that now

[fabiocav]

Indeed, looks like an issue with the implementation of the default ExtensionTypeLocator. Not directly related to the recent assembly load changes, but the issue there is triggered by some of the behavior changes.

I'm looking at a feasible workaround, but will aim to have this fixed tomorrow to release with the next drop (which we're planning for this week).

[forki]

I can confirm that downgrade to 2.0.11651-alpha solves my issues. @fabiocav can you please notify me when changes are ready and I can switch backt to beta

[fabiocav]

Absolutely! I'll update this issue when this is resolved.

[JeryLu2020]

@fabiocav any update on this issue?

[forki]

Ping

[fabiocav]

The changes have been made. The deployment is being prepared and slated to go out this week. Waiting for some additional changes going in with this payload.

[forki]

does it mean my function app will work again or doees it mean it will show better error and I need to do something?

[fabiocav]

For pure WebJobs SDK scenarios, there will be a better, detailed and actionable message.

In functions, this code is no longer in play, so your Function App should just start working with those bits.

[JeryLu2020]

@fabiocav can you kindly share the release date to us? thanks a lot

[davidebbo]

@13127755587 The deployment of the new build (2.0.11857-alpha) is now complete. Can you verify that it works for you?

[forki]

@fabiocav any ideas what we can try?

[fabiocav]

Glad to hear that moving things to bin helped.

The FSharp.Core issue is caused by a runtime restriction that adds that assembly to the list of "shared runtime dependencies", which is a bit aggressive right now.

I had a task to remove that restriction for F# dependencies and a couple of others that still remain before the last release (that was coupled with the work to update our own F# dependencies for the compilation services), but that did slip. I'll have that done and can make those updates available for local testing. I'll also see if we can push that as quick update to 2.0.11888 to make it available on Azure.

[forki]

That would be awesome. Fabio Cavalcante <[email protected]> schrieb am Di., 26. Juni 2018, 19:34:

…

Glad to hear that moving things to bin helped. The FSharp.Core issue is caused by a runtime restriction that adds that assembly to the list of "shared runtime dependencies", which is a bit aggressive right now. I had a task to remove that restriction for F# dependencies and a couple of others that still remain before the last release (that was coupled with the work to update our own F# dependencies for the compilation services), but that did slip. I'll have that done and can make those updates available for local testing. I'll also see if we can push that as quick update to 2.0.11888 to make it available on Azure. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2881 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNL0WK4NgI9VWkgk9wUztLOZmL7JMks5uAnC1gaJpZM4UIPi-> .

[JeryLu2020]

@fabiocav thank you Fabio for the continous help!!

[forki]

@fabiocav any news about this?

[forki]

ping

[davidebbo]

We are planning a 2.0 update next week (primarily to fix the http response issue mentioned here). @fabiocav, can you confirm whether the change you referred to above will also be part of that release?

[fabiocav]

Indeed, it will. Working to have that out ASAP. Will again update this thread once that process begins.

[forki]

Hugs! Fabio Cavalcante <[email protected]> schrieb am Do., 5. Juli 2018, 18:39:

…

Indeed, it will. Working to have that out ASAP. Will again update this thread once that process begins. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2881 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNA-1WN7kCoLWqbUuLi5Ydbt7hwUrks5uDkFUgaJpZM4UIPi-> .

[forki]

ping. any news here?

[fabiocav]

Apologies for the delay here. Those bits have been ready for a while and blocked on deployment activities. This is finally moving forward and we have the announcement out here: Azure/app-service-announcements#124

[forki]

awesome thank you. Waiting for bits to show up on my function apps now ;-)

[forki]

@davidebbo how long does it usally take until the bits land in Europe West?

[fabiocav]

David is unavailable for the next few days, but the deployment is completing today.

[TurtleBeach]

With respect, which deployment?? There are at least 3 reasonably current runtimes in various threads, and it is increasingly hard to identify what MS considers "THE" current runtime, and which one is currently being deployed, to where. To wit: 2.0.11857 was a fallback I had to use because 2.0.11888 broke HTTP returns; 2.0.11888 seems to sometimes be referred to as 1.0.11888; in any case it is deployed in US EAST and at least partially works, though I reported an issue today with 2.0.11888 sometimes not reading JAVA_OPTS correctly; 2.0.11933 supposedly fixes the HTTP return issue but the gitHub issue was closed a few minutes ago and we have been unable to build a Java function to use the 2.0.11933 runtime locally (we keep getting 2.0.11888, even after following MS instructions).
It would be helpful to have some consistent and clear statements about what specific runtime is deployed where.
Thanks.

[pragnagopa]

I have update the title to include full version for the runtime. This issue is now fixed as part the 2.0.11933-alpha release

@TurtleBeach - Breaking changes for Java functions that require users to update functions that use HttpRequest/Response messages will be part of the next release. Annoucement Azure/app-service-announcements#122 has been posted early as a heads up on the breaking changes.

[TurtleBeach]

@pragnagopa I understand and appreciate the heads-up notice. But we tried following your instructions from another thread to get the correct maven tools to create a simple HTTP triggered function using the new runtime. That is what is not working - the local environment still shows we are using the 2.0.11888 runtime. Did I miss something? (i.e., is it actually not possible to create / run a function locally using the new 2.0.11933 right now, even using tools from the sona..... (I forget the name) path you specified?

[pragnagopa]

@TurtleBeach - To test locally with runtime 2.0.11933, please get the latest Azure functions CLI version 2.0.1-beta.32. We just published this one.

You will not be able to test java breaking changes with runtime 2.0.11933.

[TurtleBeach]

Thanks - and now understood about the breaking changes.

[TurtleBeach]

@pragnagopa Now I am thoroughly confused and disheartened. A queue-trigger function that runs at the end of potentially long running jobs running on a separate VM just crashed because the runtime failed to read a valid string property defined in JAVA_OPTS (a property - storage account name) that has been in JAVA_OPTS and in use in all my Java functions for at least 6 months. I earlier today reported this happening with an HTTP triggered function running on 2.0.11857 (due to the issue with HTTP function returns in 2.0.11888). When the queue function was deployed, the runtime in EAST US was 2.0.11888 (I specifically wanted to see the effect of WEBSITE_USE_PLACEHOLDER = 1 on cold starts) and the function has been working. In reviewing the environment when the crash occurred this evening, I was surprised to see that 2.0.11933 has been deployed in EAST US (according to the portal). You told me just a few hours ago that the necessary tools to build a function for 2.0.11933 had just been deployed. In any case, the function handles bookkeeping at the end of a job, so 21 hours of modeling building went down the drain (more to the point, the information needed to close out the job). This does not make me happy. Despite my significant investment in time and dollars in the Azure platform, I have to now question that. What is going on??

[pragnagopa]

@TurtleBeach - Opened Azure/azure-functions-java-worker#146 to figure out issue with JAVA_OPTS.

[forki]

@fabiocav thank you for all the hard work on this. This is a nice birthday present. I can confirm that runtime version 2.0.11933.0 (beta) works for me with FSharp.Core 4.5.1 and latest TaskBuilder. This is really really awesome and allows me to update my libs in my SAFE-Stack app - and that even without binding redirects or crazy assembly loading hacks. Thank you so much.

[tforkmann]

Hi there!

I can confirm as well that the runtime version 2.0.11933.0 (beta) is working for me like a charm!

@fabiocav thank you very much for you help on that!

@forki happy birthday bro!!

[fabiocav]

That's great to hear! Happy birthday! :)