SerializationException when executing UDF with dependent assemblies

[guruvonline]

My structured streaming query is calling a UDF which has few dependency dlls(some my code and some third party).

i tried running on Azure Databricks and windows standalone cluster and still get same error "System.Runtime.Serialization.SerializationException: Unable to find assembly 'myassembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null".

for azure Databricks i am using set-jar to trigger job and i have uncommented the dependency copy step in db-int.sh. all the dependent dlls are copied in dbfs.

for windows standalone cluster, i have using spark-submit and have tried --files and sparkcontext.addfile in code. but still it fails. i have tried specifying file as c:\1.dll,2.dll and file:///c:/1.d.

[rapoth]

@guruvonline: Thank you! Could you please provide a self-contained sample code and logs?

[imback82]

Also, are you using 0.3.0 or the master branch? We have some fix on assembly loading in master branch (#135). #135 doesn't support calling static functions in external libraries yet, but it handles more scenarios.

[imback82]

FYI, we will assign 'bug' label after we identify this issue as a bug.

[imback82]

#180 should fix this. Please reopen this if it doesn't address your scenario.