runtime: can't call from Go to c-shared Go library on Darwin #38692
Comments
|
Can you share the program? |
Please, help me |
Use the |
Then run the program: Can it help you? |
|
There are things I do not understand, my code runs well in ubuntu but not in macos, what's the difference here? |
|
Well that looks like a completely different panic trace.
I do not know. Without a way for us to reproduce it's going to be hard to find out. |
I have uploaded it to the driver, Can you give me your gmail? |
You can send it directly to @randall77 via email. |
andybons
added
the
NeedsInvestigation
|
Marking as release-blocker until we figure out what’s going on. |
How do I solve this problem? Please |
|
We don't know what the problem is. We will most likely need the ability to reproduce this problem ourselves. |
I have uploaded it to the driver, You can see it here |
|
I downloaded your code and tried to run it, but ran into problems. We need exact instructions on how to get to the error posted. Show us every command you issued. |
Thank you for watching, to resolve this error for macos
brew install pkg-config
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/System/Library/Frameworks/Python.framework/Versions/2.7/lib/pkgconfig |
go build -o hybrid_go.so -buildmode=c-shared hybrid_go.go
go run main.go
cd test/ && \
go run main.go |
|
Ok, I can reproduce the problem. Not sure what is going on yet. One thing to try would be to move the python initialization code out of an |
I have tried all possible cases. But does not solve this problem on mac. Everything at Ubuntu is still very good |
|
After finding a lot of ways, the only possible way is to work as sleel in python. You can see it below But it's slow :)) |
|
I got completely different stack trace with this demo program. Click to see stack trace |
I changed the code a bit to reproduce the error, but my problem didn't change |
|
Here's a simple reproducer. foo.go: main.go: This prints Something is wrong with the Go->C->Go path. When we get back to Go, we have an uninitialized environment. |
|
It would be nice if this worked but I'm not terribly surprised that it doesn't. |
ianlancetaylor
changed the title
Will my problem not be solved? |
|
cc @odeke-em @eliasnaur for ideas on a solution. |
|
@khaitranvan96kt I hope that someone will solve your problem. But it is hard in the general case, and I do not know of anybody working on it. Sorry. |
thanks you |
|
@ianlancetaylor Any insight into what it might take to fix (or work around) the darwin symbol collision issue? I'm certainly not an expert in this area, but I'm not finding any solid resources on the darwin-specific dlopen behavior. Perhaps it's related to this "two-level namespace" concept I keep seeing. If you have any pointers, I can dig into it a bit further. |
|
I'm sorry, I don't have any useful pointers. I'm not familiar with the details of shared libraries on macOS. I want to clarify that this is not an intended use case. The That said, I would not be surprised if those options also have trouble on macOS. My point in mentioning this is that if you want to work on fixing something here, I encourage you to focus on |
|
@ianlancetaylor Thanks for the clarification 👍 I do find that position interesting, since I would expect a Your doc on the different buildmodes was very helpful. If I'm understanding them properly, I was inspired by the power of As an example, I created a project to make writing Bash plugins easier by abstracting away the C-specific APIs and loading other programs as Based on your last point, focusing more on improving |
That relies on a particular semantics for symbols that occur in both the executable and the shared library, or that occur in multiple shared libraries. ELF provides fairly tight control for how to handle multiple symbol definitions, and Go tries to take advantage of that on ELF systems. But I don't know how that works on macOS. I think it would be useful to remove the requirement that buildmode=plugin and buildmode=shared use exactly the same Go version. But I don't see any plausible way to do it. I guess it might be slightly easier to tackle for plugins, but it would require inventing a completely new mechanism for plugins, and not using the system dynamic linker at all. But we are talking about months of work. |
|
I see, thank you for your responses. Maybe if someone has some intimate knowledge of the macOS dynamic library situation they can chime in. (I may poke around in my spare time, who knows. 🙂) Based on the source of |
|
Clock in. This problem has not been solved yet |
|
I encountered this issue on macOS while importing two Python extensions written in Go using Pygolo. I could reduce the problem to the following repro, completely removing Python from the picture. This is a minimal shared library exporting a dummy function: package main
import "C"
func main() {
}
//export fun
func fun() {
}This test C program loads two libraries built from the above source code and invokes the exported #include <assert.h>
#include <stdio.h>
#include <dlfcn.h>
typedef void (*fun)(void);
int main(int argc, char* argv[])
{
void *lib1 = dlopen("./lib1.so", RTLD_NOW);
if (!lib1) {
printf("%s\n", dlerror());
}
assert(lib1);
void *lib2 = dlopen("./lib2.so", RTLD_NOW);
if (!lib2) {
printf("%s\n", dlerror());
}
assert(lib2);
fun fun1 = dlsym(lib1, "fun");
assert(fun1);
fun1();
fun fun2 = dlsym(lib2, "fun");
assert(fun2);
fun2();
dlclose(lib1);
dlclose(lib2);
return 0;
}This Makefile builds the two libraries and the test program, invokes the test multiple times. In an handful of attempts the runtime explodes. GO ?= go
LIBS := lib1.so lib2.so
ITERATIONS ?= 1000
all: $(LIBS) test
for n in `seq $(ITERATIONS)`; do ./test || exit 1; printf .; done; echo ok
%.so: lib.go FORCE
$(GO) build -buildmode=c-shared -o $@ $<
clean:
rm -rf test $(LIBS) $(LIBS:.so=.h)
FORCE:
.PHONY: FORCEWhat's interesting is that with Go 1.20.12 I can run thousands of iterations without any problem (I've never seen a panic) while with Go 1.21.5 it explodes almost immediately. Same macOS system. On Linux I don't get any issue. From the Pygolo Project point of view, not being able to import multiple extensions written in Go is a limitation. We created new repo go-multi-c-shared to try to describe better what works and what doesn't, on which system and, maybe, under which condition it can made to work; ideally coordinating the effort for improving the status quo towards a possible solution. Should we maybe create a new GH issue for this? |
|
@cavokz Yeah, just skimming this issue again, that does seem different. In your case it seems specific to Go 1.21 somehow and you're loading the shared library from a C program, while the author of this issue was loading the c-shared library from Go. Thanks for posting and for the minimal reproducer! Please file a new issue so we can track it better. |
Here it is #65050. Thanks! |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
I wrote a program that combines golang vs python. I call from golang to python and from python call to golang. It works great in Ubuntu but can't work in macos.
Then run the program in macOS Catalina v10.15.3:
What did you expect to see?
Program successfully
What did you see instead?
Program panic
The text was updated successfully, but these errors were encountered: