local: fix platform-split=true option #6007

tonistiigi · 2025-06-03T04:56:06Z

Currently the option only worked when platform-split=false was set for multi-platform build but not for setting single platform build to use platform-split=true.

docker/buildx#3215

tonistiigi · 2025-06-03T05:53:00Z

@profnandaa Any idea what happened to the windows tests here?

profnandaa · 2025-06-03T06:18:09Z

@profnandaa Any idea what happened to the windows tests here?

Taking a look rn.

profnandaa · 2025-06-03T10:10:08Z

We missed that path that calls outputFS.Walk since it wasn't being hit on master.

(dlv)
> github.com/moby/buildkit/exporter/local.(*localExporterInstance).Export.func2.1() C:/dev/core-containers/buildkit/exporter/local/export.go:128 (PC: 0x224b379)
   123:                         }
   124:                         if cleanup != nil {
   125:                                 defer cleanup()
   126:                         }
   127:
=> 128:                         if !e.opts.PlatformSplit {
   129:                                 // check for duplicate paths
   130:                                 err = outputFS.Walk(ctx, "", func(p string, entry os.DirEntry, err error) error {
   131:                                         if entry.IsDir() {
   132:                                                 return nil
   133:                                         }
   
(dlv) p e.opts.PlatformSplit
true

Cross-checking with the rest of the exporters then send a patch.

Ref #4994

profnandaa · 2025-06-03T13:23:49Z

PFA the patch to your branch, ran on my fork here

0001-fix-wcow-privileges-for-local-exporter-fs-walk.patch

tonistiigi · 2025-06-03T16:29:34Z

@profnandaa So previously this code path did not run for windows?

profnandaa · 2025-06-03T20:35:53Z

@profnandaa So previously this code path did not run for windows?

Yes, did not run previously.

crazy-max · 2025-06-04T11:51:42Z

@profnandaa This test failure looks related to this commit 30b572b

https://github.com/moby/buildkit/actions/runs/15430742302/job/43437060882?pr=6007#step:8:307

=== FAIL: client TestIntegration/slice=3-4/TestExportLocalForcePlatformSplit/worker=containerd (0.89s)
    client_test.go:6905: 
        	Error Trace:	D:/a/buildkit/buildkit/client/client_test.go:6905
        	            				D:/a/buildkit/buildkit/util/testutil/integration/run.go:103
        	            				D:/a/buildkit/buildkit/util/testutil/integration/run.go:251
        	Error:      	"[]" should have 1 item(s), but has 0
        	Test:       	TestIntegration/slice=3-4/TestExportLocalForcePlatformSplit/worker=containerd
        	Messages:   	expected one files in the output directory

profnandaa · 2025-06-04T14:07:14Z

@crazy-max -- I doubt... I've tried to dig through to find where the failure is happening but can't find it since it fails just silently. Same happens even if I revert my change.

Ran the debugger till session/filesync/diffcopy_windows.go:19, but can't find anything.

buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=. --output type=local,dest=./output,platform-split=true
--
#10 exporting to client directory
#10 copying files windows(10.0.22631)/amd64
#10 copying files windows(10.0.22631)/amd64 35.6s done
#10 DONE 43.6s
--
// but then wehn you check the ./output directory, it's empty

// it writes the output correctly when platform-split=false

Any ideas?

crazy-max · 2025-06-04T14:57:55Z

#10 copying files windows(10.0.22631)/amd64
#10 copying files windows(10.0.22631)/amd64 35.6s done

Hum wonder if it could be the folder name with os version 🤔

profnandaa · 2025-06-04T18:37:10Z

Hum wonder if it could be the folder name with os version 🤔

That was my first suspicion but windows(10.0.22631)_amd64 is still a valid Windows directory. Adding explicit --opt platform=windows/amd64 still yields same results, so not really the directory...

tonistiigi · 2025-06-04T19:06:00Z

@profnandaa Do the platform directories work correctly for wcow on multi-platform builds that don't require platform-split?

profnandaa · 2025-06-04T19:11:42Z

@profnandaa Do the platform directories work correctly for wcow on multi-platform builds that don't require platform-split?

Since we don't have multi-platform build support yet, I'm unable to test this.

I was just about to write to you that since this is actually not needed at the moment coz of the single-platform support, I suggest we skip the test on Windows for now; as I continue with the investigation.

profnandaa · 2025-06-05T15:14:13Z

PFA the patch to your branch, ran on my fork here

0001-fix-wcow-privileges-for-local-exporter-fs-walk.patch

@tonistiigi -- this is besides our current issue. I just noticed that you cherry-picked a slightly older fix from my branch then (if you see the patch). I have done a refactor here to match, will be nice if you can pick that - profnandaa@780b19b

profnandaa · 2025-06-05T19:05:27Z

@profnandaa Do the platform directories work correctly for wcow on multi-platform builds that don't require platform-split?

Since we don't have multi-platform build support yet, I'm unable to test this.

I was just about to write to you that since this is actually not needed at the moment coz of the single-platform support, I suggest we skip the test on Windows for now; as I continue with the investigation.

@crazy-max @tonistiigi -- I think I've finally found where the bug is at last, in fsutil after the call to fsutil.Sent() at session/filesync/diffcopy_windows.go:19:

> github.com/tonistiigi/fsutil.(*subDirFS).Walk() C:/dev/core-containers/buildkit/vendor/github.com/tonistiigi/fsutil/fs.go:125 (PC: 0x1501600)
   120: func (fs *subDirFS) Walk(ctx context.Context, target string, fn gofs.WalkDirFunc) error {
   121:         first, rest, _ := strings.Cut(target, string(filepath.Separator))
   122:
   123:         for _, d := range fs.dirs {
   124:                 if first != "" && first != d.Stat.Path {
=> 125:                         continue
   126:                 }
   127:
   128:                 fi := &StatInfo{d.Stat.Clone()}
   129:                 if !fi.IsDir() {
   130:                         return errors.WithStack(&os.PathError{Path: d.Stat.Path, Err: syscall.ENOTDIR, Op: "walk subdir"})
(dlv) p first
"/"
(dlv) p rest
""
(dlv) p target
"/"
(dlv) p d.Stat.Path
"windows_amd64"

filepath.Separator here is \\, while target=/.

Exploring how best to fix...

profnandaa · 2025-06-05T19:32:52Z

Looks like we'll have to fix from the fsutil side.

https://github.com/tonistiigi/fsutil/blob/3f76f81301443083e015c78ea5526da2823bf419/send.go#L151

err := s.fs.Walk(ctx, "/", func(path string, entry os.DirEntry, err error) error {

Already target = "/" and we are using the platform-specific separator later on at fsutil/fs.go:121. Should we just use / instead of filepath.separator or switch the target to the platform-specific /? Tested both approaches, work okay.

diff --git a/vendor/github.com/tonistiigi/fsutil/send.go b/vendor/github.com/tonistiigi/fsutil/send.go
index e4a315638..6584ccef7 100644
--- a/vendor/github.com/tonistiigi/fsutil/send.go
+++ b/vendor/github.com/tonistiigi/fsutil/send.go
@@ -148,7 +148,7 @@ func (s *sender) sendFile(h *sendHandle) error {

 func (s *sender) walk(ctx context.Context) error {
        var i uint32 = 0
-       err := s.fs.Walk(ctx, "/", func(path string, entry os.DirEntry, err error) error {
+       err := s.fs.Walk(ctx, string(filepath.Separator), func(path string, entry os.DirEntry, err error) error {
                if err != nil {
                        return err
                }

Using "/" was causing a silent bug later on at `fs.go:121` that is expecting platform-specific separators. See discussion at moby/buildkit#6007 Fix this by using `\\` on Windows and `/` on unix. Signed-off-by: Anthony Nandaa <[email protected]>

profnandaa · 2025-06-05T21:10:30Z

client/client_test.go

+
+	expPlatform := runtime.GOOS + "_" + runtime.GOARCH
+	_, err = os.Stat(filepath.Join(destDir, expPlatform+"/"))
+	require.NoError(t, err)


how do we deal with the windows case where the build number is added to the os name? Unless you explicitly specify --opt platform=..., eg.

#10 exporting to client directory #10 copying files windows(10.0.22631)/amd64 #10 copying files windows(10.0.22631)/amd64 6.01MB 11.4s #10 copying files windows(10.0.22631)/amd64 150.83MB 16.5s #10 copying files windows(10.0.22631)/amd64 239.47MB 23.7s #10 copying files windows(10.0.22631)/amd64 239.47MB 23.9s done #10 DONE 24.2s

should we normalize that from the code?

are these () invalid names for windows?

they are not invalid, I think this change was brought in some time back here - containerd/platforms#6

what do you think about this? profnandaa@3fc6393

Currently the option only worked when platform-split=false was set for multi-platform build but not for setting single platform build to use platform-split=true. Signed-off-by: Tonis Tiigi <[email protected]>

crazy-max · 2025-06-11T09:34:02Z

Rebased to include #6017

profnandaa · 2025-06-11T09:51:53Z

Rebased to include #6017

@crazy-max -- the failure now is coz the directory is named differently on windows with a (version) ~~suffix~~, not suffix but something like windows(10.0.22631)_amd64, I'd proposed this - #6007 (comment)

crazy-max · 2025-06-11T10:57:31Z

Rebased to include #6017

@crazy-max -- the failure now is coz the directory is named differently on windows with a (version) ~~suffix~~, not suffix but something like windows(10.0.22631)_amd64,

I have updated the test to use platforms.DefaultSpec() instead of runtime consts (see last commit).

@tonistiigi Feel free to squash with your commit.

I'd proposed this - #6007 (comment)

I agree that it would be better to strip the os version but I think we should just change the key here:

buildkit/exporter/local/export.go

Line 184 in 25c7c80

eg.Go(export(ctx, p.ID, r, inp.Attestations[p.ID]))

Instead of p.ID we could use platforms.Format(p.Platform).

We can look at this as follow-up.

profnandaa · 2025-06-11T12:26:07Z

PFA the patch to your branch, ran on my fork here
0001-fix-wcow-privileges-for-local-exporter-fs-walk.patch

@tonistiigi -- this is besides our current issue. I just noticed that you cherry-picked a slightly older fix from my branch then (if you see the patch). I have done a refactor here to match, will be nice if you can pick that - profnandaa@780b19b

@crazy-max One last thing, can cherry pick this refactor instead? Tonis has picked a slightly older version from my branch, I'd thought he'll use my patch file.

crazy-max · 2025-06-11T15:12:33Z

One last thing, can cherry pick this refactor instead?

Done, thanks!

Signed-off-by: Anthony Nandaa <[email protected]>

Signed-off-by: Tonis Tiigi <[email protected]>

Signed-off-by: CrazyMax <[email protected]>

crazy-max · 2025-06-18T09:16:28Z

@ArthurFlag Needs docs follow-up in https://docs.docker.com/build/exporters/local-tar/ with some examples as well similar to what we have in BuildKit README but with Buildx: https://github.com/moby/buildkit?tab=readme-ov-file#local-directory

github-actions bot assigned tonistiigi Jun 3, 2025

github-actions bot added the area/exporter label Jun 3, 2025

tonistiigi force-pushed the fix-platform-split-true branch from 1707034 to e30a4fc Compare June 3, 2025 05:30

crazy-max mentioned this pull request Jun 3, 2025

add platform-split parameter docker/docs#22753

Merged

3 tasks

github-actions bot added the area/windows label Jun 3, 2025

github-actions bot added area/testing area/client labels Jun 3, 2025

tonistiigi marked this pull request as ready for review June 3, 2025 22:58

tonistiigi force-pushed the fix-platform-split-true branch from db3590b to 65f3ddc Compare June 4, 2025 00:33

profnandaa mentioned this pull request Jun 5, 2025

fix: send: use platform-specific root path tonistiigi/fsutil#238

Merged

profnandaa reviewed Jun 5, 2025

View reviewed changes

profnandaa mentioned this pull request Jun 6, 2025

vendor: update fsutil to master@586307ad452f #6017

Merged

tonistiigi added this to the v0.23.0 milestone Jun 9, 2025

local: fix platform-split=true option

41ee238

Currently the option only worked when platform-split=false was set for multi-platform build but not for setting single platform build to use platform-split=true. Signed-off-by: Tonis Tiigi <[email protected]>

crazy-max force-pushed the fix-platform-split-true branch from 65f3ddc to 31a8c4e Compare June 11, 2025 09:33

crazy-max approved these changes Jun 11, 2025

View reviewed changes

profnandaa approved these changes Jun 11, 2025

View reviewed changes

crazy-max force-pushed the fix-platform-split-true branch from 7c3c4fe to c818a97 Compare June 11, 2025 15:11

crazy-max force-pushed the fix-platform-split-true branch from c818a97 to d7f98a9 Compare June 11, 2025 15:28

profnandaa and others added 3 commits June 11, 2025 17:35

fix: wcow: privileges for local exporter

40ddcfc

Signed-off-by: Anthony Nandaa <[email protected]>

client: add test for forcing platform-split=true on single-arch build

3eb4e72

Signed-off-by: Tonis Tiigi <[email protected]>

test: use platforms spec instead of runtime const

db03322

Signed-off-by: CrazyMax <[email protected]>

crazy-max force-pushed the fix-platform-split-true branch from d7f98a9 to db03322 Compare June 11, 2025 15:35

crazy-max approved these changes Jun 11, 2025

View reviewed changes

tonistiigi merged commit 9b8aded into moby:master Jun 11, 2025
139 checks passed

crazy-max added the needs/docs-follow-up label Jun 11, 2025

crazy-max mentioned this pull request Jun 18, 2025

Local output omits platform specific folder docker/buildx#3215

Closed

3 tasks

This was referenced Jun 25, 2025

e2e tests fail docker/buildkit-syft-scanner#133

Closed

Regression with attestations local export since 0.23.0 #6049

Closed

local: fix platform-split=true option #6007

local: fix platform-split=true option #6007

Uh oh!

Conversation

tonistiigi commented Jun 3, 2025

Uh oh!

tonistiigi commented Jun 3, 2025

Uh oh!

profnandaa commented Jun 3, 2025

Uh oh!

profnandaa commented Jun 3, 2025

Uh oh!

profnandaa commented Jun 3, 2025

Uh oh!

tonistiigi commented Jun 3, 2025

Uh oh!

profnandaa commented Jun 3, 2025

Uh oh!

crazy-max commented Jun 4, 2025

Uh oh!

profnandaa commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

crazy-max commented Jun 4, 2025

Uh oh!

profnandaa commented Jun 4, 2025

Uh oh!

tonistiigi commented Jun 4, 2025

Uh oh!

profnandaa commented Jun 4, 2025

Uh oh!

profnandaa commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

profnandaa commented Jun 5, 2025

Uh oh!

profnandaa commented Jun 5, 2025

Uh oh!

profnandaa Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

tonistiigi Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

profnandaa Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

profnandaa Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

crazy-max commented Jun 11, 2025

Uh oh!

profnandaa commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

crazy-max commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

profnandaa commented Jun 11, 2025

Uh oh!

crazy-max commented Jun 11, 2025

Uh oh!

Uh oh!

crazy-max commented Jun 18, 2025

Uh oh!

Uh oh!

profnandaa commented Jun 4, 2025 •

edited

Loading

profnandaa commented Jun 5, 2025 •

edited

Loading

profnandaa commented Jun 11, 2025 •

edited

Loading

crazy-max commented Jun 11, 2025 •

edited

Loading