-
Notifications
You must be signed in to change notification settings - Fork 305
Support multiple workers for NODEFS /wordpress mounts #2231
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
230 commits
Select commit
Hold shift + click to select a range
fc589cb
Begin shared/writer file locking implementation
brandonpayton edf4ed9
Implement F_SETLK using cross-platform flock()
brandonpayton 203b1ec
Call builtin fcntl implementation from php_wasm.c
brandonpayton 968aaaa
Simplify async syscall
brandonpayton 8601c32
Remove unnecessary stdarg.h include
brandonpayton 071115b
[Website] return true selectSitesLoaded() only when a site is active
adamziel 5088480
Fix changelog workflow and backfill changelog entries from recent ver…
brandonpayton d00072a
Support quickly testing php-wasm CLI changes with bun (#85)
brandonpayton 87238a3
WIP: Stub a FileLockManager that will later be shared among workers
brandonpayton e00e617
Fix system call varargs reading
brandonpayton f485a8a
Avoid type error with default switch case
brandonpayton 1288e81
Merge branch 'trunk' into add-fcntl-for-nodejs
brandonpayton feff197
Move comlink API helpers from php-wasm web to universal
brandonpayton 72bb46c
Fix comlink API helpers to work for both node and web
brandonpayton 1de9f84
Fix php-wasm/web exports of comlink API helpers
brandonpayton 2448d15
Move to PHP worker thread
brandonpayton 24c9112
Remove unused import
brandonpayton 7fc9c9f
Adjust consumeAPI signature to support exposing API from parent to wo…
brandonpayton 334a6c5
Make file lock manager node-specific for now
brandonpayton 1d9f94a
Bump comlink dep to try to resolve apparent listener leak
brandonpayton ad26310
Allow exposing API to specific endpoint
brandonpayton 13b040a
Stop exporting non-existing types from php-wasm/universal
brandonpayton 0daddcc
Include package-lock.json from comlink upgrade
brandonpayton 38f4455
Expose file lock manager to worker
brandonpayton f4e6f18
Hack together a boot method for secondary workers
brandonpayton 9319e9b
Load balance requests between multiple workers
brandonpayton 6a3d9cf
Add note about broken secondary worker request routing
brandonpayton d1e82d2
Remove import of nonexistent type
brandonpayton 433bcc3
Remove some cruft and a pointless comment
brandonpayton 80ebb33
Add a couple of notes
brandonpayton 7965bbb
Respect bootWordPress() dataSqlPath option
brandonpayton efe764b
Fix secondary worker setup
brandonpayton 1292042
Isolate emscripten constant replacement to allow editor syntax checking
brandonpayton e6913a0
Explicitly throw not-implemented error on F_SETLKW
brandonpayton 3ca558d
Support upgrade/downgrading locks owned by same pid
brandonpayton 3809809
Read/write full flock struct
brandonpayton 114884c
Support resolving address for flock struct whence field
brandonpayton 61541c9
Implement file byte range locking
brandonpayton 6b200b8
Add TODO about locks and fd access mode
brandonpayton c40e8e8
Debug: Log what request a worker is selected for
brandonpayton 6dc495e
Merge branch 'trunk' into add-fcntl-for-nodejs
brandonpayton 4969242
Fix openssl library paths in php-wasm build
brandonpayton 9065ac3
Add TODO about sharing same constants
brandonpayton 9f73091
Update PHP 8.3 build for testing
brandonpayton 3ed7a4b
Add note about merging worker management with PHPProcessManager
brandonpayton 126e176
Merge branch 'trunk' into add-fcntl-for-nodejs
brandonpayton e65f362
Fix some lint errors
brandonpayton 6a458fe
WIP: Mirror /internal directory in secondary workers
brandonpayton 839d73f
Mark worker_threads as external for Vite build
brandonpayton bda5510
Fix incorrect import of with-networking.js
brandonpayton 52283a4
Stop requiring Playground CLI to manually use comlink import
brandonpayton 2169320
Mark Node.js url module as Vite external
brandonpayton d09b92b
Finish renaming to more generic FileLockManager
brandonpayton aebdbd2
Stop forcing bundler moduleResolution on entire project
brandonpayton 6e26825
Fix nested comlink import from unbuilt TS
brandonpayton cddf59e
Avoid caching VFS nodes for NODEFS
brandonpayton 2a67f77
Fix crash related to F_GETLK debug statement
brandonpayton 4a4e2c7
Add launch.json entry for attaching to existing Playground CLI
brandonpayton 47dd7aa
Fix worker boot issue
brandonpayton 2fa91a7
Remove unused import
brandonpayton 0df08cc
Adjustments for worker URL resolution and debugging
brandonpayton 826b5d3
Update PHP 8.3 build
brandonpayton ccab4e7
Improve readability and info for file locking logging
brandonpayton f14b8a7
Fix worker URL for both dev and production
brandonpayton 822e906
Back out unnecessary tsconfig changes
brandonpayton 34b9bbf
Revert unintentional change
brandonpayton 3ea3d2e
Override hashAddNode without copying original implementation
brandonpayton 79db9f8
Separate processId from runtimeId
brandonpayton b9b4ceb
Release file locks when a PHP request is shutdown
brandonpayton 1188738
Replace flock() with an async stub
brandonpayton f5a0557
Implement local flock() support
brandonpayton 64342fe
Note that we can only lock NODEFS paths and add TODO
brandonpayton 2df81fb
Address more TODOs and add more error handling
brandonpayton b2fd96f
Adjust file lock manager API to be sync
brandonpayton b77b264
Lock with respect to file descriptors and unlock on fd_close
brandonpayton 8274390
Remove unused override from esm-suffix.js
brandonpayton 77c5ec5
Multiple changes listed in commit message body
brandonpayton e95af95
Make logging show mem addrs as hex
brandonpayton a0decdd
Add temp timestamps to logging for troubleshooting
brandonpayton 875a0d9
WIP: Native flock support
brandonpayton b37ef9f
Add fx-ext types
brandonpayton 26ce568
WIP cleanup
brandonpayton 21ce6d1
WIP test updates
brandonpayton a435e48
Continue cleanup and fixes
brandonpayton 060186e
Merge byte-range lock/unlock methods
brandonpayton 9bf76b0
Fix BitInt zero comparison
brandonpayton ee10918
Update PHP version used for test
brandonpayton 42bf085
Add TODO for F_GETLK bug
brandonpayton 427f645
Fix range unlock to be able to split existing locked range.
brandonpayton fd4beda
Fix bug that led to zero-width locked ranges
brandonpayton a6a9e82
Move move to whole-file unlocked once all shared locks are gone
brandonpayton 566f497
Fix tests
brandonpayton f9d0efc
Fix treatment of special zero-length ranges
brandonpayton b78955a
Fix no-unlock-error test
brandonpayton 127d40e
Add naming TODO
brandonpayton e79de39
Recognize whole file locks in findFirstConflictingLock
brandonpayton 3a547be
Make sure file is closed before releasing locks on close
brandonpayton 4d85c78
Actually apply NODEFS caching workaround
brandonpayton dbc822c
Temporary logging
brandonpayton cba4ee8
Fix fcntl syscall to use asyncify
brandonpayton edc46eb
Fix NODEFS file lookups and locking by native path
brandonpayton 5f8b7be
Fix how errno is relayed by __syscall_fcntl64
brandonpayton 3e04008
Test native whole-file locking
brandonpayton 517b2d1
Fix native lock creation to hold onto native fd
brandonpayton 62f1866
Add temporary logging of request routing to workers
brandonpayton 785a884
Mark PHPLoaderOptions.fileLockManager as a RemoteAPI
brandonpayton 2134497
Merge branch 'trunk' into add-fcntl-for-nodejs (from private trunk)
brandonpayton c16ae62
Avoid duplicated boot logging
brandonpayton 24b5b5c
Add workaround for missing wp-config while booting additional CLI wor…
brandonpayton 2d60a70
Allow __dirname and __filename refs in unbuilt modules
brandonpayton c6c65e1
Update sqlite asyncify functions and remove other duplicates
brandonpayton bec0b6e
Fix post-merge issues
brandonpayton 31eb0ba
Add more asyncify functions for sqlite
brandonpayton 38dc446
More sqlite functions for asyncify
brandonpayton 29fa83c
WIP
brandonpayton c04e108
Comment out redundant function names in php_wasm.c
brandonpayton 2998294
Sufficient asyncify functions to run fcntl
adamziel 17594ba
Add more functions to ASYNCIFY list after re-enabling default PHP ext…
brandonpayton 142bd22
Re-enable lock releases on fd_close
brandonpayton 5a08ceb
Revert: Re-enable lock releases on fd_close
brandonpayton 2d45021
Resume using actual lock manager
brandonpayton 2b7ba6f
Resume use of js_release_file_locks
brandonpayton b4280a8
Re-enable fd_close
brandonpayton 2452009
Add temp workaround to avoid WP install in additional workers
brandonpayton e9ce309
Exploratory changes for restoring JSPI functionality
brandonpayton 626550a
Adjust fcntl library for JSPI
brandonpayton 780d1b0
Merge branch 'trunk' into add-fcntl-for-nodejs
brandonpayton 2e380ec
Some cleanup for Dockerfile and esm suffix
brandonpayton 1e2b4af
Fix flock() for non-NodeFS files
brandonpayton 6280c4c
Restore fd_close() override
brandonpayton 5275760
Remote WP install workaround during boot
brandonpayton 3296f62
Move LoadBalancer to dedicated file
brandonpayton 0992a68
Clarify comment about merging load-balancer with PHPProcessManager
brandonpayton 8135767
Put multi-worker support behind --experimentalMultiWorker flag
brandonpayton 1d35a22
Add --experimentalMultiWorker TODO
brandonpayton e089d83
Remove unused constants declaration
brandonpayton c7736a9
Leave flock override disabled until further fix
brandonpayton e078e55
Restore actual locking
brandonpayton 7a7c5ed
Address some lock manager types and TODOs
brandonpayton 4f68496
Remove some unused imports and types
brandonpayton 2b17e69
Fix tests from specifying now-unused property. Add unrelated TODO.
brandonpayton 7761fe7
Use max 64-bit address when byte-range lock length is zero
brandonpayton eedd825
Stop unnecessarily passing fd for byte range locking
brandonpayton d2aa4cf
Add more docs and address TODOs
brandonpayton 7d25ca3
Address more TODOs
brandonpayton 0305fde
Address more TODOs for Node.js lock manager
brandonpayton ed4f495
Try to improve readability of file-lock-manager-for-node module
brandonpayton 42bb243
Allow specifying num workers via experimentalMultiWorker
brandonpayton 1c45f34
Update some TODOs
brandonpayton a4847ab
Fix flock to return zero instead of boolean true
brandonpayton 7a4278e
Use more consistent naming convention
brandonpayton 637304b
Remove file op tracing used for debugging
brandonpayton a61787d
Remove debug logging from load balancer
brandonpayton 59f0bfc
Clarify name of new Emscripten library
brandonpayton b65849b
Move toward making tracing optional
brandonpayton 6b28b63
Only call js_release_file_locks() in php-wasm/node builds
brandonpayton 9619bed
Clarify some naming and trace messages
brandonpayton 1db53fc
Clarify more names
brandonpayton 5056fb6
Add more docs and address TODOs
brandonpayton 4d5f22a
Make tracing optional
brandonpayton 6f25be8
More docs and TODOs
brandonpayton 6c5804c
Some run-cli cleanup and TODOs
brandonpayton 9f8fe5d
Add minor doc TODOs
brandonpayton 194015d
Minor clarification
brandonpayton 013375a
Remove console logging from file-lock-manager-for-node
brandonpayton 4d4a812
Address remaining TODOs in run-cli
brandonpayton 019ab9c
Fix variable-width trace timestamp
brandonpayton d9087d9
Address TODO and lint error
brandonpayton d14673e
Address more lint errors
brandonpayton 2401ae8
Finish fixing lint errors
brandonpayton 06ace3b
Remove logger dependency from PHP instance module
brandonpayton 607cb31
Temporarily disable run-cli tests due to conflicts with Worker
brandonpayton 2d81ecf
Merge branch 'trunk' into add-fcntl-for-nodejs
brandonpayton e470c40
Fix php_wasm.c compilation bug due to late extern declaration
brandonpayton cbe70b4
Update php-wasm/node builds
brandonpayton 2dcad60
Remove troubleshooting code that was committed by accident
brandonpayton 86695ca
Fix type error
brandonpayton 7a800c0
Address type errors in cli-run tests
brandonpayton 255a96e
Update php-wasm-node build to be able to deal with native Node modules
brandonpayton 11fc6f6
Restore worker URL approach that worked for dev and prod builds
brandonpayton 04c79db
Clarify name of test worker
brandonpayton 8b8620f
Address esm-suffix TODOs
brandonpayton 7264edb
Wrap long comment
brandonpayton ad2ab97
Always use plain Docker output when building php-wasm
brandonpayton e52b4c3
Em library cleanup
brandonpayton 40bd596
Only define lock-related C functions when locking is enabled for the …
brandonpayton c57bb9b
Fix js_getpid() to return default value when no pid provided to php-wasm
brandonpayton a309444
Add inline comment for js_wasm_trace()
brandonpayton 27b9a2a
Remove Playground CLI attach target that is little used and not 100% …
brandonpayton 37526b1
Address php-wasm Dockerfile TODO
brandonpayton 846574c
Revert path resolution regression in Dockerfile
brandonpayton 3b76c6c
Add trailing space
brandonpayton 425f597
Document changes to load-runtime
brandonpayton 7ef15d0
Remove unimportant TODO
brandonpayton ae61316
Revert debug changes to Playground CLI project.json
brandonpayton e806a2d
Remove CLI TODOs
brandonpayton 15f04bd
Temporarily skip the cli-run tests
brandonpayton 646d7d2
Remove unnecessary changes to load-php-runtime
brandonpayton 2ca09e9
Rename ambiguous new var in api.ts
brandonpayton c227163
Fix FS node lookup bug caused by cleanup
brandonpayton 21e6aa3
Remove some cruft from worker-thread
brandonpayton 009057a
Disable fcntl and flock overrides for aysncify until we fully support it
brandonpayton cc3f2cc
Refresh JSPI builds after Dockerfile updates today
brandonpayton b7e2054
Fix locking-related conditional in hashAddNode
brandonpayton d98a8c6
Rebuild php-wasm after esm-suffix fix
brandonpayton eec72b4
Fix docs link
brandonpayton 210ed66
Address native dependency mismatch
brandonpayton 0610f7c
Attempt to fix the test-built-npm-packages error
brandonpayton 78175d6
Fix the fix to test-built-npm-packages
brandonpayton 6fac4ca
Fix prepare-playground to include Node.js version in cached-modules key
brandonpayton 255c314
Try forcing dependency reinstall for test-built-npm-packages
brandonpayton 86f7ab3
Revert "Try forcing dependency reinstall for test-built-npm-packages"
brandonpayton 6bb8744
Switch from fs-ext to os-lock library for Bun compatibility
brandonpayton 12dc402
Revert "Switch from fs-ext to os-lock library for Bun compatibility"
brandonpayton 1f61e62
Fix undefined obj in tracing
brandonpayton 5dc537a
Avoid Bun-specific issues in test-built-npm-packages
brandonpayton 90a0e91
Fix missing nvm in CI step
brandonpayton a7514d8
Adjust test-built-npm-packages test for changed RunCLIServer interface
brandonpayton dabf3da
Make PHPWorker asyncDisposable and use in testing built npm packages
brandonpayton 5214b63
Fix remote Playground disposal
brandonpayton 666c2c8
Fix let-should-be-const lint error
brandonpayton a4a116d
Avoid undefined-null-to-object error in PHP.exit
brandonpayton d8a75c3
Terminate all workers when CLI disposed
brandonpayton 8d515dc
Fix test-built-npm-packages tests
brandonpayton e948fd1
Only log resolved WP release if downloading WP
brandonpayton a886826
Test unbuilt multi-worker
brandonpayton 04fd547
Skip the sqlite-asyncify tests until we solve unexpected GH Actions t…
brandonpayton 26dc841
Fix Node version check in unbuilt-jspi target
brandonpayton f51eb69
Restore package install in unbuilt CLI test
brandonpayton 463f820
Merge branch 'trunk' into add-fcntl-for-nodejs
brandonpayton File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.