Skip to content

Integrate Binary provisioning #4671

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
codebien merged 70 commits into from
May 2, 2025
Merged

Integrate Binary provisioning #4671

Show file tree
Hide file tree
Changes from 63 commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
3b52a41
k6exec naive usage
olegbespalov Mar 12, 2025
cebb21e
Use k6's config file
olegbespalov Mar 14, 2025
6b9282c
Better handling of script args
olegbespalov Mar 17, 2025
09021bf
Libraries updates and more simplifications of the logic
olegbespalov Mar 20, 2025
890fb4b
simplify code structure
pablochacin Apr 3, 2025
3cfd619
remove redundant ExecuteWithGlobalState method
pablochacin Apr 3, 2025
96ac5d7
simplify launcher's logic
pablochacin Apr 3, 2025
4961632
improve visibility of binary provisioning process
pablochacin Apr 3, 2025
458cfe7
simplify analize logic
pablochacin Apr 4, 2025
4d40785
refactor to facilitate testing
pablochacin Apr 4, 2025
3fc4ebb
remove unnecessary error formatting logic
pablochacin Apr 4, 2025
d0e8faf
refactor os exit to improve testability
pablochacin Apr 7, 2025
92d6012
add tests
pablochacin Apr 7, 2025
cb9b3f5
fix validation of provisioning required
pablochacin Apr 7, 2025
fe0c0d9
fix linter issues
pablochacin Apr 7, 2025
d5eae2b
fix logic to check if provisioning is required
pablochacin Apr 8, 2025
4c111b7
improve comments and fix log messages
pablochacin Apr 8, 2025
524a232
update k6deps
pablochacin Apr 8, 2025
85fd837
Revert "remove ExecuteWithGlobalState method"
pablochacin Apr 9, 2025
d3de2c0
attend comments from review
pablochacin Apr 11, 2025
8fc8641
refactor launching custom build
pablochacin Apr 11, 2025
605679e
attend more review comments
pablochacin Apr 11, 2025
5d827da
inform stdin is not supported
pablochacin Apr 11, 2025
ecc2186
fix returning when no script found
pablochacin Apr 11, 2025
9df7a32
Apply suggestions from code review
pablochacin Apr 14, 2025
692508a
fix build after review comments
pablochacin Apr 14, 2025
4d2f5d7
Apply suggestions from code review
pablochacin Apr 15, 2025
e1e2148
fix renaming of launcher test fixture
pablochacin Apr 15, 2025
3cd2e7f
Merge branch 'master' into feat/binary-provisioning
pablochacin Apr 15, 2025
720d031
attend linter issues
pablochacin Apr 15, 2025
154481f
downgrade dependencies
pablochacin Apr 15, 2025
4a7c7aa
downgrade dependencies (2)
pablochacin Apr 15, 2025
9baa845
fix dependencies
pablochacin Apr 16, 2025
7faf234
fix linter issues
pablochacin Apr 16, 2025
f00debc
attend review comments
pablochacin Apr 16, 2025
4054b6e
refactor dependency analysis
pablochacin Apr 16, 2025
3fec225
refactor k6 execution
pablochacin Apr 17, 2025
64b99c0
Merge branch 'master' into feat/binary-provisioning
pablochacin Apr 17, 2025
6ebfd7d
add missing package
pablochacin Apr 17, 2025
9abc0a5
fix deps lint issue
pablochacin Apr 17, 2025
29e78f2
attend review suggestions
pablochacin Apr 22, 2025
29639c5
fix bug passing k6 environment variables
pablochacin Apr 22, 2025
69a8530
Rename runners (#4708)
codebien Apr 22, 2025
a6764fe
Fix argument scanning (#4721)
pablochacin Apr 23, 2025
528f280
Consistent naming for default executor
codebien Apr 23, 2025
bafbabf
Fix the scan of cloud login command (#4725)
pablochacin Apr 23, 2025
1d61bcc
Merge executors
codebien Apr 23, 2025
68141c0
Drop Outputs and use directly k6provider
codebien Apr 23, 2025
9872be4
Merge provision file
codebien Apr 23, 2025
ca56958
Merge extractToken
codebien Apr 23, 2025
9d3cdaf
Move to internal/cmd package
codebien Apr 23, 2025
2a6f52b
No custom logic for the Coud token
codebien Apr 23, 2025
ba9558d
Merge analyze files
codebien Apr 23, 2025
2a5e97b
Address lint issues
codebien Apr 23, 2025
e182f5a
Merge branch 'master' into feat/binary-provisioning
pablochacin Apr 23, 2025
2c3c934
Merge pull request #4712 from grafana/binpro/move-launcher
codebien Apr 23, 2025
aa987af
Merge branch 'master' into feat/binary-provisioning
codebien Apr 23, 2025
2fea117
disable manifest search (#4728)
pablochacin Apr 23, 2025
dec77b0
Skip binary provisioning for the run command (#4731)
pablochacin Apr 24, 2025
e6027d1
[Binary provisioning] Exclude fallback path (#4730)
codebien Apr 24, 2025
bdbba30
pass os stdout to subprocess
pablochacin Apr 24, 2025
b5e77b3
Relax the feature flag conditions
codebien Apr 24, 2025
0d1951e
address review comments
pablochacin Apr 24, 2025
bdfea21
handle signals (#4740)
pablochacin Apr 29, 2025
7034e39
Binary provisioning/add cache dir flag (#4743)
pablochacin Apr 30, 2025
eed742e
Skip binary provisioning for help commands (#4753)
pablochacin Apr 30, 2025
9968498
Merge branch 'master' into feat/binary-provisioning
codebien Apr 30, 2025
143403b
Merge branch 'master' into feat/binary-provisioning
codebien Apr 30, 2025
ddb7836
Merge branch 'master' into feat/binary-provisioning
codebien Apr 30, 2025
7f3b93d
Merge branch 'master' into feat/binary-provisioning
codebien May 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions cmd/execute.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,24 @@ import (
"context"

"go.k6.io/k6/cmd/state"
"go.k6.io/k6/internal/cmd"
internalcmd "go.k6.io/k6/internal/cmd"
)

// Execute the k6 command
// Execute executes the k6 command
// It only is exported here for backwards compatibility and the ability to use xk6 to build extended k6
func Execute() {
gs := state.NewGlobalState(context.Background())
cmd.ExecuteWithGlobalState(gs)

if gs.Flags.BinaryProvisioning {
internalcmd.NewLauncher(gs).Launch()
return
}

// If Binary Provisioning is not enabled, continue with the regular k6 execution path

// TODO: this is temporary defensive programming
// The Launcher has already the support for this specific execution path, but we decided to play safe here.
// After the v1.0 release, we want to fully delegate this control to the Launcher.
gs.Logger.Debug("Binary Provisioning feature is disabled.")
internalcmd.ExecuteWithGlobalState(gs)
}
40 changes: 36 additions & 4 deletions cmd/state/state.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ import (
"os"
"os/signal"
"path/filepath"
"slices"
"strconv"
"sync"

"go.k6.io/k6/lib"
Expand All @@ -21,7 +23,13 @@ import (
"go.k6.io/k6/secretsource"
)

const defaultConfigFileName = "config.json"
const (
// BinaryProvisioningFeatureFlag defines the environment variable that enables the binary provisioning
BinaryProvisioningFeatureFlag = "K6_BINARY_PROVISIONING"

defaultBuildServiceURL = "https://ingest.k6.io/builder/api/v1"
defaultConfigFileName = "config.json"
)

// GlobalState contains the GlobalFlags and accessors for most of the global
// process-external state like CLI arguments, env vars, standard input, output
Expand Down Expand Up @@ -102,7 +110,12 @@ func NewGlobalState(ctx context.Context) *GlobalState {

env := BuildEnvMap(os.Environ())
defaultFlags := GetDefaultFlags(confDir)
globalFlags := getFlags(defaultFlags, env)
globalFlags := getFlags(defaultFlags, env, os.Args)

logLevel := logrus.InfoLevel
if globalFlags.Verbose {
logLevel = logrus.DebugLevel
}

logger := &logrus.Logger{
Out: stderr,
Expand All @@ -111,7 +124,7 @@ func NewGlobalState(ctx context.Context) *GlobalState {
DisableColors: !stderrTTY || globalFlags.NoColor,
},
Hooks: make(logrus.LevelHooks),
Level: logrus.InfoLevel,
Level: logLevel,
}

return &GlobalState{
Expand Down Expand Up @@ -155,6 +168,9 @@ type GlobalFlags struct {
SecretSource []string
LogFormat string
Verbose bool

BinaryProvisioning bool
BuildServiceURL string
}

// GetDefaultFlags returns the default global flags.
Expand All @@ -164,10 +180,11 @@ func GetDefaultFlags(homeDir string) GlobalFlags {
ProfilingEnabled: false,
ConfigFilePath: filepath.Join(homeDir, "k6", defaultConfigFileName),
LogOutput: "stderr",
BuildServiceURL: defaultBuildServiceURL,
}
}

func getFlags(defaultFlags GlobalFlags, env map[string]string) GlobalFlags {
func getFlags(defaultFlags GlobalFlags, env map[string]string, args []string) GlobalFlags {
result := defaultFlags

// TODO: add env vars for the rest of the values (after adjusting
Expand All @@ -193,5 +210,20 @@ func getFlags(defaultFlags GlobalFlags, env map[string]string) GlobalFlags {
if _, ok := env["K6_PROFILING_ENABLED"]; ok {
result.ProfilingEnabled = true
}
if v, ok := env["K6_BINARY_PROVISIONING"]; ok {
vb, err := strconv.ParseBool(v)
if err == nil {
result.BinaryProvisioning = vb
}
}
if val, ok := env["K6_BUILD_SERVICE_URL"]; ok {
result.BuildServiceURL = val
}

// check if verbose flag is set
if slices.Contains(args, "-v") || slices.Contains(args, "--verbose") {
result.Verbose = true
}
Comment on lines +231 to +234
Copy link
Contributor

@codebien codebien Apr 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, are they here just for simplicity or something is blocking us from to move rootCmdPersistentFlagSet in a shared package and re-use it?

Copy link
Contributor Author

@pablochacin pablochacin Apr 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My guess is that @olegbespalov tried to minimize the blast radius of this chane set and I keep it. I'm not sure of how rootCmdPersistentFlagSet is used, but can do the change if you consider this is more convenient.

Copy link
Contributor

@codebien codebien Apr 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pablochacin I'll try to directly work on this, I leave it open as a reminder for myself.

Copy link
Contributor

@codebien codebien Apr 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pablochacin we don't have the time to address it, we should add an issue to the epic for researching and exploring the alternative solution by using pre-run as you have in the other pull request. Can you open it, plaese?


return result
}
4 changes: 4 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ toolchain go1.23.7
require (
buf.build/gen/go/prometheus/prometheus/protocolbuffers/go v1.36.5-20240802094132-5b212ab78fb7.1
github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358
github.com/Masterminds/semver/v3 v3.3.1
github.com/PuerkitoBio/goquery v1.9.2
github.com/Soontao/goHttpDigestClient v0.0.0-20170320082612-6d28bb1415c5
github.com/andybalholm/brotli v1.1.1
Expand All @@ -17,6 +18,8 @@ require (
github.com/golang/protobuf v1.5.4
github.com/google/uuid v1.6.0
github.com/gorilla/websocket v1.5.3
github.com/grafana/k6deps v0.2.6
github.com/grafana/k6provider v0.1.15
github.com/grafana/sobek v0.0.0-20250320150027-203dc85b6d98
github.com/grafana/xk6-dashboard v0.7.5
github.com/grafana/xk6-redis v0.3.3
Expand Down Expand Up @@ -77,6 +80,7 @@ require (
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/google/pprof v0.0.0-20230728192033-2ba5b33183c6 // indirect
github.com/grafana/k6build v0.5.11 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.1 // indirect
github.com/inconshreveable/mousetrap v1.0.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
Expand Down
14 changes: 12 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMT
github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 h1:mFRzDkZVAjdal+s7s0MwaRv9igoPqLRdzOLzw/8Xvq8=
github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358/go.mod h1:chxPXzSsl7ZWRAuOIE23GDNzjWuZquvFlgA8xmpunjU=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/Masterminds/semver/v3 v3.2.1 h1:RN9w6+7QoMeJVGyfmbcgs28Br8cvmnucEXnY0rYXWg0=
github.com/Masterminds/semver/v3 v3.2.1/go.mod h1:qvl/7zhW3nngYb5+80sSMF+FG2BjYrf8m9wsX0PNOMQ=
github.com/Masterminds/semver/v3 v3.3.1 h1:QtNSWtVZ3nBfk8mAOu/B6v7FMJ+NHTIgUPi7rj+4nv4=
github.com/Masterminds/semver/v3 v3.3.1/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM=
github.com/PuerkitoBio/goquery v1.9.2 h1:4/wZksC3KgkQw7SQgkKotmKljk0M6V8TUvA8Wb4yPeE=
github.com/PuerkitoBio/goquery v1.9.2/go.mod h1:GHPCaP0ODyyxqcNoFGYlAprUFH81NuRPd0GX3Zu2Mvk=
github.com/Soontao/goHttpDigestClient v0.0.0-20170320082612-6d28bb1415c5 h1:k+1+doEm31k0rRjCjLnGG3YRkuO9ljaEyS2ajZd6GK8=
Expand Down Expand Up @@ -81,6 +81,14 @@ github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/grafana/k6build v0.5.11 h1:nuYV5DOMz5PEU8/XFRcDcfgzxUlLujeYNkU7NVEbdfc=
github.com/grafana/k6build v0.5.11/go.mod h1:SA6/5cnnCLQ99IpFrqVfcuO+SxCRo/yLGtcnj3bfpII=
github.com/grafana/k6deps v0.2.6 h1:T9t5pGnL4697SJ6fCF36fNCNSFSH/kDBI9Do/W+5TqA=
github.com/grafana/k6deps v0.2.6/go.mod h1:y+pBsBemOJHML6ObuJFVgcNgO9RSfRvBklltI194aTE=
github.com/grafana/k6foundry v0.4.6 h1:fFgR72Pw0dzo8wVWyggu35SGGjdt31Dktil1bDE1MCM=
github.com/grafana/k6foundry v0.4.6/go.mod h1:eLsr0whhH+5Y1y7YpDxJi3Jv5wHMuf+80vdRyMH10pg=
github.com/grafana/k6provider v0.1.15 h1:aUStpqDMEnEL9aGCcSKmpcreHRZsr8IELna+ttKMOYI=
github.com/grafana/k6provider v0.1.15/go.mod h1:tnyPNxenfF0G5JWVXQixk7QUJIn2FbZneDfvgocttNs=
github.com/grafana/sobek v0.0.0-20250320150027-203dc85b6d98 h1:DqWI8D/A8GABIIjukZVNr0Sj4sBeewK2TmbTyiqUAZk=
github.com/grafana/sobek v0.0.0-20250320150027-203dc85b6d98/go.mod h1:FmcutBFPLiGgroH42I4/HBahv7GxVjODcVWFTw1ISes=
github.com/grafana/xk6-dashboard v0.7.5 h1:TcILyffT/Ea/XD7xG1jMA5lwtusOPRbEQsQDHmO30Mk=
Expand Down Expand Up @@ -235,6 +243,8 @@ golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.23.0 h1:Zb7khfcRGKk+kqfxFaP5tZqCnDZMjC5VtUBs87Hr6QM=
golang.org/x/mod v0.23.0/go.mod h1:6SkKJ3Xj0I0BrPOZoBy3bdMptDDU9oJrpohJ3eWZ1fY=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
Expand Down
Loading
Loading