Templating, lower_case variables and files

[mrnugget]

This builds on top of #334 and adds templating to campaign spec including lower-case variable names and access to STDOUT/STDERR.

Updated campaign spec from previous prototype:

name: templated-and-files
description: Use template variables and mount files in campaign specs

on:
  - repositoriesMatchingQuery: repo:automation-testing lang:go fmt.Sprintf("%d", :[v]) patterntype:structural -file:vendor count:5

steps:
  # Run comby over the search results in each repository:
  - run: comby -in-place -config /tmp/go-sprintf.toml -f ${{ join repository.search_result_paths "," }}
    container: comby/comby
    files:
      # Create files inside the container by specifying path and content here:
      /tmp/go-sprintf.toml: |
        [sprintf_to_strconv]
        match='fmt.Sprintf("%d", :[v])'
        rewrite='strconv.Itoa(:[v])'

  - run: echo "comby found the following problems:" >> CHANGELOG.md && cat /tmp/comby-output.txt >> CHANGELOG.md
    container: alpine:3
    files:
      # files also support templating:
      /tmp/comby-output.txt: ${{ previous_step.stdout }}

  - run: echo $MY_MODIFIED_FILES >> modified_files.txt
    container: alpine:3
    env:
      # env vars also support templating:
      MY_MODIFIED_FILES: ${{ join previous_step.modified_files "\n" }}

changesetTemplate:
  title: templated-and-files
  body: My not-first campaign!
  branch: campaign/testing-files
  commit:
    message: Code your code with templates
  published: false

Details

• ${{ .Repository.SearchResultPaths }} and ${{ repository.search_result_paths }}
• ${{ .Repository.Name }} and ${{ repository.name }} and the other fields and methods on *graphql.Repository
• ${{ .PreviousStep.ModifiedFiles }} and ${{ previous_step.modified_files }}
• ${{ .PreviousStep.AddedFiles }} and ${{ previous_step.added_files }}
• ${{ .PreviousStep.DeletedFiles }} and ${{ previous_step.deleted_files }}
• ${{ .PreviousStep.Stdout }} and${{ previous_step.stdout }}
• ${{ .PreviousStep.Stderr }} and ${{ previous_step.stderr}}
• ${{ join <list> <sep> }}, example: ${{ join repository.search_result_paths "\n" }}
• ${{ split <string> <sep> }}, example: ${{ split repository.name "/" }}

TODOs

• Support mounting local files Erik and I decided against this, since that would make server-side execution harder.
• Parse env too

[chrispine]

Awesome! I like the new lower-case names. (Do we even need the longer-form names?)

[mrnugget]

Awesome! I like the new lower-case names. (Do we even need the longer-form names?)

Well, we don't need the longer-form names, but as I said in the sync meeting: the lower-case names have a bunch of drawbacks, since they're super-dynamically-typed (everything is a map[string]interface{}) and might end in "no value" messages instead of errors, etc.

I think for a v0.1 of templated-campaign specs that's fine as long as we flag it as experimental

[LawnGnome]

I think for a v0.1 of templated-campaign specs that's fine as long as we flag it as experimental

Agreed. I also prefer the lowercase, un-.-prefixed names, as we talked about last week, but I understand the implementation concerns. (I did play around with this a bit on Friday, but couldn't come up with a better approach with the standard library, and didn't want to get into the weeds of evaluating third party libraries right now!)

I think the design here is solid, and we should move on to reviewing and landing this once you're happy with the implementation. Nice job!

[bobheadxi]

Question: does changesetTemplate get access to templating? eying using campaigns to replace the release PR-creating tooling completely one day, and some changes generate long-lived branches while some merge to master/main, e.g. in deploy-sourcegraph, the tool creates a 3.21 branch from master and opens a PR onto the new 3.21 branch from an upgrade-3.21 branch, while other repos don't

[eseliger]

So in your scenario, the 3.21 branch would be at the head of the default branch, and then upgrade-3.21 is a second PR, that actually has changes and should target 3.21?

[bobheadxi]

So in your scenario, the 3.21 branch would be at the head of the default branch, and then upgrade-3.21 is a second PR, that actually has changes and should target 3.21?

Yep - where the 3.21 branch might not yet exist (will campaigns create a branch where the base does not exist? is another question)

So I imagine templating in the changeset template would be, if repo is deploy-sourcegraph, set base branch to 3.21

[eseliger]

I see, thanks for elaborating! Currently we don't have support for creating the base branch. Setting the base branch to an existing branch, however, is a planned feature (also specifying per repo is coming).

[eseliger]

Nice 🌟 Very cool feature and makes using campaign specs so much simpler

[mrnugget]

Here's another example campaign that can be run with this prototype (adding this here so I have a link to share):

# Step 1: build the `ruplacer` Docker image
#   $ cat Dockerfile
#   FROM rust
#   RUN cargo install ruplacer
#
#   $ docker build -t ruplacer .
#
#
#   (Why use ruplacer? Because it supports `--subvert` which allows us to
#   replace `camelCase`, `snake_case`, `ThisCase`, `nocase`.)
#
# Step 2: use the src-cli prototype with templating support
#
#   $ cd src-cli && git fetch && git checkout mrnugget/templates-and-files
#   $ go build ./cmd/src -o ~/bin/src
#
# Step 3: update the `repositoriesMatchingQuery` to include or exclude file types.
#
# Step 4: run this campaign
#
#   $ src campaign preview -f update-language.campaign.yaml
name: update-language
description: This campaign changes occurrences of whitelist & blacklist to allowlist & denylist.

on:
  - repositoriesMatchingQuery: whitelist OR blacklist -file:scss$ -file:html$ repo:github.com/sourcegraph/sourcegraph

steps:
  - run: |
      cat /tmp/search-results | while read file;
      do
        ruplacer --subvert whitelist allowlist --go ${file} || echo "nothing to replace";
        ruplacer --subvert blacklist denylist --go ${file} || echo "nothing to replace";
      done
    container: ruplacer
    files:
      /tmp/search-results: ${{ join repository.search_result_paths "\n" }}

# Describe the changeset (e.g., GitHub pull request) you want for each repository.
changesetTemplate:
  title: Replace usage of whitelist & blacklist with allowlist & denylist
  body: This replaces usages 
  branch: campaigns/allowlist-denylist
  commit:
    message: Use allowlist/denylist in wording
  published: false

[mrnugget]

I think the design here is solid, and we should move on to reviewing and landing this once you're happy with the implementation. Nice job!

That's what I meant with requesting the review 😄 But I should've been clearer:

I think the implementation as it is now is fine to be merged as an experimental feature, which we will modify and tweak in the future. I checked that this branch here works with sourcegraph@main as well as this branch that knows about step.files: https://github.com/sourcegraph/sourcegraph/pull/15008. I also checked that src-cli@main works with https://github.com/sourcegraph/sourcegraph/pull/15008, so we don't run into "customer needs to upgrade because we merged an experimental feature"-problems.

I'd love to merge this as soon as possible.

[LawnGnome]

That's what I meant with requesting the review smile But I should've been clearer:

Sorry, I think I somehow thought this was still in draft when I looked at it Monday. Real review coming shortly!

[chrispine]

Excited to see this land!