gabyhelp feedback

[rsc]

I am experimenting with running a new bot @gabyhelp (which I am writing) in the Go issue tracker to try to help automate various mundane things that a machine can do reasonably well. Some aspects of the work may involve AI/LLMs; some will not.

Please thumbs up or thumbs down individual comments from the bot depending on whether they are correct/helpful or not. If the bot is getting in the way, we obviously want to know and fix it.

When the bot posts a comment, it will link back to this discussion page, which is a good place to leave non-boolean feedback. Thanks!

This discussion is also a good place to post ideas for things you wish a bot would do for you in the issue tracker.

To snoop around in all the issues where the bot has commented, you can use this search.

[rsc]

Right now, @gabyhelp is editing recently modified/posted comments and issues. Anywhere the plain text "CL nnn" appears, it changes the text to be a link to go.dev/cl/nnn. I've meant to do this for many years now, so that's the first chore.

[qiulaidongfeng]

See https://go.dev/doc/contribute#commit_messages , The rest of the description elaborates and should provide context for the change and explain what it does. Write in complete sentences with correct punctuation, just like for your comments in Go. Don't use HTML, Markdown, or any other markup language.
Does this mean that the Contribution Guide need to be revised？

[thepudds]

Hi @qiulaidongfeng, I think that section of the Contribution Guide is talking about avoiding HTML and Markdown in commit messages, vs. Russ above is talking about automatically editing GitHub issues.

[rsc]

@qiulaidongfeng The bot accidentally edited your PR description, because GitHub pretends that every PR is an Issue. The bot is only meant to be editing Issues, and I changed it to explicitly ignore PRs. Apologies!

[qiulaidongfeng]

For example :
@gabyhelp rewrite

CL 512155

to

[CL 512155](https://go.dev/cl/512155)

in #67902.

[CL 512155](https://go.dev/cl/512155)

isn't Markdown?

[rsc]

Yes, that's the PR that it accidentally edited. Again, my apologies.

[rsc]

@gabyhelp has also started posting to new issues (plus the issues created in the last two days, to get started) with lists of similar issues, if it finds any that are similar enough. If not, it will stay quiet.

Here is one success story: #67887 (comment). And here is another: #67897 (comment). This one is only a partial success because @thepudds got there first, but only because the bot gave him a head start. 😄

[dolmen]

This post by @gabyhelp is definitely useful: #67943 (comment).

[dmitshur]

In #68028 (comment), it's visible that gabyhelp currently doesn't filter out PRs in the list of potential "similar issues", which it probably should. GitHub API treats issues and PRs similarly, but PRs are a subset of CLs.

[rsc]

I think the only issue is the title (Similar Issues). In general if something is very related I think we should surface it. If an issue gets filed and gabyhelp posts hey there is a PR that seems to be about this issue, that's very helpful. The plan is to include all Gerrit CLs eventually too. (At that point we'd probably want to deduplicate GitHub PRs against Gerrit CLs.)

[dolmen]

@gabyhelp 's success is more about GitHub search being a big fail.

As an example I have just filled #68102 because a search of GOBIN didn't show anything relevant. See https://github.com/golang/go/issues?q=is%3Aissue+GOBIN+-author%3Adolmen

[benaryorg]

I agree that this feature is definitely useful, and I also agree with it being mostly useful because GitHub search is borderline useless¹ at this point.
What I'm a little confused about is the line at the bottom of the comment; "Emoji vote if this was helpful or unhelpful".
In my case it did not show any duplicates, however the post was definitely helpful in that it affirmed that I did not miss any.
Am I supposed to react to this?
In technical terms, a true positive is of course good, and a false negative is bad, but if the list contains only (close enough) false positives when there is no true positive, should that be counted as a true negative?
Maybe a different emoji for that, and a clear guidance on which emoji to use for what?

Though I understand if that'd be too much noise for people who don't have my particular set of 'tisms, just my 2¢ here.

Footnotes

1. This does of course suck on a fundamental level because I'd rather have GitHub fix its search rather than a community project having to pick up its slack, but that still makes the community project a cool thing, thanks! ↩

[magical]

I imagine this is on your list already, but in case not: another thing that would be nice would be for the bot to rewrite google-internal urls to their public counterparts.

[rsc]

Indeed. Thanks for the reminder. Done! And apologies again for having the internal-vs-external split at all. I'm just as annoyed by it as all of you. (It's literally just the same site with a different URL to force a different, stricter authentication path.)

[randall77]

It might be nice to have gabyhelp not generate emails when posting to issues. I see its updates, particularly related issues posts, as not needing a "ping". It's just additional information put there when you otherwise navigate to an issue.

[rsc]

@prattmic This should be fixed now.

[hyangah]

On the other hand I miss the github's formatting that shows whether the issues are closed or not clearly, and gets updated. Is there a way to mix both somehow?

[rsc]

Sure. Once I get better tests written I will add "(closed)" markers.

[cespare]

@rsc but that doesn't cover "and gets updated" part. I guess the "(closed)" markers will just be a snapshot of the state at whatever time the bot processed that comment?

IMO all these options are worse than going back to the initial state where we let GitHub dynamically format the issue links and sacrifice a bit of readability in the emails.

[AlexanderYastrebov]

Just found this thread and would like to +1 the idea to unfurl github links in comments.

[seankhliao]

I wonder if can be smart enough to recognize misformatted text, especially code blocks. We often see issues with no fences around code blocks or using single backticks (I assume this is from jira's formatting).

[deefdragon]

Related, I would love if it could add syntax highlighting to existing code-blocks that don't have it. It makes it so much easier for me to read when the highlighting is applied.

[rsc]

@seankhliao Can you point to specific issues with this problem? Thanks.

@deefdragon Different people have different but very strong opinions about syntax highlighting. I'm going to focus on clearer wins.

[seankhliao]

• unfenced code: Bug Report: JSON Deserialization Case Sensitivity Issue in Go #67689 , runtime: time.Sleep problem on Windows 10, but not Windows 7 or Linux #61042 , go v1.17.x breaks the (os.exec) Cmd.Start() functionality on Windows #49044
• unfenced stacktrace: net/http: Transport memory leak #43966 , Panic when trying to serve http with embedded assets #43761
• jira style(?) code blocks: crypto/rsa: can generate digital signature when hash algorithm (digest method) is set to MD5 in FIPS mode #45565

[seankhliao]

you'll need to look at the edit history for these, since I found them with a google search of "edited by seankhliao"

[ianlancetaylor]

Just a note that the similar issues list in #67861 was not helpful other than the reference to #33818. The issue #67861 was about changing sort.Sort (which we can't do), and the related issues are about other changes to sorting. We don't necessarily have to do anything here, the other issues weren't outrageously wrong, they just weren't helpful.

[cespare]

In #67948 (comment), the similar issues were not helpful.

The first listed issue, #56471, is very related -- as I wrote in the description, I created #67948 as a spin-off from #56471. Perhaps gabyhelp could be updated not to list issues which are already mentioned in the description?

The other four listed issues are superficially similar but not actually related in a useful way.

[jfrech]

I would feel alienated were I nudged to interact with an LLM by merely browsing Go development. I hope you will confine your bot to deterministic, comprehensible actions only.

[rsc]

I probably share your skepticism and concerns, but I don't fully understand the specific scenario you are concerned about. Can you say more?

[Merovius]

I think one thing I don't want to happen is a bot ask me clarifying questions, for example.

Another example of something I've seen LLMs used for which I consider insulting would be a bot re-phrasing comments.

I guess personally I'm mostly fine with AI use-cases I'd call "classification" but would feel patronized by any use-case that tries to emulate (or interlope in) human interaction.

[jfrech]

@rsc I don't envision a scenario. For me, it is a purely aesthetic matter: I have personally sworn not to interact with AI in my life. So when it becomes prevalent on this forum, I will have to leave. In this sense I feel getting alienated.

[malikwirin]

@rsc I don't envision a scenario. For me, it is a purely aesthetic matter: I have personally sworn not to interact with AI in my life. So when it becomes prevalent on this forum, I will have to leave. In this sense I feel getting alienated.

aesthetic or more like esoteric?

[mvdan]

Would this bot eventually replace the rather basic "Gopher Robot" at https://go-review.googlesource.com/q/owner:gobot@golang.org? It currently seems to rely on a few heuristics to tell PR/CL authors about common mistakes, such as in https://go-review.googlesource.com/c/build/+/591455. It seems rather similar to the bot discussed here.

[rsc]

I think in the long term yes we'd want it to be able to do everything the current Gopherbot does. Right now we're focusing on what it can do that's new.

[ianlancetaylor]

In #67934 gabyhelp correctly reported that the issue was similar to #67933. That is because they are both backport requests for the same original issue. Perhaps gabyhelp can skip backport issues.

[findleyr]

Thanks @seankhliao, you beat me to it. It would be nice if gabyhelp could recognize issues that are 'process'.

[rsc]

I already disabled it on watchflakes issues, and I will disable it on backports issues too.

I am happy to disable it on gopls release issues too. Is that any issue with title prefix "x/tools/gopls: release version v"?

[findleyr]

@rsc yes all gopls release issues follow that format.

[rsc]

Thanks. Updated to skip both backports and gopls releases:

	if strings.Contains(issue.Body, `— [watchflakes](https://go.dev/wiki/Watchflakes)`) ||
		strings.HasSuffix(issue.Title, ` backport]`) ||
		strings.HasPrefix(issue.Title, `x/tools/gopls: release version v`) {
		continue
	}

Eventually we will hopefully be able to tell the bot to do this kind of thing. 😄

[seankhliao]

also package removal requests?
#68003 (comment)

[dolmen]

The Go project has a redirector for permalinks.
Examples: https://golang.org/s/generatedcode https://go.dev/s/generatedcode (I don't know which one should be preferred nowadays).

It would be helpful if @gabyhelp could replace links to targets of the redirector (and hopefully, any previous targets if the short link has an history) by the short permalink.

[thanm]

For my newly filed issue #67953, the related issues that @gabyhelp annotated it with (in #67953 (comment)) are not really all that helpful. Yes, they are all problems with coverage, but don't really relate to the very specific problem at hand.

[rsc]

The related issues for my issue #67961 (filed using gopls bug) are all empty instances of the gopls bug template. They are 90% similar to mine. Maybe I was too concise in my report!

Probably we need to look for empty templates anyway for auto-closing and then exclude them. (It's easy to tell, the empty templates are all 99% similar to each other.)

[rsc]

Another potential thing to explore is trying to help deduplicate the automated crashes in https://github.com/golang/vscode-go/issues.

[MikeMitchellWebDev]

As a relative newcomer to filing issues, I found the Gaby bot helpful for re-categorizing the issue I am filing (i.e editing my own title after the Gaby bot makes its post) (#67984)

[findleyr]

In #67995 (comment), Gaby provided a remarkably accurate summary of the timeline of our feelings about file watching in gopls: the initial issue where we discussed the philosophy of client-side file watching, and then the most relevent issues where we struggled to work around client-side issues, culminating in the current https://go.dev/issue/67995 which I filed to explore server-side file watching.

I experienced delight :).

[mvdan]

Can @gabyhelp learn to close empty issues? This should also include issues which copied the template but did not fill it out. For example: #69795

[jba]

Filed golang/oscar#37.

[Jorropo]

Some kind of issues are not helpful to give recommendations #70376 (comment) altho it's mostly harmless
could be very smart like by predicting click through rate and not posting if it would be too low or as simple as hardcoding access:

[prattmic]

While I agree that linking to other approver issues isn't helpful, it actually is helpful that it linked to your prior may-start-trybots issue. (Though just having the issue title isn't great since it isn't obvious that it is your issue).

[Jorropo]

Fair enough

Though just having the issue title isn't great since it isn't obvious that it is your issue

Yes got me

[millerresearch]

@gabyhelp not helpful on issue #70484 reporting on a bug which was introduced by a specific CL. Maybe if the title includes "after CL ..." or "since CL ..." it's not worth looking for other issues?

[hyangah]

Thanks! I see gabyhelp's comment doesn't bring new info in this specific case.

On the other hand, when there are already existing reports or the breakage was actually a reoccurring bug, or if the mentioned CL turned out to be causing other issues, the related issue feature could surface them and be helpful (of course, if that can be done correctly).

[ALTree]

In #70571 (a spec-related issue), in the doc section, the bot linked both the current and the historical go1.17 spec at https://go.dev/doc/go1.17_spec. The 1.17 spec page is only there for historical reasons and in general I don't think there's any benefit in linking it in a discussion about the current version of the language; and it could create confusion.

[hyangah]

Thanks! Filed golang/oscar#63

[seankhliao]

In #70569 (comment) it linked to the same issue it was commenting in.

[hyangah]

Thanks! Filed golang/oscar#62

[Jorropo]

Example of messages that would help if it added codeblocks & formated:

• range generics list type error #70842
• runtime: "SIGILL: illegal instruction" at arm64 v8.0 and use go routing #70987

[jba]

You may have noticed that @gabyhelp is now labeling issues. Feedback welcome.

[jba]

Bug is gone.

[seankhliao]

#71362 i think should be a bug report (like some others when people are too lazy to fill out the bug report template), and I would have expected the bot to make the same judgement (rather than just copying the proposal label).

[seankhliao]

I feel like the labels it manages can be somewhat redundant with all the other labels we have

perhaps it could instead manage issue types rather than labels?
https://docs.github.com/en/issues/tracking-your-work-with-issues/configuring-issues/managing-issue-types-in-an-organization

[apparentlymart]

I've noticed that on some issues it has added the "LibraryProposal" label without also adding the "Proposal" label.

I'm not sure if that was intentional, but I'd personally like there to be one label that is present on all proposals regardless of what they affect, even if there's also more specific labels like "LibraryProposal", just because that makes it easier to quickly construct a filter to help answer the question "was there already a proposal for something like this?" while excluding all of the bug reports, etc.

[dolmen]

@apparentlymart A bug may be related to a proposal without being a proposal. For example #74757 is a bug related to the #ToolProposal feature, but isn't a proposal.

[AndrewHarrisSPU]

The gabybot summary of 71203 was pretty interesting. The summary seems useful (not impressive, not sharp, not insightful, but plausibly useful). It captured topics well, although I thought it missed some interesting or convincing posts and seemed a bit long. Of course it's very, very cool that this exists and wouldn't be written otherwise.

There are further issues specifically with the "issue comment" citations that seemed worth detailing: Out of 27 (19 unique, I think) "issue comment" citations made by gabybot:

1. Sometimes there is a poor conceptual relationship between the point gabyhelp is on, and the content in mentioned posts:

• discussion themes sub 1: kortshack mention
• discussion themes sub 1: dominikh mention (also, link is marked off-topic)
• discussion themes sub 2: thiht mention

(These are only the cases I thought were objectively clear. There are also fuzzier cases where a conceptual relationship was IMHO too thin, but not absent.)

2. Sometimes gabybot has made reference errors, mentioning the correct participant but linking to the wrong post/participant:

• discussion Themes sub 2: misdirect: doggedOwl mention links to ianlancetaylor comment (doggedOwl comment appears to be distant)
• next steps: earthboundkid mention links to mzzsfy comment (mzzsfy comment is immediately prior to earthboundkid comment)
• next steps: puellanivis mention links to jba comment (puellanivis comment is immediately prior to jba comment)

3. Errata:

• spams mentions of the original post early on (7 times)
• discussion themes sub 3: ConradIrwin post linked twice
• next steps: offtopic: godcong mention links to off-topic comment (ironically, comment suggests AI summary)

[apparentlymart]

I was kinda amused when this summary appeared while I had already been working on a manual summary in an earlier comment.

Subjectively, I think the generated summary was easier to consume than the huge list of links I ended up with. However, I must admit that I didn't actually try following most of the links because I thought I could guess what they were linking to based on what I already had in my head from writing my summary manually.

The "off-by-one" errors in the linking are unfortunate, but I did still enjoy the text summary of topics that had already been raised since GitHub had already hidden most of the discussion and so the issue degenerated into the same comments made over and over again. It seems like that slowed a little after the summary appeared and the discussion switched to some new topics, but the summary is now also hidden by GitHub and so duplicate remarks seem to have started again. It's unfortunate that GitHub doesn't have a concept of "pinned comments" that are prioritized for display regardless of how many other comments there are.

[rogpeppe]

FWIW as someone that hasn't read the full discussion but is familiar with other similar previous proposals, I felt that the gabybot summary was useful and saved me a bunch of time. I feel that I know where the discussion has gone and know enough to decide whether I would like to contribute or dive deeper.

[seankhliao]

I think it should ignore any issues closed/locked as spam
#72051 (comment)

[firelizzard18]

I would find a search feature very useful. GitHub's issue search is not very useful, and it soulds like Gaby's database includes a lot more than just issues. I can't count the number of times there's some obscure feature or issue report or something that I remember reading, but barely, and I have a really hard time finding it again. Even if it was as simple as typing in a blurb and having Gaby treat it as a new issue to check for duplicates would be valuable.

[randall77]

Gaby can sometimes be late and redundant to a conversation. For example, on #73253 @thepudds posted saying "I sent CL 663795 for discussion", and then Gaby commented a day or so later mentioning that CL as a "related code change".
Probably not worth the comment if a human already posted that observation.

[toqueteos]

Regarding the CL NNNN "magic text replacement", GitHub does already provide ways to do that without a bot.

They call it Autolink references

It's accessible to repo admins from https://github.com/ORG/REPO/settings/key_links

[jakebailey]

Not for CL NNNN, only CLNNNN or CL-NNNN (no spaces allowed).

[dolmen]

GabyHelp has added the ToolProposal label to #74786, but this isn't relevant. I have not even a mention of "tool" in the title or description.

Could it be that it added that label because the issues I have opened recently are related to the ToolProposal? If yes, this is incorrect behavior.

[dolmen]

Cc: @jba

[AGWA]

It would nice if gabyhelp comments were automatically deleted if they receive a sufficient number of negative reactions (maybe numPositive - numNegative <= -2). That way unhelpful comments wouldn't clutter up the issue.

(I got this idea from Reddit bots whose comments self-destruct if they are downvoted.)