Insights into the growth and maintenance of our Patterns collection

[spier]

Our InnerSource Patterns are maintained by a working group within the InnerSource Commons, often called Patterns WG for short. That Patterns WG is creating a report about anything that has happened related to our patterns once a quarter, to give the Board of the InnerSource Commons visibility into what is going on. These reports have also prove valuable for the Patterns WG itself as it motivates us to pause and look back over the last couple of months every once in a while :)

As an example, see the 2023-05 report.

In these reports we also include some measurable things like

• how many new patterns were proposed (or published in our book)
• how many patterns were improved? (like where did we discover further organizations that are using this pattern)
• etc

However as some of these measures were hard to collect, we did not do this consistently in our reports.

Now I read about the Issue Metrics GitHub Action, and was wondering if we could use that to generate some of the metrics from our reports automatically? See also this related blog post about data-driven insights by GitHub.

[spier]

@zkoppert any recommendations about a useful test run that we could do to find out if the Issue Metrics GitHub Action would be helpful for the Patterns WG?

As preparation for a test run, defining the goal seems like a good activity :)

This could include:

• create a list of metrics that we have tracked in previous reports of the Patterns WG to the board
• (bonus) add some metrics related to the maintenance of our repo that could be helpful to understand how we as maintainers can better support our contributors

Then we could try to generate some of these metrics, using the issue metrics GHA.

Thoughts?

[zkoppert]

I totally agree that first finding a goal is great! Listing those out as the top priorities and then finding tools to support that is best. For example, "Goals: Be responsive (respond to pr and issues in a few days or less), 3 pattern improvements every quarter, etc.

Happy to help automate the metrics part once there are clear goals!

[spier]

@zkoppert I might work on this as part of hacktoberfest. Would you have a bit of time to help me on this?

[zkoppert]

Yeah! Definitely!

[spier]

Awesome @zkoppert.

If you could help me do a first iteration of some metrics on this repo itself, then I am sure I can "learn by doing" and figure things out from there.

Data to track

• number of new Structured patterns - that are new .md files created in the 2-structured folder
• number of new Initial patterns - that are new .md files created in the 1-initial folder
• number of new Known Instances for an existing pattern - PR merged containing this label
• Time to first response
• new issues - for this it would be interesting to only see the ones opened by people that are not on the maintainer team (might be complicated by us using the CODEOWNERS rather than a GitHub team for this)
• new PRs - for this it would be interesting to only see the ones opened by people that are not on the maintainer team (might be complicated by us using the CODEOWNERS rather than a GitHub team for this)

If any of these are complicated, let's skip them. Starting with the easy ones is fine.

Reporting periods

Optimally we would track this data on a quarterly basis, as we report quarterly from the patterns WG to the board.
The quarters we use are non-standard though. We use these months:

• 2-4
• 5-7
• 8-10
• 11-1

Again, if this makes it complicated, let's just track things on a monthly basis.

How to get started?

Would you have time to create a first version of this, that I can build upon?
Or would you prefer to do it the other way around? (i.e. I try to do this, and then ask you for PR review and help?)

[zkoppert]

I can get started on this! I'll let you know which pieces ended up be hard or awkward and thus skipped. :) I should be able to get started on this next week. I've got it marked on my calendar to remind me. 😄

[zkoppert]

Opened a pull request to get started on automating the metrics listed above. #593 It is ready for review. I'll open a new PR later with more metrics as I get time.

[spier]

@zkoppert I had to do minor fixes but now the GHA runs through:
https://github.com/InnerSourceCommons/InnerSourcePatterns/actions/runs/6473958588

The two issues created were #596 and #597.

[zkoppert]

Looks like it is working! I can start working on the other metrics (number of new Structured patterns, number of new Initial patterns, number of new Known Instances for an existing pattern) later this week. 🚀

[spier]

Awesome.

I just realized that all metrics have to work of issues/PRs, right?
At least in order to use the issue-metrics GHA.

We do have labels that we use for PRs that work on initial or structured patterns.
However with those we would not know if a pattern is new in that state or if it would be an improvement to an existing pattern. We could ignore that difference for now though.

Also one general question about the output of the GHA:
Would we always create one new issue for each metric that we want to track?

[zkoppert]

I can write something that doesn't use issue-metrics GHA in order to get the number of patterns by looking at the files in the repo instead of having to rely on pull requests and labels.

We could merge the outputs of the two GitHub Actions into a single report issue if that is preferred. Let me know!

[spier]

Let's go for the native issue-metrics behavior only. i.e. only search queries (using PRs/labels).
We can still get more fancy later :)

Andy for merging: Yes, getting the report for everything in a single issue would be great.
(would also make for a nice documentation addition to the issue-metrics README)