How did you implement the reporting pipeline?

[HHobeck]

Hi @gasparnagy!

I really like the transparency reports/statistics you publish for Reqnroll. We’re discussing something similar for GitVersion and I was wondering: how did you implement the reporting pipeline?

Mainly curious about:

• which analytics backend(s) you use,
• how the data is collected,
• where are you hosting your infra (self-hosting, third party)
• whether the reports are generated automatically and
• if any part of the implementation is reusable or open source?

We’re especially interested in automated monthly report generation (possibly via GitHub Actions) containng the following information:

• .NET Framework Usage
• CI/CD Server Usage
• Visual Studio version usage

We’d love to learn from your experience before reinventing the wheel 🙂

• Implement time-boxed CLI telemetry with opt-out controls GitTools/GitVersion#4927 (comment)

Cheers!

[gasparnagy]

Hello @HHobeck!

Sorry for the delayed response, happy to help.

• which analytics backend(s) you use,

We use Azure Application Insights. I think earlier we used Google Analytics. I haven't been there where the decision for Azure Application Insights was done, and I have not checked the alternatives, but Application Insights works fine in general.

• how the data is collected,

Basically Azure Application Insights provides a REST endpoint where you need to send the events in a specific (simple) format. Each event has a name, some common properties, custom properties (key/value pairs) and custom metrics (key/number pairs).

You can send these to the endpoint via simple HttpClient calls (this is what we do from Reqnroll runtime) or there is also a Microsoft.ApplicationInsights.WorkerService library that simplifies that (this is what we use from VS extension). If dependencies are not a problem then this latter approach is better.

These endpoints are public, no special authentication is required to send data to them.

We generally have two endpoints: the debug and the prod. By default the code is configured to send to the debug, but for release builds we reconfigure it to use the prod endpoint with a build task.

This way the debugging, local testing does not influence the stats and we can use the debug endpoint to test telemetry itself.

• where are you hosting your infra (self-hosting, third party)

Azure Application Insights can only be hosted in Azure (i think), currently it is hosted within my own azure subscription, but my plan is to move it out to a standalone one. But it is not free, in the last 8 months we have spent ~500 USD, which has been paid by my company so far, but the plan is to pay it from the sponsorship we get.

• whether the reports are generated automatically and

The reports are not generated automatically, but the ones based on Azure Application Insights could be. Application Insights supports KQL (Kusto Query Language) that is an SQL like query language. For each sub-report, like ".NET Framework Usage", I have a KQL query that splits out the numbers, that I just include to the monthly post.

Application Insights has an own API, so I'm sure those queries could be executed from the API as well, and that part of the report could be generated automatically, but I haven't spent the time for that yet (and our report also contains Google Analytics website data and some sponsorship summary that is from a third source, so for us it is a bit more complicated).

• if any part of the implementation is reusable or open source?

The telemetry in VS extension is at https://github.com/reqnroll/Reqnroll.VisualStudio/tree/main/Reqnroll.VisualStudio/Analytics,
in the Reqnroll runtime at https://github.com/reqnroll/Reqnroll/tree/main/Reqnroll/Analytics. Both are a bit outdated, probably some cleanup would be useful, but feel free to use it.

The KQL queries are not in the project, but I can share them with you if it helps.

We’re especially interested in automated monthly report generation (possibly via GitHub Actions) containing the following information:

• .NET Framework Usage
• CI/CD Server Usage
• Visual Studio version usage

You can find the related data sending details in the codebase above, and I can share the KQL queries.

Just as an example, here is how the CI/CD server usage query looks like:

let nowDate = now();
let toDate = make_datetime(datetime_part('year', nowDate), datetime_part('month', nowDate), 1);
let fromDate = datetime_add('month', -1, toDate);
let mainTable = customEvents
    | where timestamp between(fromDate .. toDate)
    | where isempty(operation_SyntheticSource)
    | where name == "Running Reqnroll project"
;
let frameworkByProject = mainTable
    | extend ["HashedAssemblyName"] = tostring(customDimensions["HashedAssemblyName"])
    | extend BuildServerName = tostring(customDimensions["BuildServerName"])
    | extend BuildServerName = iif(isempty(BuildServerName), "<undefined>", BuildServerName)
    | extend BuildServerName = iif(BuildServerName == "GitLab CI/CD", "GitLab CI", BuildServerName)
    | project HashedAssemblyName, BuildServerName
    | distinct *
;
let totalRows = toscalar(frameworkByProject
    | summarize count());
let totalRowsDefined = toscalar(frameworkByProject
    | where BuildServerName != "undefined"
    | summarize count());
frameworkByProject
| summarize Count = count() by BuildServerName
| order by Count desc
| extend Percentage = round(todouble(Count) * 100 / todouble(totalRows), 1)
| extend PercentageOfDefined = round(todouble(Count) * 100 / todouble(totalRowsDefined), 1)

We can have an online chat where I can show the things live, if that helps. Drop me a mail to gaspar🐌specsolutions.eu.