You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm developing data-driven evaluations using Langfuse (see #6860) that looks like this
publicclassSimpleQuestionsScenario{privatestaticreadonlyReportingConfigurationReportingConfiguration=LangfuseReportingConfiguration.Create(/* ... */);[TestCaseSource(nameof(DownloadDataset))]publicasyncTaskEvaluate(DatasetItemitem){awaitusingvarscenarioRun=awaitReportingConfiguration.CreateScenarioRunAsync(scenarioName:DatasetName,iterationName:item.Id!,additionalTags:[]);varresponse=/* ... */EvaluationResult result =awaitscenarioRun.EvaluateAsync(messages,response,[newTraceIdContext(traceId)]);}
I realized that IEvaluationResultStore.WriteResultsAsync is always called with a single ScenarioRunResult in the ScenarioRun.DisposeAsync.
I would like to score an entire Langfuse dataset run (MEAI execution), meaning have a callback when all my iterations are ran that has access to all the ScenarioRunResult. Is there a concept like this in MEAI?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
I'm developing data-driven evaluations using Langfuse (see #6860) that looks like this
I realized that
IEvaluationResultStore.WriteResultsAsync
is always called with a singleScenarioRunResult
in theScenarioRun.DisposeAsync
.I would like to score an entire Langfuse dataset run (MEAI execution), meaning have a callback when all my iterations are ran that has access to all the
ScenarioRunResult
. Is there a concept like this in MEAI?Beta Was this translation helpful? Give feedback.
All reactions