[QA] Uploading a answer-file without specifying the dataset

[RicardoUsbeck]

As a user, I would like to be able to upload answer-files without specifying the target dataset.

Solution: Make a search through the uploaded answer-file for the question stemming from the evaluation dataset. If the uploaded answer is not there, vote it as 0 in all metrics otherwise evaluate.

[MichaelRoeder]

I am not convinced that this is a valuable feature. I see two solutions to implement that and both have drawbacks.

1. search every question individually in all available datasets.

   • That does not fit to the workflow of GERBIL
     • Can not map the result to a single dataset
     • Breaks goals like reproducability
   • Would have to instantiate all datasets for the search
   • Special case: how to handle a question that is available in more than one dataset?

2. For every question in the result file, search through all questions from all datasets. Choose the dataset with the most questions matching the result file as the dataset for the evaluation

   • Would have to instantiate all datasets for the search
   • Special case: how to handle an uploaded file that matches more than 1 dataset an the number of matching queries is equal for all of them?

[RicardoUsbeck]

With respect to being a valuable feature: I imagine the following scenario: some crazy guy comes up with a chess QA engine and a chess QA dataset. Pretty crazy, right? He wants to test his system's answers (formatted as a QALD JSON file since he has no public IP address or URL) against his dataset (formatted as a QALD JSON file).

My solution: for each question in the dataset(s), you look into the uploaded file, whether there is a question with the same string (basically treating the file as a system).

[MichaelRoeder]

Ok, so you still have a mapping to a dataset but instead of mapping it to one of the listed datasets, you would like to map it to the uploaded dataset. Yes, that makes it much clearer! In this case it would make sense. Although your example scenario is really unrealistic.

What about the following idea: next to the known datasets in the drop down menu, we could add a uploaded dataset line. If this is chosen, the user has to upload a dataset - otherwise the submit button can not be pressed.

[RicardoUsbeck]

Yes! There should be "next to the known datasets in the drop down menu [an] a uploaded dataset line. If this is chosen, the user has to upload a dataset - otherwise the submit button can not be pressed."

However, what about the scenario that someone, e.g. Semantic Web lecture students, have no way to deploy a web service but are asked to evaluate there system with GERBIL QA? Then we could <FILL IN TEXT>

[TortugaAttack]

unrealistic?
I genuinely had to supervise such student groups :D

But i do not understand the answer file upload drop box either way.
As a user you have to choose the dataset either way or not?
I think without the answer file dropbox and using just the dataset dropbox/upload it is much more intuitive. But just my two cents.

[RicardoUsbeck]

(https://github.com/dice-group/semanticchess)

[MichaelRoeder]

@RicardoUsbeck your second last comment simply ends within the sentence. Maybe you want to finalize your thought because I do not really understand it 😉

Nikit will create an easy solution based on what we already have.

[RicardoUsbeck]

It actually ate my parameter. Thus, Nikit can go ahead

[MichaelRoeder]

Proposed solution:

1. The user uploads a dataset
2. The dataset is added to the list of datasets for which a solution can be uploaded
3. The user uploads the answer file and chooses the uploaded datasets from the drop down menu

The main disadvantage of this approach is that it is counter-intuitive because the user has to add the dataset first, scroll back and upload the answer file.

Apart from that, it has to be made sure that the backend can handle the data.

[RicardoUsbeck]

We initially did introduce that drop-down list, because we didn't want to do a n times m search to find the matching question-answer-pairs from the dataset and the system answer, right? So maybe, for the case of an uploaded dataset, the backend should do that sacrificing runtime efficiency?

[MichaelRoeder]

But it is our runtime on our server 😢
If the user wants to sacrifice runtime on his/her own machine, I am fine with that 😄

I think there should be not too hard to adapt it to work as before. We simply have to add this special case to our backend, I guess 🤔

[RicardoUsbeck]

You mean @nikit91 has to add this special case 😄