Skip to content

Fixed averaged battle points calculation #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jan 31, 2022

Conversation

ImogenBits
Copy link
Collaborator

Currently the points for the averaged battles are calculated incorrectly.
The current formula for the valuations where a higher valuations means you get a higher share of the point is
valuation = (len(ratios) / sum(ratios)) / len(ratios) which simplifies to valuation = 1 / sum(ratios). So to get more points you want a smaller sum of approximation ratios.
This schema breaks when some, but not all, of the iterations were not solved by the solving team. Then the list of approximation ratios will have 0 entries since that is the default value assigned to unsolved instances. There is error handling for completely unsolved instances, but not for ones that are sometimes solved and sometimes not.
With this calculation it is better to e.g. not solve the problem 9 out of 10 times, but solve it very badly with an approx. ratio of 9 once (valuation 1/9), than to solve it perfectly every time (valuation 1/10).

As this is clearly not intended behaviour I made this PR to fix that bug and also incorporate a small amount of cleanup changes.
The new formula works by calculating the average reciprocal of the approx. ratios and setting unsolved instances to 0. This means that an unsolved instance will always be worse than a badly solved one. I have decided on this schema instead of eg giving the unsolved instances a very high approx. ratio because in the case of a team solving 9/10 instances very well but not solving 1 they should not be punished too harshly.

The rest of this PR is not essential to this fix, but should improve overall code readability in the future. The variable points_per_iteration does not actually track the iterations, but rather the number of rounds. These two terms are not used interchangably throughout the rest of the codebase so they should be differentiated here too.
The other change is that previously each combination of teams had their points calculation done twice as it was done per permutation instead. This does not change the output as it previously was done twice and the value halved.

@Benezivas
Copy link
Collaborator

Thank you for the pull request, the code does indeed not behave as intended, due to brackets being set incorrectly. The intended way was

valuation = len(ratios) / (sum(ratios) / len(ratios))

which still does not do what it is supposed to do, as the len(ratios) still includes the 0.0 ratios.

The basic idea for calculation is even given in the docstring:

        The valuation of an averaged battle is the number of successfully
        executed battles divided by the average competitive ratio of successful
        battles, to account for failures on execution.

I would like to extend a little bit on the two competing ways of calculating the valuation.
As an example, compare the results [1.0, 2.0, 0.0, 0.0, 2.0] and [1.0, 2.0, 3.0, 0.0, 2.0] and their valuations:

res int   sugg
1   9/5   2/5
2   16/8  7/15

So both reward a team having a ratio compared to having none.

This does of course change if we replace the added ratio of 3.0 by a much bigger value. Then the intended valuation tends quickly to zero while the suggested one treats it like it was a zero ratio.

Thus, I think your suggestion is overall better. Teams should always be rewarded for having a competitive ratio compared to none. If the added competitive ratio is bad, it is better if it does not add anything positive to the valuation rather than actively punishing a team.

@Benezivas Benezivas self-assigned this Jan 31, 2022
@Benezivas Benezivas changed the base branch from main to develop January 31, 2022 10:12
@Benezivas Benezivas merged commit 04072d2 into Algorithmic-Battle:develop Jan 31, 2022
@Benezivas Benezivas mentioned this pull request Jan 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants