-
Notifications
You must be signed in to change notification settings - Fork 3
rework problem and battle wrapper classes #73
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
oh i forgot to mention this today and it's a really weird bug to track down if you run into it yourself, the old style of testing docker images don't work with the new docker interaction anymore! because we now use drive mounts the files that we want in the output need to be written there during the execution of the container, so
won't work, but
will |
I have finally got a machine with Two things I have noticed while getting the machine up and running on this PR:
More feedback will come once I have time to properly dig into these changes. Thank you already for tackling this bigger rework. |
on the first point: it still does this, but only when using verbose logging, I thought that was the previous behaviour too but you're right that it's not. on the second: yeah, the ui display is pretty broken at the moment (even before this PR), but fixing most of the issues would require pretty extensive reworks that I'd like to only do when I know what the web server side of things actually looks like. |
I have started playing around with the new problem model, currently trying to port one of the If I want to upper bound the |
If I'm understanding you correctly you're trying to do something like this: class MyProblem(ProblemModel):
size: int
class MySolution(SolutionModel):
val: int = Field(le=size) This is indeed not possible. class MyProblem(ProblemModel):
size: int
class MySolution(SolutionModel):
val: int
def check_semantics(self, instance: MyProblem, size: int) -> bool:
return self.val <= instance.size But this is a good opportunity to consider this kinda situation more abstractly: Really this is an issue of where we draw the line between "syntactically" incorrect and "syntactically" correct, but "semantically" incorrect outputs. This distinction seems pretty arbitrary and I think it ultimately is highly situational. The same property like the output being a complete graph might be entirely "semantic" for most graph problems, but for things like TSP it is a "syntactic" requirement on the input. |
I agree, the separation between syntax and semantic of a problem instance is very blurry, also in the old format. The proposed solution of doing the size checks in the "semantics" section is fine, a renaming to validation seems appropriate. I will try to port one or two more problems that I remember to have esoteric behaviour (TSP wanting to give the solving team more slack during verification than the generating team, Path bounded Vertex Cover needing a certificate intermediate solution) and then come back to this PR. Feel free to do necessary changes meanwhile, I do not mind adapting rewritten problems again. |
I am currently trying to implement a problem that is trying to distinguish between a solver solution and a generator solution in order to determine the quality of a solution: The solver is supposed to have an advantage in the calculation. The current draft of the problem file is the following:
The main issue lies in the question of how to access the attributes of the Since we are not in a simple subclass relation between |
Nevermind, I can simply pass |
I think I have played around sufficiently with the new I did not have the opportunity to play around with the new battle wrappers and to implement a new one, but the proposed changes seem reasonable and I do not see an issue in merging them into the rc right now. Thank you very much for the changes. I will port more problems to the new format over time, for which we should also discuss a standard of how to format the documentation soon. The READMEs currently still emulate the generated pdfs we use for the lab courses and are not very pretty. Since the |
This PR completes the changes to the problem and battle wrapper classes we discussed. as they are pretty extensive we'll probably have to talk about them in person more, but I'll give a basic overview here:
From the point of view of the Problem class I essentially kept the same interface but simplified it greatly. Instead of having various methods for each step of the parsing and verifying process and dealing with the raw data streams we now have only 4 basic methods:
decode
takes a path to the folder where the docker container placed the problem instance (or a solution to one) and parses it into a problem objectencode
does the reverse, taking a path to a folder where the container expects the data and places a representation of itself therecheck_semantics
verifies that the constructed object is semantically correctcalculate_score
returns a value in [0, 1] that tells us how well a solution performs for a problem instanceAll of them take not only the raw data / object but also other relevant info such as instance size or what role we're encoding things for. We also provide default implementations for all of them so that in practice the end user has to rarely touch them.
Of note here are the
ProblemModel
andSolutionModel
classes and theScored
protocol. The exact details of how they work is a little hard to explain and very pythonic but if you look at the new pairsum and biclique problem definitions the semantics should be pretty intuitive.I also slightly modified what we expect problems to look like. Previously the generator always output a problem instance and a solution and then the solver's task was to provide a better solution. We now just expect the generator to create a problem instance and for the solver to provide a solution. This lets us implement basic decision problems directly (though the
calculate_score
method might be rather hard to write efficiently then), and even a natural approach to problems in NP \cap co-NP where we don't need the generator to provide a witness and can rely entirely on the solvers.To still make this style of problem with a generator provided solution easier to write I added the
with_solution
flag, with it set (the default) the problem behaves more or less like the current ones.The side of the battle wrappers also changed significantly. The broad takeaway here is an additional layer of abstraction over raw docker images, the
Program
class and its childrenGenerator
andSolver
. These essentially handle all of the low level details of parsing between the file systems and the python environment. A battle wrapper gets passed a generator and a solver and can then call theirrun
methods with the appropriate data. We also provide arun_programs
method on the base wrapper that safely does precisely that (essentially behaving like a much more powerfulFightHandler
) and returns an object summarizing the results.Of note here is that these methods don't take just the instance size or generated instance, but also docker config options such as runtime, space, and number of cpu cores. These don't need to be passed most of the time, the program objects already have the settings as parsed from the cli/config file, they just enable the wrappers to not only adjust the instance size during their run but also other parameters. This lets us make wrappers that eg test the generator to see how long it takes to generate an instance too hard for the solver.
They also take arguments specifying the
battle_data
. This is essentially any data that we can pass to a container and retrieve from it, as specified by the wrapper instead of the program. This lets us make wrappers that eg let each team remember some fixed amount of data between runs, lets the teams adjust their strategy based on how well they are doing, or even based on previous instances or solutions.