111 create yaml template for problem submission #121
Conversation
Dvermetten
left a comment
Dvermetten
left a comment
There was a problem hiding this comment.
Maybe it would be useful to have some sort of comments to describe what the fields are / what would be valid input, specifically for the cases where there is a difference between quoted and non-quoted input. Example:
variables: # information about the input variables
types: continuous # can be one of (continuous, integer, binary, mixed)
conditional: 'no' # whether there are conditional dependencies between variables, 'yes' or 'no'
dimensionality: scalable # number of input variables, either as a number (in quotes) or scalable
CIGbalance
left a comment
CIGbalance
left a comment
There was a problem hiding this comment.
Thanks for this! The comments are mostly easy fixes or not that serious.
But I am mostly wondering of how this is supposed to be used? It is called a template, but it has BBOB-specific entries? I think this increases the risk of biasing the results.
I think I would suggest doing a template with a lot more comments (like @Dvermetten suggested) and then we can use this as real data as well as link to it as an example?
| - name: | ||
| short: BBOB | ||
| full: Real-Parameter Black-Box Optimization Benchmarking | ||
| suite/generator/single: suite |
There was a problem hiding this comment.
I still think it would be best to enter problems separately, but I understand the necessity of streamlining this process. So if we want to allow entering suites, maybe we have different templates? Because some values might only make sense for a suite? Otherwise, for a suite, a lot of the input would be "varies" for number of objectives, variables, constraints, dynamic, noise, etc?
There was a problem hiding this comment.
Yes, it would be nice to have problems separately, I have also prototyped how it might work to have both the suite and component problems, see here:
Line 1107 in 84b34b6
I am not sure about separate templates for suites or problems, but we can think about this and discuss what makes sense. I would imagine, e.g., the BBOB sphere still has a bunch of "varies", because it is scalable in the number of variables for example. For a suite, I would want an exhaustive list (e.g., noise: yes, no / optional, or something like that) of the options, rather than a vague "varies". I will describe this in a comment in the template.
| conditional: 'no' | ||
| dimensionality: scalable | ||
| constraints: | ||
| present: 'no' |
There was a problem hiding this comment.
This is confusing to enter, because they have boundary/box: yes.
I would suggest removing present, it should be clear from the rest of the values.
There was a problem hiding this comment.
Yes, makes sense, we can also automatically derive the 'present' field from other fields later if we need it.
There was a problem hiding this comment.
Actually, for backwards compatibility with the existing template it might make sense to keep it for the moment. What do you think @CIGbalance
template.yaml
Outdated
| implementations: | ||
| - name: COCO | ||
| link: https://github.com/numbbo/coco | ||
| languges: 'C, Python' |
There was a problem hiding this comment.
Technically, COCO has way more interfaces.
Also typo in languages.
There was a problem hiding this comment.
Fixed typo.
Can you add those interfaces to the problems.yaml ?🙂
| - name: COCO | ||
| link: https://github.com/numbbo/coco | ||
| languges: 'C, Python' | ||
| evaluation time: 'less than a second' |
There was a problem hiding this comment.
This is going to be wild to analyse if we don't give some structure
There was a problem hiding this comment.
What would you suggest? @CIGbalance
My initial idea would be to ask for something like:
Enter approximate time with units and greater/less than symbol if relevant, e.g.: <1s, or 2h5m3s
Are there good standards we can refer to/use?
There was a problem hiding this comment.
We did something similar for the real-world benchmarks questionnaire. There we had less than a second/minute/hour/day, or more than a day.
Perhaps the 2h5m3s format is nicer, so we can automatically group things as desired in whatever analysis code is used. (Probably with some optional symbols like <, >, ~.)
Thoughts?
|
Work in progress ... ` Please enter the relevant information.Fields that are not relevant can be left empty.
|
template.yaml
Outdated
| number: '1' | ||
| types: single | ||
| variables: | ||
| types: continuous |
There was a problem hiding this comment.
Type of variables should be continuous or discrete or mixed (mixed-integer)
| variables: | ||
| types: continuous | ||
| conditional: 'no' | ||
| dimensionality: scalable |
There was a problem hiding this comment.
scalable or fixed number or interval / ranges (e.g. 5-11,19,85)
What do we expect the practitioner to add here? I'd prefer the exact number or range if available.
| present: 'no' | ||
| soft: '0' | ||
| hard: '0' | ||
| boundary/box: 'yes' |
There was a problem hiding this comment.
We don't consider discrete, categorical variables here, right?
That's fine with me but we should be aware of the missing thing.
In case of non-categorical variables, we expect the number of boxes to be equal to the number of variables. Could it be different?
| soft: '0' | ||
| hard: '0' | ||
| boundary/box: 'yes' | ||
| permutation: 'no' |
There was a problem hiding this comment.
A yes would expect the variables to be discrete, right?
In the case of a mixed-integer problem, can there be a permutation present in the (parts of) discrete part?
Not the regular case, I'd assume, but again we should be aware of the possibility if it exists.
| dynamic: | ||
| present: 'no' | ||
| types: '' | ||
| noise: 'no' |
There was a problem hiding this comment.
With respect to consistency, I suggest to follow the present, type logic here as well. This also keeps things simple.
We should ask practitioners to fill the other info field with corresponding information in case this is needed, e.g. for the dynamics, the noise, etc.
Draft yaml template for review