The fact that you can find non-discontinuous metrics that improve gradually is basically not interesting, and doesn't support the underlying hypothesis, because there is no evidence the non-discontinuous metrics are things that matter or we want to achieve.
It is true that the metrics used are sometimes harsh judges, but the question is which metrics we want.
For example, they complain about exact string match for math problems.
15 + 23 is 38, and saying that "3<something>" is better than "2<something>" is simply wrong. The reality is the large models don't get it wrong, and the small models do. Yes, you can come up with metrics that make it look more continuous, but as I said, it's not really interesting or useful to say that one wrong answer is "better" than another in this case. the only interesting sideshow would be if your chosen metric somehow demonstrates progress towards the end goal metric.
That is why exact string match is used.
It would be more interesting if it was within the limits of floating point math or something (IE one gave you 34.99999998 due to rounding error, and one gave you 35.0"), but that's not what we are talking about.
But yes, i'm sure you can come with some theoretical notion of partial credit for almost any metric, and use it to display that almost anything is more continuous.
Would you assign partial credit to a model whose answers to questions about which numbers are prime are "kind of right"?
I'm sure you can come up with a metric that does, and makes it look like the model gets better as it scales up.
But unless the underlying way in which it is getting better is also a useful measure of progress towards the end metric (being able to correctly answer which number are prime), it doesn't matter at all.
Solving 2-sat in P-time is easy. Solving 3-sat in p-time would be worth a turing award.
Can we define an arbitrary 2.1 SAT that is in the middle? Maybe (oh god now i've forgotten how hard weird approximations to 3-sat are, hopefully this isn't wrong), but i'd argue it's not super interesting, nor does it necessarily get you any farther in your goal.
It is true that the metrics used are sometimes harsh judges, but the question is which metrics we want.
For example, they complain about exact string match for math problems.
15 + 23 is 38, and saying that "3<something>" is better than "2<something>" is simply wrong. The reality is the large models don't get it wrong, and the small models do. Yes, you can come up with metrics that make it look more continuous, but as I said, it's not really interesting or useful to say that one wrong answer is "better" than another in this case. the only interesting sideshow would be if your chosen metric somehow demonstrates progress towards the end goal metric.
That is why exact string match is used.
It would be more interesting if it was within the limits of floating point math or something (IE one gave you 34.99999998 due to rounding error, and one gave you 35.0"), but that's not what we are talking about.
But yes, i'm sure you can come with some theoretical notion of partial credit for almost any metric, and use it to display that almost anything is more continuous.
Would you assign partial credit to a model whose answers to questions about which numbers are prime are "kind of right"? I'm sure you can come up with a metric that does, and makes it look like the model gets better as it scales up. But unless the underlying way in which it is getting better is also a useful measure of progress towards the end metric (being able to correctly answer which number are prime), it doesn't matter at all.
Solving 2-sat in P-time is easy. Solving 3-sat in p-time would be worth a turing award.
Can we define an arbitrary 2.1 SAT that is in the middle? Maybe (oh god now i've forgotten how hard weird approximations to 3-sat are, hopefully this isn't wrong), but i'd argue it's not super interesting, nor does it necessarily get you any farther in your goal.