0

I'm currently getting started in integrating JMH into a Scala Project and would like to implement a way to do adaptive warmups. What do I mean by apative warmups? Instead of defining a fixed number of warmup iterations, I would like to automatically determine the number of warmups it takes to get a stable runtime and then start measuring.

Has anyone already implemented this feature or is there a good reason why nobody did?

My naive attempt would be running a benchmark without warmup iterations, checking when runtimes seem to stabilize and if they don't try again with more iterations until the output seems stable. This number of iterations would then be used as the warmup iteration value when doing the actual measurements.

Improve this question
1
  • 2
    The problem is at "when warmup seem to stabilize" - you need a human who runs things several times to make sense of the results, because there might be a case that something seems to stabilized but a few seconds more and the numbers change again. Additionally, for reproducibility and any meaningful comparison (e.g. for regressions) you need to have reproducible environment - same hardware, no other tasks, same settings. If the number of iterations changes between runs, it's apples to oranges. If that number changes someone should take a closer look why - automating it kinda defeats the purpose
    – Mateusz Kubuszok
    Commented Dec 20, 2023 at 19:24
Related questions
4427
How do I avoid checking for nulls in Java?
4749
How do I read / convert an InputStream into a String in Java?
4115
How do I generate random integers within a specific range in Java?
Load 7 more related questions Show fewer related questions

0

Reset to default

Browse other questions tagged or ask your own question.