MemSpeed: Re-use allocated memory

[Erik Schnetter]

Allocating memory is slow. In thorn MemSpeed, we should re-use memory that has been allocated instead of allocating new memory for each benchmark.

Keyword:

[Roland Haas]

Maybe related to this. The MemSpeed test takes lots of memory, namely 64GB on bethe:

23714 rhaas     20   0   34.4g  32.0g  17016 R  47.4 12.7   3:09.85 cactus_sim
26084 rhaas     20   0   33.4g  32.0g  17308 R  42.1 12.7   1:39.39 cactus_sim

It really should use less memory (<200MB would be preferable) to avoid eg trashing when running on someones laptop or workstation.

Is the amount of memory used fixed or proportional to the memory available on the host (bethe has lots: 263867020)?

[Erik Schnetter]

MemSpeed needs to allocate sufficiently much memory to ensure that the measurement is not handled by the L3 cache. By default, it allocates 1/4 of 1 NUMA node worth of memory. There is an option to allocate even more memory so that the inter-NUMA memory speed can also be measured.

If you run one or two instances of MemSpeed simultaneously, there should be no swapping. I do this regularly on my laptop.

It should be possible to limit this further to e.g. at most 10x the last-level cache size.

[Roland Haas]

That may be a good idea. The MemSpeed test on 2 MPI ranks on my workstation (96GB of memory, each MemSpeed ranks uses 12GB) is also quite slow, taking about 10 minutes to finish.