Limit physical memory per process

Question

I am writing an algorithm to perform some external memory computations, i.e. where your input data does not fit into main memory and you have to consider the I/O complexity.

Answer 1

Kernel mem= boot parameter limits how much memory in total OS will use.

This is almost never what user wants.

For physical memory, there is RSS rlimit aka RLIMIT_AS.