I start a new bash shell, and execute:
ulimit -m 102400
ulimit -a
\"
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
schedu
According to the man page for setrlimit:
RLIMIT_RSS
Specifies the limit (in pages) of the process's resident set (the number of virtual pages resident in RAM). This limit only has effect in Linux 2.4.x, x < 30, and there only affects calls to madvise(2) specifying MADV_WILLNEED
You probably want to set the virtual memory size instead, via ulimit -v
You can restrict the resident memory using cgroups. See Resident Set Size (RSS) limit has no effect