I start a new bash shell, and execute:
ulimit -m 102400 ulimit -a \" core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited schedu
You can restrict the resident memory using cgroups. See Resident Set Size (RSS) limit has no effect