I was receiving ICMP packets but did not see them go out. The problem was related to the traffic traversing multiple interfaces and reverse path filtering being on by default...
I've enabled martian source logging first:
$ echo 1 >/proc/sys/net/ipv4/conf/eth2/log_martians
Then there are several options for what to do with them... I'm enabling loosely handling them:
$ sysctl net.ipv4.conf.all.rp_filter=2
See these for details:
- http://lartc.org/howto/lartc.kernel.html
- https://access.redhat.com/site/solutions/53031