I would try using VPC ACLs for that. First of all, ELBs inside VPC can use Security Groups but they only specify a traffic you allow in and out of an ELB. To actually block a traffic coming from a certain IP - an ACL would be the best.
For that to work - a pair of a public (internet-facing) and internal ELBs need to be used with internal ELB protected by subnet ACL DENY rules.
