How can I make an IP/VPC whitelist for an API in API Gateway?

Question

We have an API in API Gateway connected to a lambda function. The API has three stages (Dev/Stage/Prod), an API key (required) and a usage plan (connected to all three stages).

We're trying to restrict traffic to this API so that Stage/Prod is only accessible from our servers from within our VPC, and Dev is only accessible from our office IP. We have tried using the Resource Policy below, but it doesn't work. Stage/Prod is still accessible from our office IP.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "execute-api:Invoke",
            "Resource": "arn:aws:execute-api:eu-west-1:{{accountId}}:{{apiId}}/*"
        },
        {
            "Effect": "Deny",
            "Principal": "*",
            "Action": "execute-api:Invoke",
            "Resource": [
                "arn:aws:execute-api:eu-west-1:{{accountId}}:{{apiId}}/Stage",
                "arn:aws:execute-api:eu-west-1:{{accountId}}:{{apiId}}/Prod"
            ],
            "Condition": {
                "StringNotEquals": {
                    "aws:SourceVpc": "{{vpcId}}"
                }
            }
        },
        {
            "Effect": "Deny",
            "Principal": "*",
            "Action": "execute-api:Invoke",
            "Resource": "arn:aws:execute-api:eu-west-1:{{accountId}}:{{apiId}}/Dev",
            "Condition": {
                "NotIpAddress": {
                    "aws:SourceIp": "{{ipAddress}}"
                }
            }
        }
    ]
}

We have replaced our real values with handlebars {{}}.

What are we doing wrong? Cheers!