问题
I have a vm that sits in front of the cluster. Currently it is running HAProxy (with use-proxy-protocol: "true"). My end goal is to allow the pods associated with the default backend to be able to read the actual source client source IP.
Here's a sample log of with use-proxy-protocol turned on:
10.244.0.0 - [10.244.0.0] - - [10/Jan/2018:23:06:42 +0000] "GET /platform/ping HTTP/1.1" 200 16 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 367 0.002 [upstream-default-backend] 10.244.3.101:80 16 0.002 200
10.244.0.0 - [10.244.0.0] - - [10/Jan/2018:23:06:59 +0000] "GET /platform/ping HTTP/1.1" 200 16 "-" "curl/7.54.0" 91 0.074 [upstream-default-backend] 10.244.3.101:80 16 0.074 200
10.244.0.0 - [10.244.0.0] - - [10/Jan/2018:23:09:51 +0000] "PROXY TCP4 127.0.0.1 127.0.0.1 43088 80" 400 173 "-" "-" 0 0.001 [] - - - -
10.244.0.0 - [10.244.0.0] - - [10/Jan/2018:23:09:59 +0000] "PROXY TCP4 127.0.0.1 127.0.0.1 43092 80" 400 173 "-" "-" 0 0.001 [] - - - -
10.244.0.0 - [10.244.0.0] - - [10/Jan/2018:23:10:09 +0000] "PROXY TCP4 127.0.0.1 127.0.0.1 43096 80" 400 173 "-" "-" 0 0.002 [] - - - -
I0110 23:11:42.050971 5 controller.go:211] backend reload required
I0110 23:11:42.054732 5 event.go:218] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"7539f546-f599-11e7-bee6-fa163e2f1153", APIVersion:"v1", ResourceVersion:"127044", FieldPath:""}): type: 'Normal' reason: 'UPDATE' ConfigMap ingress-nginx/nginx-configuration
I0110 23:11:42.138901 5 controller.go:220] ingress backend successfully reloaded...
127.0.0.1 - [127.0.0.1] - - [10/Jan/2018:23:11:56 +0000] "GET /platform/ping HTTP/1.1" 200 16 "-" "curl/7.47.0" 86 0.003 [upstream-default-backend] 10.244.3.101:80 16 0.003 200
142.xx.xxx.xx - [142.xx.xxx.xx] - - [10/Jan/2018:23:15:50 +0000] "GET / HTTP/1.1" 500 21 "-" "curl/7.47.0" 78 0.020 [upstream-default-backend] 10.244.3.101:80 21 0.020 500
142.xx.xxx.xx - [142.xx.xxx.xx] - - [10/Jan/2018:23:16:02 +0000] "GET /platform/bitcoin HTTP/1.1" 200 45 "-" "curl/7.47.0" 94 0.165 [upstream-default-backend] 10.244.3.101:80 45 0.165 200
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:16:16 +0000] "GET / HTTP/1.1" 500 21 "-" "curl/7.54.0" 78 0.002 [upstream-default-backend] 10.244.3.101:80 21 0.002 500
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:16:30 +0000] "GET /platform/bitcoin HTTP/1.1" 200 45 "-" "curl/7.54.0" 94 0.002 [upstream-default-backend] 10.244.3.101:80 45 0.002 200
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:16:43 +0000] "GET /platform/bitcoin HTTP/1.1" 200 45 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 370 0.049 [upstream-default-backend] 10.244.3.101:80 45 0.049 200
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:16:44 +0000] "GET /favicon.ico HTTP/1.1" 404 9 "http://142.xx.xxx.xx/platform/bitcoin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 324 0.013 [upstream-default-backend] 10.244.3.101:80 9 0.013 404
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:17:04 +0000] "GET /platform/bitcoin HTTP/1.1" 200 45 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 370 0.002 [upstream-default-backend] 10.244.3.101:80 45 0.002 200
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:17:07 +0000] "GET /platform/ping HTTP/1.1" 200 16 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 367 0.002 [upstream-default-backend] 10.244.3.101:80 16 0.002 200
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:17:56 +0000] "GET /platform/ping HTTP/1.1" 200 16 "-" "curl/7.54.0" 91 0.002 [upstream-default-backend] 10.244.3.101:80 16 0.002 200
Logs from 1/10/18 10:17 PM to 1/10/18 11:17 PM UTC
142.xx.xxx.xx is the IP of the HAProxy vm
216.249.49.20 is an external IP coming from the university. As you can see, the ingress pod can read external IP's passed from HAProxy with use-proxy-protocol: "true" Just fine.
But when I curl the address of HAProxy vm, I get:
demonfuse@Williams-MacBook-Pro ~/N/K/NGINX> curl 142.xx.xxx.xx/platform/ping
pong2 10.244.2.6
10.244.2.6 is the IP of the ingress pod. I am confident ingress-nginx at this point has the real source IP.
Is there a way to forward the headers and real source IP to pods behind ingress-nginx via configmaps? From what I can tell here it most of it should be turned on by default.
How to reproduce:
- Install ingress-nginx on brand new cluster following the guide here
- Redirect traffic from HAProxy / external load balancer to ingress-nginx
- Go script
as follows:
import (
"github.com/kataras/iris"
"github.com/kataras/iris/context"
//...
)
func main() {
app := iris.New()
app.Get("/platform/ping", func(ctx context.Context) {
fmt.Println("connected with " + ctx.RemoteAddr() + "!")
ctx.WriteString("pong2 " + ctx.RemoteAddr())
})
//...
app.Run(iris.Addr(":80"), iris.WithoutServerError(iris.ErrServerClosed))
}
Additional info:
Environment: Internet -> Dedicated HAProxy VM -> Bare metal OVH K8S Cluster (1 master, 2 worker)
configmap.yaml
apiVersion: v1
data:
proxy-set-headers: "ingress-nginx/custom-headers"
use-proxy-protocol: "true"
kind: ConfigMap
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app: ingress-nginx
custom_headers.yaml
apiVersion: v1
data:
X-Forwarded-For: "142.xx.xxx.xxx"
kind: ConfigMap
metadata:
name: custom-headers
namespace: ingress-nginx
haproxy config
global
maxconn 4096
log 127.0.0.1 local0 notice
maxconn 2000
user haproxy
group haproxy
defaults
log global
mode http
retries 3
option redispatch
maxconn 2000
timeout connect 5000
timeout client 50000
timeout server 50000
frontend TestServerTest
bind 142.xx.xxx.xxx:80
mode tcp
default_backend TestServernodes
backend TestServernodes
mode tcp
server TestServer01 142.xx.xxx.xxx:80 send-proxy
Where and how did I made a mistake?
I have attempted a combination of X-Forwaded-For with the internal ingress pod IP, the external IP associated with the ingress service, and the public IP of the HAProxy vm. So far curling the external IP of the HAProxy still returns pong2 10.244.2.6 (internal IP of the ingress pod)
回答1:
I figured it out! The problem lies in the Iris web framework and has little to nothing to do with ingress-nginx.
The solution is to read the remote headers manually in ctx.Application().ConfigurationReadOnly().GetRemoteAddrHeaders(). By default the Iris framework does not check for X-Forwarded-For and X-Real-Ip
Hopefully this will be useful for those running reverse proxies to and from Kubernetes.
来源：https://stackoverflow.com/questions/48215498/kubernetes-ingress-nginx-preserve-source-ip