Ask questionsSupport Istio
Currently, Kuberhealthy looks up the calling pod's IP to validate that it is an allowed checker pod. If the source IP connecting to the check reporting handler is not a valid pod IP, then the lookup will fail and the pods will not be allowed to check in.
This is explained by
nstott on Slack:
nstott 6:15 AM Hey All, qq about the latest kuberhealthy, I just installed this and am hitting some odd bugs. This is installed on a kube 1.14, using istio. so all the pods have envoy side cars. it seems that the check pods can't report in, and hit a deadline and they try to get reaped (which fails) The kuberhealthy deployment reports errors like time="2020-06-03T04:38:07Z" level=warning msg="was unable to find calling pod with remote IP 127.0.0.1 while watching for duration" time="2020-06-03T04:38:08Z" level=info msg="web: 07d39a5f-a917-4871-bdfe-910a69a207f2 Failed to look up pod by IP: 127.0.0.1:38656 Failed to fetch source pod with IP 127.0.0.1 after trying for 1m0s" the checker pod logs errors like ime="2020-06-03T13:10:17Z" level=info msg="Error reporting failure to Kuberhealthy servers: bad status code from kuberhealthy status reporting url: %!s(int=400)%!(EXTRA string=400 Bad Request)" time="2020-06-03T13:10:17Z" level=error msg="Error running Pod Restarts check: bad status code from kuberhealthy status reporting url: %!s(int=400)%!(EXTRA string=400 Bad Request)" I think this is due to the RemoteAddr here https://github.com/Comcast/kuberhealthy/blob/0f294378c580dd313631effe99cedfc1349233fd/cmd/kuberhealthy/kuberhealthy.go#L953 any chance we could key this by something like x-forwarded-for (if it exists)
It seems like a temporary work-around here would be to disable sidecar injection on the Kuberhealthy namespace.
Answer questions nstott
Out of curiosity, what's the reason for using remoteAddr there?
is it basically to guarantee that the right pod is responding? we might be able to do the same thing with a unique pre-shared key, between the kh server, and the check pod