Closed
Description
Is this a BUG REPORT or FEATURE REQUEST?:
Uncomment only one, leave it on its own line:
/kind bug
/kind feature
What happened:
I'm not able to request the logs of my pods from non master nodes.
Example:
$ kubectl logs redis-deployment-5697dff764-k6pns -v=9
I0306 09:18:36.559814 24182 loader.go:357] Config loaded from file /etc/kubernetes/kubelet.conf
I0306 09:18:36.581218 24182 round_trippers.go:417] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.9.3 (linux/amd64) kubernetes/d283541" https://192.168.99.2:6443/api/v1/namespaces/default/pods/redis-deployment-5697dff764-k6pns
I0306 09:18:36.589393 24182 round_trippers.go:436] GET https://192.168.99.2:6443/api/v1/namespaces/default/pods/redis-deployment-5697dff764-k6pns 200 OK in 7 milliseconds
I0306 09:18:36.589735 24182 round_trippers.go:442] Response Headers:
I0306 09:18:36.589956 24182 round_trippers.go:445] Content-Type: application/json
I0306 09:18:36.590163 24182 round_trippers.go:445] Content-Length: 2461
I0306 09:18:36.590359 24182 round_trippers.go:445] Date: Tue, 06 Mar 2018 09:18:36 GMT
I0306 09:18:36.590758 24182 request.go:873] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"redis-deployment-5697dff764-k6pns","generateName":"redis-deployment-5697dff764-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/redis-deployment-5697dff764-k6pns","uid":"673e0953-2112-11e8-a5d9-0238074c8285","resourceVersion":"885","creationTimestamp":"2018-03-06T07:45:56Z","labels":{"app":"redis","pod-template-hash":"1253899320"},"ownerReferences":[{"apiVersion":"extensions/v1beta1","kind":"ReplicaSet","name":"redis-deployment-5697dff764","uid":"673cc513-2112-11e8-a5d9-0238074c8285","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"volume","hostPath":{"path":"/data/redis","type":"DirectoryOrCreate"}},{"name":"default-token-nxvgw","secret":{"secretName":"default-token-nxvgw","defaultMode":420}}],"containers":[{"name":"redis","image":"redis","ports":[{"containerPort":6379,"protocol":"TCP"}],"resources":{},"volumeMounts":[{"name":"volume","mountPath":"/data"},{"name":"default-token-nxvgw","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"node-192-168-99-3","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}]},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-03-06T07:47:00Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-03-06T07:47:06Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-03-06T07:46:59Z"}],"hostIP":"10.0.2.15","podIP":"10.67.1.163","startTime":"2018-03-06T07:47:00Z","containerStatuses":[{"name":"redis","state":{"running":{"startedAt":"2018-03-06T07:47:06Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"redis:latest","imageID":"docker-pullable://redis@sha256:0bf55b655c461a6f41a7c83c407e1e90be9dbb9aa41831e93cc1a96621530669","containerID":"docker://4e0d97e508b57252cdd18e388b06f20e54d6995ed2a852290eea7290d890293d"}],"qosClass":"BestEffort"}}
I0306 09:18:36.667654 24182 round_trippers.go:417] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.9.3 (linux/amd64) kubernetes/d283541" https://192.168.99.2:6443/api/v1/namespaces/default/pods/redis-deployment-5697dff764-k6pns/log
I0306 09:18:36.671694 24182 round_trippers.go:436] GET https://192.168.99.2:6443/api/v1/namespaces/default/pods/redis-deployment-5697dff764-k6pns/log 404 Not Found in 4 milliseconds
I0306 09:18:36.671723 24182 round_trippers.go:442] Response Headers:
I0306 09:18:36.671730 24182 round_trippers.go:445] Content-Length: 280
I0306 09:18:36.671739 24182 round_trippers.go:445] Date: Tue, 06 Mar 2018 09:18:36 GMT
I0306 09:18:36.671745 24182 round_trippers.go:445] Content-Type: application/json
I0306 09:18:36.671797 24182 request.go:873] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the server could not find the requested resource ( pods/log redis-deployment-5697dff764-k6pns)","reason":"NotFound","details":{"name":"redis-deployment-5697dff764-k6pns","kind":"pods/log"},"code":404}
I0306 09:18:36.672037 24182 helpers.go:201] server response object: [{
"metadata": {},
"status": "Failure",
"message": "the server could not find the requested resource ( pods/log redis-deployment-5697dff764-k6pns)",
"reason": "NotFound",
"details": {
"name": "redis-deployment-5697dff764-k6pns",
"kind": "pods/log"
},
"code": 404
}]
F0306 09:18:36.672070 24182 helpers.go:119] Error from server (NotFound): the server could not find the requested resource ( pods/log redis-deployment-5697dff764-k6pns)
I don't see anything in the logs about that.
What you expected to happen:
The logs.
How to reproduce it (as minimally and precisely as possible):
I bootstraped the cluster using kubeadm. I'm not sure how you could reproduce it, here's the config I use (no special config files):
Kubelet:
/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.67.0.4 --cluster-domain=cluster.local --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cadvisor-port=0 --rotate-certificates=true --cert-dir=/var/lib/kubelet/pki
apiserver:
kube-apiserver --requestheader-group-headers=X-Remote-Group --advertise-address=192.168.99.2 --service-account-key-file=/etc/kubernetes/pki/sa.pub --secure-port=6443 --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --enable-bootstrap-token-auth=true --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-allowed-names=front-proxy-client --client-ca-file=/etc/kubernetes/pki/ca.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --insecure-port=0 --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota --tls-private-key-file=/etc/kubernetes/pki/apiserver.key --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --service-cluster-ip-range=10.66.0.0/16 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-username-headers=X-Remote-User --authorization-mode=Node,RBAC --etcd-servers=http://127.0.0.1:2379
Anything else we need to know?:
The issue here #38178 but I tried with/without --enable-debugging-handlers=true
which yields the same result.
Environment:
- Kubernetes version (use
kubectl version
):v1.9.3
- Cloud provider or hardware configuration: Vagrant
- OS (e.g. from /etc/os-release):
Ubuntu 16.04.3 LTS
- Kernel (e.g.
uname -a
):Linux master-192-168-99-2 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
- Install tools:
kubeadm
- Others:
vim 7.4.1689
Thanks!