Skip to content

pods/log: NotFound resource #60835

Closed
Closed
@xtuc

Description

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened:

I'm not able to request the logs of my pods from non master nodes.

Example:

$ kubectl logs redis-deployment-5697dff764-k6pns -v=9

I0306 09:18:36.559814   24182 loader.go:357] Config loaded from file /etc/kubernetes/kubelet.conf
I0306 09:18:36.581218   24182 round_trippers.go:417] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.9.3 (linux/amd64) kubernetes/d283541" https://192.168.99.2:6443/api/v1/namespaces/default/pods/redis-deployment-5697dff764-k6pns
I0306 09:18:36.589393   24182 round_trippers.go:436] GET https://192.168.99.2:6443/api/v1/namespaces/default/pods/redis-deployment-5697dff764-k6pns 200 OK in 7 milliseconds
I0306 09:18:36.589735   24182 round_trippers.go:442] Response Headers:
I0306 09:18:36.589956   24182 round_trippers.go:445]     Content-Type: application/json
I0306 09:18:36.590163   24182 round_trippers.go:445]     Content-Length: 2461
I0306 09:18:36.590359   24182 round_trippers.go:445]     Date: Tue, 06 Mar 2018 09:18:36 GMT
I0306 09:18:36.590758   24182 request.go:873] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"redis-deployment-5697dff764-k6pns","generateName":"redis-deployment-5697dff764-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/redis-deployment-5697dff764-k6pns","uid":"673e0953-2112-11e8-a5d9-0238074c8285","resourceVersion":"885","creationTimestamp":"2018-03-06T07:45:56Z","labels":{"app":"redis","pod-template-hash":"1253899320"},"ownerReferences":[{"apiVersion":"extensions/v1beta1","kind":"ReplicaSet","name":"redis-deployment-5697dff764","uid":"673cc513-2112-11e8-a5d9-0238074c8285","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"volume","hostPath":{"path":"/data/redis","type":"DirectoryOrCreate"}},{"name":"default-token-nxvgw","secret":{"secretName":"default-token-nxvgw","defaultMode":420}}],"containers":[{"name":"redis","image":"redis","ports":[{"containerPort":6379,"protocol":"TCP"}],"resources":{},"volumeMounts":[{"name":"volume","mountPath":"/data"},{"name":"default-token-nxvgw","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"node-192-168-99-3","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}]},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-03-06T07:47:00Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-03-06T07:47:06Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-03-06T07:46:59Z"}],"hostIP":"10.0.2.15","podIP":"10.67.1.163","startTime":"2018-03-06T07:47:00Z","containerStatuses":[{"name":"redis","state":{"running":{"startedAt":"2018-03-06T07:47:06Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"redis:latest","imageID":"docker-pullable://redis@sha256:0bf55b655c461a6f41a7c83c407e1e90be9dbb9aa41831e93cc1a96621530669","containerID":"docker://4e0d97e508b57252cdd18e388b06f20e54d6995ed2a852290eea7290d890293d"}],"qosClass":"BestEffort"}}
I0306 09:18:36.667654   24182 round_trippers.go:417] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.9.3 (linux/amd64) kubernetes/d283541" https://192.168.99.2:6443/api/v1/namespaces/default/pods/redis-deployment-5697dff764-k6pns/log
I0306 09:18:36.671694   24182 round_trippers.go:436] GET https://192.168.99.2:6443/api/v1/namespaces/default/pods/redis-deployment-5697dff764-k6pns/log 404 Not Found in 4 milliseconds
I0306 09:18:36.671723   24182 round_trippers.go:442] Response Headers:
I0306 09:18:36.671730   24182 round_trippers.go:445]     Content-Length: 280
I0306 09:18:36.671739   24182 round_trippers.go:445]     Date: Tue, 06 Mar 2018 09:18:36 GMT
I0306 09:18:36.671745   24182 round_trippers.go:445]     Content-Type: application/json
I0306 09:18:36.671797   24182 request.go:873] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the server could not find the requested resource ( pods/log redis-deployment-5697dff764-k6pns)","reason":"NotFound","details":{"name":"redis-deployment-5697dff764-k6pns","kind":"pods/log"},"code":404}
I0306 09:18:36.672037   24182 helpers.go:201] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server could not find the requested resource ( pods/log redis-deployment-5697dff764-k6pns)",
  "reason": "NotFound",
  "details": {
    "name": "redis-deployment-5697dff764-k6pns",
    "kind": "pods/log"
  },
  "code": 404
}]
F0306 09:18:36.672070   24182 helpers.go:119] Error from server (NotFound): the server could not find the requested resource ( pods/log redis-deployment-5697dff764-k6pns)

I don't see anything in the logs about that.

What you expected to happen:

The logs.

How to reproduce it (as minimally and precisely as possible):

I bootstraped the cluster using kubeadm. I'm not sure how you could reproduce it, here's the config I use (no special config files):

Kubelet:

/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.67.0.4 --cluster-domain=cluster.local --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cadvisor-port=0 --rotate-certificates=true --cert-dir=/var/lib/kubelet/pki

apiserver:

kube-apiserver --requestheader-group-headers=X-Remote-Group --advertise-address=192.168.99.2 --service-account-key-file=/etc/kubernetes/pki/sa.pub --secure-port=6443 --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --enable-bootstrap-token-auth=true --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-allowed-names=front-proxy-client --client-ca-file=/etc/kubernetes/pki/ca.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --insecure-port=0 --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota --tls-private-key-file=/etc/kubernetes/pki/apiserver.key --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --service-cluster-ip-range=10.66.0.0/16 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-username-headers=X-Remote-User --authorization-mode=Node,RBAC --etcd-servers=http://127.0.0.1:2379

Anything else we need to know?:

Related #45911, #29480

The issue here #38178 but I tried with/without --enable-debugging-handlers=true which yields the same result.

Environment:

  • Kubernetes version (use kubectl version): v1.9.3
  • Cloud provider or hardware configuration: Vagrant
  • OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS
  • Kernel (e.g. uname -a): Linux master-192-168-99-2 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools: kubeadm
  • Others: vim 7.4.1689

Thanks!

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.sig/api-machineryCategorizes an issue or PR as relevant to SIG API Machinery.sig/nodeCategorizes an issue or PR as relevant to SIG Node.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions