profile
viewpoint

Ask questionscontainerd can't pull image from Github Docker Package Registry

Using the new github docker registry containerd kubernetes can't pull image but using docker engine based k8s works fine.

Steps to reproduce the issue:

  1. Create a secret with github docker registry token Follow instructions here: https://help.github.com/en/articles/configuring-docker-for-use-with-github-package-registry#authenticating-to-github-package-registry

Using kubectl

kubectl create secret docker-registry regcred --docker-server=https://docker.pkg.github.com --docker-username=<user | org>--docker-password=15650cad4e8a6602284255f7caf76134eb977b45 --docker-email=<email>
  1. Create pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: private-reg
spec:
  containers:
  - name: private-reg-container
    image: docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go
  imagePullSecrets:
  - name: regcred
  1. Create a pod
kubectl apply -f pod.yaml

Describe the results you received: Errors for the Pod pulling image

31s         Normal    Pulling               Pod             pulling image "docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go"
31s         Warning   Failed                Pod             Failed to pull image "docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go": rpc error: code = Unknown desc = failed to resolve image "docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go:latest": no available registry endpoint: docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go:latest not found
31s         Warning   Failed                Pod             Error: ErrImagePull
3s          Normal    BackOff               Pod             Back-off pulling image "docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go"
3s          Warning   Failed                Pod             Error: ImagePullBackOff

Describe the results you expected: Pod is in Running State Here is the output when running same scenario on minikube with docker engine

  Normal  Pulling    2m58s  kubelet, minikube  Pulling image "docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go"
  Normal  Pulled     2m48s  kubelet, minikube  Successfully pulled image "docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go"
  Normal  Created    2m48s  kubelet, minikube  Created container private-reg-container
  Normal  Started    2m47s  kubelet, minikube  Started container private-reg-container

Output of containerd --version: I'm running on IKS here is the version 1.2.6 when running kubectl get nodes -o wide

kubectl get nodes -o wide
NAME             STATUS   ROLES    AGE   VERSION       INTERNAL-IP      EXTERNAL-IP      OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
10.187.176.105   Ready    <none>   18d   v1.13.5+IKS   10.187.176.105   169.*.*.*   Ubuntu 18.04.2 LTS   4.15.0-47-generic   containerd://1.2.6
containerd/containerd

Answer questions tamalsaha

@dmcgowan , the 406 issue is fixed but now there is a new issue. I built from master and this is what I got.

root@ubuntu-s-1vcpu-1gb-sfo2-01:~# ./ctr version
Client:
  Version:  v1.3.0-rc.3-14-ga668365e
  Revision: a668365eca1ddcdc3046e04d8a6421f956ddefc7

Server:
  Version:  v1.3.0-rc.3-14-ga668365e
  Revision: a668365eca1ddcdc3046e04d8a6421f956ddefc7
  UUID: e6f60ed5-dabd-42b7-a809-cfaa70d68b38



root@ubuntu-s-1vcpu-1gb-sfo2-01:~# ./ctr --debug images pull --user stashed:****** docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64
DEBU[0000] fetching                                      image="docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64"
DEBU[0000] resolving                                     host=docker.pkg.github.com
DEBU[0000] do request                                    host=docker.pkg.github.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.3.0-rc.3-14-ga668365e request.method=HEAD url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64"
DEBU[0000] fetch response received                       host=docker.pkg.github.com response.header.cache-control=no-cache response.header.content-security-policy="default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com" response.header.content-type="application/json; charset=utf-8" response.header.date="Fri, 27 Sep 2019 00:20:22 GMT" response.header.expect-ct="max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"" response.header.referrer-policy="origin-when-cross-origin, strict-origin-when-cross-origin" response.header.server=GitHub.com response.header.set-cookie="has_recent_activity=1; path=/; expires=Fri, 27 Sep 2019 01:20:22 -0000" response.header.status="401 Unauthorized" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.vary=X-PJAX response.header.www-authenticate="Basic realm=\"GitHub Docker Registry\"" response.header.x-content-type-options=nosniff response.header.x-frame-options=deny response.header.x-github-backend=Kubernetes response.header.x-github-request-id="DE78:3E15:4290:372C3:5D8D55C5" response.header.x-request-id=6eb2c9bf-6b62-4a78-93d9-c11edd61c11d response.header.x-runtime=0.004865 response.header.x-runtime-rack=0.012802 response.header.x-xss-protection="1; mode=block" response.status="401 Unauthorized" url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64"
DEBU[0000] Unauthorized                                  header="Basic realm=\"GitHub Docker Registry\"" host=docker.pkg.github.com
DEBU[0000] do request                                    host=docker.pkg.github.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.3.0-rc.3-14-ga668365e request.method=HEAD url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64"
DEBU[0000] fetch response received                       host=docker.pkg.github.com response.header.cache-control="max-age=0, private, must-revalidate" response.header.content-security-policy="default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com" response.header.content-type="application/vnd.docker.distribution.manifest.v2+json; charset=utf-8" response.header.date="Fri, 27 Sep 2019 00:20:22 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="W/\"3e803b76cf7154c538fe10333409779b\"" response.header.expect-ct="max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"" response.header.referrer-policy="origin-when-cross-origin, strict-origin-when-cross-origin" response.header.server=GitHub.com response.header.set-cookie="has_recent_activity=1; path=/; expires=Fri, 27 Sep 2019 01:20:22 -0000" response.header.status="200 OK" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.vary=X-PJAX response.header.x-content-type-options=nosniff response.header.x-frame-options=deny response.header.x-github-backend=Kubernetes response.header.x-github-request-id="DE78:3E15:4291:372C4:5D8D55C6" response.header.x-github-user=1gtm response.header.x-request-id=98c91ee8-03f1-4bcd-aa73-3989f5d195cd response.header.x-runtime=0.334852 response.header.x-runtime-rack=0.374636 response.header.x-xss-protection="1; mode=block" response.status="200 OK" url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64"
DEBU[0000] no Docker-Content-Digest header, fetching manifest instead  host=docker.pkg.github.com
DEBU[0000] do request                                    host=docker.pkg.github.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.3.0-rc.3-14-ga668365e request.method=GET url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64"
DEBU[0001] fetch response received                       host=docker.pkg.github.com response.header.cache-control="max-age=0, private, must-revalidate" response.header.content-security-policy="default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com" response.header.content-type="application/vnd.docker.distribution.manifest.v2+json; charset=utf-8" response.header.date="Fri, 27 Sep 2019 00:20:22 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="W/\"3e803b76cf7154c538fe10333409779b\"" response.header.expect-ct="max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"" response.header.referrer-policy="origin-when-cross-origin, strict-origin-when-cross-origin" response.header.server=GitHub.com response.header.set-cookie="has_recent_activity=1; path=/; expires=Fri, 27 Sep 2019 01:20:22 -0000" response.header.status="200 OK" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.vary=X-PJAX response.header.x-content-type-options=nosniff response.header.x-frame-options=deny response.header.x-github-backend=Kubernetes response.header.x-github-request-id="DE78:3E15:4292:372C6:5D8D55C6" response.header.x-github-user=1gtm response.header.x-request-id=11499acf-a7c6-45c2-aa25-2200eb0798af response.header.x-runtime=0.090834 response.header.x-runtime-rack=0.114095 response.header.x-xss-protection="1; mode=block" response.status="200 OK" url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64"
DEBU[0001] resolved                                      desc.digest="sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7" host=docker.pkg.github.com
DEBU[0001] fetch                                         digest="sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1373
DEBU[0001] do request                                    digest="sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/v1.3.0-rc.3-14-ga668365e request.method=GET size=1373 url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7"
DEBU[0001] fetch response received                       digest="sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.cache-control=no-cache response.header.content-security-policy="default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com" response.header.content-type="application/json; charset=utf-8" response.header.date="Fri, 27 Sep 2019 00:20:22 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.expect-ct="max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"" response.header.referrer-policy="origin-when-cross-origin, strict-origin-when-cross-origin" response.header.server=GitHub.com response.header.set-cookie="has_recent_activity=1; path=/; expires=Fri, 27 Sep 2019 01:20:22 -0000" response.header.status="404 Not Found" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.vary=X-PJAX response.header.x-content-type-options=nosniff response.header.x-frame-options=deny response.header.x-github-backend=Kubernetes response.header.x-github-request-id="DE78:3E15:4293:372C8:5D8D55C6" response.header.x-github-user=1gtm response.header.x-request-id=6a772f9f-fe3a-4781-a711-dd9fb6117766 response.header.x-runtime=0.041982 response.header.x-runtime-rack=0.049381 response.header.x-xss-protection="1; mode=block" response.status="404 Not Found" size=1373 url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7"
DEBU[0001] do request                                    digest="sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/v1.3.0-rc.3-14-ga668365e request.method=GET size=1373 url="https://docker.pkg.github.com/v2/stashed/stash/stash/blobs/sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7"
DEBU[0001] fetch response received                       digest="sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.cache-control=no-cache response.header.content-security-policy="default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com" response.header.content-type="application/json; charset=utf-8" response.header.date="Fri, 27 Sep 2019 00:20:23 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.expect-ct="max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"" response.header.referrer-policy="origin-when-cross-origin, strict-origin-when-cross-origin" response.header.server=GitHub.com response.header.set-cookie="has_recent_activity=1; path=/; expires=Fri, 27 Sep 2019 01:20:23 -0000" response.header.status="404 Not Found" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.vary=X-PJAX response.header.x-content-type-options=nosniff response.header.x-frame-options=deny response.header.x-github-backend=Kubernetes response.header.x-github-request-id="DE7A:2065:56B2:39BCD:5D8D55C7" response.header.x-github-user=1gtm response.header.x-request-id=c8bcc730-f8eb-457c-bc1a-0c2a59522173 response.header.x-runtime=0.042270 response.header.x-runtime-rack=0.053240 response.header.x-xss-protection="1; mode=block" response.status="404 Not Found" size=1373 url="https://docker.pkg.github.com/v2/stashed/stash/stash/blobs/sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7"
ctr: failed to copy: httpReaderSeeker: failed open: could not fetch content descriptor sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7 (application/vnd.docker.distribution.manifest.v2+json) from remote: not found
$ docker pull docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64
v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64: Pulling from stashed/stash/stash
092586df9206: Already exists 
6b8bc148141c: Pull complete 
6ceec8dc461a: Pull complete 
53b0d3a7e6a2: Pull complete 
bf8fa31a6e81: Pull complete 
Digest: sha256:3e803b76cf7154c538fe10333409779bae20b4c19e406d5e929313c3268a51a7
Status: Downloaded newer image for docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64
docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-14-g20b3eab6-dbg_linux_amd64
useful!
source:https://uonfu.com/
answerer
Tamal Saha tamalsaha @AppsCode Las Vegas, NV https://www.linkedin.com/in/tamalsaha/ Founder @appscode. Please email me to reach out!
Github User Rank List