[Bug 1037830] New: Error pulling image, Authentication is required / Server error: 404

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1037830

Bug ID: 1037830
           Summary: Error pulling image, Authentication is required /
                    Server error: 404
           Product: Fedora EPEL
           Version: el6
         Component: docker-io
          Assignee: [e-mail]
          Reporter: [e-mail]
        QA Contact: [e-mail]
                CC: [e-mail], [e-mail],
                    [e-mail], [e-mail],
                    [e-mail]

Description of problem:

Version-Release number of selected component (if applicable):
docker-io-0.7.0-14.el6.x86_64
Red Hat Enterprise Linux Server release 6.5 (Santiago)

How reproducible:
Always

Steps to Reproduce:
1. Run `docker -d -b none -D`
2. In another terminal, run `docker pull mattdm/fedora`

Actual results:
# docker pull mattdm/fedora
Pulling repository mattdm/fedora
1bdd590de956: Error pulling image (f20) from mattdm/fedora, Authentication is
required.
64e0619c473e: Error pulling image (latest) from mattdm/fedora, Authentication
is required.
97fc5bf7f8d4: Error pulling image (f20rc3.small) from mattdm/fedora,
Authentication is required.
a567e6f3d26a: Error pulling image (f19) from mattdm/fedora, Authentication is
required.
2013/12/03 21:15:36 Server error: 404 trying to fetch remote history for
mattdm/fedora

Expected results:
(Taken from my F20 system which works perfectly)
# docker pull mattdm/fedora
Pulling repository mattdm/fedora

1bdd590de956: Download complete
a567e6f3d26a: Download complete
64e0619c473e: Download complete
97fc5bf7f8d4: Download complete

Additional info:
There doesn't seem to be anything useful in the daemon output by default.  I've
added debug statements in various places to try and understand what's going on.
 I'll elaborate in future comments when I can better organize what I'm seeing. 
Just wanted to get this out there incase it's something obvious :)

[Vincent Batts]

Lokesh,

I chatted with the reporter, and puzzled by his issue. I'm not able to
reproduce.

vb
On Dec 3, 2013 4:22 PM,  wrote:
> https://bugzilla.redhat.com/show_bug.cgi?id=1037830
>         Bug ID: 1037830
>        Summary: Error pulling image, Authentication is required /
>                 Server error: 404
>        Product: Fedora EPEL
>        Version: el6
>      Component: docker-io
>       Assignee: [e-mail]
>       Reporter: [e-mail]
>     QA Contact: [e-mail]
>             CC: [e-mail], [e-mail],
>                 [e-mail], [e-mail],
>                 [e-mail]
>
>
>
> Description of problem:
>
> Version-Release number of selected component (if applicable):
> docker-io-0.7.0-14.el6.x86_64
> Red Hat Enterprise Linux Server release 6.5 (Santiago)
>
> How reproducible:
> Always
>
> Steps to Reproduce:
> Run `docker -d -b none -D`
> In another terminal, run `docker pull mattdm/fedora`
>
> Actual results:
> # docker pull mattdm/fedora
> Pulling repository mattdm/fedora
> 1bdd590de956: Error pulling image (f20) from mattdm/fedora, Authentication
> is
> required.
> 64e0619c473e: Error pulling image (latest) from mattdm/fedora,
> Authentication
> is required.
> 97fc5bf7f8d4: Error pulling image (f20rc3.small) from mattdm/fedora,
> Authentication is required.
> a567e6f3d26a: Error pulling image (f19) from mattdm/fedora, Authentication
> is
> required.
> 2013/12/03 21:15:36 Server error: 404 trying to fetch remote history for
> mattdm/fedora
>
> Expected results:
> (Taken from my F20 system which works perfectly)
> # docker pull mattdm/fedora
> Pulling repository mattdm/fedora
>
> 1bdd590de956: Download complete
> a567e6f3d26a: Download complete
> 64e0619c473e: Download complete
> 97fc5bf7f8d4: Download complete
>
> Additional info:
> There doesn't seem to be anything useful in the daemon output by default.
>  I've
> added debug statements in various places to try and understand what's
> going on.
>  I'll elaborate in future comments when I can better organize what I'm
> seeing.
> Just wanted to get this out there incase it's something obvious :)
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
> _______________________________________________
> golang mailing list
> [e-mail]
> https://lists.fedoraproject.org/mailman/listinfo/golang

[Bug 1037830] Error pulling image, Authentication is required / Server error: 404

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1037830

--- Comment #1 from John Eckersberg  ---
Ok, some more detail at least on the "Authentication is required" bit.

GetRemoteHistory contains the following line:

req.Header.Set("Authorization", "Token "+strings.Join(token, ", "))

However, token is a slice with only a single element, and this single string
appears to be a previously joined list of tokens.  Most importantly is that it
appears to have been joined with "," instead of ", " and that makes a
difference.

Here's two curl calls to illustrate the difference.

The first doesn't contain the extra space after the comma, and this is the way
the code as written is sending the request.  You can see it returns a 401
status code (Unauthorized).

$ curl -i
https://cdn-registry-1.docker.io/v1/images/a567e6f3d26a5736d00a5e9be9a609b44...
--header 'Authorization: Token
signature=545980deb5d0238c5c1fdbe4bc20867932fa8aee,repository="mattdm/fedora",access=read'
HTTP/1.1 401 Unauthorized
Server: cloudflare-nginx
Date: Wed, 04 Dec 2013 00:28:12 GMT
Content-Type: application/json
Content-Length: 41
Connection: keep-alive
Set-Cookie: __cfduid=de296b6a53b52e2ff5883a0e6b303fc811386116891989;
expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/; domain=.docker.io; HttpOnly
expires: -1
www-authenticate: Token
pragma: no-cache
cache-control: no-cache
x-docker-registry-version: 0.6.2
x-docker-registry-config: prod
CF-RAY: d74600eeb9c03f4

{   
    "error": "Requires authorization"
}

The second call adds that extra space, and completes successfully.

$ curl -i
https://cdn-registry-1.docker.io/v1/images/a567e6f3d26a5736d00a5e9be9a609b44...
--header 'Authorization: Token
ignature=545980deb5d0238c5c1fdbe4bc20867932fa8aee, repository="mattdm/fedora",
access=read'
HTTP/1.1 200 OK
Server: cloudflare-nginx
Date: Wed, 04 Dec 2013 00:28:26 GMT
Content-Type: application/json
Content-Length: 74
Connection: keep-alive
Set-Cookie: __cfduid=d493453bb50165de54a9a9cc1021b890d1386116906549;
expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/; domain=.docker.io; HttpOnly
last-modified: Thu, 01 Jan 1970 00:00:00 GMT
pragma: no-cache
cache-control: public, max-age=31536000
expires: Thu, 04 Dec 2014 00:28:26 GMT
x-docker-registry-version: 0.6.2
x-docker-registry-config: prod
CF-RAY: d746069e53c03fa

[   
    "a567e6f3d26a5736d00a5e9be9a609b44ab13f0e43fc466f45beaa7ea9054766"
]

I'm assuming that part of the bug is that the token string slice shouldn't ever
be passed in like this.  I still need to trace it back and figure out where
it's getting joined ahead of time.  That's probably the right place to fix it. 
I see nine places in registry.go where it sets the Authorization header in this
manner.

[Bug 1037830] Error pulling image, Authentication is required / Server error: 404

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1037830

--- Comment #2 from John Eckersberg  ---
I see at least a few patches on master upstream that are related to this:

https://github.com/dotcloud/docker/commit/045989e3d824f577cd90a6386b66a5814e...

https://github.com/dotcloud/docker/commit/ec4863ae5582d8e7a9dccb57d7f5f3e21c...

https://github.com/dotcloud/docker/commit/3b5010e90bd51ba3630f48091ed70111d2...

https://github.com/dotcloud/docker/commit/3f921639894d3eca72c9ae105d3ad4f386...

[Bug 1037830] Error pulling image, Authentication is required / Server error: 404

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1037830

--- Comment #3 from John Eckersberg  ---
I've provisioned a clean RHEL 6.5 VM and cannot reproduce this issue there, so
whatever the issue is seems to be isolated to this one system.

Also of note is that I get the exact same error if I use the docker binary as
provided upstream here -
https://docs.docker.io/en/latest/installation/binaries/ - so whatever is going
on isn't a packaging issue.

[Bug 1037830] Error pulling image, Authentication is required / Server error: 404

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1037830

--- Comment #4 from Lokesh Mandvekar  ---
I see, this didn't occur on my rhel instance either, but I saw a 404 on my
rawhide machine for docker-io built using go1.2, need to check

[Bug 1037830] Error pulling image, Authentication is required / Server error: 404

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1037830

--- Comment #5 from John Eckersberg  ---
I think you'll get the 404 at the end if any of the downloads fail.  I think
this patch upstream fixes that part:

https://github.com/dotcloud/docker/commit/d47507791e14908e78cf38d415a9863c9e...

[Bug 1037830] Error pulling image, Authentication is required / Server error: 404

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1037830

--- Comment #6 from Lokesh Mandvekar  ---
(In reply to John Eckersberg from comment #5)
> I think you'll get the 404 at the end if any of the downloads fail.  I think
> this patch upstream fixes that part:
>
> https://github.com/dotcloud/docker/commit/
> d47507791e14908e78cf38d415a9863c9ef75c5e

This commit seems to have been included since 0.7.1. I built 0.7.2-2 few mins
back, and I guess it should go into testing sometime tomorrow. Could you test
it out please. Get it here:
http://kojipkgs.fedoraproject.org//packages/docker-io/0.7.2/2.fc20/x86_64/do...

[Bug 1037830] Error pulling image, Authentication is required / Server error: 404

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1037830

Daniel Walsh  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |CLOSED
         Resolution|---                         |CURRENTRELEASE
        Last Closed|                            |2014-05-28 14:19:03