Re: F28 System Wide Change: Golang 1.10a

Hi,

With the new Go 1.10 I get

+ echo /usr/share/gocode/src/github.com/hashicorp/go-discover/cmd/discover/main.go
+ install -m 0755 -vd
/builddir/build/BUILDROOT/golang-github-hashicorp-discover-0-0.5.0.20180108git7642001.fc28.llt.x86_64/usr/bin
install: creating directory
'/builddir/build/BUILDROOT/golang-github-hashicorp-discover-0-0.5.0.20180108git7642001.fc28.llt.x86_64/usr/bin'
+ install -m 0755 -vp _bin/discover
/builddir/build/BUILDROOT/golang-github-hashicorp-discover-0-0.5.0.20180108git7642001.fc28.llt.x86_64/usr/bin/
'_bin/discover' ->
'/builddir/build/BUILDROOT/golang-github-hashicorp-discover-0-0.5.0.20180108git7642001.fc28.llt.x86_64/usr/bin/discover'
+ /usr/lib/rpm/find-debuginfo.sh -j4 --strict-build-id -m -i --build-id-seed
0-0.5.0.20180108git7642001.fc28.llt --unique-debug-suffix
-0-0.5.0.20180108git7642001.fc28.llt.x86_64 --unique-debug-src-base
golang-github-hashicorp-discover-0-0.5.0.20180108git7642001.fc28.llt.x86_64 --run-dwz
--dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 -S debugsourcefiles.list
/builddir/build/BUILD/go-discover-7642001b443a3723e2aba277054f16d1df172d97
extracting debug info from
/builddir/build/BUILDROOT/golang-github-hashicorp-discover-0-0.5.0.20180108git7642001.fc28.llt.x86_64/usr/bin/discover
dwz: dwz.c:8790: write_die: Assertion `refd != NULL' failed.
/usr/lib/rpm/find-debuginfo.sh: line 518:  1870 Aborted                 (core dumped) dwz
$dwz_opts ${dwz_files[@]}
/usr/lib/rpm/sepdebugcrcfix: Updated 0 CRC32s, 1 CRC32s did match.


in golang-github-hashicorp-discover

Any idea if it's due to Go 1.10 or another fedora-devel change?

Regards,

-- 
Nicolas Mailhot

F28 System Wide Change: Golang 1.10a

[Jakub Cajka]

----- Original Message -----
>  From: "nicolas mailhot"
> <nicolas.mailhot(a)laposte.net&gt;
>  To: golang(a)lists.fedoraproject.org, "Jakub Cajka" <jcajka(a)redhat.com&gt;,
> "Development discussions related to Fedora"
>  <devel(a)lists.fedoraproject.org&gt;
>  Sent: Friday, January 12, 2018 4:32:22 PM
>  Subject: Re: F28 System Wide Change: Golang 1.10a
>  
>  Hi,
>  
>  With the new Go 1.10 I get
>  
>  + echo
>  /usr/share/gocode/src/github.com/hashicorp/go-discover/cmd/discover/main.go
>  + install -m 0755 -vd
>
> /builddir/build/BUILDROOT/golang-github-hashicorp-discover-0-0.5.0.20180108git7642001.fc28.llt.x86_64/usr/bin
>  install: creating directory
>
> '/builddir/build/BUILDROOT/golang-github-hashicorp-discover-0-0.5.0.20180108git7642001.fc28.llt.x86_64/usr/bin'
>  + install -m 0755 -vp _bin/discover
>
> /builddir/build/BUILDROOT/golang-github-hashicorp-discover-0-0.5.0.20180108git7642001.fc28.llt.x86_64/usr/bin/
>  '_bin/discover' ->
>
> '/builddir/build/BUILDROOT/golang-github-hashicorp-discover-0-0.5.0.20180108git7642001.fc28.llt.x86_64/usr/bin/discover'
>  + /usr/lib/rpm/find-debuginfo.sh -j4 --strict-build-id -m -i --build-id-seed
>  0-0.5.0.20180108git7642001.fc28.llt --unique-debug-suffix
>  -0-0.5.0.20180108git7642001.fc28.llt.x86_64 --unique-debug-src-base
>  golang-github-hashicorp-discover-0-0.5.0.20180108git7642001.fc28.llt.x86_64
>  --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 -S
>  debugsourcefiles.list
>  /builddir/build/BUILD/go-discover-7642001b443a3723e2aba277054f16d1df172d97
>  extracting debug info from
>
> /builddir/build/BUILDROOT/golang-github-hashicorp-discover-0-0.5.0.20180108git7642001.fc28.llt.x86_64/usr/bin/discover
>  dwz: dwz.c:8790: write_die: Assertion `refd != NULL' failed.
>  /usr/lib/rpm/find-debuginfo.sh: line 518:  1870 Aborted                 (core
>  dumped) dwz $dwz_opts ${dwz_files[@]}
>  /usr/lib/rpm/sepdebugcrcfix: Updated 0 CRC32s, 1 CRC32s did match.
>  
>  
>  in golang-github-hashicorp-discover
>  
>  Any idea if it's due to Go 1.10 or another fedora-devel change?
>  
>  Regards,
>  
>  --
>  Nicolas Mailhot
>   
I would suspect debug info extraction based on the log, turning off debug package should
workaround the issue. Other question if debug info extraction fails due to bug in it or if
it is broken by broken debug info in golang binaries. I haven't stumbled on this while
rebuilding distro with 1.10 beta1(does the build pass in f27?), this package is not on the
list of failed builds(still processing them).

JC

F28 System Wide Change: Golang 1.10a

Hi Jakub

I'm not sure if the package exists in devel, of if we're unbundling it from some
other package, or if we're executing unit tests previously ignored

The core dump does not stop the package build

Will investigate some more…

Regards,

-- 
Nicolas Mailhot

F28 System Wide Change: Golang 1.10a

Hi,

So, to be more scientific, I did a complete rebuild from scratch of my Go spec stash in
rawhide, both with Go 1.10 and forcing the previous Go 1.9.2 (about 300 packages, some of
which are not supposed to build yet because I still need to finish the unbundling and
bootstraping of docker in addition to consul, etcd, cobra and runc)

I have 58 more packages failing with Go 1.10 (though some are probably fallouts that
depend on a packages Go 1.10 does not like). Starting with Google’s own
golang.og/x/debug!

Maybe Go 1.10 is not ready for use yet?

Build logs of both runs by PM, they take around 100 MiB each uncompressed

Regards,

-- 
Nicolas Mailhot

F28 System Wide Change: Golang 1.10a

[Jakub Cajka]

----- Original Message -----
>  From: "nicolas mailhot"
> <nicolas.mailhot(a)laposte.net&gt;
>  To: "Jakub Cajka" <jcajka(a)redhat.com&gt;
>  Cc: golang(a)lists.fedoraproject.org, "Development discussions related to Fedora"
> <devel(a)lists.fedoraproject.org&gt;
>  Sent: Monday, January 15, 2018 2:22:26 PM
>  Subject: Re: F28 System Wide Change: Golang 1.10a
>  
>  Hi,
>  
>  So, to be more scientific, I did a complete rebuild from scratch of my Go
>  spec stash in rawhide, both with Go 1.10 and forcing the previous Go 1.9.2
>  (about 300 packages, some of which are not supposed to build yet because I
>  still need to finish the unbundling and bootstraping of docker in addition
>  to consul, etcd, cobra and runc)
>  
>  I have 58 more packages failing with Go 1.10 (though some are probably
>  fallouts that depend on a packages Go 1.10 does not like). Starting with
>  Google’s own golang.og/x/debug!
>  
>  Maybe Go 1.10 is not ready for use yet? 
Have you investigated the build failures to have such strong claim?

I'm finishing going through distro wide rebuild(all ~500 packages) and roughly 74
packages are failing to build. Nearly all build failures are caused by
errors(inefficiencies) in the packages code or stale/inactive packagers, with no direct 
relation to the go compiler itself. I see less than 1% of packages that look like
candidates for golang bugs.

I will post results when I finish(in ~1-2 days).

JC

> 
> Build logs of both runs by PM, they take around 100 MiB each uncompressed
> 
> Regards,
> 
> --
> Nicolas Mailhot
> _______________________________________________
> golang mailing list -- golang(a)lists.fedoraproject.org
> To unsubscribe send an email to golang-leave(a)lists.fedoraproject.org
>

F28 System Wide Change: Golang 1.10a

De: "Jakub Cajka" 

>  Have you investigated the build failures to have such strong claim?
I, obviously hadn't have the time to investigate each failure

However, given the builds ran in parallel on the same computer, pointing to the same
repositories (with the exception of Go 1.9 forcing in one case), and a large part of the
spec stash completely replaces the Fedora Go package baseline with up to date code (often
the upstream last commit or release) that actually runs unit tests I'm pretty sure you
can not blame the failures on outdated packages.

On the contrary I suspect some of the packages that pass in your tests are obsolete code
with disabled unit tests so you're not seeing all the problems.

Regards,
-- 
Nicolas Mailhot

F28 System Wide Change: Golang 1.10a

[Jakub Cajka]

----- Original Message -----
>  From: "nicolas mailhot"
> <nicolas.mailhot(a)laposte.net&gt;
>  To: golang(a)lists.fedoraproject.org
>  Sent: Monday, January 15, 2018 2:48:36 PM
>  Subject: Re: F28 System Wide Change: Golang 1.10a
>  
>  De: "Jakub Cajka"
>  
>  > Have you investigated the build failures to have such strong claim?
>  
>  I, obviously hadn't have the time to investigate each failure
>  
>  However, given the builds ran in parallel on the same computer, pointing to
>  the same repositories (with the exception of Go 1.9 forcing in one case),
>  and a large part of the spec stash completely replaces the Fedora Go package
>  baseline with up to date code (often the upstream last commit or release)
>  that actually runs unit tests I'm pretty sure you can not blame the failures
>  on outdated packages. 
I'm not generally blaming it on outdated packages(although there are some), I'm
mostly blaming it on code that is not following best coding
practices(https://tip.golang.org/doc/go1.10#test) or assume some not guaranteed/universal
assumptions.

>  
>  On the contrary I suspect some of the packages that pass in your tests are
>  obsolete code with disabled unit tests so you're not seeing all the
>  problems. 
Can you back it up it? Or is it just your assumption? Actually most of the failures approx
70% is occurring in tests(see up mentioned link).

JC

> 
> Regards,
> --
> Nicolas Mailhot
> _______________________________________________
> golang mailing list -- golang(a)lists.fedoraproject.org
> To unsubscribe send an email to golang-leave(a)lists.fedoraproject.org
>

F28 System Wide Change: Golang 1.10a

> De: "Jakub Cajka" 
> > From: "nicolas mailhot"  
>  I'm not generally blaming it on outdated packages(although there
> are some), I'm mostly blaming it > on code that is not following best coding
> practices(https://tip.golang.org/doc/go1.10#test) or
>  assume some not guaranteed/universal assumptions. 
Well, given that Google’s own golang.org/x/debug fails, I suspect the beast coding
practices Go 1.10 relies on are not that well prevalent.

> > On the contrary I suspect some of the packages that pass in your
> tests are
> > obsolete code with disabled unit tests so you're not seeing all the
> > problems. 
>  Can you back it up it? Or is it just your assumption? Actually most
> of the failures approx 70% is > occurring in tests(see up mentioned link). 
Thus, counting the tests which had not been written yet in the obsolete versions currently
packaged in rawhide, plus all the tests which had never been activated there, it is not
surprising current code with unit tests on sees more failures

And getting rawhide to the point one can actually build current Go apps on it requires
replacing all this legacy code with current versions

Regards

-- 
Nicolas Mailhot

F28 System Wide Change: Golang 1.10a

[Jakub Cajka]

----- Original Message -----
>  From: "nicolas mailhot"
> <nicolas.mailhot(a)laposte.net&gt;
>  To: golang(a)lists.fedoraproject.org
>  Sent: Monday, January 15, 2018 3:47:00 PM
>  Subject: Re: F28 System Wide Change: Golang 1.10a
>  
>  
>  >De: "Jakub Cajka"
>  >> From: "nicolas mailhot"
>  
>  > I'm not generally blaming it on outdated packages(although there are some),
>  > I'm mostly blaming it > on code that is not following best coding
>  > practices(https://tip.golang.org/doc/go1.10#test) or
>  > assume some not guaranteed/universal assumptions.
>  
>  Well, given that Google’s own golang.org/x/debug fails, I suspect the beast
>  coding practices Go 1.10 relies on are not that well prevalent. 
If you don't share how, why it fails we can be just speculating(maybe it is bug in
your packaging, build setup). I wouldn't assume google in name == it always works ;).
JC

> 
> >> On the contrary I suspect some of the packages that pass in your tests are
> >> obsolete code with disabled unit tests so you're not seeing all the
> >> problems.
> 
> > Can you back it up it? Or is it just your assumption? Actually most of the
> > failures approx 70% is > occurring in tests(see up mentioned link).
> 
> Thus, counting the tests which had not been written yet in the obsolete
> versions currently packaged in rawhide, plus all the tests which had never
> been activated there, it is not surprising current code with unit tests on
> sees more failures
> 
> And getting rawhide to the point one can actually build current Go apps on it
> requires replacing all this legacy code with current versions
> 
> Regards
> 
> --
> Nicolas Mailhot
> _______________________________________________
> golang mailing list -- golang(a)lists.fedoraproject.org
> To unsubscribe send an email to golang-leave(a)lists.fedoraproject.org
>

F28 System Wide Change: Golang 1.10a

----- Mail original -----
De: "Jakub Cajka" 

>  If you don't share how, why it fails we can be just
> speculating(maybe it is bug in your > packaging, build setup). I wouldn't assume
> google in name == it always works ;). 
I sent you the logs by PM ("only" 5 MiB compressed with xz -9). Did you not
receive them ?

Regards,

-- 
Nicolas Mailhot

F28 System Wide Change: Golang 1.10a

[Jakub Cajka]

----- Original Message -----
>  From: "nicolas mailhot"
> <nicolas.mailhot(a)laposte.net&gt;
>  To: golang(a)lists.fedoraproject.org
>  Sent: Monday, January 15, 2018 6:16:07 PM
>  Subject: Re: F28 System Wide Change: Golang 1.10a
>  
>  
>  
>  ----- Mail original -----
>  De: "Jakub Cajka"
>  
>  > If you don't share how, why it fails we can be just speculating(maybe it is
>  > bug in your > packaging, build setup). I wouldn't assume google in name ==
>  > it always works ;).
>  
>  I sent you the logs by PM ("only" 5 MiB compressed with xz -9). Did you not
>  receive them ? 
Yes I did, and I replied to you. 

Thinking about it now as both logs are ~100M text blobs, could you give me an list of
packages that have failed for you(ideally, but no necessarily with fragments of log where
the failure is visible) and srpm link so I can reproduce/investigate the issue?(Although
I'm not promising that I will get to it this or next week as I'm quiet busy at the
moment.)

JC

> 
> Regards,
> 
> --
> Nicolas Mailhot
> _______________________________________________
> golang mailing list -- golang(a)lists.fedoraproject.org
> To unsubscribe send an email to golang-leave(a)lists.fedoraproject.org
>

F28 System Wide Change: Golang 1.10a

----- Mail original -----
De: "Jakub Cajka" <jcajka(a)redhat.com&gt;
À: golang(a)lists.fedoraproject.org
Envoyé: Mardi 16 Janvier 2018 10:16:19
Objet: Re: F28 System Wide Change: Golang 1.10a




> >> De: "Jakub Cajka"
> > From: "nicolas mailhot"
>  De: "Jakub Cajka"
>  
>  > If you don't share how, why it fails we can be just speculating(maybe it is
>  > bug in your > packaging, build setup). I wouldn't assume google in name ==
>  > it always works ;).
>  
> > I sent you the logs by PM ("only" 5 MiB compressed with xz -9). Did you
> not
> > receive them ? 
>  Yes I did, and I replied to you.  
I see your message now, I had missed it, sorry :(

>  Thinking about it now as both logs are ~100M text blobs, could you
> give me an list of packages
>  that have failed for you(ideally, but no necessarily with fragments of log where the
> failure is
>  visible)  
1. You have a summary of the successes and failures at the end of the log files
2. each failed build run of a package ends with 'Done processing $spec (failure)'

So it's pretty easy to check what package failed and then locate the part of the log
dealing with the last build attempt of this package.

Then check if it's because of a missing dep or a go 1.10 specific problem (of course
the missing dep may be because of the dep failed due to go 1.10). If it's not obvious
the go 1.9 logs provide the logs of the same builds with go 1.9

>  and srpm link so I can reproduce/investigate the issue? 
I'll try to revive my old apache instance to give you access to the files. Ideally I
should redo a full copr run but that takes days to run even only on devel x86_64 (and I
haven't checked how to force go 1.9 in copr to provide a comparison point). Local full
build 'only' takes 8 to 10 hours. On go 1.10 it may be faster so many things fail
:(

I'd love to work on go 1.10 but right now my priority is to finish the bootstraping of
the baseline on go 1.9 so there's no package in the set waiting on a dep or a dep
update to build in full mode. The big remaining bits are docker and maybe kubernetes (some
deps want up to date kubernetes libs, not sure yet if it requires updating kubernetes
itself to latest stable release). Though it is getting easier so many bits have been
updated there are fewer weird failures due to an obsolete part when creating a new
package.

Regards,

-- 
Nicolas Mailhot

F28 System Wide Change: Golang 1.10a

[Jakub Cajka]

----- Original Message -----
>  From: "nicolas mailhot"
> <nicolas.mailhot(a)laposte.net&gt;
>  To: golang(a)lists.fedoraproject.org
>  Sent: Tuesday, January 16, 2018 11:16:44 AM
>  Subject: Re: F28 System Wide Change: Golang 1.10a
>  
>  
>  
>  ----- Mail original -----
>  De: "Jakub Cajka" <jcajka(a)redhat.com&gt;
>  À: golang(a)lists.fedoraproject.org
>  Envoyé: Mardi 16 Janvier 2018 10:16:19
>  Objet: Re: F28 System Wide Change: Golang 1.10a
>  
>  
>  
>  
>  >>> De: "Jakub Cajka"
>  >> From: "nicolas mailhot"
>  > De: "Jakub Cajka"
>  > 
>  > > If you don't share how, why it fails we can be just speculating(maybe it
>  > > is
>  > > bug in your > packaging, build setup). I wouldn't assume google in name
>  > > ==
>  > > it always works ;).
>  > 
>  >> I sent you the logs by PM ("only" 5 MiB compressed with xz -9). Did
> you
>  >> not
>  >> receive them ?
>  
>  > Yes I did, and I replied to you.
>  
>  I see your message now, I had missed it, sorry :(
>  
>  > Thinking about it now as both logs are ~100M text blobs, could you give me
>  > an list of packages
>  > that have failed for you(ideally, but no necessarily with fragments of log
>  > where the failure is
>  > visible)
>  
>  1. You have a summary of the successes and failures at the end of the log
>  files
>  2. each failed build run of a package ends with 'Done processing $spec
>  (failure)'
>   
Ah... missed that, I have been just quickly scanning the log's first few 100 lines.
What tool are you using for the build? mockchain?

>  So it's pretty easy to check what package failed and then locate
> the part of
>  the log dealing with the last build attempt of this package. 
Easy but extremely time consuming. I have nicely structured output from mockchain(so no
need to writer parser for log file(s)), still it is extremely time consuming, especially
when verifying the issues.

>  
>  Then check if it's because of a missing dep or a go 1.10 specific problem (of
>  course the missing dep may be because of the dep failed due to go 1.10). If
>  it's not obvious the go 1.9 logs provide the logs of the same builds with go
>  1.9
>  
>  > and srpm link so I can reproduce/investigate the issue?
>  
>  I'll try to revive my old apache instance to give you access to the files.
>  Ideally I should redo a full copr run but that takes days to run even only
>  on devel x86_64 (and I haven't checked how to force go 1.9 in copr to
>  provide a comparison point). Local full build 'only' takes 8 to 10 hours. On
>  go 1.10 it may be faster so many things fail :(
>  
>  I'd love to work on go 1.10 but right now my priority is to finish the
>  bootstraping of the baseline on go 1.9 so there's no package in the set
>  waiting on a dep or a dep update to build in full mode. The big remaining
>  bits are docker and maybe kubernetes (some deps want up to date kubernetes
>  libs, not sure yet if it requires updating kubernetes itself to latest
>  stable release). Though it is getting easier so many bits have been updated
>  there are fewer weird failures due to an obsolete part when creating a new
>  package.
>  
>  Regards,
>  
>  --
>  Nicolas Mailhot
>  _______________________________________________
>  golang mailing list -- golang(a)lists.fedoraproject.org
>  To unsubscribe send an email to golang-leave(a)lists.fedoraproject.org
>   
One more thing popped on my mind, I can't really investigate the issues if you have
out of the Fedora dependencies(maybe you could fork the packages in pagure??). If you will
give me failures that you want to be investigated(with log and srpm), I will be happy to
take a look(that do look like golang bugs, no broken deps, code issues, etc.).

JC

F28 System Wide Change: Golang 1.10a

De: "Jakub Cajka" 
> > 1. You have a summary of the successes and failures at the end of
> the log
> > files
> > 2. each failed build run of a package ends with 'Done processing $spec
> > (failure)' 
 
>  Ah... missed that, I have been just quickly scanning the log's
> first few 100 lines. What tool are you using for the build? mockchain? 
It's a quick and dirty script that is a bit smarter than mockchain (knows about
bootstraping and release bumping), but with pretty raw logging facilities. I can attach it
if you want.

I will look at mockchain but given its man page it is far from evident it is possible to
wrap around it in a bootstraping context (and Go packages absolutely need this given all
the cycles between them)

> > So it's pretty easy to check what package failed and then
> locate the part of
> > the log dealing with the last build attempt of this package. 
>  Easy but extremely time consuming. I have nicely structured output
> from mockchain(so no need to writer parser for log file(s)), still it is extremely
>  time consuming, especially when verifying the issues. 
I understand, at this point my Go toolchain makes creating and updating Go packages
trivial. The time consuming phase is build log analysis (especially to sort what unit
tests are relevant and what unit tests can not work in a mock env).

Regards,

-- 
Nicolas Mailhot