Minitest 5 update

[Josef Stribny]

Hi Rubyists,

as you have probably noticed we would like to update Rails framework to the latest version (4.1) when it's released[1].
The biggest task for us in this change is the update of Minitest to version 5. This would require test suites
of many gems that depends on Minitest to be fixed.

Here are the most significant changes of Minitest 5 taken from the changelog[2]:

* 12 major (oft incompatible) changes:

* Renamed MiniTest to Minitest. Your pinkies will thank me. (aliased to MiniTest)
  * Removed MiniTest::Unit entirely. No more manager objects.
  * Added Minitest::Runnable. Everything minitest can run subclasses this.
  * Renamed MiniTest::Unit::TestCase to Minitest::Test (subclassing Runnable).
  * Added Minitest::Benchmark.
    * Your benchmarks need to move to their own subclass.
    * Benchmarks using the spec DSL have to have "Bench" somewhere in their describe.
  * MiniTest::Unit.after_tests moved to Minitest.after_tests
  * MiniTest::Unit.autorun is now Minitest.autorun. Just require minitest/autorun pls.
  * Removed ParallelEach#grep since it isn't used anywhere.
  * Renamed Runnable#__name__ to Runnable#name (but uses @NAME internally).
  * Runnable#run needs to return self. Allows for swapping of results as needed.

That means the minimal changes needed to be done for every minitest test suite include:

* rename MiniTest::Unit::TestCase to Minitest::Test
* change require 'minitest/unit' (or 'test/unit') to require 'minitest/autorun'
* delete MiniTest::Unit.autorun (or use Minitest.autorun)

For some really simple test suites that might be enough. Look at the changes done in Rails 4.1[3]
or in my own little gem[4] for how the necessary changes may look like. For the others some additional
changes might be required.

Another problem are gems that depends on Gem::TestCase (like RubyGems plugins)
since Gem::TestCase is inherited from MiniTest::Unit::TestCase. I guess this can't be easily
solved at the moment, but we probably don't have many gems like that, or are we?

So what do you think of this change? Should we address it by updating upstream test suites?
Do you know about other issues that this can cause and I forgot to mention?

Josef

[1] https://fedoraproject.org/wiki/Changes/Ruby_on_Rails_4.1
[2] https://github.com/seattlerb/minitest/blob/master/History.txt#L137
[3] https://github.com/rails/rails/commit/3073c531983de243219fb55be93fbcebfdd9c4...
[4] https://github.com/strzibny/ruby-ares/commit/a1017ce59eeb8bfb5256b6e7789d957...

[Mamoru TASAKA]

Hello, Josef:

Josef Stribny wrote, at 03/21/2014 09:11 PM +9:00:
> Hi Rubyists,
>
> as you have probably noticed we would like to update Rails framework to the latest version (4.1) when it's released[1].
> The biggest task for us in this change is the update of Minitest to version 5. This would require test suites
> of many gems that depends on Minitest to be fixed.
>
> Here are the most significant changes of Minitest 5 taken from the changelog[2]:

<snip>
> So what do you think of this change? Should we address it by updating upstream test suites?
> Do you know about other issues that this can cause and I forgot to mention?

The easiest solution is to just renaming the current minitest rpm as rubygem-minitest4
(i.e. submitting a new review request), or instead introducing rubygem-minitest5 and at some
time renaming rubygem-minitest5 to rubygem-minitest, as we have once already done for
rspec version 1 vs 2, or I am currently doing for rubygem-goocanvas (version 2, depending
on gtk3) vs rubygem-goocanvas1 (depending on gtk2). As basically rubygems supports
installation of multiple versions of gems with the same name, this should be easily
done (basically just changing srpm name should be enough).

Note that some packages (I maintain) already requires minitest 5 for testsuite (and
currently I have to revert changes), e.g.

http://pkgs.fedoraproject.org/cgit/rubygem-net-http-persistent.git/commit/?i...
http://pkgs.fedoraproject.org/cgit/rubygem-mechanize.git/commit/?id=59b668ec...
http://pkgs.fedoraproject.org/cgit/rubygem-hoe.git/commit/?id=eade41e72919f2...

Regards,
Mamoru

[Josef Stribny]

Hi,
> The easiest solution is to just renaming the current minitest rpm as rubygem-minitest4
> (i.e. submitting a new review request), or instead introducing rubygem-minitest5 and at some
> time renaming rubygem-minitest5 to rubygem-minitest, as we have once already done for
> rspec version 1 vs 2, or I am currently doing for rubygem-goocanvas (version 2, depending
> on gtk3) vs rubygem-goocanvas1 (depending on gtk2). As basically rubygems supports
> installation of multiple versions of gems with the same name, this should be easily
> done (basically just changing srpm name should be enough).

This seems like a safe path that we also shortly discussed with Vit. We could probably need
it for some packages, although I would probably like try to propose minitest 5 change to
upstream which could speed up the adoption.
> Note that some packages (I maintain) already requires minitest 5 for testsuite (and
> currently I have to revert changes), e.g.

True, I think I was also already reverting something.

Josef

[Ken Dreyer]

On Fri, Mar 21, 2014 at 10:18 AM, Mamoru TASAKA
 wrote:
> Note that some packages (I maintain) already requires minitest 5 for
> testsuite (and
> currently I have to revert changes), e.g.
>
> http://pkgs.fedoraproject.org/cgit/rubygem-net-http-persistent.git/commit/?i...
> http://pkgs.fedoraproject.org/cgit/rubygem-mechanize.git/commit/?id=59b668ec...
> http://pkgs.fedoraproject.org/cgit/rubygem-hoe.git/commit/?id=eade41e72919f2...

rubygem-escape_utils (not yet in Fedora) was also converted upstream
to Minitest 5. Here's what I used in %prep to switch back to the old
Minitest 4 syntax:

# Get the major version number of the Minitest gem
minitest=$(ruby -r 'minitest/unit' \
  -e "puts Minitest::Unit::VERSION.split('.')[0]")
if [ $minitest > 5 ]; then
  # Conditionally correct Minitest usage, for Minitest versions < 5.0.0.
  # For example, at least Fedora 20 has Minitest 4.x.
  for f in $(find test -type f); do
    sed -i "s/Minitest::Test/Minitest::Unit::TestCase/g" $f
  done
fi

This should be forward-compatible with Minitest 5 when it lands in Fedora 21.

- Ken

[Vít Ondruch]

Dne 21.3.2014 17:18, Mamoru TASAKA napsal(a):
> Hello, Josef:
>
> Josef Stribny wrote, at 03/21/2014 09:11 PM +9:00:
> > Hi Rubyists,
> >
> > as you have probably noticed we would like to update Rails framework
> > to the latest version (4.1) when it's released[1].
> > The biggest task for us in this change is the update of Minitest to
> > version 5. This would require test suites
> > of many gems that depends on Minitest to be fixed.
> >
> > Here are the most significant changes of Minitest 5 taken from the
> > changelog[2]:
>
> <snip>
> > So what do you think of this change? Should we address it by updating
> > upstream test suites?
> > Do you know about other issues that this can cause and I forgot to
> > mention?
> >
>
> The easiest solution is to just renaming the current minitest rpm as
> rubygem-minitest4

This is option of course. The question for me is what is bigger hassle?
Try to migrate as many libraries as we can to minitest5 in some
reasonable time or proactively introduce minitest4 version?

We can try to upgrade minitest to version 5 and if that proves, that it
causes more issues then it solves, then we can do review for
rubygem-minitest4.

Also note that no matter what we will do, there will always be available
the rubygem-minitest subpackage provided by Ruby's SRPM. So may be
explicitly specifying the version < 5 could be enough?

Vít

[Vít Ondruch]

Dne 21.3.2014 17:59, Vít Ondruch napsal(a):
> Dne 21.3.2014 17:18, Mamoru TASAKA napsal(a):
> > Hello, Josef:
> >
> > Josef Stribny wrote, at 03/21/2014 09:11 PM +9:00:
> > > Hi Rubyists,
> > >
> > > as you have probably noticed we would like to update Rails framework
> > > to the latest version (4.1) when it's released[1].
> > > The biggest task for us in this change is the update of Minitest to
> > > version 5. This would require test suites
> > > of many gems that depends on Minitest to be fixed.
> > >
> > > Here are the most significant changes of Minitest 5 taken from the
> > > changelog[2]:
> >
> > <snip>
> > > So what do you think of this change? Should we address it by updating
> > > upstream test suites?
> > > Do you know about other issues that this can cause and I forgot to
> > > mention?
> > >
> > The easiest solution is to just renaming the current minitest rpm as
> > rubygem-minitest4
>
> This is option of course. The question for me is what is bigger hassle?
> Try to migrate as many libraries as we can to minitest5 in some
> reasonable time or proactively introduce minitest4 version?
>
> We can try to upgrade minitest to version 5 and if that proves, that it
> causes more issues then it solves, then we can do review for
> rubygem-minitest4.
>
> Also note that no matter what we will do, there will always be available
> the rubygem-minitest subpackage provided by Ruby's SRPM. So may be
> explicitly specifying the version < 5 could be enough?
>
> Vít

So I put togehter testing build of minitest 5.x in Copr [1] and testing
this build in mock using mock config (see attachment). With this
configuration, I had available rubygem-minitest-4.7.0-2.fc20 (the latest
minitest available in Fedora, not the one bundled in Ruby) as well as
rubygem-minitest-5.3.1-1.fc21. I was testing with
rubygem-net-http-persistent and I was able to build the gem against
minitest 4 as well as minitest 5. For build against minitest 4, I had to
use:

BuildRequires:    rubygem(minitest) < 5

For build against minitest 5, I used unversioned require, revert the
minitest 4 changes in %prep section and replace testrb by:

ruby -Ilib -e 'Dir.glob "./test/test_*.rb", &method(:require)'

So to conclude:

* I'd go with update to minitest 5 (probably tomorrow, unless you'll be
fast enough to point out some weak points ;)
* This means a lot of FTBFS packages, but to fix this, we can use BR:
rubygem(minitest) < 5 temporarily or better to fix the compatibility and
submit patch upstream. But please note that Ruby ships with minitest
4.3, which is a bit older version then the 4.7 currently available. I
hope that will not cause any additional issues.
* If something goes really wrong, we can review rubygem-minitest4, but I
hope this wont be needed.
* We should consider to update packaging guidelines to suggest something
like "ruby -Ilib -e 'Dir.glob "./test/test_*.rb", &method(:require)'"
instead of testrb. testrb does not handle nested directories anyway,
which Dir.glob handles just fine.

Thoughts?

Vít

[1] http://copr.fedoraproject.org/coprs/vondruch/rubygem-minitest/

[Vít Ondruch]

Dne 1.4.2014 12:46, Vít Ondruch napsal(a):
> I'd go with update to minitest 5 (probably tomorrow, unless you'll be
>
> fast enough to point out some weak points ;)

rubygem-minitest-5.3.1-1.fc21 is now available in Rawhide [1].

Vít

[1] http://koji.fedoraproject.org/koji/taskinfo?taskID=6701679

[Ken Dreyer]

On Thu, Apr 3, 2014 at 1:31 AM, Vít Ondruch  wrote:
> Dne 1.4.2014 12:46, Vít Ondruch napsal(a):
> > I'd go with update to minitest 5 (probably tomorrow, unless you'll be
> >
> > fast enough to point out some weak points ;)
>
> rubygem-minitest-5.3.1-1.fc21 is now available in Rawhide [1].

My first gem package broke yesterday on this :) rubygem-geoip-1.4.0-1

I've submitted patches upstream for Minitest 5 support, and
rubygem-geoip-1.4.0-2 builds successfully.

I did have to migrate away from testrb in %check in order to make my
backwards-compatibility-with-mintest-4 hack succeed. I don't know what
testrb was doing differently, but it's probably a good idea to remove
testrb in any regard.

By the way, do you have a list of all the gems that failed to build
with Minitest 5? I'd like to take a look and see if I can help fix
some of them (particularly the ones that are mine... :)

- Ken

On Thu, Apr 3, 2014 at 2:31 AM, Vít Ondruch  wrote:
> Dne 1.4.2014 12:46, Vít Ondruch napsal(a):
> > I'd go with update to minitest 5 (probably tomorrow, unless you'll be
> >
> > fast enough to point out some weak points ;)
>
> rubygem-minitest-5.3.1-1.fc21 is now available in Rawhide [1].
>
> Vít
>
> [1] http://koji.fedoraproject.org/koji/taskinfo?taskID=6701679
> _______________________________________________
> ruby-sig mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/ruby-sig

[Vít Ondruch]

Dne 4.4.2014 18:19, Ken Dreyer napsal(a):
> On Thu, Apr 3, 2014 at 1:31 AM, Vít Ondruch  wrote:
> > Dne 1.4.2014 12:46, Vít Ondruch napsal(a):
> > > I'd go with update to minitest 5 (probably tomorrow, unless you'll be
> > >
> > > fast enough to point out some weak points ;)
> >
> > rubygem-minitest-5.3.1-1.fc21 is now available in Rawhide [1].
>
> My first gem package broke yesterday on this :) rubygem-geoip-1.4.0-1
>
> I've submitted patches upstream for Minitest 5 support, and
> rubygem-geoip-1.4.0-2 builds successfully.
>
> I did have to migrate away from testrb in %check in order to make my
> backwards-compatibility-with-mintest-4 hack succeed. I don't know what
> testrb was doing differently, but it's probably a good idea to remove
> testrb in any regard.

Using testrb, you can specify the directory from which you should
execute the test suite. In theory, it used to be less hassle then
fiddling with "ruby -e" or rake. But we used "ruby -e" on many places
anyway, since tesrb cant find test files recursively in folder structure.

And yes, testrb is looking for minitest 4 classes, so it does not work
with minitest 5. It would need to be fixed, but testrb is provided by
ruby package .....
> By the way, do you have a list of all the gems that failed to build
> with Minitest 5? I'd like to take a look and see if I can help fix
> some of them (particularly the ones that are mine... :)

I expect every package using minitest is FTBFS now (except the one
you've fixed ;), so this should be your list:

repoquery --repoid=rawhide-source --arch=src --whatrequires
'rubygem(minitest)' --qf="%{NAME}"

Vít
> Ken
>
> On Thu, Apr 3, 2014 at 2:31 AM, Vít Ondruch  wrote:
> > Dne 1.4.2014 12:46, Vít Ondruch napsal(a):
> > > I'd go with update to minitest 5 (probably tomorrow, unless you'll be
> > >
> > > fast enough to point out some weak points ;)
> >
> > rubygem-minitest-5.3.1-1.fc21 is now available in Rawhide [1].
> >
> > Vít
> >
> > [1] http://koji.fedoraproject.org/koji/taskinfo?taskID=6701679
> > _______________________________________________
> > ruby-sig mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/ruby-sig
>
> ruby-sig mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/ruby-sig

[Vít Ondruch]

Dne 3.4.2014 09:31, Vít Ondruch napsal(a):
> Dne 1.4.2014 12:46, Vít Ondruch napsal(a):
> > I'd go with update to minitest 5 (probably tomorrow, unless you'll be
> >
> > fast enough to point out some weak points ;)
>
> rubygem-minitest-5.3.1-1.fc21 is now available in Rawhide [1].
>
> Vít

Ok, so building rubygem-bson_ext for Ruby 2.1, I am facing following issue:

DEBUG backend.py:978:  ['/usr/bin/yum-builddep', '--installroot', '/var/lib/mock/f21-ruby-2052943-374761/root/', '/var/lib/mock/f21-ruby-2052943-374761/root///builddir/build/SRPMS/rubygem-bson_ext-1.9.2-3.fc21.src.rpm']
DEBUG util.py:331:  Executing command: ['/usr/bin/yum-builddep', '--installroot', '/var/lib/mock/f21-ruby-2052943-374761/root/', '/var/lib/mock/f21-ruby-2052943-374761/root///builddir/build/SRPMS/rubygem-bson_ext-1.9.2-3.fc21.src.rpm', '--setopt=tsflags=nocontexts'] with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'PROMPT_COMMAND': 'echo -n "<mock-chroot>"', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
DEBUG util.py:281:  Getting requirements for rubygem-bson_ext-1.9.2-3.fc21.src
DEBUG util.py:281:   --> ruby-devel-2.1.1-18.fc21.x86_64
DEBUG util.py:281:   --> ruby-libs-2.1.1-18.fc21.x86_64
DEBUG util.py:281:   --> rubygems-devel-2.2.2-18.fc21.noarch
DEBUG util.py:281:   --> ruby-2.1.1-18.fc21.x86_64
DEBUG util.py:281:   --> 1:rubygem-activesupport-4.0.3-1.fc21.noarch
DEBUG util.py:281:   --> rubygem-bson-1.9.2-1.fc21.noarch
DEBUG util.py:281:   --> rubygem-json-1.8.1-18.fc21.x86_64
DEBUG util.py:281:   --> rubygem-minitest-5.3.1-1.fc21.noarch
DEBUG util.py:281:  Error: Package: 1:rubygem-activesupport-4.0.3-1.fc21.noarch (build)
DEBUG util.py:281:             Requires: rubygem(minitest) < 5
DEBUG util.py:281:             Available: rubygem-minitest-4.7.5-18.fc21.noarch (build)
DEBUG util.py:281:                 rubygem(minitest) = 4.7.5-18.fc21
DEBUG util.py:281:             Installing: rubygem-minitest-5.3.1-1.fc21.noarch (build)
DEBUG util.py:281:                 rubygem(minitest) = 5.3.1
DEBUG util.py:281:   You could try using --skip-broken to work around the problem
DEBUG util.py:281:   You could try running: rpm -Va --nofiles --nodigest
DEBUG util.py:371:  Child return code was: 1

In short, activesupport has following requires:

Requires: rubygem(minitest) >= 4.2
Requires: rubygem(minitest) < 5

where bson_ext:

BuildRequires: rubygem(minitest)

and yum can't correctly figure out, which version of minitest to install.

Now the question is how to proceed. There are two possibilities coming 
to my mind:

* Work on the Rails first.
* There is approx. 20 packages depending on AS, so as a temporary 
measure, we might specify minitest < 5 dependency for them and revisit 
later.

I am inclined to skip such gems for now and work on the Rails. Any other 
ideas?

Vít

[Josef Stribny]

> Work on the Rails first.

Let's work on Rails, it's a work that needs to be done anyway.
I will start to do local updates, there shouldn't be any significant
issues.

Josef

testrb dropped from upstream Ruby

[Vít Ondruch]

Dne 1.4.2014 12:46, Vít Ondruch napsal(a):
> We should consider to update packaging guidelines to suggest something
>
> like "ruby -Ilib -e 'Dir.glob "./test/test_*.rb", &method(:require)'"
> instead of testrb. testrb does not handle nested directories anyway,
> which Dir.glob handles just fine.

Just for the record, testrb was already dropped from next version of Ruby:

https://bugs.ruby-lang.org/projects/ruby-trunk/repository/revisions/46011

Vít

[Vít Ondruch]

Just for the record, the list of packages using minitest is not exactly
short:

$ repoquery --repoid=rawhide --repoid=rawhide-source --arch=src
--whatrequires 'rubygem(minitest)' --qf="%{NAME}" | wc -l
203

Vít

Dne 21.3.2014 13:11, Josef Stribny napsal(a):
> Hi Rubyists,
>
> as you have probably noticed we would like to update Rails framework to the latest version (4.1) when it's released[1].
> The biggest task for us in this change is the update of Minitest to version 5. This would require test suites
> of many gems that depends on Minitest to be fixed.
>
> Here are the most significant changes of Minitest 5 taken from the changelog[2]:
>
> 12 major (oft incompatible) changes:
> Renamed MiniTest to Minitest. Your pinkies will thank me. (aliased to MiniTest)
> Removed MiniTest::Unit entirely. No more manager objects.
> Added Minitest::Runnable. Everything minitest can run subclasses this.
> Renamed MiniTest::Unit::TestCase to Minitest::Test (subclassing Runnable).
> Added Minitest::Benchmark.
> Your benchmarks need to move to their own subclass.
> Benchmarks using the spec DSL have to have "Bench" somewhere in their describe.
> MiniTest::Unit.after_tests moved to Minitest.after_tests
> MiniTest::Unit.autorun is now Minitest.autorun. Just require minitest/autorun pls.
> Removed ParallelEach#grep since it isn't used anywhere.
> Renamed Runnable#__name__ to Runnable#name (but uses @NAME internally).
> Runnable#run needs to return self. Allows for swapping of results as needed.
>
> That means the minimal changes needed to be done for every minitest test suite include:
> rename MiniTest::Unit::TestCase to Minitest::Test
> change require 'minitest/unit' (or 'test/unit') to require 'minitest/autorun'
> delete MiniTest::Unit.autorun (or use Minitest.autorun)
>
> For some really simple test suites that might be enough. Look at the changes done in Rails 4.1[3]
> or in my own little gem[4] for how the necessary changes may look like. For the others some additional
> changes might be required.
>
> Another problem are gems that depends on Gem::TestCase (like RubyGems plugins)
> since Gem::TestCase is inherited from MiniTest::Unit::TestCase. I guess this can't be easily
> solved at the moment, but we probably don't have many gems like that, or are we?
>
> So what do you think of this change? Should we address it by updating upstream test suites?
> Do you know about other issues that this can cause and I forgot to mention?
>
> Josef
>
> [1] https://fedoraproject.org/wiki/Changes/Ruby_on_Rails_4.1
> [2] https://github.com/seattlerb/minitest/blob/master/History.txt#L137
> [3] https://github.com/rails/rails/commit/3073c531983de243219fb55be93fbcebfdd9c4...
> [4] https://github.com/strzibny/ruby-ares/commit/a1017ce59eeb8bfb5256b6e7789d957...
>
> ruby-sig mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/ruby-sig

[Vít Ondruch]

For inspiration, there were used some interesting approaches to execute 
the Minitest 4.x testsuite against Minitest 5.x:

http://pkgs.fedoraproject.org/cgit/rubygem-idn.git/tree/rubygem-idn.spec
http://pkgs.fedoraproject.org/cgit/rubygem-raindrops.git/tree/rubygem-raindr...

But still, please discuss this topic with upstream preferably, since in 
Ruby 2.2, there might not be shipped any testing framework:

https://bugs.ruby-lang.org/issues/9711

Vít