Topic: Why are test cases not run in the order they are defined?

Leading on from my previous thread, I also note that unit tests (and presumably also functional tests) are not executed in the order in which they are defined in the test class. Why not? It's not unreasonable to assume that they would be.

It seems logical that unit tests are defined to start off simple and get more complex. Thus the simpler ones get run first so that fixing them hopefully resolves problems in later tests.

Also, why isn't data from yml test fixtures loaded into the database in the order it is defined in the yml file?

Gareth

Last edited by Guff (2007-03-06 08:45:17)

Re: Why are test cases not run in the order they are defined?

I believe that tests are run in alphabetical order.

Don't know about fixtures order...  But does it matter?

Regards,
Richard

PlanetGPS.net
RogueWaveMarine.com
===================================

Re: Why are test cases not run in the order they are defined?

Guff wrote:

Leading on from my previous thread, I also note that unit tests (and presumably also functional tests) are not executed in the order in which they are defined in the test class. Why not? It's not unreasonable to assume that they would be.

They get run in alphabetical order, but this shouldn't matter. Unit tests are designed to always start with a clean slate before every test, so the environment should be exactly the same. Nothing should stick around between the individual tests so it shouldn't matter if one is run before another.


Guff wrote:

Also, why isn't data from yml test fixtures loaded into the database in the order it is defined in the yml file?

Not sure about this one. Are you specifying the "id" column in the fixtures? I haven't experienced this issue. However, I don't think it should really be an issue. If you don't specify a sort order in an SQL query the results aren't guarenteed to return in any order (IIRC). So, if you rely on them being in a certain order you need to do an "order" in the SQL query.

Railscasts - Free Ruby on Rails Screencasts

Re: Why are test cases not run in the order they are defined?

Ryan
It matters if the transaction doesn't roll back properly, as per my other post.

as to the YML, again, it certainly matters if records refer to each other and there are foreign key contraints: as in my other post. nd yes, I am specifying id, as it is also parent_id

Guff

Last edited by Guff (2007-03-06 12:19:32)

Re: Why are test cases not run in the order they are defined?

Guff wrote:

It matters if the transaction doesn't roll back properly, as per my other post.

Yeah, fixing the rollback problem should solve this as well then.

Guff wrote:

as to the YML, again, it certainly matters if records refer to each other and there are foreign key contraints: as in my other post. nd yes, I am specifying id, as it is also parent_id

Hmm, I see what you mean now. Unfortunately I'm not sure how to solve it as I don't have any experience with foreign key constraints (I keep all the constraints in Ruby).

BTW, here's a ticket on the subject. Also see this wiki article

Railscasts - Free Ruby on Rails Screencasts

Re: Why are test cases not run in the order they are defined?

ryanb wrote:

They get run in alphabetical order, but this shouldn't matter. Unit tests are designed to always start with a clean slate before every test, so the environment should be exactly the same. Nothing should stick around between the individual tests so it shouldn't matter if one is run before another.

And what about functional tests ? beacause the test_destroy is running before the test_edit.. so the item is not found!

Is it a good habit to have tests naming convention similar to those of migrations?

def test_001_edit # A method name couldn't start by a number
  assert..
end

def test_002_destroy
  assert..
end


any comment ?

Re: Why are test cases not run in the order they are defined?

Tests should be completely self contained, meaning what happens in one test shouldn't depend on what happens in another test. The environment should be reset at the beginning of each test. It's bad practice to number the tests and you shouldn't rely on tests to be done in this order.

Railscasts - Free Ruby on Rails Screencasts

Re: Why are test cases not run in the order they are defined?

ryanb wrote:

Tests should be completely self contained

Do you mean that i should reinsert the element after deleting it ?

ryanb wrote:

...The environment should be reset at the beginning of each test.

So how to let rails do that ? I know that it's reset each rake test:xxx but how to reset it each method execution?

Re: Why are test cases not run in the order they are defined?

mrorm wrote:

Do you mean that i should reinsert the element after deleting it ?

Nope, tests shouldn't worry about resetting the environment. This should be done in the setup/teardown methods.

mrorm wrote:

So how to let rails do that ? I know that it's reset each rake test:xxx but how to reset it each method execution?

It should be done automatically for you. Is it not?

Railscasts - Free Ruby on Rails Screencasts

Re: Why are test cases not run in the order they are defined?

No, it isn't, because after the test_destroy, the test_destroy raise an error and when commenting the first, every thing is fine.

Re: Why are test cases not run in the order they are defined?

If you post the code I may be able to help. It's hard to say what's wrong without seeing it.

Railscasts - Free Ruby on Rails Screencasts

Re: Why are test cases not run in the order they are defined?

The errors happens just with scaffolded code (./script/generate scaffold book for exp.).

The test:

  def test_destroy
    assert_nothing_raised {
      User.find(@first_id)
    }

    post :destroy, :id => @first_id
    assert_response :redirect
    assert_redirected_to :action => 'list'

    assert_raise(ActiveRecord::RecordNotFound) {
      User.find(@first_id)
    }
  end


the errors:

Loaded suite /usr/local/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/rake_test_loader
Started
........................EEEEEE
Finished in 0.287782 seconds.

  1) Error:
test_edit(UsersControllerTest):
ActiveRecord::RecordNotFound: Couldn't find User with ID=1
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1031:in `find_one'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1014:in `find_from_ids'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:419:in `find'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:424:in `find'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:501:in `users'
    ./test/functional/users_controller_test.rb:15:in `setup_without_fixtures'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:576:in `setup'

  2) Error:
test_index(UsersControllerTest):
ActiveRecord::RecordNotFound: Couldn't find User with ID=1
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1031:in `find_one'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1014:in `find_from_ids'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:419:in `find'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:424:in `find'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:501:in `users'
    ./test/functional/users_controller_test.rb:15:in `setup_without_fixtures'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:576:in `setup'

  3) Error:
test_list(UsersControllerTest):
ActiveRecord::RecordNotFound: Couldn't find User with ID=1
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1031:in `find_one'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1014:in `find_from_ids'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:419:in `find'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:424:in `find'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:501:in `users'
    ./test/functional/users_controller_test.rb:15:in `setup_without_fixtures'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:576:in `setup'

  4) Error:
test_new(UsersControllerTest):
ActiveRecord::RecordNotFound: Couldn't find User with ID=1
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1031:in `find_one'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1014:in `find_from_ids'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:419:in `find'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:424:in `find'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:501:in `users'
    ./test/functional/users_controller_test.rb:15:in `setup_without_fixtures'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:576:in `setup'

  5) Error:
test_show(UsersControllerTest):
ActiveRecord::RecordNotFound: Couldn't find User with ID=1
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1031:in `find_one'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1014:in `find_from_ids'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:419:in `find'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:424:in `find'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:501:in `users'
    ./test/functional/users_controller_test.rb:15:in `setup_without_fixtures'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:576:in `setup'

  6) Error:
test_update(UsersControllerTest):
ActiveRecord::RecordNotFound: Couldn't find User with ID=1
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1031:in `find_one'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1014:in `find_from_ids'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:419:in `find'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:424:in `find'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:501:in `users'
    ./test/functional/users_controller_test.rb:15:in `setup_without_fixtures'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/fixtures.rb:576:in `setup'

30 tests, 71 assertions, 0 failures, 6 errors


Commenting the this test or renaming it (test_zdestroy), makes tests running without any error / failure.

Re: Why are test cases not run in the order they are defined?

Are you using fixtures? Is that test suite set up to load the fixtures?

I believe the fixtures should be reset automatically after each test. What database are you using? Perhaps if it doesn't support rollbacks this may be the problem.

Are you doing anything special in the "setup" method?

Railscasts - Free Ruby on Rails Screencasts

Re: Why are test cases not run in the order they are defined?

ryanb wrote:

Are you using fixtures? Is that test suite set up to load the fixtures?

Yes, I'm using fixtures:

fixtures :users

and the @first_id used in the test is written in the setup test and used by almost all test methods:

@first_id = users(:mahmoud).id

ryanb wrote:

I believe the fixtures should be reset automatically after each test. What database are you using? Perhaps if it doesn't support rollbacks this may be the problem.

I'm using MySQL:

Server version: 5.0.37
Protocol version: 10
Server: Localhost via UNIX socket
Tables Type: MyISAM

ryanb wrote:

Are you doing anything special in the "setup" method?

Here my setup:

  def setup
    @controller = UsersController.new
    @request    = ActionController::TestRequest.new
    @response   = ActionController::TestResponse.new

    @first_id = users(:mahmoud).id
  end

Re: Why are test cases not run in the order they are defined?

Hmm, I'm wondering if the users are getting a new "id" every time the fixtures are recreated. Are you specifying the "id" column in the users fixtures?

I'm not sure what else the problem could be.

Railscasts - Free Ruby on Rails Screencasts

Re: Why are test cases not run in the order they are defined?

Yes I do sad but I really hoped that I don't found it specified in the fixtures!

Can you please send a sample project were scaffolded tests are running fine? just to make a comparison.

Re: Why are test cases not run in the order they are defined?

If you just run the scaffold generator that comes with tests which work properly.

Railscasts - Free Ruby on Rails Screencasts

Re: Why are test cases not run in the order they are defined?

Here a sample app for illustration (try to rename the test_zdestroy to test_destroy).

Re: Why are test cases not run in the order they are defined?

It works without error for me. I wonder if the rollback isn't working on your database? Check out the test.log file and see if there are rollback statements and if there's any error.

Railscasts - Free Ruby on Rails Screencasts

Re: Why are test cases not run in the order they are defined?

Even with renaming the test_zdestroy to test_destroy ??