Tests in bats, to what extent?

Development
1

There is a pending patch in the nest for converting most tests to “bats”. I don’t see what are the pros and cons of that, except that I don’t like to write anything non-trivial in bash. Maybe we should convert just a few tests, to show that it is possible to write tests for pijul without knowing rust. How easy is bats to learn? It seems that adding an

@test
{
 command1 --flags
 command2 --options
}

is enough to create a test. If so, that seems like a great thing for accepting more contributions in the form of tests.

Now, we need to see if it’s possible to have these tests and tests written in rust coexist, and how to both types of tests with only one command.

2

I understand your reluctance to use bash, because I agree that it’s a huge pain to maintain non-trivial bash scripts. But I actually think bats is surprisingly great, and well suited to testing the pijul command line tool. Here are some plusses

  • the tests are much more concise. I converted all the tests in pijul/src/tests.rs to bats (since I did the conversion, a couple more appeared in tests.rs). It went from 1400+ lines of rust to 300ish lines of bash.
  • the tests are easy to understand. For example, each test now is quite short (whereas some of the rust ones were 100+ lines each)
  • the tests are easy to write: modulo some asserts and a few convenience shortcuts, they look almost exactly like how you would interact with pijul on the command line.
  • related to the previous points, one of the places where bash shines is in interacting with the filesystem, and the pijul tests interact heavily with the filesystem.
  • the tests give better coverage. Even though I basically converted them one-to-one, I still found several bugs by doing the conversion. That’s because there were problems with the argument parsing, and the in-rust tests bypassed that.

If it were up to me, I would put all the pijul tests in bash, and use rust tests for libpijul only. That’s because (IMO) pijul should be a relatively thin interface to the functionality in libpijul and so the pijul tests are primarily integration tests and interface tests. But if you want rust and bash tests to coexist, just replace “cargo build” in pijul/tests/run_tests.sh by “cargo test”. Then “sh tests/run_tests.sh” will run all the tests.

3

Thanks for the detailed overview. Do you know if it’s possible to run kcov with these shell tests? Otherwise, it probably makes sense to convert/duplicate some of the tests as rust tests of libpijul.

4

It turns out to be easy to hook up kcov – I’ve pushed a patch that does it.

5

I’m all for bats tests. @joeneeman, I believe you forgot one extra point: bats tests can serve as better documentation than Rust ones.

I’ll try to merge your patch as soon as the Nest allows me to do that (this is not easy at the moment).

1 Like