Introduction to CheckPy

Installation

pip install checkpy

Writing and running your first test

First create a new directory to store your tests somewhere on your computer. Then navigate to that directory, and create a new file called helloTest.py. CheckPy discovers tests for a particular source file (i.e. hello.py) by looking for a test file starting with a corresponding name (hello) and ending with test.py. So CheckPy uses helloTest.py to test hello.py.

Now open up helloTest.py and insert the following code:

import checkpy.tests as t
import checkpy.lib as lib
import checkpy.assertlib as asserts

@t.test(0)
def exactlyHelloWorld(test):
    def testMethod():
        output = lib.outputOf(test.fileName)
        return asserts.exact(output.strip(), "Hello, world!")

    test.test = testMethod
    test.description = lambda : "prints exactly: Hello, world!"

Next, create a file called hello.py somewhere on your computer. Insert the following snippet of code in hello.py:

print("Hello, world!")

Now there’s only one thing left to do. We need to tell CheckPy where the tests are located. You can do this by calling CheckPy with the -register flag and by providing an absolute path to the directory helloTest.py is located in. Say helloTest.py is located in \Users\foo\bar\tests\ then call CheckPy like so:

checkpy -register \Users\foo\bar\tests\

Alright, we’re there. We got a test (helloTest.py), a Python file we want to test (hello.py), and we’ve told CheckPy where to look for tests. Now navigate over to the directory that contains hello.py and call CheckPy as follows:

checkpy hello

Writing simple tests in CheckPy

Tests in CheckPy are instances of checkpy.tests.Test. These Test instances have several abstract methods that you can implement or rather, overwrite by binding a new method. These methods are executed when CheckPy runs a test. For instance you have the description method which is called to produce a description for the test, the timeout method which is called to determine the maximum alotted time for this test, and ofcourse the test method which is called to actually perform the test. This Test instance is automatically provided to you when you decorate a function with the checkpy.tests.test decorator. In our hello-world example this looked something like:

@t.test(0)
def exactlyHelloWorld(test):

Here the t.test decorator (t is short for checkpy.tests) decorates the function exactlyHelloWorld. This causes CheckPy to treat exactlyHelloWorld as a test creator function. That when called produces an instance of Test. The t.test decorator accepts an argument that is used to determine the order in which the result of the test is shown to the screen (lowest first). The decorator then passes an instance of Test to the decorated function (exactlyHelloWorld). It is up to exactlyHelloWorld to overwrite some or all abstract methods of that one instance of Test that it receives.

Lets start with the necessities. CheckPy requires you to overwrite two methods from every Test instance. These methods are test and description. The description method should produce the description, that can be just a string, for the user to see. In our hello-world example we used this description method:

test.description = lambda : "prints exactly: Hello, world!"

Depending on whether the test fails or passes, the user sees this string in red or green respectively. The other method we have to overwrite, the test method, should return True or False depending on whether the tests passes or fails. You are free to implement this method in any which way you want. CheckPy just offers some useful tools to make your testing life easier. Again, looking back at our hello-world example, we used this test method:

def testMethod():
    output = lib.outputOf(test.fileName)
    return asserts.exact(output.strip(), "Hello, world!")

test.test = testMethod

So what’s going on here? Python doesn’t support multi statement lambda functions. This means that if you want to use multiple statements, you have to resort to named functions, i.e. testMethod(), and then bind this named function to the respective method of the Test instance. You can put the above test method in a single statement lambda function, but readability will suffer from it. Especially once we move on to some more complex test methods.

Now there are just 2 lines of code in this testMethod. First we take the output of something called test.fileName. test.fileName just refers to the name of the file the user wants to test, CheckPy will automatically set this for you. lib.outputOf is a function that gives you all the print-output of a python file. Thus what this line of code does is simply run the file that the user wants to test, and then return the print output as a string.

The line return asserts.exact(output.strip(), "Hello, world!") is equivalent to return output.strip() == "Hello, world!". The checkpy.assertlib module, that is renamed to asserts here, simply offers a collection of functions to perform assertions. These functions do nothing more than return True or False.

That’s all there is to it. You simply write a function and decorate it with the test decorator. Overwrite a couple of methods of Test, and you’re good to go.

Testing functions

Let’s make life a little more exciting. CheckPy can do a lot more besides simply running a Python file and looking at print output. Specifically CheckPy lets you import said Python file as a module and do all sort of things with it and to it. Lets focus on Functions for now.

For an assignment on (biological) virus simulations, we asked students to do the following:

Write a function generateVirus(length). This function should accept one argument length, that is an integer representing the length of the virus. The function should return a virus, that is a string of random nucleotides ('A', 'T', 'G' or 'C').

This is just a small part of a bigger assignment that ultimately moves towards a simulation of viruses in a patient. We can use CheckPy to test several aspects of this assignment. For instance to test whether only the nucleotides ATGC occurred we wrote the following:

@t.test(10)
def onlyATGC(test):
    def testMethod():
            generateVirus = lib.getFunction("generateVirus", test.fileName)
            pairs = "".join(generateVirus(10) for _ in range(1000))
            return asserts.containsOnly(pairs, "AGTC")

    test.test = testMethod
    test.description = lambda : "generateVirus produces viruses consisting only of A, T, G and C"

To test whether the function actually exists and accepted just one argument, we wrote the following:

@t.test(0)
def isDefined(test):
  def testMethod():
    generateVirus = lib.getFunction("generateVirus", test.fileName)
    return len(generateVirus.arguments) == 1

  test.test = testMethod
  test.description = lambda : "generateVirus is defined and accepts just one argument"

Testing programs with arguments

Taking a closer look at the checkpy.lib module we find three functions that allow you to interact with dynamic components and results from the program we are testing. All these functions (outputOf, getFunction and getModule) take in the same optional arguments that let you change the dynamic environment in which the code is tested. Zooming in on outputOf:

def outputOf(
            fileName,
            src = None,
            argv = None,
            stdinArgs = None,
            ignoreExceptions = (),
            overwriteAttributes = ()
    ):
  """
  fileName is the file name you want the stdout output of
  src can be used to ignore the source code of fileName, and instead use this string
  argv is a collection of elements that are used to overwrite sys.argv
  stdinArgs is a collection of arguments that are passed to stdin
  ignoreExceptions is a collection of exceptions that should be ignored during execution
  overwriteAttributes is a collection of tuples (attribute, value) that are overwritten
      before trying to import the file
  """

Lets see what we can do with this. For an assignment we asked students to write a program that prints out how many liters of water were used while showering. The program should prompt the user for the number of minutes they shower, and then print out many liters of water were used. We told them 1 minute of showering equaled 12 liters used.

For this assignment we wrote the following test:

@t.test(10)
def oneLiter(test):
  def testMethod():
    output = lib.outputOf(
        test.fileName,
        stdinArgs = [1],
        overwriteAttributes = [("__name__", "__main__")]
    )
    return asserts.contains(output, "12")

  test.test = testMethod
  test.description = lambda : "1 minute equals 12 bottles."

The above test runs the student’s file, pushes the number 1 in stdin and sets the attribute __name__ to "__main__". It does not ignore any exceptions, that means that CheckPy will fail the test if an exception is raised and kindly tell the user what exception was raised. Argv is set to the default (just the program name).

Customizing output

An instance of Test has a couple of methods that you can use to show the user exactly what you want the user to see. We have already seen the .description() method that you can overwrite with a function that should produce the description of the test. This description then turns green or red, with a happy or sad smiley depending on whether the test fails or passes. Besides the .description() method you also find the .success(info) and .fail(info) methods. These methods take in an argument called info and should produce a message for the user to read when the test succeeds or fails respectively. This message is printed directly under the description. Take the following test:

@t.test(0)
def failExample1(test):
  def testMethod():
    output = lib.outputOf(test.fileName)
    line = lib.getLine(output, 0)
    return asserts.numberOnLine(42, line)

  test.test = testMethod
  test.description = lambda : "demonstrating the use of .fail()!"
  test.fail = lambda info : "could not find 42 in the first line of the output"

The above test looks for the number 42 on the first line of the output. If the test fails it will print that it could not find 42 in the output. Okay, this is a little boring, CheckPy just prints a static description if the test fails. So let’s spice things up. Take the following test:

@t.test(0)
def failExample2(test):
  def testMethod():
    output = lib.outputOf(test.fileName)
    line = lib.getLine(output, 0)
    return asserts.numberOnLine(42, line), lib.getNumbersFromString(line)

  test.test = testMethod
  test.description = lambda : "demonstrating the use of .fail()!"
  test.fail = lambda info : "could not find 42 on the first line of output, only these numbers: {}".format(info)

This test also looks for 42 on the first line of the output. If this test fails however it will also print what numbers it did find on that one line of output. Here’s what’s happening. The .test() method can return a second value besides simply a boolean indicating whether the passed. This value is passed to the .fail(info) and .success(info) methods. So you can use this second return value to customize what .fail(info) and .success(info) do. Here the implementation of .fail(info) simply prints out whatever info it receives.