Test Coverage

Writing unit tests for your application lets you check that the codeyou wrote works the way you expect. Flask provides a test client thatsimulates requests to the application and returns the response data.

You should test as much of your code as possible. Code in functions onlyruns when the function is called, and code in branches, such as ifblocks, only runs when the condition is met. You want to make sure thateach function is tested with data that covers each branch.

The closer you get to 100% coverage, the more comfortable you can bethat making a change won’t unexpectedly change other behavior. However,100% coverage doesn’t guarantee that your application doesn’t have bugs.In particular, it doesn’t test how the user interacts with theapplication in the browser. Despite this, test coverage is an importanttool to use during development.

Note

This is being introduced late in the tutorial, but in your futureprojects you should test as you develop.

You’ll use pytest and coverage to test and measure your code.Install them both:

  1. $ pip install pytest coverage

Setup and Fixtures

The test code is located in the tests directory. This directory isnext to the flaskr package, not inside it. Thetests/conftest.py file contains setup functions called fixtures_that each test will use. Tests are in Python modules that start withtest, and each test function in those modules also starts withtest_.

Each test will create a new temporary database file and populate somedata that will be used in the tests. Write a SQL file to insert thatdata.

tests/data.sql

  1. INSERT INTO user (username, password)
  2. VALUES
  3.   ('test', 'pbkdf2:sha256:50000$TCI4GzcX$0de171a4f4dac32e3364c7ddc7c14f3e2fa61f2d17574483f7ffbb431b4acb2f'),
  4.   ('other', 'pbkdf2:sha256:50000$kJPKsz6N$d2d4784f1b030a9761f5ccaeeaca413f27f2ecb76d6168407af962ddce849f79');
  5.  
  6. INSERT INTO post (title, body, author_id, created)
  7. VALUES
  8.   ('test title', 'test' || x'0a' || 'body', 1, '2018-01-01 00:00:00');

The app fixture will call the factory and pass test_config toconfigure the application and database for testing instead of using yourlocal development configuration.

tests/conftest.py

  1. import os
  2. import tempfile
  3.  
  4. import pytest
  5. from flaskr import create_app
  6. from flaskr.db import get_db, init_db
  7.  
  8. with open(os.path.join(os.path.dirname(__file__), 'data.sql'), 'rb') as f:
  9.     _data_sql = f.read().decode('utf8')
  10.  
  11.  
  12. @pytest.fixture
  13. def app():
  14.     db_fd, db_path = tempfile.mkstemp()
  15.  
  16.     app = create_app({
  17.         'TESTING': True,
  18.         'DATABASE': db_path,
  19.     })
  20.  
  21.     with app.app_context():
  22.         init_db()
  23.         get_db().executescript(_data_sql)
  24.  
  25.     yield app
  26.  
  27.     os.close(db_fd)
  28.     os.unlink(db_path)
  29.  
  30.  
  31. @pytest.fixture
  32. def client(app):
  33.     return app.test_client()
  34.  
  35.  
  36. @pytest.fixture
  37. def runner(app):
  38.     return app.test_cli_runner()

tempfile.mkstemp() creates and opens a temporary file, returningthe file object and the path to it. The DATABASE path isoverridden so it points to this temporary path instead of the instancefolder. After setting the path, the database tables are created and thetest data is inserted. After the test is over, the temporary file isclosed and removed.

TESTING tells Flask that the app is in test mode. Flask changessome internal behavior so it’s easier to test, and other extensions canalso use the flag to make testing them easier.

The client fixture callsapp.test_client() with the applicationobject created by the app fixture. Tests will use the client to makerequests to the application without running the server.

The runner fixture is similar to client.app.test_cli_runner() creates a runnerthat can call the Click commands registered with the application.

Pytest uses fixtures by matching their function names with the namesof arguments in the test functions. For example, the test_hellofunction you’ll write next takes a client argument. Pytest matchesthat with the client fixture function, calls it, and passes thereturned value to the test function.

Factory

There’s not much to test about the factory itself. Most of the code willbe executed for each test already, so if something fails the other testswill notice.

The only behavior that can change is passing test config. If config isnot passed, there should be some default configuration, otherwise theconfiguration should be overridden.

tests/test_factory.py

  1. from flaskr import create_app
  2.  
  3.  
  4. def test_config():
  5.     assert not create_app().testing
  6.     assert create_app({'TESTING': True}).testing
  7.  
  8.  
  9. def test_hello(client):
  10.     response = client.get('/hello')
  11.     assert response.data == b'Hello, World!'

You added the hello route as an example when writing the factory atthe beginning of the tutorial. It returns “Hello, World!”, so the testchecks that the response data matches.

Database

Within an application context, get_db should return the sameconnection each time it’s called. After the context, the connectionshould be closed.

tests/test_db.py

  1. import sqlite3
  2.  
  3. import pytest
  4. from flaskr.db import get_db
  5.  
  6.  
  7. def test_get_close_db(app):
  8.     with app.app_context():
  9.         db = get_db()
  10.         assert db is get_db()
  11.  
  12.     with pytest.raises(sqlite3.ProgrammingError) as e:
  13.         db.execute('SELECT 1')
  14.  
  15.     assert 'closed' in str(e.value)

The init-db command should call the init_db function and outputa message.

tests/test_db.py

  1. def test_init_db_command(runner, monkeypatch):
  2.     class Recorder(object):
  3.         called = False
  4.  
  5.     def fake_init_db():
  6.         Recorder.called = True
  7.  
  8.     monkeypatch.setattr('flaskr.db.init_db', fake_init_db)
  9.     result = runner.invoke(args=['init-db'])
  10.     assert 'Initialized' in result.output
  11.     assert Recorder.called

This test uses Pytest’s monkeypatch fixture to replace theinit_db function with one that records that it’s been called. Therunner fixture you wrote above is used to call the init-dbcommand by name.

Authentication

For most of the views, a user needs to be logged in. The easiest way todo this in tests is to make a POST request to the login viewwith the client. Rather than writing that out every time, you can writea class with methods to do that, and use a fixture to pass it the clientfor each test.

tests/conftest.py

  1. class AuthActions(object):
  2.     def __init__(self, client):
  3.         self._client = client
  4.  
  5.     def login(self, username='test', password='test'):
  6.         return self._client.post(
  7.             '/auth/login',
  8.             data={'username': username, 'password': password}
  9.         )
  10.  
  11.     def logout(self):
  12.         return self._client.get('/auth/logout')
  13.  
  14.  
  15. @pytest.fixture
  16. def auth(client):
  17.     return AuthActions(client)

With the auth fixture, you can call auth.login() in a test tolog in as the test user, which was inserted as part of the testdata in the app fixture.

The register view should render successfully on GET. On POSTwith valid form data, it should redirect to the login URL and the user’sdata should be in the database. Invalid data should display errormessages.

tests/test_auth.py

  1. import pytest
  2. from flask import g, session
  3. from flaskr.db import get_db
  4.  
  5.  
  6. def test_register(client, app):
  7.     assert client.get('/auth/register').status_code == 200
  8.     response = client.post(
  9.         '/auth/register', data={'username': 'a', 'password': 'a'}
  10.     )
  11.     assert 'http://localhost/auth/login' == response.headers['Location']
  12.  
  13.     with app.app_context():
  14.         assert get_db().execute(
  15.             "select * from user where username = 'a'",
  16.         ).fetchone() is not None
  17.  
  18.  
  19. @pytest.mark.parametrize(('username', 'password', 'message'), (
  20.     ('', '', b'Username is required.'),
  21.     ('a', '', b'Password is required.'),
  22.     ('test', 'test', b'already registered'),
  23. ))
  24. def test_register_validate_input(client, username, password, message):
  25.     response = client.post(
  26.         '/auth/register',
  27.         data={'username': username, 'password': password}
  28.     )
  29.     assert message in response.data

client.get() makes a GET requestand returns the Response object returned by Flask. Similarly,client.post() makes a POSTrequest, converting the data dict into form data.

To test that the page renders successfully, a simple request is made andchecked for a 200 OK status_code. Ifrendering failed, Flask would return a 500 Internal Server Errorcode.

headers will have a Location header with the loginURL when the register view redirects to the login view.

data contains the body of the response as bytes. Ifyou expect a certain value to render on the page, check that it’s indata. Bytes must be compared to bytes. If you want to compareUnicode text, use get_data(as_text=True)instead.

pytest.mark.parametrize tells Pytest to run the same test functionwith different arguments. You use it here to test different invalidinput and error messages without writing the same code three times.

The tests for the login view are very similar to those forregister. Rather than testing the data in the database,session should have user_id set after logging in.

tests/test_auth.py

  1. def test_login(client, auth):
  2.     assert client.get('/auth/login').status_code == 200
  3.     response = auth.login()
  4.     assert response.headers['Location'] == 'http://localhost/'
  5.  
  6.     with client:
  7.         client.get('/')
  8.         assert session['user_id'] == 1
  9.         assert g.user['username'] == 'test'
  10.  
  11.  
  12. @pytest.mark.parametrize(('username', 'password', 'message'), (
  13.     ('a', 'test', b'Incorrect username.'),
  14.     ('test', 'a', b'Incorrect password.'),
  15. ))
  16. def test_login_validate_input(auth, username, password, message):
  17.     response = auth.login(username, password)
  18.     assert message in response.data

Using client in a with block allows accessing context variablessuch as session after the response is returned. Normally,accessing session outside of a request would raise an error.

Testing logout is the opposite of login. session shouldnot contain user_id after logging out.

tests/test_auth.py

  1. def test_logout(client, auth):
  2.     auth.login()
  3.  
  4.     with client:
  5.         auth.logout()
  6.         assert 'user_id' not in session

Blog

All the blog views use the auth fixture you wrote earlier. Callauth.login() and subsequent requests from the client will be loggedin as the test user.

The index view should display information about the post that wasadded with the test data. When logged in as the author, there should bea link to edit the post.

You can also test some more authentication behavior while testing theindex view. When not logged in, each page shows links to log in orregister. When logged in, there’s a link to log out.

tests/test_blog.py

  1. import pytest
  2. from flaskr.db import get_db
  3.  
  4.  
  5. def test_index(client, auth):
  6.     response = client.get('/')
  7.     assert b"Log In" in response.data
  8.     assert b"Register" in response.data
  9.  
  10.     auth.login()
  11.     response = client.get('/')
  12.     assert b'Log Out' in response.data
  13.     assert b'test title' in response.data
  14.     assert b'by test on 2018-01-01' in response.data
  15.     assert b'test\nbody' in response.data
  16.     assert b'href="/1/update"' in response.data

A user must be logged in to access the create, update, anddelete views. The logged in user must be the author of the post toaccess update and delete, otherwise a 403 Forbidden statusis returned. If a post with the given id doesn’t exist,update and delete should return 404 Not Found.

tests/test_blog.py

  1. @pytest.mark.parametrize('path', (    '/create',    '/1/update',    '/1/delete',))def test_login_required(client, path):    response = client.post(path)    assert response.headers['Location'] == 'http://localhost/auth/login'
  2. def test_author_required(app, client, auth):
  3. # change the post author to another user
  4. with app.app_context():
  5.     db = get_db()
  6.     db.execute('UPDATE post SET author_id = 2 WHERE id = 1')
  7.     db.commit()
  9. auth.login()
  10. # current user can't modify other user's post
  11. assert client.post('/1/update').status_code == 403
  12. assert client.post('/1/delete').status_code == 403
  13. # current user doesn't see edit link
  14. assert b'href="/1/update"' not in client.get('/').data
  15. @pytest.mark.parametrize('path', (    '/2/update',    '/2/delete',))def test_exists_required(client, auth, path):    auth.login()    assert client.post(path).status_code == 404

The create and update views should render and return a200 OK status for a GET request. When valid data is sent in aPOST request, create should insert the new post data into thedatabase, and update should modify the existing data. Both pagesshould show an error message on invalid data.

tests/test_blog.py

  1. def test_create(client, auth, app):
  2.     auth.login()
  3.     assert client.get('/create').status_code == 200
  4.     client.post('/create', data={'title': 'created', 'body': ''})
  5.  
  6.     with app.app_context():
  7.         db = get_db()
  8.         count = db.execute('SELECT COUNT(id) FROM post').fetchone()[0]
  9.         assert count == 2
  10.  
  11.  
  12. def test_update(client, auth, app):
  13.     auth.login()
  14.     assert client.get('/1/update').status_code == 200
  15.     client.post('/1/update', data={'title': 'updated', 'body': ''})
  16.  
  17.     with app.app_context():
  18.         db = get_db()
  19.         post = db.execute('SELECT * FROM post WHERE id = 1').fetchone()
  20.         assert post['title'] == 'updated'
  21.  
  22.  
  23. @pytest.mark.parametrize('path', (
  24.     '/create',
  25.     '/1/update',
  26. ))
  27. def test_create_update_validate(client, auth, path):
  28.     auth.login()
  29.     response = client.post(path, data={'title': '', 'body': ''})
  30.     assert b'Title is required.' in response.data

The delete view should redirect to the index URL and the post shouldno longer exist in the database.

tests/test_blog.py

  1. def test_delete(client, auth, app):
  2.     auth.login()
  3.     response = client.post('/1/delete')
  4.     assert response.headers['Location'] == 'http://localhost/'
  5.  
  6.     with app.app_context():
  7.         db = get_db()
  8.         post = db.execute('SELECT * FROM post WHERE id = 1').fetchone()
  9.         assert post is None

Running the Tests

Some extra configuration, which is not required but makes runningtests with coverage less verbose, can be added to the project’ssetup.cfg file.

setup.cfg

  1. [tool:pytest]
  2. testpaths = tests
  3.  
  4. [coverage:run]
  5. branch = True
  6. source =
  7.     flaskr

To run the tests, use the pytest command. It will find and run allthe test functions you’ve written.

  1. $ pytest
  2.  
  3. ========================= test session starts ==========================
  4. platform linux -- Python 3.6.4, pytest-3.5.0, py-1.5.3, pluggy-0.6.0
  5. rootdir: /home/user/Projects/flask-tutorial, inifile: setup.cfg
  6. collected 23 items
  7.  
  8. tests/test_auth.py ........                                      [ 34%]
  9. tests/test_blog.py ............                                  [ 86%]
  10. tests/test_db.py ..                                              [ 95%]
  11. tests/test_factory.py ..                                         [100%]
  12.  
  13. ====================== 24 passed in 0.64 seconds =======================

If any tests fail, pytest will show the error that was raised. You canrun pytest -v to get a list of each test function rather than dots.

To measure the code coverage of your tests, use the coverage commandto run pytest instead of running it directly.

  1. $ coverage run -m pytest

You can either view a simple coverage report in the terminal:

  1. $ coverage report
  2.  
  3. Name                 Stmts   Miss Branch BrPart  Cover
  4. ------------------------------------------------------
  5. flaskr/__init__.py      21      0      2      0   100%
  6. flaskr/auth.py          54      0     22      0   100%
  7. flaskr/blog.py          54      0     16      0   100%
  8. flaskr/db.py            24      0      4      0   100%
  9. ------------------------------------------------------
  10. TOTAL                  153      0     44      0   100%

An HTML report allows you to see which lines were covered in each file:

  1. $ coverage html

This generates files in the htmlcov directory. Openhtmlcov/index.html in your browser to see the report.

Continue to Deploy to Production.