Pylons Project Testing

The Pylons Project is a collection of open-source web application technologies developed in Python. Together, these applications form an ecosystem that facilitates the development of Python applications. Some of the web applications included in the Pylons Project are:

• Pyramid: This is a web framework that enables you to start your project quickly and simply, while supporting extensibility at later stages.
• Deform: This is a library for building web forms.
• Colander: This is a library for validating and deserializing data structures.
• WebOb: This is a library for working with HTTP requests and responses in Python. It provides a simple, consistent interface for parsing and generating HTTP messages, making it easier to build web applications that handle complex HTTP interactions.
• Webtest: This lets you test your Pylons application code without starting up a server.

As applications built using any of the Pylons Project frameworks are Python compliant, you can test them using testing tools that are designed for Python. In this post, we will discuss the following approaches to testing Pylons Project applications:

• Unit testing
• Integration testing
• End-to-end testing

Unit Testing

Unit testing serves as the first layer of verification for the application. It is a good practice to try to write both positive and negative unit tests for as many functions as possible. Unit tests should be lightweight and fast, hence they should not have dependencies. This means that dependencies, such as API calls, data requirements, UI interactions, or any others, will be mocked. Therefore, you can run your tests in isolation as needed. One important point to remember is that every unit test should focus on a single output. Therefore, your functions should be written to produce a single output.

Testing using Unittest

Unittest is a very popular choice for unit testing Python code. It supports the creation and maintenance of test fixtures, test suites, and provides a test runner to execute test cases. Using the assert methods provided by Unittest, you can verify the expected outcomes. These assertions are available in the TestCase class.

Here's a simple example of fetching a name and checking if it returns the expected value.

import unittest
from myapp.model import MyModel

class TestMyModel(unittest.TestCase):
  def test_get_name(self):

    name = 'Alice'
    mymodel = MyModel(name=name)

    # Call the get_name method and check that it returns the expected name
    expected_name = name
    actual_name = mymodel.get_name()
    self.assertEqual(expected_name, actual_name)

Testing using Pytest

Pytest is another widely used option for writing unit tests. It supports running Unittest and Nose test cases. Pytest offers many features such as a rich set of assertions, a plugin ecosystem, fixtures, mocking and patching using Pytest's monkeypatch fixture, parameterized testing, and test coverage plugins. Pytest needs to be added as a requirement in the setup.py file along with other configurations. For applications that use asynchronous programming, Pytest offers a plugin called pytest-asyncio, which can be used for testing. However, this plugin might be more useful for integration or functional testing, rather than unit testing.

Using Pyramid test utilities

If you're using the Pyramid framework to develop your application, you can utilize pyramid.testing to access other testing utilities such as:

• TestConfig: This class in pyramid.testing enables you to configure your application and its components in a test environment. It can be used to register views, routes, templates, and other application dependencies, as well as to set up custom configuration settings for testing.
• DummyRequest: This class in pyramid.testing allows the creation of a mock request object for testing. You can set attributes such as the method, path, headers, and params to simulate various types of requests.
• DummyResource: This class in pyramid.testing allows the creation of a mock resource object for testing. You can set attributes such as _parent_, _name_, and _acl_ to simulate the behavior of different types of resources.
• MockTemplate: This class in pyramid.testing allows the creation of a mock template object for testing. You can set attributes such as name, renderer, and implementation to simulate the behavior of your application's actual templates.
• setUp and tearDown methods: These methods can be used to establish and dismantle the testing environment before and after each test. For example, you can use setUp to create a database connection and tearDown to close the connection.
• TestingAuthenticationPolicy: This class provides a convenient way to test authentication and authorization in Pyramid applications.

These examples demonstrate the variety of testing utilities and helpers that Pyramid provides. By leveraging these utilities and combining them with other testing frameworks like Unittest or Pytest, you can create comprehensive tests to ensure your application behaves as expected under a range of conditions.

Test case example

Let's consider testing a function written in Pyramid. Here's an example function:

from pyramid.httpexceptions import HTTPForbidden

def view_fn(request):
  if request.has_permission('edit'):
    raise HTTPForbidden
  return {'greeting':'hello'}

Now, here's a unit test for this function, using both Unittest for writing the test and Pytest for running it. This test also uses Pyramid's own test utilities:

import unittest
from pyramid import testing

class MyTest(unittest.TestCase):
  def setUp(self):
    self.config = testing.setUp()

  def tearDown(self):
    testing.tearDown()

  def test_view_fn_forbidden(self):
    from pyramid.httpexceptions import HTTPForbidden
    from my.package import view_fn
    self.config.testing_securitypolicy(userid='hank', permissive=False)
    request = testing.DummyRequest()
    request.context = testing.DummyResource()
    self.assertRaises(HTTPForbidden, view_fn, request)

  def test_view_fn_allowed(self):
    from my.package import view_fn
    self.config.testing_securitypolicy(userid='hank', permissive=True)
    request = testing.DummyRequest()
    request.context = testing.DummyResource()
    response = view_fn(request)
    self.assertEqual(response, {'greeting':'hello'})

This test class, MyTest, inherits from unittest.TestCase. If it's properly set up, it will be available when Pytest runs. The test_view_fn_forbidden tests view_fn when the current user doesn't have 'edit' permissions per the security policy. A dummy request and dummy resource are created using Pyramid's testing module, and then we assert that calling view_fn with this request raises an HTTPForbidden exception.

The second test method, test_view_fn_allowed, checks that the security policy grants access. A dummy request and dummy resource are created using Pyramid's testing module, thenview_fn is called with this request. We then assert that the response returned by view_fn is a dictionary with a single key 'greeting' and value 'hello'.

Thus, this test class is testing the behavior of the view_fn function under two different security policies (permissive and non-permissive). It does this by creating dummy requests and resources using Pyramid's testing module, and by asserting that the function behaves as expected under each condition.

Testing using Nose

Nose is a testing framework for the Python programming language that extends the capabilities of Python's built-in unittest module. It provides additional features and plugins that make it easier to write, discover, and run tests. Nose can also integrate with other Python tools such as coverage and profiling tools.

You can easily install Nose for your project with the following command:

$ easy_install -U nose

Nose will automatically discover and run all tests in the directory and its subdirectories.

Testing using WebTest

WebTest is a versatile testing library particularly beneficial for testing WSGI applications. TestApp, a key component of WebTest, serves as a wrapper for WSGI applications. You can employ WebTest for writing unit tests for APIs, non-HTTP protocols like SMTP and FTP, webhooks, OAuth clients, HTTP-based services, and RESTful services. The following example demonstrates how to use WebTest's TestApp to validate the response body of an API call:

from webtest import TestApp
from myapp import main  # replace "myapp" with the name of your application

def test_api_view():

  app = TestApp(main())

  response = app.get('/api')

  # Check that the response status code is 200 OK
  assert response.status_code == 200

  # Check that the response content type is application/json
  assert response.content_type == 'application/json'

  # Check that the response body contains the expected JSON data
  expected_data = {'message': 'Hello, world!'}
  assert response.json == expected_data

Integration Testing

Integration testing enables you to verify that various units of code are operating correctly in conjunction. Such tests are not required for every function but are essential for scenarios involving integration between different components such as the database, UI views, or other network-related elements. Writing these tests serves as a valuable precursor to end-to-end testing as they examine the junctions between different components. These tests should be more lightweight than end-to-end tests and often use both mocking and actual resources. For instance, to test if an API is writing data into the database upon a button click, you can invoke the API and the database write operation while mocking the UI button click.

It's a good practice to set up test data or a test database to ensure there are no impacts on the production environment.

Much like unit test creation, you can utilize Unittest, Nose, Pytest, and Pyramid utilities to write integration tests. The following example uses them to test if a database write operation occurs upon a button click:

from unittest import TestCase
from pyramid import testing
from nose.plugins.attrib import attr
from myapp import main
import mysql.connector

@attr(integration=True)
class IntegrationTests(TestCase):
  def setUp(self):
    # Connect to the test database
    self.cnx = mysql.connector.connect(
      user='testuser',
      password='testpassword',
      host='localhost',
      database='testdb'
    )
    
    # Create a test application
    app = main({}, **settings)
    self.testapp = TestApp(app)
    
    def tearDown(self):
      # Clean up the test database and application
      cursor = self.cnx.cursor()
      cursor.execute('DROP DATABASE testdb')
      cursor.close()
      self.cnx.close()
      testing.tearDown()
    
    def test_button_click_writes_to_database(self):
      # Simulate a button click
      request = testing.DummyRequest(post={'foo': 'bar'})
      response = self.testapp.post('/my_view', params=request.params)
      assert 'Success' in response
      
      # Verify that the data is written to the database
      cursor = self.cnx.cursor()
      cursor.execute('SELECT * FROM mytable WHERE foo = "bar"')
      result = cursor.fetchone()
      self.assertIsNotNone(result)
      cursor.close()

In this example, Nose's attr decorator labels the test as an integration test. We use the pyramid.testing utilities to create a test application and dummy request object. The setUp method creates a test application by invoking the main function with necessary configuration settings, and stores a reference to the test application in self.testapp. The tearDown method calls testing.tearDown() to clean up any test fixtures.

Finally, in the test_button_click_writes_to_database method, a button click is simulated by creating a dummy request with POST data and sending it to our view function using self.testapp.post(). An assertion checks that the response contains the string 'Success'. Finally, the data is verified to have been written to the database by querying the database session and looking for a record with the correct 'foo' value.

You can also use WebTest's TestApp to write integration tests. In the example below, we use the TestApp class from WebTest to simulate HTTP requests to our Pylons application. A test application is created in the setUp method and stored in self.testapp.

In the test_button_click_writes_to_database method, we simulate a button click by sending a POST request to the my_view URL with a 'foo' parameter set to 'bar'. We then assert that the response contains the string 'Success'. Finally, we verify that the data has been written to the database by querying the database using the DBSession object and checking that a row with the correct value for 'foo' exists.

from unittest import TestCase
from webtest import TestApp
from pyramid import testing

from myapp import main

class IntegrationTests(TestCase):
  def setUp(self):
    # Create a test application
    app = main({}, **settings)
    self.testapp = TestApp(app)
  
  def tearDown(self):
    # Clean up the test application
    testing.tearDown()
  
  def test_button_click_writes_to_database(self):
    # Simulate a button click
    response = self.testapp.post('/my_view', {'foo': 'bar'})
    
    # Verify that the response contains the expected string
    assert 'Success' in response
    
    # Verify that the data was written to the database
    with transaction.manager:
      result = DBSession.query(MyModel).filter(MyModel.foo == 'bar').first()
      self.assertIsNotNone(result)

End-to-End Testing

Unit and integration tests help check the functioning of the code. But that is not enough. You also need to make sure when the end user interacts with your application, it is able to operate smoothly while fulfilling business cases. This is where end-to-end testing comes into play. Since the end user’s perspective is given more importance when formulating these tests, it is good to include manual testers and business team members for gaining perspective. You can perform end-to-end testing for your Pylons project using:

• WebTest
• Robot framework
• testRigor

WebTest

Using WebTest, you can create functional test cases that will help you verify whether different modules work properly. You can simulate HTTP requests and responses, different types of web browsers, including sending browser-specific headers and inspecting browser-specific behaviour and also simulate cookies and session data. Here is an example of adding an item to the cart on Amazon while checking if the pages load without issues.

import unittest
from webtest import TestApp

class TestAddToCart(unittest.TestCase):
  def setUp(self):
  
    self.app = TestApp("http://www.amazon.com")
  
  def test_add_to_cart(self):
  
    response = self.app.get("/s?k=python+book")
    
    # assert that the search results page loaded successfully
    self.assertEqual(response.status_code, 200)
    self.assertIn("Python book", response)
    
    response = response.click("Python Crash Course")
    
    # assert that the product page loaded successfully
    self.assertEqual(response.status_code, 200)
    self.assertIn("Python Crash Course", response)
    
    form = response.forms["addToCart"]
    response = form.submit()
    
    # assert that the item was added to the cart successfully
    self.assertEqual(response.status_code, 200)
    self.assertIn("Added to Cart", response)
    
    # check that the item was added to the cart
    response = self.app.get("/gp/cart/view.html/ref=lh_cart")
    
    # assert that the cart page loaded successfully
    self.assertEqual(response.status_code, 200)
    
    # assert that the item is present in the cart
    self.assertIn("Python Crash Course", response)

if __name__ == "__main__":
  unittest.main()

You can read more about the various functions of WebTest and their uses over here.

Robot Framework

This open-source test automation framework utilizes a keyword-driven testing approach. It is used for acceptance testing and Robotic Process Automation (RPA). It comes with a rich library of keywords for creating test cases. However, when creating custom keywords, you need to write Python or Java code to define their actions. This requirement could pose a challenge when automating tests. Furthermore, you need a proper setup before using this framework, which can be time-consuming compared to cloud-based platforms that only require login credentials. Another point to consider is identifying UI elements in your Robot scripts; you still need to specify HTML attributes for identification.

testRigor

testRigor is an AI-driven, codeless test automation platform, presenting a more efficient solution for automating your end-to-end tests. It allows manual testers to develop complex test automation without requiring coding skills and spend almost no time on maintenance.

The innovative feature of testRigor lies in its natural language processing capability that enables the creation of test cases resembling plain English sentences. It offers excellent scalability and advanced AI functionality, including generative AI test creation, simplified test creation and execution, and minimal maintenance. These features set testRigor apart from other tools in the market.

In the context of UI testing, testRigor allows you to use relative locations to identify UI elements, making the process more intuitive. For instance, a button present to the right of the search bar can be referred to as 'click on "Add to cart" to the right of "Search"'. However, you also have the flexibility to specify HTML attributes of UI elements, if preferred.

Unlike many tools requiring you to write explicit conditions to wait for elements to load, testRigor's intelligent design automatically waits for the element to appear. The platform supports conditional execution using 'if' statements and offers a library of validations for creating assertions, which are a crucial part of testing.

Moreover, testRigor provides comprehensive solutions for testing web, native and hybrid mobile applications, native desktop applications, and more. It integrates seamlessly with third-party tools for test management, CI/CD, and issue management, making it a robust choice for modern software testing needs.

Here is an example of a testRigor test case, mirroring the one shown in the 'end-to-end test using WebTest' section:

check that page contains "Amazon"
enter "Python book" in "Search"
click on "Search button" to the right of "Search"
check that page contains "Python Crash Course"
click "Python Crash Course"
check that page contains "Add to cart"
click "Add to cart"
check that page contains "Added to cart!"
click "Cart" at the top of the page
check that page contains "Python Crash Course"

It's quite impressive, isn't it? With testRigor, you even have the ability to combine different platforms within a single test. For instance, you can start a test in a mobile app and finish it in a browser. These capabilities position testRigor as a superior choice for end-to-end testing.

The tool has a lot more to offer. You can read more here.

Conclusion

Using different frameworks and utilities provided by the Pylons Project, you can create competent and effective applications. This post provides some recommendations for automation testing tools that you can use when writing and executing test cases. Remember, the ultimate choice of tool depends on your project and team's requirements.