Implementing Snapshot Testing In Tasty For Enhanced Test Tree Pre-processing

by James Vasile 77 views

Introduction

Hey guys! Today, we're diving deep into the world of testing, specifically focusing on how to implement snapshot testing within the Tasty framework. Snapshot testing, a super cool variation of golden testing, offers a lightweight and speedy way to add new tests. This article will guide you through the process, discussing the challenges and solutions encountered while integrating snapshot testing into Tasty's pre-processing test trees. We'll explore how to identify placeholders in your source code, rewrite files with the actual values, and manage multiple replacements efficiently. So, buckle up and let's get started!

The Challenge: Integrating Snapshot Testing with Tasty

To make snapshot testing tick, the main goal is identifying those spots in the source code where placeholders exist. Once found, these placeholders need to be replaced with the Show string representation of the actual value. Now, here’s the catch: if a file has more than one placeholder, you've got to replace them all in one go. Why? Because if you do replacements one test at a time, the line numbers might get jumbled up in later tests. It’s like trying to solve a puzzle where the pieces keep moving! So, what's the most logical approach here?

The initial thought was to use withResource to create an IORef. This IORef would then be passed to each test using an Option. The idea is to gather all the replacements needed and then rewrite each file just once during the cleanup phase of withResource. This sounds pretty neat, right? You wrap all the tests with withResource and you're good to go. But hold on, there's a twist.

The tasty-discover Dilemma

The real challenge arises when you want to use tasty-discover. As far as I can tell, tasty-discover doesn’t let you wrap all tests with withResource directly. So, what’s the workaround? Well, you can introduce Ingredients. Now, before you start thinking about reporters or managers, let me clarify: we just want to wrap the test tree with withResource. This means tweaking Tasty a bit.

The Solution: Enhancing Tasty with a TestWrapper

The proposed solution involves adding a third constructor to the Ingredient data type. This new constructor, TestWrapper, will allow us to wrap the test tree with custom logic. Here’s how it looks:

data Ingredient = ...
                | TestWrapper [OptionDescription] (OptionSet -> TestTree -> TestTree)

Along with this, a wrapIngredients function is introduced to apply the wrapper:

wrapIngredient :: Ingredient -> OptionSet -> TestTree -> TestTree
wrapIngredient (TestReporter _ _) _ testTree = testTree
wrapIngredient (TestManager _ _) _ testTree = testTree
wrapIngredient (TestWrapper _ fn) opts testTree = fn opts testTree

wrapIngredients :: [Ingredient] -> OptionSet -> TestTree -> TestTree
wrapIngredients ins opts testTree = 
  foldl' (\t i -> wrapIngredient i opts t) testTree ins

This function iterates through the ingredients and applies the appropriate wrapper function. The wrapIngredient function checks the type of ingredient and applies the corresponding transformation to the test tree. For TestWrapper, it applies the provided function (fn) to the test tree, allowing us to inject our custom logic.

Modifying Test.Tasty.CmdLine

To make this work, you need to tweak Test.Tasty.CmdLine. The original code looks something like this:

case tryIngredients ins opts testTree of
  ...

This needs to be modified to include the wrapIngredients function:

-  case tryIngredients ins opts testTree of
+  case tryIngredients ins opts (wrapIngredients ins opts testTree) of

By wrapping the test tree with the ingredients, we ensure that our custom logic is applied before the tests are executed. This is crucial for setting up resources and cleaning up afterwards.

Implementing the snapshots Ingredient

Now comes the fun part: defining the Ingredient that does the actual work. Let’s call it snapshots. This ingredient will use WithResource to manage the resources needed for snapshot testing. Here’s the code:

snapshots :: Ingredient
snapshots = TestWrapper [] $ \_ tt ->
  WithResource (ResourceSpec resOpen resClose) $ \getRes ->
    PlusTestOptions (optFn getRes) tt
  where
    optFn getRes = setOption (UpdatesRef (Just getRes))
    resOpen = newIORef []
    resClose (ref :: IORef [Update]) = do
      [...]

Let's break this down:

  • snapshots :: Ingredient: Defines the ingredient named snapshots.
  • TestWrapper [] $ \_ tt -> ...: Uses the TestWrapper constructor, taking an empty list of option descriptions ([]) and a function that transforms the test tree (tt).
  • WithResource (ResourceSpec resOpen resClose) $ \getRes -> ...: Wraps the test tree with resource management using WithResource. It takes a ResourceSpec that defines how to open (resOpen) and close (resClose) the resource. The getRes function provides access to the resource.
  • PlusTestOptions (optFn getRes) tt: Adds test options to the test tree. The optFn function returns a function that sets the UpdatesRef option with the resource.
  • optFn getRes = setOption (UpdatesRef (Just getRes)): Defines the function to set the UpdatesRef option.
  • resOpen = newIORef []: Defines the resource opening action, which creates a new IORef to store updates.
  • resClose (ref :: IORef [Update]) = do [...]: Defines the resource closing action, which performs the necessary cleanup, such as rewriting the source files with the updated values.

The resOpen function initializes an IORef to hold updates, and resClose handles the cleanup. The crucial part here is how resClose rewrites the source files with the updated values. This ensures that the snapshots are persisted after the tests run.

Writing Snapshot Tests

With the snapshots ingredient in place, writing snapshot tests becomes a breeze. Here’s an example:

tasty_snapTest :: Snap String
tasty_snapTest = Snap (reverse "foo") $ do
  snapshotHole

When this test runs for the first time, the source code is automatically rewritten to:

tasty_snapTest :: Snap String
tasty_snapTest = Snap (reverse "foo") $ do
  """
  "oof"
  """

From then on, every subsequent run checks if the output of `reverse