VELOCITY, FRAMEWORKS & PLUGINS
In this chapter, you will learn:
- The thinking behind Velocity
- Changes to the development workflow
- How it works
- Choosing the right framework(s)
WHAT IS VELOCITY?
Velocity was born out of a meeting between Mike Risse, Adrian Lanning, Joshua Owens, Abigail Watson, Robert Dickert and myself, each of us having played our own roles in the Meteor testing story. When we came together, we proposed the following goals for a unified testing framework:
- Simple installation
- An easy Meteor-way workflow
- A one stop shop for all Meteor testing
- Flexible for the community to evolve
And that’s exactly what happened. The Meteor Development Group saw the above and deemed Velocity as the official testing framework for Meteor 1.0. See the video of our intro talk here.
Simple Installation
Assuming you want to test using Mike Risse’s (excellent) Mocha-web:
$ meteor add mike:mocha
You can now get started, or you can add more test frameworks, plugins and reporters as you will shortly see.
It so happens that mike:mocha is a test framework that has a dependency on both velocity:core and on velocity:html-reporter, so you don’t need to install any other packages.
A Reactive Test Runner
When installed, the velocity:core package monitors any files that change under the /tests directory in your project and/or detects when Meteor restarts. When one (or both) of these things happen, the package informs the test framework(s) to re-run the tests. Once the framework(s) have run their tests, they report back to Velocity which publishes results. Reporters observe these results and make them accessible to you. All of this happens reactively whilst you are developing.
If you’re familiar with Karma, you’ll notice it has some similarities to Velocity. In fact, Velocity’s architecture is an evolution of RTD, which is also a test runner and was built on top of Karma. Velocity, however, was written from the ground up with Meteor’s inner workings in mind.
Unified Testing
Prior to Velocity, the main testing players were TinyTest, Mocha-web, Laika and RTD, and you were left to choose between them, oftentimes having to give up one feature to gain another. With Velocity, you no longer have to make that choice and can use a combination of frameworks. Once you have opted to test with Mocha-web for integration testing, for example, you would be able to simultaneously include Tinytest for package testing.
Also, plugins that were only available in one framework, such as linting or code coverage, can now be shared. These are big wins as they allow you to collect and utilize the best possible toolset for your project.
Extensible Framework
Since Velocity is the test runner, it allows package authors to write separate frameworks and use a common API amongst them. The end product is a consistent user journey for you as the developer.
Much like many Meteor packages are wrappers around other NPM packages, Velocity frameworks are currently wrappers around existing testing technologies- with a lot of plumbing to make them work reactively.
There are a lot of tried and tested JS testing frameworks that are potentials for porting to Meteor, like CasperJS and Chai, and even non-JS frameworks can be integrated, like JMeter and more.
The scope of this book won’t cover how to write Velocity frameworks and plugins, but hopefully you can see the potential that Velocity has opened up for testing Meteor applications.
If you would like to get involved, you can join the Velocity-core group.
GETTING STARTED
Please note: This book is being written while Velocity is still under heavy development, and as such, Velocity is buggy. As a member of the Velocity-core team, I’ll be fixing as many of these errors as possible, and they should be resolved by the time this book is completed. Please do report any issues you encounter.
The New Testing & Development Workflow
Let’s start with the Meteor TODO’s app in Meteor:
# Create the example and go into it
$ meteor create --example todos
$ cd todos
# Add the Mocha testing framework from Mike Risse
$ meteor add mike:mocha
# Start testing (and coding)
$ meteor
You’ll see the app running on http://localhost:3000 and you’ll also notice a little dot in the top right corner:
Clicking this dot will reveal the following screen:
Since you have just started, you don’t yet have any tests. The html-reporter has a feature that detects the non-existence of test files and allows you to create sample tests from that framework. When the “Add mocha sample tests” button is clicked, the following files should appear in the project directory:
/tests
/tests/mocha/client/sampleClientTest.js
/tests/mocha/server/sampleServerTest.js
Notice that this framework has opted to place all its tests under the /tests/mocha directory.
You will also notice the html-reporter changes to this:
The results of the sample tests are displayed, and here they are shown as passing. Now you can collapse the reporter by clicking the dot again, as seen here:
The green dot in the corner is letting you know that all the tests are currently passing. Let’s see what happens if you break a test. Edit the file sampleClientTest.js, change the line chai.assert.equal(5,5); to chai.assert.equal(5,6); and save it. You should quickly see this:
The dot is red and pulsating, letting you know there’s something wrong. Clicking it will show the details of the failure:
The failing test is reported with the stack trace.
Now, for fun, let’s return the test back to chai.assert.equal(5,5); and save it whilst the html-reporter is open. The report will reactively return to a green state.
This is how Velocity adds testing to the Meteor development workflow and makes it easy to work with test-first workflows such as TDD (Test Driven Development ) and BDD (Behaviour Driven Development). Although you are using the html-reporter here, there is nothing stopping you from switching to another reporter. At this time, not many other options exist, but you can expect some to surface in the near future such as audible reporters, system notifications, and console printers.
The Html-Reporter Buttons
Additional information to help users better understand errors and tests is made available through the buttons on the left.
“Show passing tests”
By default, the reporter shows an aggregate pass result if all tests are passing, and it only expands the failing tests. This button opens all the results at once as shown below:
“Show logs”
Velocity exposes an API for frameworks to post their logs. This button may reveal more details about the error if a framework uses Velocity’s logging feature.
“Show files”
Velocity watches files under the /tests directory on the behalf of frameworks. This button shows you what Velocity has detected and is useful for tracking down issues with tests not running as expected, such as a file not being named as a framework expects it to be.
“Show iframe”
Mocha is an integration testing framework. The client portion of this integration testing runs inside an iframe. This button reveals the iframe. This is also a peek into the inner workings of Velocity. To understand the need for the iframe, you need to take a peak under the hood.
UNDER THE HOOD
Mirror Mirror
Tests are necessarily destructive. That is, in the setup, execute and verify stages, they can clear databases and add or remove data. For tests to be able to do this without being affected by user actions, and for a user to continue being able to develop an application uninterrupted, Velocity creates a mirror of the running application and tests using this mirror.
The mirror is a physical copy running an entirely separate Meteor command on a different port with a different database. The copy is made through an rsync process that happens whenever the main app restarts. This in turn triggers the mirror app to restart.
Frameworks that require client testing and need to access the mirror are currently using either iframes to run the tests on or are instantiating a browser, headless or otherwise.
Mirrors are another area that are currently in flux within Velocity. The concept is valid, though its implementation is not the final solution.
Lifecycle and Collections
Velocity has a lifecycle that makes some heavy use of collections. The lifecycle works as follows:
- Velocity and installed frameworks are packages that start with your app.
- Frameworks register themselves with Velocity and provide options. A particular option worthy of note is the regex used to match files in the
/testsdirectory. - Velocity starts a mirror or mirrors (based on the framework options above) and updates metadata about each mirror in the
VelocityMirrorscollection. Some frameworks wait for this information to be present before they commence testing. - Velocity’s file watcher monitors the
/testsdirectory and inserts a document containing the path, as well as the target framework, intoVelocityTestFiles. - The mirror’s source files are synchronized from the main app, and any fixtures are copied to the mirror. In some cases, the test frameworks copy the tests themselves into the mirror. These changes cause the mirror to restart, since the mirror is a separate Meteor application that has its own file-watching capability.
- Frameworks subscribe to or observe the
VelocityTestFilescollection and filter documents that have theframeworkfield set to the framework’s name. If any file additions/removals/modifications happen, the framework may choose to run only the changed tests, or it may re-run all of them. In either case, when the testing run is finished, the frameworks post their results back to Velocity. Frameworks may also choose to post their log entries to Velocity. - Velocity inserts the results and other useful metadata (such as the framework name and any exceptions) into the
VelocityTestReportscollection. It also updates theVelocityAggregateReportscollection which is used to store the completion status of frameworks. - Whilst the framework ran the tests, any logs that were submitted to Velocity are added to the
VelocityLogscollection, again with relevant metadata about the framework and test run. - Reporters subscribe to or observe the
VelocityTestReports,VelocityAggregateReportsandVelocityLogscollections and present the information they need. - A user changes a file in the application code or the some test files. The cycle repeats from step 4.
Take-Home Notes:
- Velocity is an OpenSource initiative by the community for the community
- Velocity is supported by the MDG and is the official testing solution for Meteor 1.0
- Test frameworks run the actual tests and they use Velocity as their core
- Mirrors are a copy of your application, running on a different port
THE COMMUNITY
There are three players in the Velocity community: the core team, test package authors, and users. The core team creates and maintains the core packages to support package authors, whilst package authors create and maintain their own frameworks and plugins. The users benefit!
The Core Packages
The Velocity GitHub organization is where you will find the core packages. These are:
velocity:core
This is the core test runner. All test frameworks have a dependency on this package.
velocity-ci
This package provides continuous integration support for Velocity. It’s an NPM module that launches Velocity and runs the client tests using PhantomJS.
velocity:html-reporter
Adding this package provides you with an in-app reporter of test results, logs, and other useful information. You can see this in the Getting Started section of this chapter.
The Available Test Frameworks and Plugins
Below is a list of the currently available Velocity packages. These are frameworks you can use today.
You’ll notice there are two types of unit tests in Meteor: In-context and isolated. Isolated unit tests are true unit tests in that the application code or system under test (SUT) is loaded into a VM without any other code. In-context tests are different. The SUT is at unit-level, however the entire Meteor context is loaded. Whilst this is adequate for most needs, it has the possibility for error and the purists will have a thing or two to say about that!
mike:mocha
TDD, BDD Integration, In-Context Unit
This test framework uses Mocha and Chai, and it supports both client and server-side integration testing. It requires a mirror and iframe for client-side integration tests.
STATUS: Most complete framework. It does one thing and it does it really well. Combines mocha with meteor and gives you server and client side integration testing.
sanjo:jasmine
BDD, Integration, In-Context Unit, Isolated Unit
This test framework uses Jasmine 2.0 and features client-side integration testing with server-side unit testing. The unit-testing portion utilizes auto-stubbing. Support for client-side unit testing and server-side integration testing are planned. This framework also requires a mirror and iframe for client-side integration tests.
STATUS: Most ambitious framework. It addresses the extremely difficult issue of isolated unit testing and adds auto-stubbing to it! As such, it requires a more advanced understanding of testing to use at this early stage.
clinical:nightwatch
End-to-end Browser Automation
A CLI-based Nightwatch.js wrapper, this framework uses Selenium webdriver and supports SauceLabs & BrowserStack. Nightwatch.js is a tried and tested UI testing solution and sports a proprietary syntax for defining test cases.
STATUS: Runs as a shell script and reports results back to Velocity. Doesn’t use Velocity’s test cycle . Not currently compatible with the velocity-ci command.
nblazer:casperjs
End-to-end Headless Browser Automation
A testing framework that uses the popular CasperJS navigation and scripting utility. CasperJS can use either PhantomJS(webkit-based) or SlimerJS (gecko-based) headless browsers. This framework can be used standalone or used within other test-frameworks such as meteor-jasmine or meteor-mocha-web.
STATUS: Ready since it wraps an existing proven technology and is integrated into the Velocity architecture.
numtel:velocity-tinytest
TDD, Integration, In-Context Unit
Whereas TinyTest was designed for packages, this framework allows you to use TinyTest against your application directly. This means you can place files under your project /test directory instead of inside packages.
STATUS: Very new framework that creates a new testing approach using the existing familiar TinyTest.
xolvio:coverage
Code Coverage
This is a plugin that instruments the mirror code using Istanbul and provides code coverage reports for any framework that uses a mirror.
STATUS: Still in alpha, needs a 1.0 update and rigorous testing with user apps.
velocity:meteor-stubs
Unit Testing Add-on
Contains a set of stubs for the core Meteor objects that are used both in the server and client.
Status: Stable. Currently only Jasmine supports these stub files.
Frameworks and Packages in Progress
These are currently being developed and will soon be ready to join the available frameworks:
spacejamio:munit
TDD, BDD, Integration, Package Testing, In-Context Unit
Munit is an existing test package that is being ported to Velocity. It extends the official yet simplistic TinyTest framework and augments it with SinonJS and Chai. There is also talk of bringing Mocha and Jasmine support into this framework.
xolvio:webdriver
End-to-end Browser Automation
The vision behind this framework is to bring tight integration between webdriver browser automation and the Jasmine or Mocha test frameworks.
xolvio:cucumber
ATDD, BDD, End-to-end Browser Automation, Integration, Unit
Cucumber is an industry standard framework for specification-by-example testing. This package brings cucumber to Meteor, allowing you to write features and step definitions using Gherkin syntax coupled with any testing framework.
Other Planned Frameworks and Plugins:
These are items that are on the horizon that are known from talking with Velocity team members and planning the roadmap:
meteor-jshint
Static Code Analysis
JSHint is a popular code analysis package in the javascript world. It analyzes the code based on rules defined in configuration files and fails if standards are not met. This plugin is a Velocity port of JSHint.
browser-launcher
Velocity Extension
This extension allows you to run your integration tests using real browsers instead of an iframe or PhantomJS.
console-reporter
Velocity Extension
This extension reports the status of tests via the console or terminal. This is useful if you do not wish to clutter your app with the html-reporter.
You can browse the Velocity roadmap on Trello for a more detailed peek into what’s planned and what’s in progress.
CHOOSING A FRAMEWORK
Today there are a handful of Velocity-compatible frameworks to use, some overlapping in function, others distinct.
The short answer
You should use either meteor-mocha-web or meteor-jasmine if you want to get reactive in-app reporting goodness today. Wasn’t that short?!
The long answer
As you might expect, this is not a clear-cut scenario, and you have to determine what type of testing capability you need. You will recall from the test-boundaries section in the Fundamentals chapter that an application is made up of small systems (unit boundary-level) that work together to create larger systems (integration boundary-level), which work together to create even larger systems (end-to-end boundary-level). In theory, all you need is a framework that support all the levels and their nuances. Sadly, that framework doesn’t exist (yet), so you need to identify the systems in your application that need testing the most and the framework that will provide the means to test those systems at their boundary-level.
Below is a list of system types typically found in applications as well as the types of tests that are suited for them. See if your application has these systems to get a feel of the type of testing needs your app has.
Contains Custom Algorithms
Algorithms have a high number of execution paths based on the state that enters them. Well-designed algorithms will do one thing and do it very well. This means you want to focus your SUT to be at the unit level, so you’ll need a framework that supports unit testing. If you’re using a library that provides algorithms as opposed to writing your own, then you’ll probably need to apply integration testing.
Uses External APIs
Twillo is a messaging service that allows people to send SMS messages. The integration of Twillo would typically start with a Meteor method call, which would send Twillo servers requests via REST or similar. The SUT boundary here can be drawn between the Meteor method call and the REST call being made at the back-end. This is an integration boundary and would be best tested with integration testing frameworks, as you want to be sure you’re exercising the API correctly.
Uses UI Widgets
Javascript includes are a popular means of adding functionality to websites, such as UserVoice, which allows your site to collect feedback from your users. These integrations are usually best tested with a UI testing framework, as you want to ensure they appear on the page correctly and a user can interact with them.
Has a Complicated UI
Counterintuitively, a complicated UI would benefit just as much from unit testing as it would from UI testing. This is because there are typically a high number of heuristics in complicated UI’s. Consider a drag & drop application that deals with many boundary cases. You would want a combination of unit tests and UI tests to make sure you have good coverage.
Is Mostly Made from Custom Packages
If all the code you write is in package form and the main app is a collection of packages, then you’ll likely want to use a framework that supports package testing. It is possible, however, to structure your packages in such a way that they are testable using an integration or UI framework. You’ll see how to do this in the Testing Packages chapter.
Still confused?
Even the most seasoned testers don’t always know where to start, and sometimes a test may start at the UI and change mid way to an integration or unit level. It’s important to know that the above are guidelines to get you started with a test. The most important part of testing is to actually write tests and reap the benefits. So even if you do draw the boundary of your SUT at the integration level when it could be done better at a unit level, know that both the code and tests will evolve and both will require refactoring over time. As you learn more about testing in this book, you will develop the skills required to refactor tests. You will learn how to practically shift the focus of the SUT up and down the boundary levels as needed, and you will learn the “smells” that tell you when you should reconsider your approach.
Take Home Notes
- There are core packages, test frameworks and test plugins
- Test frameworks and plugins are created and maintained by package authors
- Some Frameworks are stronger than others at specific boundary levels
- Even if in doubt, start writing a test knowing that you can and probably will refactor with time
