As are most questions worth asking, the answer is nuanced and could be boiled down to it depends. I’ll show you my way of thinking and where it’s coming from. Your goals and constraints are likely different, so the final conclusions can differ as well.
My basic assumptions are:
- I want the application to have the highest quality possible—instead of moving fast and break things, I aim to move slow and not mess up the production;
- I plan for long term—I don’t believe in starting from scratch, and I hope the things I build will be used in 10 years and beyond; and
- I care about developers’ experience—I want reliable tests, with possibly a fast feedback loop for developers. Both locally and on continuous integration (CI).
So keeping this in mind, let’s take a look at automated quality tools.
End-to-end
The name end-to-end (E2E) comes from the nature of the testing: we are checking the application from the frontend to the backend. The tests are interacting with the application through the graphical user interface (GUI)—so we are testing the frontend as it runs in the browser. At the same time, we have the server and database running on the backend—through interactions in the browser, we can test whether the backend behaves the way we expect.
E2E tests are meant to simulate user behavior, so the scenarios we test with them will be similar to what the user could do in the application. An example scenario for an online shop:
- Log in as a customer,
- Find a product,
- Add it to the cart,
- Go to the checkout page,
- Expect one product in the cart and a total equal to its price.
Because the interface you use for interacting with the application is a graphical interface built for the user, your application is always testable this way. At most, you will need to add some attributes to help identify elements of the interface you want to interact with. The complexity of the tests depends mostly on the complexity of the workflows you have in the application.
Example libraries that allow you to create E2E test:
- Cypress
- Playwright
- Nightwatch
- (deprecated) Protractor for Angular applications
Unit tests
Unit tests are so named because what we are testing is a single unit of code: a class, a function, or any type of entity defined by the framework you use. You test those units by interacting with the interface they provide, and by mocking parts of code that they use.
Unit tests test the code from the code level. Example test scenario:
- Create the order object and call it
testOrder,
- Add a product object to the order,
- Expect
testOrder.totalPrice, testOrder.totalTax, and testOrder.totalQuantity to match expected values
This type of test is closely integrated with your code. Depending on how the code is structured, the ease of writing unit tests will vary. It can range from rather easy to almost impossible. It can be especially challenging to add unit tests to existing code that was written without testing in mind.
Example libraries you can use to write unit tests in JavaScript:
Feature overlap
In a way, those two types of testing overlap completely: in both places, set up expectations for the code and make sure they are met. Anything meaningful that happens inside your code units will eventually find its way up to the user interface—and there you can check it with E2E. Besides that, most E2E libraries allow you to mock backend calls, or even isolate parts of your code—so you can precisely recreate very subtle scenarios.
On the other hand, there are complex user interface (UI) cases that cannot be tested in a unit test. Unit tests usually check only what is returned by JS—without calculating the whole screen with HTML, CSS, and JS in place. With unit tests, you cannot test whether a button is clickable on a given screen.
So, if unit tests have those limitations while E2E can cover everything that our unit tests does, does it mean we only need E2E in our application?
Downsides of end-to-end
There is an E2E test suite that I’m happy about—it’s relatively fast (350+ tests run in 15 minutes), stable (random failures happen about once per each five runs), and it does a good job of catching regressions. But even good E2E tests cost plenty of development time—when they are created, run and maintained. Let’s see a few causes for why it’s this way.
Complex set up
The tests I’ve mentioned require:
- an HTTP server that hosts frontend files,
- a backend server up and running, and
- two databases, preset with data that the tests require to run.
Thanks to Docker and containerization, it’s relatively easy to share the whole stack among developers and CI server.
On the top of the requirements listed above, CI introduces a few other moving parts:
- a CI server that starts jobs.
- CI agents that run the job—run in Docker containers as well.
- for some time, we had a CI agents coordinator that started and stopped the CI agent based on demand for the CI server.
In summary, to run E2E on CI, there are few layers of cloud instances: Docker containers running inside Docker containers—in short, plenty of things. Usually, you just need one thing to fail to get the whole test run to break. This creates false-positive failures, which—if they happen too often—will train the whole team to ignore failing E2E—exactly the opposite of the behavior we would like to see.
Slow in execution
Most E2E I’ve seen in my test suite takes about 5 to 15 seconds to run. Not too bad, but even on the lower end of the range, 350 tests would take half an hour to pass while running them one after the other. Total execution time can be lowered by running tests in parallel. This again brings few downsides:
- it complicates the setup even more.
- it may require some changes in tests to avoid collision if each test runner talks to the same backend.
Slow in writing
E2E and relatively slow in terms of writing too. While developing, the execution time we discussed above introduces delays in the developer’s feedback loop. Something like 5–15 seconds is not much, but those seconds add up, and it makes staying in the productive zone more difficult.
Strength of unit tests
At the same time, unit tests have quite a few important strengths.
Fast
Unit tests are very fast. Because you interact directly with code, you don’t suffer delays introduced by the:
- browser,
- application,
- backend server, or
- databases.
I maintain a suite of 3200 unit tests that run in about 30 seconds—only about 1/100 of a second for a test. At this speed, you can rerun relevant tests each time you change code and keep getting almost immediate feedback. This is something that helps a lot when you are doing test-driven development (TDD): when you write code only after writing a test that checks for the expected behavior.
Interactive specification
In a way, well written unit tests become an interactive specification—one that explains how the code is supposed to work and can be run to see whether those expectations are met. Plenty of challenges in coding are due to communication issues, and most of the communication happens in writing—especially the communication between past developers who wrote the code months ago and present developers who maintain it now.
For written communication to work, you need the reading to at least happen. It’s difficult to get others to read, especially when the communication happens across time.
If you compare ‘specification in comments’:
if (order.quantity < 0) {
order.quantity = 0
}
To the specification in unit tests:
it(‘should reset quantity to 0 if negative’, () => {
order.setQuantity(-1);
expect(order.quantity).toEqual(0)
})
I am much more comfortable trusting a future developer to notice conflicting changes when there are tests in place. Failing tests at least force you to see what’s up there, while comments can be easily ignored. Besides that, sometimes you are uncertain whether the comments are still up-to-date.
Helps you design units
Writing unit tests, and especially TDD, will impact the way you write code. As your units get bigger and take on more responsibility, the testing becomes exponentially more complicated. This will give you a slight but constant push towards keeping the responsibilities well distributed across various units of your code. Over time, this will add up to a visible difference in how the logic is structured. I wrote more about the impact of testing on the application architecture in another article.
What goes where
So, if I keep both unit tests and E2E, how do I decide what should be tested with which tool?
Smoke tests
Smoke tests are a crude test to see whether the application even starts. The name comes from testing hardware—if you connect your new device to power, and the smoke goes off, you don’t need to test any further. These tests are a perfect case for E2E—at a minimum, you want each of your pages to open successfully on the screen when the user tries to visit.
Happy path for the user stories
Happy path is when everything is in its right place, and we don’t have to deal with exceptions or errors. The stock is in place, credit card payment is accepted, and email address is valid. It’s good to have those cases covered by E2E because happy paths cover the main reason for the application to exist. Successful transactions are the reason why users go to online stores and are why the company built the application in the first place.
Painful bugs that often affect users
For each workflow that you can cover with E2E, there are hundreds of ways it can go wrong. That’s why I usually don’t dive too much into covering edge (error) cases with my E2E—that would be a lot of work. But for a subtle issue that was made through manual testing and got to production, it’s worth evaluating whether it should be tested with E2E to prevent this kind of regression from happening again. This way, you can avoid the risk of making a bad impression on your customers by making them suffer from the same issue coming back after it was fixed. And you implement additional test automatization in places that are clearly lacking coverage with manual testing.
Subtle details of implementation
There are many important yet subtle behaviors that would be very difficult to test directly from a GUI. For example:
- Rounding prices correctly when you apply discounts.
- Setting translation to a correct language based on a combination of browser settings, user data, cookies, etc.
- Weird cases that should never happen in your application in the normal user session. For example, data that was saved to
localStorage with an older version of the data structure, and you wish to make sure it’s migrated and works correctly in the current version.
Those cases fit perfectly in unit tests.
Everything else
If you do TDD, you are expected to test everything—and the only realistic way of doing it is unit testing.
Want to learn more?
If you are interested in learning more about testing, or other topics relevant for beginner programmers, you can sign up here to get occasional updates about my new content.
