Author: Alan Page

I was nothing short of blown away over the past few days, when some comments I made on twitter about UI automation caused a lot of folks to raise their eyebrows.

Here’s the tweet in question.

I’m not against discussions on the invalidity of the test automation pyramid.

If you don’t like it, you use whatever model you want as long as it suggests you write AS FEW UI TESTS AS POSSIBLE.

seriously – stop your infatuation with UI tests

— Alan Page (@alanpage) May 3, 2018

Feedback (blowback?) ranged from accusations of harmful blanket statements to lectures on how what I really meant was “checking”, not testing – with a handful of folks who seemed worried that the world I was describing was too scary compared to the world where they lived. Testing evolves at different speeds in different places, so the last point, at least, was expected. But I stand by the sentiment in that tweet.

The test automation pyramid is a model for thinking about distribution of your tests. Mike Cohn first (I think) wrote about it here. In my opinion (and observation), there is an unhealthy obsession in software testing with writing “automation” (where “automation” means UI automation – i.e. using tools like Selenium to manipulate the UI to test the system under test). A few folks recently have complained about the pyramid (“it’s not a pyramid, it’s a triangle!”; “There are more than 3 types of testing”, etc.).

“All Models are wrong, some are useful” — George Box

It’s a good model, with a lot of practical application. Two key takeaways from the pyramid model are:

• Write tests at the lowest level where they can find the bug
• Minimize the amount of top/UI level tests

I’m passionate about the second point for three main reasons:

1. UI Tests are flaky. This was true 25 years ago when I first wrote UI automation, and it’s true today. They’re just not as reliable and trustworthy as lower level tests.
2. Despite the fact that reliable UI Tests are difficult to write, we (industry) seem to think that UI automation is a reasonable entry point to the world of coding for “manual” testers. UI automation is a horrible way to start programming. Shell (or other) scripts to help set up test environments or generate test data would be a much better use of time (and achieve more success) than learning to code by writing UI tests.
3. UI tests are s l o w. This is fine if you have a handful of tests, but a huge issue if you have hundreds, or thousands of UI tests.
   Note – if you have a large amount of testing that can only be done at the UI level, that’s a big red testability issue you should probably address before investing in expensive testing.

Let’s assume for a moment that the problems with UI automation stability have been solved (companies like testim.io have used ML to make some strides in this area, and despite the entry path problem I mentioned above, there is improvement in automation tools and tester skills). If we go with this assumption, then point #1 – and possibly point #2 above are no longer an issue.

Point #3, however, is not solvable. Tests that automate the UI are slow. Way slow. Like a glacier stuck in molasses slow. I once wrote a UI based networking test to create a folder, share it, connect to it, write files to it, delete files, and then unshare the folder. That test took a little less than two minutes. Problem was, that I needed to test that process for every character possible in isolation (due to issues with DBCS code pages on non-Unicode Windows where details would fill pages of no longer relevant information). On Chinese windows, for example, this was (IIRC), somewhere near 8000 characters.

I wrote an API level test that tested the entire code page – including varying lengths of folder and share names that ran in under 5 minutes (and less than a minute for Western code pages). Of course, we still did spot checking (both exploratory, and via some UI automation), but testing at the level closest to where we could find bugs was the most efficient – both in proximity and speed.

Another view of tests I like is the size model from google. Rather than dwell too much on what makes a test a unit or integration test, think of tests in sizes – where tests of a certain duration are classified at different levels. This model works well (and solves the pyramid complaints I’ve seen recently), but it doesn’t have a visualization.

So – without further babbling, I created this alternate view – The Test Automation Snowman.

Use it, or ignore it. But I still beg you to consider writing far fewer UI based tests.

This is a brainstorm rather than a well thought out (or thought out at-all) blog post. I briefly considered a tweet stream, but wanted to give my few followers a break.

I’ve been thinking a bit recently about what it means to be productive on a software team – or how that productivity is viewed, tracked, or measured. I’m not going to dive into performance reviews, as those have a lot more moving parts and dynamics – but just the general view of how we view result – or output from any individual, and any organization.

Traditionally, “output” is a bullet point list of things-you-did. This is output – or production. It’s certainly measurable, and if someone has no output or results, there’s likely an issue. But there’s another kind of output that doesn’t get talked about enough. What about the person who makes other people better? The person who coaches, mentors, and finds ways to make people around them productive?

Jessica Kerr, calls this attribute ‘generativity’

generativity: the difference between team output and growth with me vs without me
> productivity: personal output #organizationalPsychology

— Jessica Kerr (@jessitron) December 20, 2016

I like the term, and have begun using it recently…which is probably why I’ve been thinking more about this over the past few days.

Some points to consider (and potentially discuss):

• Output is linear. I can produce 0 things, and I can produce n things. More things == more production. (Note that all produced things are not equal).
• Generativity can be positive or negative. I can help the team achieve more than the sum of their parts, or I can be an asshole and hold everyone back.
• You can be both Generative and Productive. While you can be Productive without being generative, I don’t know that it’s a good thing to be Generative and not Productive. More on this below the bullets.
• Generativity requires leadership – this means that generative people tend to be more experienced (and that less experienced people may focus entirely on output).
• For those who care about measuring, Productivity is much easier to measure than Generativity.

I don’t know of a system that’s done a decent job figuring out how to measure the non-tangible output (Generativity) of team members. I’ve seen highly generative (IMO) people viewed poorly because their output was difficult to measure (which is why I included the third bullet above), but I think generative people are critical for success in most teams. Of course, the best generative people I know also produce some measurable results as well, so perhaps that combination is necessary.

• Quote I’m pondering: “Anyone operating with a theory of leadership that assumes that experts know what is best, and that then the leadership problem is basically a sales problem in persuasion, is in our experience doomed at best to selling partial solutions at high cost” – Heifetz, et al in The Practice of Adaptive Leadership.
  I’ve been (re) studying adaptive leadership quite a bit recently, and hit home on a few of my poor-leadership experiences.
• What I’m reading – The Five Levels of Leadership, but John Maxwell. I’ve read one of Maxwell’s other books, but I’m finding this one really valuable in forming a model in how leaders grow into better leaders.
• I enjoyed (and shared with my team at Unity) this article on Continuous Improvement and Feature Branching. Well worth the read.
• There was a time when PSP (Personal Software Process) was (well, not popular, but far more interesting than it should have been). I thought of PSP when I read this article on tracking time, but completely without the icky feeling of bean-counting that PSP always gave me. In fact, I’m going to try tracking what I do for a month purely for self-reflection on my own prioritization. Definitely interesting.
• Finally, for all of my readers who enjoy my love of wine, there’s this article fro National Geographic – Our 9,000-Year Love Affair With Booze