Coding with TestProject

This is my third post on TestProject. The previous posts are Getting Started with Test Project and Experimenting with Mobile Automation.

The last thing I wanted to explore a bit with TestProject is directly writing some code for some (potentially) more advanced testing. Currently, the TestProject SDK supports Java, but SDKs for JavaScript, Python, C#, and Groovy are under construction. Fortunately, Java is a language I’m at least partially competent in, so I feel comfortable giving it a shot.

The Application Under Test

To keep things simple, I just used the TestProject Example page at

To start, I cloned the latest samples from GitHub, and grabbed the TestProject Developer SDK. Note that there’s a link on the bottom of the SDK page link above that allows you to copy your developer key. You’ll need that to get started as well.

The next steps are documented well in the for the web project (TestProject Java SDK – Quick Start for Web). Short version is that to get the samples to run, you need to make sure the build.gradle file knows where the SDK jar file lives, and add your developer key to the test runner java file.

I could write a series of blog posts with details on gradle and java build command lines, but I probably wouldn’t answer questions here as fast as a google search. Once you can build the test and test runner you can upload the test to TestProject using the New Test Wizard to upload your .jar file.

For this blog post, I’ll be playing with the BasicTest sample. I cloned the sample repo, and built it as is.

My Test Package now shows up on the Web page (note that I temporarily changed the test name to match the full namespace name of the test – and that the package includes the Extended Test from the sample project as well. From here, I can add tests and run them the same as we did with the Recorded / Designed tests discussed in the previous posts.

Coded Automation

The coding constructs in TestProject should feel pretty familiar to anyone who’s written Selenium or a derived language before. You instantiate a web driver, tell it to do things with elements, and write some verification code.

The entire basic test is this:


I think it’s pretty straightforward to read – even if you’re not a “coder”.

These two lines:

…start WebDriver and navigate to our page-under test.

The next two lines initialize the Page Objects on the profile page and fill out the fields there. (please – if you write web automation, use Page Objects. TestProject gets serious points from me for including Page Objects in their sample) – and then login to the page.

And finally, there’s some verification to ensure the page is saved.

This is a sample, but it’s worth mentioning that the oracle for this test isn’t great.

As testers, we’ve all seen pages that display the proper text, but don’t do _everything_ we expect. On one hand, I’d lean toward a more robust verification – OTOH, writing a massively complex verification function is part of why I get scared of a lot of UI automation.

Play Time

Cool so far, but let’s muck with this a bit and think about how we can use the power of the computer to do a bit more testing.

In a much longer blog post, I would create an array of structures containing names, addresses, etc. and loop through them ensuring that all could be entered correctly. I’d pick canonical examples from different locales to ensure (or help with) globalization, and probably add a handful of contrived examples as well (sql injection, cross site scripting, etc.). Note that I did try logging in with a user name of <script>alert(‘ruh-roh’);</script> with no problems.


First thing to do is wrap the whole function in a loop. Other than adding a loop construct, I’ll track a count of failures (so I can just return pass / fail once).


I wrote a tiny helper function to create a random string from characters in the ISO Latin Alphabet. I could just as easily used UTF-16.

I’ll use this to create a random first name and last name of (arbitrary) lengths 10 and 20.

At this stage, I have a test that loops, trying different names…but there’s one thing interesting on the following page I want to investigate.

The web page grabs the name I entered on the previous screen and displays it. I want to write a test to verify that the name is correct.

Probably the easiest thing to do at this step is right click on the text, choose Inspect (assuming Chrome), and then either grab the id and find the text element by id, or choose Copy and Copy XPath (I find XPath to often be ugly, but in this case it’s very clean).

Based on that bit of investigation, I added the following bit of code to

It grabs the full text of the greeting element, and returns true if the string contains the name. Pretty darn straightforward.

My slightly more interesting basic test now looks like this.

Of course, there are thousands of other test and verification options open at this stage, but to me, the TestProject stuff has been extremely easy to use and highly intuitive.

But Wait… Addons

I’m not going to be able to do the idea of Addons justice, but to me, this is a big differentiator with TestProject.

Addons are sharable components that perform a specific task. My dumb name generator could be spruced up to actually draw from a database of real names and be uploaded as a shareable component that anyone can use. I expect, in fact, that as TestProject grows, that the number of Addons – and their value will increase immensely.

Also note that the jRand addon from the TestProject Community addons is a nice shortcut for a lot of this particular test scenario. It generates a random (and usable) value for just about any sort of entry field you could think of, including sentences, paragraphs, birthdays, addresses, altitude, credit card details and more.

There’s definitely a lot of potential for good code reuse with the addons concept.

And Finally…

I covered a lot, but not nearly as much as I should have. If you play with the coded tests in TestProject, I suggest going through the docs on github, and not my walkthrough above, because I probably forgot just enough steps (some on purpose) to throw you into a pit of despair.

But I do suggest giving this framework a shot. As always, remember that my suggestion is to automate at the Web UI level only to verify things that can only be found at the UI level. Used correctly, I think this automation framework and supported web tools can give a lot of value to a lot of software teams.

Five for Friday – August 17, 2018

I made a quick trip to Iceland this week, where I still had time to come across a few interesting things to share.

Experimenting with Mobile Automation

As I type this, I realize I have a mildly embarrassing gap in my testing career. I don’t recall ever writing any automation for mobile applications.

Maybe that’s not exactly true – nearly 20 years ago, I wrote some automation in C++ to test Windows CE, and some applications that ran on it…and technically that was an OS used to power Windows Phone for a while (among other devices) – so I guess the gap is that I have never played with Appium. We used Appium when I worked on MS Teams for (some?) mobile automation, but I wasn’t directly involved.

Today is my chance.

TestProject Experiment Number Two

After my success playing with recorded test automation (I really want to call this recorded test assistance) for web, I decided to try the same thing using TestProject to test a mobile application.

Getting Started – The Setup

I was mentally prepared to install the Android SDK (or at least ADB), and spend a bunch of time getting a connection set up, but holy crap – was it always this easy?

I already have the TestProject agent running on my computer (for running the web tests), so all I had to do was select Android as my target, and everything else just worked.

Overall, the setup wizard is extremely easy to follow, and worked amazingly well. My phone was already in developer mode, so all I had to do was configure USB debugging, (optionally) disable a few animation features (for testability), and the TestProject web app connected to my phone and showed a mirror on screen in seconds. I thought for sure that I’d spend an hour or more getting this going, but I was wrong.

Granted – I don’t have much experience here other than half a dozen sessions debugging over ADB, so this sort of thing may be the norm now, so all I can say is setting up a remote development session can’t get much better than this.

The wizard is pretty much the same as the web wizard with one cool addition. Once the connection is configured, I can browse the device packages to choose which one I want to test:

I decided it would be fun to play with some automation for the Twitter app. There’s an option to reset the app before starting, and there’s even a default step added to reset, but for my experiment, I deleted that step.

As in my tests on, I’m just going to record selecting the top 4 elements from right to left and see what happens.

Note that the steps here are able to pull more friendly names from the package, so the test steps are pretty readable as is.

Creating Tests

Using the same test strategy as I did with web tests, I then added validation to ensure that the correct screen showed in the right place.

For each screen, I just used the “Validations” design element to confirm the existence of a particular control. There are probably better ways to do this, but for my quick test, it’s fine.

Similar to what I wrote for web tests, I have some actions with validation.

The Element Locator

If you’ve written automation in Appium or Selenium (or other frameworks that rely on IDs, XPaths, etc.) you are already familiar with the game we play to find elements.

One of the other handy features of TestProject is a simple, but useful element locator that makes it easy to discover if the element you’re working with is currently visible. In the example below, I’ve clicked the Find button in the test designer to show me the Twitter Home button.

Pretty cool!

As a nice aside, notice that the screenshot above (unlike the other screenshots), my phone screen fills the emulator (vs. the other screenshots where it fills 70% or so of the emulator screen). The TestProject team fixed the issue practically moments after I reported it.

A Few Words About Automation

Let me be the first to point out that this isn’t particularly great automation, and that the validation I’m doing probably isn’t what needs to be tested at the UI level for Twitter.

When you write UI automation, use it to find bugs that can only be found at the UI level. If it’s a special visual effect that can only be detected via human eyes, then use human eyes to validate the effects.

Of equal importance is your strategy. Don’t start by deciding what to automate. Instead, start with what you want to test – and then use automation to assist your testing. I’ll refer again to my article on the TestProject site that expands on these ideas in more depth.

All that said, it seems to me that I could very well use TestProject for proper targeted UI tests where they were needed, and that it would serve me well.


I’m still surprised by how easy it was to get up and running testing a mobile app, and the test steps were pretty straightforward and intuitive.

One of the areas where mobile automation really helps is in making sure applications work across a variety of devices. I only have one phone, but TestProject does support working with multiple connected devices to a single desktop. You just connect a USB Hub with a suite of devices, and you have an instant test lab. It’s also worth mentioning that an account can have multiple agents assigned to it. This enables executing tests not only on your desktop (and devices connected to it), but also on remote agents (and devices connected to them) in different locations anywhere in the world. It could be fun to see if TestProject could target a big virtual and real mobile test farm like BitBar, but there’s plenty of flexibility and potential right out of the box.

As with the web tests, I’d love to see a way to view / edit the underlying code, but my curiosity was less with mobile apps – perhaps because my knowledge of Appium is much less than my knowledge of Selenium.

Overall, it seems like a pretty usable and robust choice for most mobile automation tasks.

One more investigation to go where I’ll see how TestProject does with some coded tests.

Five for Friday – August 10, 2018

  • Jerry Weinberg passed away earlier this week, and I’ve taken some time to re-read parts of many of his books, where I’ve been constantly reminded of his leadership knowledge. I grabbed this quote from Becoming a Technical Leader to share
    People don’t become leaders because they never fail. They become leaders because of the way they respond to failure.
  • I spent a chunk of my early career in globalization testing (which reminds me that my last wikipedia edit was to fix this section of the software testing article). Netflix, in yet another valuable post, describe their approach to pseudo-localization.
  • We all knew this was coming. Here’s what you need to know about California’s new data privacy laws.
  • Modern Testing Principles overlap a reasonable amount with the Three Ways of Devops – which is probably why I found this article on the Three Ways of DevSecOps interesting.
  • And finally, speaking of security, here’s something to remind you how much we all suck at software security. How I gained commit access to Homebrew in 30 minutes.

Want to be a Modern Tester with me?

I posted this other places, but forgot to put it here. If you, or someone you know is interested an capable of embracing and driving Modern Testing principles with a fantastic group of developers, producers and other technical staff, and are in (or open to moving to) Austin, Texas, check out or share this position on my team.

Here’s the description

Apply on the site but drop me a note on twitter (@alanpage), or email me (alan at if you apply, and I’ll fast track the application.

Getting Started with TestProject

I wrote an article on the perils of Test Automation (Something’s Rotten with Test Automation) for TestProject last year. I’ve had a bit of time since then to look into their product since then (currently in beta), and thought it was worth sharing a bit about it, as I think it’s a pretty good product.

This post is a walk-through of my experience creating a simple web test. I plan to write a few follow-up posts to cover mobile device testing and finally a bit on my experiences playing with their Java SDK.

Please note: I am accepting a small stipend from TestProject for posting this, but the article ideas and content are entirely mine, and I would not write this if I didn’t endorse the product.

What is TestProject?

On the surface, TestProject is yet-another web and mobile automation tool. It supports recorded tests (but better than any test recording frameworks I’ve used before), as well as coded tests, which can be uploaded and run from the TestProject web site.

Yet Another Automation Framework?

We’ve had a recent resurgence of web automation frameworks, but for reasons I’ll describe in this post – as well as in the next two follow-ups, TestProject has a lot of promise, and a lot of potential. It encapsulates Selenium and Appium (no separate download needed), will eventually support at least five programming languages, and makes it extremely simple to share and schedule tests.

Better yet, the price is right. For most users, it’s FREE. The free package includes 5 agents, SDK support, 30 days of history, and some other goodies. For those who need the Professional version, it’s $8 a month per agent – which is still ridiculously affordable.

But…The Weasel Doesn’t Like UI Automation

If you’ve read my posts, seen my talks, or followed me on Twitter, you’ve likely seen me speak out against UI automation. To be clear, I’ve spoken out (and will speak out) against bad UI automation, and against trying to do too much testing in the UI.

A few targeted tests that verify critical parts of the UI are not just something I support, but a pretty damn good idea. I think TestProject is a good solution for this task for many teams.

Recorded Tests – A Walkthrough

I have always been (very) leery of recorded tests, but somewhere when I wasn’t paying attention, the tools got pretty good – or at least much better. It’s still possible to create some pretty bad tests via recording tools, but I’ve learned that at least it’s possible now to create reasonable tests with recording tools.

To try things out, I made up a (slightly convoluted) test scenario where I wanted to make sure that all of the menu pages on went to the right page.

My “test plan” is to click each link, and make sure it goes to where it’s supposed to go (probably by verifying an expected element exists).

Here’s a quick walkthrough of the setup:

Step 1 – Create a new project

Step 2 – Create a Test

Pretty straightforward – click the big blue “NEW TEST” button – select

At this stage, I could go straight into “Design” mode – but we’ll go here later after we get the basics done via the recorder.

The TestProject Agent

One thing I really like about TestProject is the TestProject agent – it’s cross-platform (I’m testing it now on my Ubuntu system), and it encapsulates Selenium or Appium as needed for the target platform. Once the agent is up and running, the TestProject web site will see it’s running and connect.

When the recorder page comes up, it looks (something) like this:

I moved the recorder window so I could see the menu items, and then clicked them from right to left. That gives me a not-very-readable output like this:

I can play this back, and it runs well. But at this point, there are two BIG problems. The first issue is that there’s no validation. We’ll add that in a few minutes, but first, I’d like to document the “Click A <Link>” names and make them a little more debuggable.

My preference would be to rename these, but that’s not an option yet in TestProject – but to aid in debugging, I can at least add a comment.

One cool thing to note here is that I can view and edit the current selector, and select by ID, Classname, Tagname, Name or (the default) Xpath. Consistently named labels are essential for writing maintainable automation, so this is something that I think will be pretty handy.

I won’t show it here, but I ended up changing the selection method for each validation, and all worked exactly as I expected.


I found creating the validation step to be pretty straightforward too. Just create a new test step, and have it verify that a specific element contains specific text. For those of us who have written coded web automation, it’s pretty easy to picture what Selenium code is running under the hood. I also think that this is indirectly a pretty reasonable way for non-coders to learn how Selenium works. Probably not TestProject’s intent, but I think there’s potential there.

I can run my tests from here, and the green bar on the left side shows that all of my test steps passed.

From here, I created a Job on the TestProject site, set the job to run on Chrome and Firefox, and added this test to the job.

Test Results

Not super exciting results, but certainlynot bad results for 30 minutes of “test development” with a new tool.


I mentioned above that I gave every action a one second timeout (note that it doesn’t wait or sleep a second – one second is just the length of time I’ve instructed the framework to wait for an element.

What happens if I decide the timeout should be 2 seconds -or half a second. It’s preferrable to change things like this in one place, and fortunately, TestProject allows you to add parameters. I added one called my_timeout, and now, if I ever need to change it, I can change the timeout in once place.


When you write tests, it’s a good idea to test what happens when they fail. I changed the “Validate “About Angry Weasel” test to look for the test “About Angry easel”. As expected, the test failed, and gave me reasonably actionable information to debug the failure from.

Final Notes (for now)

Overall, I didn’t find much to hate. Some things I’d like to see in future updates include:

  • Ability to rename steps for easier debugging
  • There are a lot of “are you sure you want to do this” type dialogs that I already feel a need to suppress
  • Given that I can mentally “see” what the underlying code looks like, it would be nice to generate code as well for more subtle tweaks.

As I said in the intro, I have a few more posts planned to walk through more of what I learn while learning the tool, but happy to hear your feedback on the tool or on my approach in using it if you have them.

Specialists and Bottlenecks

A few episodes ago on AB Testing, we talked about the Theory of Constraints and related topics. A lot of us frequently see “QA” or “Test” as a bottleneck, even on Agile teams – where several changes may be waiting for signoff from the designated quality or testing specialist before being deployed or marked as done.

If you listen to the podcast, you know I’m against this level of specialization. I’m all for people who have specialization – but things fall apart if they’re the only one who can do a “thing”. I like to imagine the extreme case of specialization where one person just writes design docs, one person writes code, but only internal functions; because another person writes external APIs, and another person tweaks all of the code for performance, and another person writes test cases, and another person runs them; unless they’re automated, then it’s another person…

At this point some of you are laughing because this sounds ridiculous, but others are thinking, “that’s pretty close to my current team – wtf?”

Today, I made a quick (or at least I planned for it to be quick) run to Home Depot to pick up a part. I knew exactly where the part was (aisle 40 – don’t ask how I knew that), so I jogged back, grabbed the part, and went to the front. I jumped in the self checkout line, scanned the part, shoved my credit card in the slot, and got ready to grab my receipt.

But the price was wrong. Or it was right, technically, but I was charged twice. Luckily, there was a cashier right there, so I explained that I scanned once, but was charged twice.

She stared for a moment, and said, “I can’t fix that. You’ll need to go to customer service.” Apparently, I needed the specialist. Or another specialist at least.

I went down to the customer service desk, where I was behind a customer with a cart full of stuff and a lot of questions. I waited 10 minutes for my meeting with the specialist, but it wasn’t moving very quickly, so I walked out of the main store area to the Returns counter to ask the returns specialist about it (the Returns counter is past the check out area to make (I assume) it easier to return stuff without entering the store.

Eventually, the Returns Specialist was able to help me, but he had a very difficult time. Apparently, he knew how to do returns, but since I wasn’t technically returning anything he thought it seemed like a problem that Customer Service should help with.

I try to be a quick thinker, and since my quick trip to the store was pushing twenty minutes in-store time, I grabbed my receipt and part, jogged back to aisle 40 to grab a duplicate part, returned to the counter with both, and I was able to return the item I didn’t want, and all was well.

On one hand, this is just another story of customer service gone south, but at the core, I see it as a flaw of too much specialization – and I think it’s important that we all recognize when we see these things around us. If your software team is ever blocked by a specialist – whether it’s waiting for a tester, waiting for a decision, or even waiting for a deployment, there’s a lesson to be learned, and a potential improvement to be made.

Five for Friday – August 3, 2018

Time again to share a few things I found interesting this week.

  • I’ve been reading Bad Blood (currently about 30% of the way through). So far, it’s a pretty damning investigative review of Theranos, but I’m still waiting for the plot twist :}. Wikipedia tells me that Jennifer Lawrence will star in the film adaptation.
  • Steve Denning is at it again with this article on Agile Is Not Just Another Management Fad.
  • A wonderful article on “Delivery Teams” from Janet Gregory.
  • This (yet another) article on the failure of open plan offices is making the rounds. Buried in the end of the article is this quote:

    “And probably least bad is small team rooms of fewer than ten people, preferably fewer than six. I’ve sat together with really small teams before and that’s been OK. Some people who don’t like the open office at all might even still enjoy this configuration.”

    This setup (a “team room”) has been, in my experience the hands down winner of the most productive work environment I’ve ever worked in – and I hate that it’s lumped into the valid arguments that go against barnyard style work seating.
  • Finally, a nice article on critical thinking and problem solving that I’ll probably re-read and refer to frequently.

Five for Friday – July 27, 2018

It was a busy week, but my list of potential links was long. Here are 5 of the things I came across this week I felt were worth sharing.

  • First off, PNSQC published a small interview with me in promotion of my talk at their conference in October. Alan Page on Tools, Technical Skills, and Modern Testing
  • In case you were under a rock and missed it, this link on Testing in Production the Netflix Way includes both a summary, a transcript, and a video of Nora Jones (I don’t think this is the first time I’ve linked to one of her talks). If you do anything with services or the web, you should check it out.
    By the way – the tech world needs more twenty-minute talks. The hour long presentation should be the exception, and not the norm. Someone remind me to write a full blog on this someday.
  • This article on The changing role of QA managers in the Agile environment shouldn’t be a surprise to anyone who follows me, but sharing anyway so you can share with those less enlightened.
  • Sadly, this article on This is what fear in leadership looks like captured the motivations of too many bad managers I’ve worked with (and for).
  • And finally, another article from one of my heroes, Steve Denning that is fascinating, yet haunting. How Agile Helped Elect Donald Trump

Five for Friday – July 20, 2018

Another Friday list to share:

  • I finished off a few books while flying this week, including Lean Change Management, which is one of those books that makes you want to read a bunch more books. It’s full of models and references, and, given that so much of my role is about leading change and helping others lead change, is probably something I’ll keep close at hand.
  • Ronan Mehigan sent me a link to an article on Specialists vs. Generalists. I like the article, but it also reminds me why I like the notion of Generalizing Specialists and Specializing Generalists so much.
  • Maybe I’m late to the party, but I just discovered I’ve found a nice handful of articles there, including the following bullet…
  • I’m relatively new (just a few years) to Mac, and very new (a few months) to using Ubuntu as my daily machine, so I really appreciated The Shell Introduction I Wish I Had.
  • Given the state of US politics and media, I generally defer to or for a (hopefully) more neutral view, but part of my daily reading continues to be the wonderful daily summary on What The Fuck Just Happened Today