I read this article over the weekend about five emerging trends in software testing – Test Automation; Rise of mobile and cloud; Emphasis on security; Context-driven testing; and More business involvement.
I fully acknowledge that I work in a software development environment that isn’t like many others, but while reading the article, I really didn’t feel like any of those areas are “emerging” – all are fully emerged already. Sure, the trends are interesting to testers, but emerging? I could waste some space rebutting or commenting on the areas above, but instead, let me offer some alternate trends that I see inside of MS and from some of my colleagues who work elsewhere.
Fuzzier Role Definitions. I don’t really like the terms “whole team approach” or “combined engineering”, but I do see software teams really figuring out how to work better together and leverage every team members strengths effectively. Great testers are working as Test Specialists and working much more broadly across the team. I expect the “lines” between software disciplines to fade even more in the future.
Developers Own More Testing. You can call them “checks” if you wish (I call them “short tests”, but software developers are beginning to own much bigger portions of traditional software testing. This is a good thing – it ensures that daily code quality is high, and gives test specialists a high quality product to work with.
Testing Live Sites. Mock-test environments typically do a poor job representing production environments. Other than brief sanity checks for the most critical components, many web service teams just roll their new bits straight to production, and then run their tests against the live system. With a good monitoring system (including the ability to stage rollouts and automatically roll back if needed), this is a safe, efficient, and frankly, practical method for testing services.
Data is HUGE. Many software teams have figured out that the best way to get an accurate representation of how customers use software is collect and analyze data from those same customers. A whole lot of traditional test activities can be replaced by product instrumentation, and an efficient method for getting product instrumentation back to the team for analysis. On a lot of teams, last year’s testers are this year’s data analysts and data scientists. While not every tester is cut out for this role, this move to data analysis is a strong trend on a lot of software teams.
To critique myself for a moment, I think a lot of readers could say that none of these points are emerging either. That’s a fair point, since I know teams that have been doing everything above for years…but I’m just now seeing some of these trends “emerge” on multiple teams (and not just those testing web services or sites).
What trends do you see? Did I miss anything huge? Have the above four points already reached the tipping point of emergence?
I feel that what an individual sees as history, current trends, emerging trends or the future depends on where they sit in the testing maturity continuum.
There are a couple of items that I can list. Don’t know if you will agree with them.
1 – Crowd-sourced testing is becoming ever more popular. Many people who work as testers in organizations also work as crowd-testers.
2 – Increased support is being sought by testers (and provided by others) on the many testing forums and social media.
Thanks for listing the trends above. Very useful.
#1 is definitely a trend – thanks for mentioning that.
Is #2 more true for testing than it is for other knowledge work? I could make a reasonable argument that there’s even more of that happening for programmers.
As always, thanks for the comments.
I worry about generalizing the concept of “Testing Live Sites”. While that is completely appropriate (and as you point out, efficient) for some sites, it’s also completely inappropriate for other sites. If your site was for online banking, would you still advocate rolling the new bits straight to production, and then running tests against the live system?
Actually, yes. Of course if the changes could have any security implications, I’d test those first (note, I said, “Other than brief sanity checks for the most critical components” – and security IS critical). I’d also make sure I had really good monitoring tools around login to ensure it worked correctly even after rolling out the changes.
But if I’m making improvements to how checking account activity looks, or what information I show on the main page (or want to do A/B testing of either), I would definitely do it in production (and in the case of A/B, I’d /have/ to do it in prod).
Alan, agree on security, but I think Joe Strazzere was talking about behavior correctness as well. For e.g. banking, you can’t just roll the new bits out without a lot of business logic testing with the attitude “well, if the numbers are wrong, customers will report it.” !! For this, you need a good set of regression tests, and you need to run them often and fast e.g. with MetaAutomation.
“Checks” is an important and useful distinction with “tests.” If you were to write a manual test, have a person (sentient) run it, call that value A. Suppose it’s then automated, then run, call that value B. No matter how it’s automated, B is very different than A. The biggest difference is that A is sentient, B is not sentient. B you can run often and fast with high repeatability (if it’s at all well-written), but not A.
I agree with Jon Bach and others that if a test is automated, it’s not a test anymore. Actually, “automated test” is a contradiction in terms.
Using the term “check” emphasizes that fact and reduces exposure to the mistake of asserting that once a test is “automated” you never have to run it manually again. Not true 🙂
Yikes.
I currently work for a financial services company. The auditors would choke on their checklists if I told them we roll the new bits straight to production and test them there. And the first time one of our clients (or their end-users) reported a bug against this newly rolled code, I’d be spending my days and nights writing Root Cause Analyses and trying to figure out who pays for the inappropriate costs their end users just incurred.
I guess we’ll have to agree to disagree that “Testing Live Sites” as you have defined is acceptable generally.
Given I have zero experience in finance, I trust that you are right on this, and that I’m stretching my experience a bit too far.
But humor me- – If the (non-secure) home page of a bank had an offer to open a new account, could you do A/B experiments of different offerings to see which drove more business? Could you roll changes out more quickly to the non-secure parts of the site?
“If the (non-secure) home page of a bank had an offer to open a new account, could you do A/B experiments of different offerings to see which drove more business?”
Perhaps. But if you broke a bank’s login page (even for a little while) that would be a *bad thing * I suspect.
My applications aren’t like that. We host applications for our large financial institution clients. These web applications are integrated within the websites of these financial firms and serve as the basis for things like mutual fund purchases. Failures here could cost some real money!
If these applications are down or failing for any reason, the calls quickly escalate up the management chain. There’s no way either my management or our clients would accept “we are testing a change in production” as a reason here.
Sorry – that simply wouldn’t fly in my context.
Thanks for the additional context. You’re right – even rolling out changes to tiny portions of the live audience (typical approach) wouldn’t fly at all in your world. I get that (now).
Given your context, are you able to simulate your production environment pretty closely in your test environment? Normally, the inability to do that well is what drives teams to do testing in prod. I realize that even if your environment wasn’t close, you’d still need to test locally extensively first, but I wonder if you ever run into issues that *only* reproduce in the production environment.
Alan and Joe, you’re both right. Alan, you can run your A/B tests on non-critical items, that makes good business sense. Joe, you still need to check the numbers on a banking site before exposing it to actual customers.
On analytics, I’m not completely in the loop, I can see where it’s very useful, but I don’t see how this could possibly substitute for well-designed regression checks, plus boundary cases, equivalence classes etc. where critical data is concerned. Run the checks pre-ship and live both, probably, and there’s some perf data available too with those tests.
With regards to testing live sites, I also worry about generalising. Such a process would hopefully be backed up with a solid suite of fast automated tests on every level that go beyond a sanity check (depending on what you mean by sanity check, but I’ve often heard the term used to mean a small suite of end to end tests). In addition, incremental rollouts can help mitigate risk when taking this approach and I think this type of mitigation is important to mention, as it’s not a reckless approach but a considered cost-benefit analysis.
It’s also suitable for some types of tests like the ones you mentioned in the above comment, but not others. It’s important to differentiate between running an experiment for user behaviour and just throwing mostly untested code into the wild and seeing what happens. Not that you’d ever do the latter, but that’s why I think it’s important to clarify.
Those are very fair points Trish – I think there’s a lot *more* testing that can happen in prod, but it’s important to note that there’s a whole bunch of stuff that’s better done preprod. I think there are probably some heuristics I can come up with to help figure out how to make good decisions.
Chaos engineering is definitely a new trend (and this falls into the testing live sites theme that you pointed out). However, with chaos engineering the unique distinction is execution faults into the system with a controlled blast radius to ferret out issues that customers may run into.
Another area that I see emerging is testing software leveraging artificial intelligence / machine learning. IEEE will have its first conference next April dedicated just to this topic.
Keep in mind that this is a post from over 3 years ago. The topics you mention are certainly emerging today.
Thank you for calling me out so nicely on the show! Much appreciated. I definitely missed the post date on the blog – I should have had my coffee first. Maybe I will be able to look back in 2022 and see if my comment was relevant then. I love your show! I am a huge fan.