Artificial Intelligence In Software Testing

Artificial Intelligence in Software Testing

Artificial intelligence isn’t a new concept, in fact, it dates back to the 1940s with McCullouch and Pitts' formal design for Turing-complete "artificial neurons." Fast-forward some 76 years later, improved computing power and an abundance of data have fueled the revolution of AI to handle and solve some of the world’s most difficult problems. 

Over the last decade, we have heard how artificial intelligence will revolutionize medicine with machines accurately diagnosing diseases within seconds, self-driving cars maneuvering all sorts of obstacles to get you from point A to B safely, and AI taking on a multitude of tasks, both large and small, in virtually every industry. But, what about the field of software testing? Can we create machines that think and work like human testers? What will this mean for them and their careers? This paper focuses on artificial intelligence and its impact on the software testing industry. The topic at hand is rather complicated, so it would be impossible to cover all aspects in this paper. So, I will provide an overview. First, highlight how AI will disrupt testing in the future by discussing some of the advantages. Next, I will discuss some of the challenges we may face as we adopt the technology. I will then summarize my key findings after reviewing articles, studies, and videos on the topic and provide my opinions.

Andrew Ng, perhaps the most respected person in Applied AI, says “Pretty much anything that a normal person can do in less than a second, we can now automate with AI.” And, that could be true when you look at AI and its role in software testing. Think about how many manual testing steps or verifications could be done in about a second. Artificial Intelligence gives machines the ability to make decisions and work like humans without having to program a set of complete instructions. A training set in machine learning includes many input and output values, much like testing where we provide an input and then an expected output when writing a unit test, for example. The machine using the data to train is later able to make decisions on its own. Therefore, the more you feed it the smarter it gets. Based on the articles I have reviewed, AI is not just a dream, but is already transforming testing in the following ways and it’s pretty mind-blowing. For example, it’s being used today to auto-generate testing scripts where AI can fill out forms, click on-screen links, log in and out of apps, etc. According to the article “How AI is Transforming Testing Today,” these scripts are generating 80% of code reliability, which in my opinion, is pretty impressive. I am stunned at how these bots can automatically fill out forms, and perform many time-consuming, mundane tests human testers would complete during a manual test. These bots are not only faster, but can operate on a much larger scale. For example, these AI bots are able to create thousands of users and test them all surpassing the current number of manual tests a human can perform.

Moreover, anytime code is changed, AI agents can automatically rebuild the tests ensuring a commitment to continuous quality. Imagine the time that could be saved in every sprint and how analysts could take on more productive stories such as reviewing requirements, planning, communicating, etc. Regression testing can take a lot of time to retest code affected by a recent change. And, changes happen often in the software development life cycle. So, having an AI-powered system is not only testing smarter but more efficiently as releases, which usually occur monthly, can now happen weekly.

Aside from automatically generating test scripts, AI is intelligently optimizing the tests it runs to quickly identify risks and defects in code. In the article “Why AI and Machine Learning Will Redefine Software Testing in 2019,” AI is not only said to be more efficient than a human, but less error-prone. While completing some homework assignments, specifically the ones regarding equivalence classes, and designing test cases, I found myself missing critical tests, and often times making simple mistakes. I had a hard time keeping track of variables on the excel spreadsheet and often confused the cells when performing a number of repetitive tasks such as calculating the total amount based on shipping. Automated testing, according to the article, helps by “performing the same steps accurately every time.” As learned in class, a simple error, such as a simple miscalculation can bring down a rocket, cost a company millions of dollars, or even worse bring down an aircraft killing hundreds of people. Any testing method that can improve accuracy and detect defects and risks faster is worth exploring as it can save lives, time, and money.

Other benefits of automated testing using AI include increasing overall test coverage. While we may never achieve 100% coverage and 100% defect-free code, as discussed in class, writing meaningful tests to improve coverage increases scope and depth to improve software quality. Automated testing allows testers to run “1000+ different test cases in every test run providing coverage is not possible with manual tests.” Furthermore, these tests can run at a lower level examining memory, file contents, internal program states, and data tables, which is remarkable. Therefore, automated testing improves accuracy, efficiency, and coverage while improving software quality and reducing costs and delivery time.

But, new technology brings a set of challenges. Of course, there is a learning curve. Testers may find the new technology intimidating because of unfamiliarity forcing some to feel desperate to learn new skills. It will likely highlight why ongoing education is vital in this business. Those who are unwilling to learn new skills will be left behind, others will reap the benefits of staying employed. While the bots work no different - generating a number of inputs and outputs putting them into a BlackBox (machine learning tool) and training - the technology ultimately only does what it is told to do. According to the article “Can Artificial Intelligence replace humans in Software Testing?,”  a problem “that cannot be solved manually cannot be solved by an AI tool as well.” After all, we are the ones training the machine. This is not to say these testing bots can’t think on their own because they can. In fact, from what I’ve discovered they can find new paths and are open to change since the tests aren’t hard-coded. The machine is only as good as how you train it. 

Another downfall is the cost - it isn’t cheap - therefore leaving out smaller businesses or companies that simply do not have a sufficient budget to incorporate the new testing technology. And, finally, one of the biggest downfalls, I have discovered in my research is the impact it could have on the livelihood of millions of testers around the globe. Or will it? I was shocked to learn most of the articles I read conclude testers will not be replaced, but their roles will change. AI testing tools will allow them to work more efficiently. That’s great news. The consensus, AI testing tools aren’t smart enough, just yet, to develop test plans, monitor progress, evaluate results, and take corrective actions when necessary. Humans still need to handle those tasks. While AI can definitely help with designing and building test cases and data, the communication aspect is missing. Humans still need to communicate with stakeholders. This is not to say these AI bots can’t develop these skills 20 years from now, but the experts say it won’t happen anytime soon. In the article “Does AI replace or assist?,” John Bates, the CEO of TestPlant says “AI isn’t a robot coming to take your job, but is a smart assistant.” He says these AI bots will predict and spot defects as code changes faster, but won’t outright replace all human testers, which is a sigh of relief for many.  

To summarize my findings, there are many ways AI in testing is beneficial: increases productivity and reduces deployment times, improves test accuracy, spots and predicts bugs, defects, and risks faster, surpasses and go way beyond the limits of human manual testing by exploring test cases virtually impossible for humans to test, adjusts and adapts to code changes faster than a human. While the thought of AI can be scary for some testers, they should not be fearful that the bots will take over their jobs. Instead, they should think of them as helpful friends, giving human testers more time to focus on other tasks such as planning, improving product ideas and designs, reviewing tests, meeting with clients and business stakeholders, and taking corrective action when necessary.

While preparing to write this paper, I stumbled across an awesome product called AppDiff (test.ai), which touts itself as the first AI mobile testing platform. Its service is pretty revolutionary. Imagine uploading your application, the binary to the cloud, robots then wake up and begin figuring out ways to test. Spinning up thousands of copies, the bots walk through the application and its thousands of lines of code, just like a human end-user, finding bugs, UX issues, crashes, error dialogs, and generating a full test report in under an hour. The robot “a virtual brain” automatically generates tens of thousands of test cases based on training thousands of inputs and outputs. Mind you, I said, thousands, which is impossible for a human to do. The company behind the product, which was founded just a few years ago in 2015, has already raised $2.5 million in seed funding. The company’s co-founder Justin Liu says testing that would normally take five days to complete can now be done in less time than it would take for someone to grab a cup of coffee. Not bad!

I am personally excited about the future and the way we are using AI technology to solve problems and enhance lives. While some people may have a negative view of AI, I am optimistic about what lies ahead in the future. Yes, it is true, technology can be used for bad (especially if biased), but there is much good that can be achieved. Having learned about testing and its processes and concepts over the last 10-weeks, I am fascinated by the power of machine learning. I can see how these new tools could make my life easier as a tester. But, the only concern I have is accuracy. While, based on my research, the technology is said to be more accurate than humans, I still have my doubts about putting all of my trust into a machine. I guess that’s normal for us humans to feel. After all, machine learning works on data. If the data is biased or flawed, it could throw everything off. So, I don’t know if I could truly rely on such technology 100% just yet, especially when testing software that has the potential to harm people. I definitely feel there needs to be at least a human supervising the entire process and working alongside the bots, whether it's performing random testing and verifying if the bot is working correctly, etc. Overall, I found this topic fascinating to research and had a great time learning about what is here and what we could expect to see in the future.

 Sources:

https://www.softwaretestingnews.co.uk/ai-replace-assist/

https://devops.com/how-ai-is-transforming-testing-today/

https://www.business2community.com/business-intelligence/why-ai-and-machine-learning-will-redefine-software-testing-in-2019-02204554

https://www.webomates.com/blog/artificial-intelligence-and-software-testing/