Automation Analysis using Artificial Intelligence by Sathish Venugopal

Here is an article by one of our Top 5 Budding Data Scientists, Sathish Kumar Venugopal. Read about Automation Analysis using Artificial Intelligence. 

Business Problem:

Automation coverage has been increased significantly by adding more no. of scripts in the last few years. When we execute all the scripts, QA members have to spend a lot of hours in analyzing automation failure results, so there came a need to find out a solution to automate this analyzing capability and reduce manual effort spent on analyzing the script failures.

Problem Background:

In our organization we have around 12 product line and we have now covered 75% of functionality through automation scripts (Which is appx. 400*12=4800 test scripts) and this must be executed weekly 2 times and the result has to be analyzed every time we execute.

Normally it takes around 6 hrs for 2 person in 1 product line i.e (2*6=12 hrs)

So for 12 product line (12*12=144 hrs ) * 2 times in a week = 288 hrs/week

1 month 4*288 hrs = 1152 hrs has been spent for analyzing the automation script results.

Process followed in analyzing the automation result file.

When we execute the automation suite, we get the script executed result report in an email to QA member, then each member will start going through all the failed test case manually and analyze whether it is a “Script” Issue OR “Application” Issue OR “Performance” issue OR “Environment” Issue OR “Other” issue.

After a complete analysis of all the failed scripts, he has to categorize the issues into different buckets as mentioned above and take respective action. Ex. If it is an “Application” issue he has to report a bug-in-bug reporting tool.

We have to develop a system that automatically analyzes the failures and categories into different buckets. Which will reduce team members analyzing time.

Data Collection & Preparation

Started collecting different script failures during each run of our scripts and created a repository of repeated failures of the issues and continued this process for 2 months and stored all these data in Github.

Categorized the identified issues into different issue buckets and segregated relevant data into a CSV file.

EDA of our Data

To start with considered only limited number of failure for analysis and wanted to focus mainly on the following failures

• • Application issue

• • Performance issue

• • Script/code issue

• • Environment issues

• • Other issue

Developed code which reads the automation executed result file and validate on each failure.

Below are the conditions with which the script will validate each script failure

Application issue: Fails when it is not matching with the expected results

Performance issue: Fails when it waits for more than X no. sec (Load issue)

Script issue: Fails when it is not able to identify any of application object (Ex. Xpath)

Environment Issue: Fails when there is no enough Disk space OR Memory space in the server

Other issues: All other failures we categorized into this bucket.

Using our code it compares the script failures with the repository which we have already created (CSV file) and it continue to increments the failure count and categorizes these issues into different buckets, finally it sends a report as shown in below format.

Algorithm Selection

Based on the problem and the conditions which we need to validate, it clearly shows that it is a classification problem and these classifications have to be done using some conditions. First which strikes my mind was the “Decision Tree” algorithm, tried with the Decision Tree algorithm in our data set.

Accuracy Prediction

Even though we have categorized different issues into different buckets, we wanted to predict the accuracy of classification using the Decision Tree algorithm, when base model prediction applied, got an accuracy score of around 52%. This shows still I need to fine-tune my data to properly categorize the failures.

Conclusion

This project still not yet completed and still in progress, need to try with other algorithms and see whether the accuracy score can be improved.

Future Action:

We need to fine-tune on the parameters and data to improve the prediction score.