Skip to content
BY-NC-ND 3.0 license Open Access Published by De Gruyter May 5, 2015

Neuro-Fuzzy Modeling for Multi-Objective Test Suite Optimization

  • Zeeshan Anwar EMAIL logo , Ali Ahsan and Cagatay Catal

Abstract

Regression testing is a type of testing activity, which ensures that source code changes do not affect the unmodified portions of the software adversely. This testing activity may be very expensive in, some cases, due to the required time to execute the test suite. In order to execute the regression tests in a cost-effective manner, the optimization of regression test suite is crucial. This optimization can be achieved by applying test suite reduction (TSR), regression test selection (RTS), or test case prioritization (TCP) techniques. In this paper, we designed and implemented an expert system for TSR problem by using neuro-fuzzy modeling-based approaches known as “adaptive neuro-fuzzy inference system with grid partitioning” (ANFIS-GP) and “adaptive neuro-fuzzy inference system with subtractive clustering” (ANFIS-SC). Two case studies were performed to validate the model and fuzzy logic, multi-objective genetic algorithms (MOGAs), non-dominated sorting genetic algorithm (NSGA-II) and multi-objective particle swarm optimization (MOPSO) algorithms were used for benchmarking. The performance of the models were evaluated in terms of reduction of test suite size, reduction in fault detection rate, reduction in test suite execution time, and reduction in requirement coverage. The experimental results showed that our ANFIS-based optimization system is very effective to optimize the regression test suite and provides better performance than the other approaches evaluated in this study. Size and execution time of the test suite is reduced up to 50%, whereas loss in fault detection rate is between 0% and 25%.

1 Introduction

Software development is a labor-intensive and human-centric activity, which frequently results in buggy code. The software testing phase in software development life cycle helps to find faults, builds confidence about the software system, and shows that the system performs the user needs correctly. According to the experimental studies, the cost of software testing activities is at least 50% of the total software development cost [45].

In order to remove all the faults during the testing phase, a diverse set of testing methods are used. These testing methods are divided into several groups such as specification-based techniques, code based, fault based, usage based, techniques based on the nature of the application, and techniques based on the software engineer’s intuition [19]. However, the regression testing method is the most widely researched area due to its impact in software testing.

When a test case fails, the fault is corrected, and the new software is re-run with the previous failed test case. But re-testing may only help to ensure that the fault has been fixed correctly, but it is possible that the change made on the software introduces new faults, which cannot be detected by that test case. Therefore, all the test cases must be used to check the unexpected side effects, a type of testing known as regression testing. The size of the regression test suite grows due to the addition of new test cases, which are prepared based on the new implemented features. Therefore, regression test execution duration increases day by day, and this time-consuming process need to be managed properly to perform it in a cost-effective way.

Researches in literature exemplify the time-consuming nature of regression testing activity. An industrial collaborator of Elbaum et al. [18] stated that a test suite of software having 20,000 lines of code requires 7 weeks to run. Carlson et al. [12] stated that the entire regression testing process of Microsoft Dynamics Ax product requires several days for executing test cases and several more days for analyzing the results. Do et al. [17] specified that a regression test suite of a product in a software company includes 30,000 functional test cases, and regression tests take 1000 machine hours. When we think of the nightly build (Nightly build means that every night, everything that has been checked into source control is built.) practice that is widely used in the current agile software development approaches (i.e., Scrum, XP, Evo, etc.), we can state that it is possible to check the nightly builds with specific test cases by reducing the number of test cases in the test suite. Otherwise, if all the test cases are executed, it will not be possible to check the build within one night, and it will not be possible to use this crucial practice. Therefore, test cases that can identify more faults must be selected, and the remaining test cases must be removed from the test suite. This approach is called test suite reduction (TSR) or test suite minimization (TSM). By using this approach, regression tests can be executed cost effectively. When human efforts such as monitoring the tests, the management of test problems, and the reorganization of the test environment are considered, it is obvious that the testing cost will be high even if the test duration is short. According to the recent studies, the optimization of regression test suite is mandatory [5, 34, 41, 44, 49].

Three main approaches have been proposed in literature to optimize the regression test suites and execute the regression tests in a cost-effective manner: TSR, regression test selection (RTS), and test case prioritization (TCP) [8, 51]. While both the TSR and RTS techniques reduce testing time, they can omit some significant test cases that can detect certain types of faults; hence, this causes an increase in the software cost. TCP techniques identify the ideal order of the test cases and maximize the earlier detection of faults or coverage rate. For example, even for a simple project that includes 20 test cases, there will be 20! = 2.432.902.008.176.640.000 different orderings.

Most of the papers in literature are based on a single objective. However, it is also necessary to optimize the regression test suite under several conflicting constraint [52]. Yoo and Harman [52] used three algorithms for the Pareto efficient multi-objective test case selection and reported that greedy algorithms that are effective for single-objective problems are not always Pareto efficient for a multi-objective test case selection problem. Unsafe optimization is risky because some important test cases may be skipped, and faults that are detected by only these test cases may not be detected. In order to solve this problem, researchers proposed the multi-objective evolutionary algorithms (MOEA) [55] to make the optimization process safe.

Comparative analysis of three algorithms (multi-objective genetic algorithms (MOGA), non-dominated sorting genetic algorithm (NSGA-II), and multi-objective particle swarm optimization (MOPSO)) was performed for the regression test suite optimization. It was observed that these algorithms are not safe for regression test suite optimization (RTO) because fault detection rate decreases dramatically, and many important test cases are not included in the optimized test suite [7] because MOEA-based algorithms select or deselect test cases based on the discrete values [20]. Despite all efforts and available techniques, human limitations prevent MOEA-based approaches to guarantee the optimization because expert judgment is not incorporated in MOEA-based approaches, and software testing requires expert judgment to optimize the test suite [6]. The RTO is NP-complete problem and soft computing-based techniques such as fuzzy logic, neural networks, and genetic algorithms can be used to solve this multi-objective problem [22]. Multi-objective optimization approaches can be used in different software testing problems, and Assunção et al. [10] evaluated NSGA-II, SPEA2, and PAES multi-objective evolutionary algorithms for the integration problem.

Alakeel [3], Haider et al. [20], and Anwar et al. [6] have successfully used fuzzy logic for RTO, but the selection of membership function parameters and fuzzy inference rules are not automated. Human intervention is required in the previously implemented approaches [3, 6, 20]. Recently, Xu et al. [50] used a fuzzy expert system to select some portion of test cases and showed that defect detection rate is improved by fuzzy system. Human experts may make errors in the selection of inference rules [3, 20]. Because the membership functions’ parameter values are static and are not updated automatically, the values of the membership functions’ parameter must be changed for each regression test suite [5].

The fuzzy logic-based systems’ flaws as discussed above can be addressed by introducing the learning capability in the fuzzy system. This is known as neuro-fuzzy modeling (NFM) [27]. NFM uses learning algorithms of neural networks [25] for parameters and rule tuning. Parameters are adjusted until an optimal set of parameters is reached. This makes the system self-adapting [47], and therefore, there is no need for the human intervention.

The rest of the paper is organized as follows: In Section 2, related work is presented. Section 3 presents the mathematical model to solve the RTO problem with the adaptive neuro-fuzzy inference system (ANFIS). Experimental studies are explained in Section 4. Section 5 discusses the analysis of the proposed model. In Section 6, the proposed approach is compared against four computational intelligence techniques used in the RTO problem. The conclusion and future work is presented in Section 7.

2 Related Work

Computational intelligence (CI) is widely used for optimization problems. Artificial neural networks, artificial immune systems, genetic algorithms, evolutionary algorithms, fuzzy systems, and hybrid intelligent systems are some approaches, which are used for computation intelligence [33]. Sumathi and Surekha [43] define CI as a successor of artificial intelligence that has biological inspirations to solve complex problems. Researchers have used various CI techniques for the RTO.

Li et al. [32] applied five algorithms on six programs for the test suite prioritization, and it was reported that genetic algorithms performs well for this problem. Zhongsheng [54] collected user session data from web server, and this data was divided into meaningful user sessions. Zhongsheng proposed genetic algorithms to optimize the test suite. Error coverage ratio and the cost of test run were used as fitness functions. This approach was effective to reduce testing time [54]. Nachiyappan et al. [37] used block coverage and execution time as objectives to propose a genetic algorithm-based test suite reduction approach. This approach was applied on three projects to reduce test suite size. Yoo et al. [52] used coverage, cost, and fault history parameters to design and evaluate a Pareto efficient multi-objective test suite selection technique. The proposed technique was compared with greedy, additional greedy, NSGA-II, and vNSGA-II methods. They concluded that the greedy algorithm is not efficient for the multi-objective optimization problem [52]. Anwar et al. [7] analyzed three algorithms (MOGA, NSGA-II, and MOPSO) for the optimization of regression test suite. The proposed approach was applied on two case studies, and it was reported that these algorithms are not safe for the regression test suite optimization because fault detection rate decreases, and many fault-finding test cases were not included into the reduced test suite. It was also proposed that expert judgment is required for test suite optimization. Kaur et al. [30] used maximum faults covered in minimum time and total code coverage [29] in conjunction with genetic algorithms for the test suite prioritization problem and reported that genetic algorithms are effective for regression test suite prioritization. Kaur et al. [28] used execution time as an objective function to prioritize the regression test suite by using the bee colony algorithm. This algorithm was compared with various algorithms and shown that their model provides better results [28].

Simão and Senger [14] used the adaptive resonance theory (ART) self-organizing neural networks to reduce the regression test suite. Their approach was compared with random selection, and the proposed model is 34% better in precision and 20% better in recall [14]. Xu et al. [53] used the fuzzy expert system for black box-based testing methods to select some test cases. Schedule factor, defect impact factor, and coverage factor were selected as objective functions. The rules for inference were formed after the interview with experts. Three releases were used to tune the system, and the fourth release was optimized. It was reported that the fuzzy system can optimize the test suite to reduce the time and find the faults earlier. Alakeel [3] used the fuzzy logic to prioritize the regression test suite. The fuzzy logic was employed to measure the effectiveness of test cases to violate assertion. This method was not validated to check its effectiveness. Haider et al. [20] proposed the tri-objective reduction method based on the fuzzy logic for regression test suite minimization. Performance, throughput, and coverage were selected as objective functions. However, this approach was not validated on any case study. Anwar et al. [6] used the fuzzy logic Sugeno model for the regression test suite minimization. The model was applied on two case studies, and 50% reduction in the size of regression test suite was achieved. The problem of selection of the fuzzy rules was solved in this paper by using the Sugeno model (the Sugeno model uses maximum possible rules by using grid partitioning, and there is no need of human expert to define or select fuzzy rules). But the membership functions’ parameter values are static and are not updated automatically in this research. The values of the membership functions’ parameter must be changed for each regression test suite [5].

In literature, most of the studies performed for the RTO are coverage based. However, code is not always available. Therefore, we need optimization approaches for black box-based testing methods. Only a few studies address this problem. Our research fills this gap, and the proposed model is applied on the black box-based testing method. Expert judgment improves the performance of the RTO problem. The fuzzy logic has been used to incorporate expert judgment for this problem. However, the RTO by using fuzzy logic cannot be fully automated, and the membership function parameters (Membership function parameters represent the spread and center of membership function.) are not optimized. This study also fills this gap, and the optimization of membership parameters is achieved by using the NFM. The NFM eliminates human intervention, and the RTO process becomes self-adapting.

3 Neuro-Fuzzy Modeling, ANFIS, and RTO

The NFM, known as the neuro-fuzzy system (NFS), is composed of neural networks and fuzzy inference system. Neural networks are used to recognize patterns and adopt themselves to changing environments, and fuzzy logic incorporates human knowledge to perform inference and decision making in NFSs [27]. The types of NFS can be found in [2, 11]. The ANFIS is one of the most popular types and has less root mean square error (RMSE) [43] compared to the other NFS. The ANFIS architecture consists of five layers that have specific purposes [25, 11] and two types of nodes that are fixed and adaptive. Fixed nodes are represented by circle, and adaptive nodes are shown by squares. The ANFIS’s detail and calculations performed on each layer of ANFIS can be seen in [5, 25, 11].

The ANFIS can be implemented using grid partitioning (GP) (GP generates fuzzy rules by using all combinations of membership functions of all inputs.) and subtractive clustering (SC) (SC is a fast algorithm to estimate the number of clusters and cluster centers). In ANFIS-GP, a single output Sugeno-type FIS is generated using grid partitioning on data. ANFIS-SC generates an initial model for ANFIS training by applying subtractive clustering on data. If we have a clear idea about the input clusters, then the ANFIS-GP is preferred. Otherwise, the ANFIS-SC is a good choice [24]. Hybrid learning algorithms are used in the ANFIS to estimate the parameters. The learning algorithm consists of forward and backward pass. The premise parameters are kept fixed, and consequent parameters are estimated in the forward pass, and the similar approach is performed in the backward pass. Input is propagated, and error is calculated by least square in the forward pass. In the second pass, this error is backpropagated, and the premise parameters are updated by gradient descent method [25, 11].

Dynamic and static testing are methods of software testing. White box and black box testing approaches are in the group of dynamic testing [20]. The NFM can be applied for the optimization of both of these approaches. The objective functions used by the researchers for the optimization of regression test suites are listed in [5, 8]. These objective functions were collected from the content analysis of papers [9, 20, 21, 3840, 42, 48, 53, 54]. In this research, we optimize the regression test suites for the black box testing method. Input variables for the ANFIS network are selected from the content analysis of papers [53, 54, 9, 21, 38, 40, 42, 48]. The following input variables are used in our research because these are important factors for regression testing. The details and procedure for the calculation of these input variables from test history can be seen in [7].

  • Faults detected by a test case are determined by the fault detection rate (FDR). Formula 1 shows how to compute this value.

    (1)FDR=FaultsDetected/Total Faults (1)
  • Time taken by a test case to execute is execution time (ET) and can be determined by inserting the timer function in test scripts.

  • Number of requirements covered by a test case is requirement coverage (RC) and can be determined with formula 2.

    (2)RC=requirementscovered/totalrequirements (2)
  • Requirement failure impact (RFI) measures the impact when a requirement is not implemented correctly. The tester can assign it during the requirement analysis phase on a scale of 0–1.

ANFIS inputs and output are represented by fuzzy sets, and each set contains linguistic variables like low, medium, and high. L, M, and H represent low, medium, and high, respectively.

(3)FDR={L,M, H} (3)
(4)ET={L,M, H} (4)
(5)Rc={L,M, H} (5)
(6)RFI={N,M, C} (6)

N, M and C represent normal, medium, and critical, respectively.

Eighty-one rules were automatically generated by ANFIS because there are four inputs, and each input has three linguistic variables. The sample calculation performed on each layer of ANFIS for RTO for two objectives (FDR and ET) is given below:

Layer 1: The membership grades of each input are generated, and the node function (The node function specifies the function used at that node.) is determined on this layer by using the following formulas:

(7)O1,i=μAi(FDR)fori=1,2 (7)
(8)O1,i=μBi2(ET)fori=3,4 (8)

where FDR is the input to node i, and Ai is the linguistic label. O1,i is the membership function of Ai. It specifies the degree to which given FDR satisfies the quantifier Ai. For the bell-shaped function, (The bell-shaped function is a direct generalization of the Cauchy distribution used in probability.) O1,i can be determined by the following equation:

(9)μA(x)=11+|xciai|2bi (9)

{ai, bi, ci} is the parameter set. The bell-shaped function varies as the values of these parameters change.

Layer 2: On this layer, the rule firing strength (The rule firing strength is also called the degree of fulfillment of the fuzzy rule.) is calculated by multiplying the incoming signals. The FDR and ET rule firing strength is calculated by the following formulas:

(10)O2,i=wi=μAi(FDR)μBi(ET),i=1,2 (10)
(11)O=j=1nμ(FDRj) (11)

Layer 3: On this layer, the ratio of the ith rule firing strength is calculated by the summation of all the rule firing strength to generate a normalized firing strength.

(12)O3,i=w¯i=wiw1+w2 (12)

Layer 4: On this layer, the contribution of each rule to the overall output is determined.

(13)O4,i=w¯ifi=w¯i(piFDR+qiET+ri) (13)

w¯i is the output of layer 3, and {pi, qi, ri} is the parameter set; parameters in this layer are referred as consequent parameters.

Layer 5: This layer computes the overall output by the summation of all incoming signals in the form of crisps value.

(14)O5,i=iw¯ifi=iwifiiwi (14)

The model diagram for the solution of the RTO using Sugeno is given in Figure 1. It consists of four inputs and one output.

Figure 1: ANFIS-GP Model Diagram.
Figure 1:

ANFIS-GP Model Diagram.

The ANFIS-GP architecture for the solution of the RTO using ANFIS-GP is given in Figure 2. It consists of four inputs, three linguistic variable for each input, 81 rules, and one output.

Figure 2: ANFIS-GP Architecture for RTO.
Figure 2:

ANFIS-GP Architecture for RTO.

The suitability of a test case to become a part of the optimized regression test suite is the output of the ANFIS. The ANFIS gives only one output variable and does not classify it into linguistic labels. Therefore, the Mamdani model [35] is implemented to classify the suitability into normal, moderate, and best as given in equation (14). The output having the best suitability is selected, which means that only those test cases will be included into the regression test suite that is best suitable. The architecture of the Mamdani model for the classification of suitability is shown in Figure 3. Those test cases that have output values >0.863 are best-suited test cases. The rule viewer of the best-suited test cases is shown in Figure 4 [6].

Figure 3: Mamdani Model to Select the Best Suited Test Cases [6].
Figure 3:

Mamdani Model to Select the Best Suited Test Cases [6].

Figure 4: Rule Viewer for The Best Suited Test Cases Case Study [6].
Figure 4:

Rule Viewer for The Best Suited Test Cases Case Study [6].

(15)To=S={N,M, B} (15)

where N, M, and B are normal, moderate, and best, respectively.

3.1 AFIS-Based System for RTO

The system diagram of our RTO software is given in Figure 5. This system consists of three main modules, namely, Test Management module, Optimization module, and ANFIS module. The Test Management module is used to save test cases and test history. The Optimization module reads the test execution history from the database and generates the training data for the ANFIS module. The ANFIS module read training data to generate the ANFIS model, tune parameters of membership functions, and generate selection rules. The ANFIS module is further divided into the training module and the inference module.

Figure 5: RTO System Diagram.
Figure 5:

RTO System Diagram.

4 Case Studies

The ANFIS-based proposed approach for the optimization of the regression test suites was implemented in Matlab [Mathworks Inc., Natick, MA, USA (www.mathworks.com)] [36], and two case studies were performed on the Previous Date Problem [1] (Case Study 1) and Siemens Print Tokens [23] (Case Study 2) datasets. Four experiments were performed to optimize the regression test suite. Two experiments were performed on each case study. The ANFIS-GP and ANFIS-SC models were used in each experiment. The Previous Date Problem consists of 33 test cases. These test cases were designed using the Equivalent Class Partitioning (The process of dividing input domain of program into equivalent classes.) and the Boundary Value Analysis (The input/output value that is on the edge of equivalent classes.). The Siemens Print Tokens is a lexical analyzer and consists of 18 procedures and seven faulty versions. The test cases of Print Tokens were created using the TSL Tool (A complier that generates executable test scripts from specifications and functional characteristics of software.). Test suites of Print Tokens, Code and Faulty versions are available at SIR (Software-artefact Infrastructure Library (http://sir.unl.edu/portal/index.php). The experimental setup for Print Tokens can be found in [16]. The flow of experiments for each case study is shown in Figure 6.

Figure 6: Flow of Experiments.
Figure 6:

Flow of Experiments.

4.1 Case Study 1: Experiment with ANFIS-GP

The ANFIS-GP model for the RTO was implemented as a Matlab Script file, and a solution was generated. The test history from a previous run of the test suite was selected as the input to the model for training, and we generated the optimized test suite. The flow of Matlab Script file is given in Figure 7.

Figure 7: Solving RTO with ANFIS flow [16].
Figure 7:

Solving RTO with ANFIS flow [16].

The 70% data was used for training, and the 30% data was reserved for testing. Fifty training epochs were completed. The graphs of the initial membership functions FDR, ET, RC, and RFI are shown in Figure 8.

Figure 8: ANFIS initial Membership Functions for Case Study 1.
Figure 8:

ANFIS initial Membership Functions for Case Study 1.

After completing the training parameters, the values of membership functions are adjusted and shown in Figure 9. The difference in initial and final membership functions can be observed from figures.

Figure 9: ANFIS-GP Final Membership Functions for Case Study 1.
Figure 9:

ANFIS-GP Final Membership Functions for Case Study 1.

The surface plots generated by the ANFIS-GP for Case Study 1 are shown in Figure 10. These surface plots represent the relationship between various inputs and outputs.

Figure 10: Surface Plots of ANFIS-GP for Case Study 1.
Figure 10:

Surface Plots of ANFIS-GP for Case Study 1.

4.2 Case Study 1: Experiment With ANFIS-SC

In this experiment, the ANFIS-SC is used to find the solution of the RTO. The data is the same as those used in experiment 1, and 40 training epochs were completed; the training graph is shown in Figure 11 and Sugeno Model for ANFIS-SC is given in Figure 12.

Figure 11: ANFIS-SC Training Error Plot for Case Study 1.
Figure 11:

ANFIS-SC Training Error Plot for Case Study 1.

The ANFIS-SC automatically generated the Sugeno model that consists of four inputs that is divided into five membership functions and one output.

Figure 12: Sugeno Model of ANFIS-SC for Case Study 1.
Figure 12:

Sugeno Model of ANFIS-SC for Case Study 1.

The ANFIS-SC structure shown in Figure 13 is very simple compared to the ANFIS-GP, and only five rules were created for inference. Surface plots of the various inputs and output are shown in Figure 14.

Figure 13: ANFIS-SC Structure for Case Study 1.
Figure 13:

ANFIS-SC Structure for Case Study 1.

Figure 14: Surface plot ANFIS-SC for Case Study 1.
Figure 14:

Surface plot ANFIS-SC for Case Study 1.

4.3 Case Study 2: Experiment With ANFIS-GP

In this experiment, the test history after running the test suite of Print Tokens was used to optimize the regression test suite. The experimental setup is the same as that used in experiment 1. The graphs of the initial and final membership functions after generating the Sugeno model for the ANFIS are shown in Figures 15 and 16, respectively.

Figure 15: ANFIS-GP Initial Membership Functions for Case Study 2.
Figure 15:

ANFIS-GP Initial Membership Functions for Case Study 2.

Figure 16: ANFIS-GP Final Membership Functions for Case Study 2.
Figure 16:

ANFIS-GP Final Membership Functions for Case Study 2.

The surface plots of the ANFIS-GP for Case Study 2 are shown in Figure 17.

Figure 17: Surface Plots of ANFIS-GP for Case Study 2.
Figure 17:

Surface Plots of ANFIS-GP for Case Study 2.

4.4 Case Study 2: Experiment With ANFIS-SC

The experimental setup and training data for this experiment is the same as those used in Experiment 2; the ANFIS-SC structure is shown in Figure 18. The ANFIS-SC divided the data into four inputs and three membership functions for each input. Three rules were automatically formed by the ANFIS-SC.

Figure 18: ANFIS-SC Structure for Case Study 2.
Figure 18:

ANFIS-SC Structure for Case Study 2.

The surface plots to show the relationship among various inputs and output are shown in Figure 19.

Figure 19: Surface Plots of ANFIS-SC for Case Study 2.
Figure 19:

Surface Plots of ANFIS-SC for Case Study 2.

5 Analysis

The proposed approach for the optimization of the regression test suite is analyzed in terms of the RMSE [4, 26]. The difference in the desired output and the output of the model is known as error. We can determine the error using the following formula:

(16)e(k)=y(k)y(k) (16)

where e(k) is the error, y(k) is the output of function, and y′(k) is the output of Sugeno after evaluation. We cannot measure the error of the Sugeno model because the output of the training data is not available; instead, we are finding the initial output from the Sugeno model to use it in our experiments.

5.1 Error in ANFIS-GP

RMSE was calculated in Matlab using the Fuzzy Logic Toolbox and Matlab Script Files. The RMSE of the ANFIS-GP for both case studies is shown as follows:

  • Case Study 1 Training RMSE = 0.00024809

  • Case Study 1 Test RMSE = 0.3664

  • Case Study 2 Training RMSE = 0.00212639

  • Case Study 2 Test RMSE = 0.0406191

The ANFIS-GP training error is low and acceptable. This means that the training process is completed successfully, but the test error of Case Study 1 is high because of small data. The ANFIS-GP error plots are shown in Figure 20. The desired output is represented by “o,” and the ANFIS output is represented by “+.”

Figure 20: ANFIS-GP Error Plots.
Figure 20:

ANFIS-GP Error Plots.

5.2 Error in ANFIS-SC

The ANFIS-SC error plots for both case studies are shown in Figure 21. The training error of Case Study 1 is higher than Case Study 2 because of the small data set.

Figure 21: ANFIS-SC Error Plots.
Figure 21:

ANFIS-SC Error Plots.

  • Case Study 1 Training RMSE = 0.015099

  • Case Study 1 Test RMSE = 0.12262

  • Case Study 2 Training RMSE = 0.0093015

  • Case Study 2 Test RMSE = 0.12805

6 Benchmarking

The proposed technique for the optimization of the regression test suites was compared with four techniques of computational intelligence known as fuzzy logic [35, 46], MOGA [31], NSGA-II [15], and MOPSO [13]. The test suites of the Previous Date Problem and Siemens Print Tokens are used for validation. The fuzzy logic, MOGA, NSGA-II, and MOPSO are already implemented for regression test suite optimization in various studies. The fuzzy logic is used for the regression test suite optimization in [3, 6, 20, 53]. We used the implementation of [6] to validate our approach. The Sugeno model for the RTO is given in Figure 22. The other details of the experiment can be seen in [6].

Figure 22: Sugeno Model for RTO [1].
Figure 22:

Sugeno Model for RTO [1].

We used the implementation of [7] to optimize the regression test suites with MOGA, NSGA-II, and MOPSO. The experimental setup and other details of the experiments can be found in [7].

6.1 Optimized Test Suite Pareto Fronts

Pareto fronts are given in Figures 23 and 24 for Case Study 1 and Case Study 2, respectively. After analyzing the Pareto fronts, we observed that the results of MOPSO and MOGA are better compared to NSGA-II for regression test suite optimization. The MOGA Pareto front is better than MOPSO because in MOGA Pareto fronts, ineffective solutions lie in a uniform region.

Figure 23: Pareto Front Generated for Case Study 1 [7].
Figure 23:

Pareto Front Generated for Case Study 1 [7].

Figure 24: Pareto Fronts Generated for Case Study 2 [7].
Figure 24:

Pareto Fronts Generated for Case Study 2 [7].

6.2 Optimized Test Suite Evaluation

Four evaluation metrics such as the percentage of test suite reduction, percentage of fault detection loss, percentage of requirement coverage loss, and percentage reduction in test suite execution time are defined in [57], and we used these evaluation metrics for benchmarking. The fuzzy logic and the ANFIS produce a single solution, whereas the evolutionary algorithms produce multiple solutions. TOPSIS [24] is used to get a single solution from the evolutionary algorithms. The final solution produced is further classified into not suited, suited, and best suited test cases using the fuzzy selection as described in Section 3.6. The evaluation results are shown in Figures 25 and 26 for both case studies.

Figure 25: Case Study 1 ANFIS-GP and ANFIS-SC comparison.
Figure 25:

Case Study 1 ANFIS-GP and ANFIS-SC comparison.

Figure 26: Case Study 2 ANFIS-GP and ANFIS-SC comparison.
Figure 26:

Case Study 2 ANFIS-GP and ANFIS-SC comparison.

Comparison of ANFIS-GP and ANFIS-SC for Case Study 1 with four techniques is given in Figure 24. The size of the regression test suite can be reduced up to 55%, but reduction in the fault detection rate is 25% because there are few fault-finding test cases, and the size of the training data is small. Similarly, the reduction in the execution time of the test cases is up to 50%, and the reduction in requirement coverage of the test suite is 70%. The ANFIS is better than fuzzy logic to reduce the test suite size and the test suite execution time. FDR loss with MOGA is 50% that is much higher than ANFIS and fuzzy logic. The ANFIS and fuzzy logic are safe techniques for reduction compared to MOGA, NSGA-II, and MOPSO.

A large training data is available for Case Study 2. Therefore, we achieved good results compared to the results of Case Study 1. We achieved up to 50% reduction in size and execution time of the test suite without any loss in fault detection rate. The results are slightly better than the fuzzy logic. Reduction in requirement coverage is about 40% that is acceptable. FDR loss with MOGA and MOPSO are also 0%, but requirement coverage reduction of these techniques is 64.41% and 77.1% respectively, much higher than ANFIS. Therefore, the ANFIS is a safe technique for reduction compared to the fuzzy logic, MOGA, NSGA-II, and MOPSO. The comparison graph of all techniques is shown in Figure 25.

6.3 Optimized Test Suite

Box plots of both case studies are drawn to visualize the test suite optimized by each technique. This analysis gives us an idea about the test cases, which are included into the optimized test suites. We can observe from the plots that the optimized test suite of fuzzy logic and ANFIS is better than in FDR, ET, RC, and RFI compared to the optimized test suites of MOGA, NSGA-II, and MOPSO. The box plots of Case Study 1 are shown in Figure 27, and the box plots of Case Study 2 are shown in Figure 28.

Figure 27: Case Study 1 Box Plots.
Figure 27:

Case Study 1 Box Plots.

Figure 28: Case Study 2 Box Plots.
Figure 28:

Case Study 2 Box Plots.

7 Conclusion and Future Works

According to the experimental results, we can conclude that the ANFIS-GP and ANFIS-SC can be used for the optimization of the regression test suite, and they achieve good results when the training data is large. The proposed technique can act like human experts and perform inference for future version of the software. The ANFIS not only reduces the size of the test suite but also considers the test cases that have high execution time with high fault detection rate and requirement coverage. It also finds the best test cases to become a part of the optimized test suite. Therefore, the ANFIS is a safe technique for the optimization of the regression test suite. The ANFIS is flexible enough to optimize multiple objectives, but other multi-objective techniques are not suitable to optimize more than three objectives [20]. The ANFIS can reduce the size of the test suite up to 50%, fault detection loss is between 0 and 25%, and the requirement coverage loss is between 40% and 70%, and 50% reduction in execution time can be achieved. This makes ANFIS a safe technique for the optimization. We will work to compare ANFIS-based regression test suite optimization with the other multi-objective algorithms like eMOGA, SPEA-II, and HPSO. We will also work to compare the proposed approach with neural network and support vector machines (SVM). We are also planning to validate the proposed technique on industrial projects.

Bibliography

[1] K. K. Aggarwal and Y. Singh, A book on software engineering, New Age International (P) Ltd.; Publishers, 4835/24, Ansari Road, Daryaganj, New Delhi, 2001.Search in Google Scholar

[2] A. Ajith, Neuro fuzzy systems: state-of-the-art modeling techniques. Connectionist models of neurons, learning processes, and artificial intelligence, pp. 269–276, Springer, Berlin Heidelberg, 2001.10.1007/3-540-45720-8_30Search in Google Scholar

[3] A. M. Alakeel, A fuzzy test cases prioritization technique for regression testing programs with assertions, in: ADVCOMP 2012: The Sixth International Conference on Advanced Engineering Computing and Applications in Sciences, pp. 78–82.Search in Google Scholar

[4] A. Al-Hmouz, J. S., R. Al-Hmouz and J. Yan, Modeling and simulation of an adaptive neuro-fuzzy inference system (ANFIS) for mobile learning, IEEE Trans. Learn. Technol.5 (2012), 226–237.10.1109/TLT.2011.36Search in Google Scholar

[5] Z. Anwar, Neuro-fuzzy modeling based regression test suite optimization, MSc. Dissertation, Center for Advanced Studies in Engineering (CASE), 2013.Search in Google Scholar

[6] Z. Anwar and A. Ahsan, Multi-objective regression test suite optimization with fuzzy logic, in: 16th International Multi-Topic Conference (INMIC-2013), pp. 95–100, December 2013.10.1109/INMIC.2013.6731331Search in Google Scholar

[7] Z. Anwar and A. Ahsan, Comparative analysis of MOGA, NSGA-II and MOPSO for regression test suite optimization, Int. J. Softw. Eng. (IJSE)7 (2014), 41–56.Search in Google Scholar

[8] Z. Anwar and A. Ahsan, Exploration and analysis of regression test suite optimization, ACM SIGSOFT Softw. Eng. Notes39 (2014).10.1145/2557833.2557841Search in Google Scholar

[9] E. Ashraf, A. Rauf and K. Mahmood, Value based regression test case prioritization, in: Proceedings of the World Congress on Engineering and Computer Science, Vol. 1, 2012.Search in Google Scholar

[10] W. K. G. Assunção, T. E. Colanzi, S. R. Vergilio and A. Pozo, A multi-objective optimization approach for the integration and test order problem, Inf. Sci.267 (2014), 119–139.10.1016/j.ins.2013.12.040Search in Google Scholar

[11] A. T. Azar, Adaptive neuro-fuzzy systems, fuzzy systems, ISBN 978-953-7619-92-3, pp. 216, INTECH, Croatia, February 2010.Search in Google Scholar

[12] R. Carlson, H. Do and A. Denton, A clustering approach to improving test case prioritization: an industrial case study, in: Proceedings of IEEE International Conference on Software Maintenance (ICSM), September 2011.10.1109/ICSM.2011.6080805Search in Google Scholar

[13] C. A. Coello, G. T. Pulido and M. Salazar Lechuga, Handling multiple objectives with particle swarm optimization, IEEE T. Evol. Comput.8 (2004), 256–279.10.1109/TEVC.2004.826067Search in Google Scholar

[14] A. da Silva Simão and L. J. Senger, A technique to reduce the test case suites for regression testing based on a self-organizing neural network architecture, in: Proceedings of the 30th Annual International Computer Software and Application Conference (COMPSAC’06), 2006.10.1109/COMPSAC.2006.103Search in Google Scholar

[15] K. Deb, S. Agrawal, A. Pratab and T. Meyarivan, A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: NSGA-II, in: Proceedings of the Parallel Problem Solving from Nature VI Conference, Lecture Notes in Computer Science No. 1917, pp. 849–858, Springer, Paris, France, 2000.Search in Google Scholar

[16] H. Do, S. Elbaum and G. Rothermel, Supporting controlled experimentation with testing techniques: an infrastructure and its potential impact. Empir. Softw. Eng.10 (2005), 405–435.10.1007/s10664-005-3861-2Search in Google Scholar

[17] H. Do, S. Mirarab, L. Tahvildari and G. Rothermel, The effects of time constraints on test case prioritization: a series of controlled experiments, IEEE Trans. Softw. Eng.36 (2010) 593–617.10.1109/TSE.2010.58Search in Google Scholar

[18] S. G. Elbaum, P. Kallakuri, A. G. Malishevsky and G. Rothermel and S. Kanduri, Understanding the effects of changes on the cost-effectiveness of regression testing techniques, Journal of Software Testing, Verification, and Reliability, 13 (2001), 65–83.10.1002/stvr.263Search in Google Scholar

[19] Guide to the software engineering body of knowledge – SWEBOK, A. Abran, P. Bourque, R. Dupuis and J. W. Moore (Eds.), IEEE Press, Piscataway, NJ, USA, 2004.Search in Google Scholar

[20] A. A. Haider, S. Rafique and A. Nadeem, Test suite optimization using fuzzy logic, in: 8th International Conference of Emerging Techniques (ICET), 8th October 2012.10.1109/ICET.2012.6375440Search in Google Scholar

[21] M. Harman, Making the case for MORTO: multi objective regression test optimization, in: Software Testing, Verification and Validation Workshops (ICSTW), 2011 IEEE Fourth International Conference on. IEEE, 2011.10.1109/ICSTW.2011.60Search in Google Scholar

[22] H.-Y. Hsu and A. Orso, MINTS: a general framework and tool for supporting test-suite minimization, in: Proceedings of the 31st IEEE and ACM SIGSOFT International Conference on Software Engineering (ICSE 2009), Vol 1, pp. 419–429, 2009.10.1109/ICSE.2009.5070541Search in Google Scholar

[23] M. Hutchins, et al. Experiments of the effectiveness of dataflow-and control flow-based test adequacy criteria, in: Proceedings of the 16th international conference on Software engineering, IEEE Computer Society Press, 1994.Search in Google Scholar

[24] J. Jablonský, Software support for multiple criteria decision making problems, Management4 (2009), 029–034.Search in Google Scholar

[25] J.-S. R. Jang, ANFIS: adaptive-network-based fuzzy inference system, IEEE Trans. Syst. Man Cybern.23, (1993), 665–685.10.1109/21.256541Search in Google Scholar

[26] J.-S. R. Jang, Input selection for ANFIS learning. Fuzzy Systems, 1996, in: Proceedings of the Fifth IEEE International Conference on., Vol. 2, IEEE, 1996.Search in Google Scholar

[27] J.-S. R. Jang, C.-T. Sun and E. Mizutani, Neuro-fuzzy and soft computing: a computational approach to learning and machine learning, Prentice Hall Upper Saddle River, NJ 07458, ISBN: 0-13-261066-3, 1997.Search in Google Scholar

[28] A. Kaur, A bee colony optimization algorithm for code coverage test suite prioritization, Int. J. Eng. Sci. Technol. (IJEST), 3 (2011), 2786–2795.Search in Google Scholar

[29] A. Kaur and S. Goyal, A genetic algorithm for regression test case prioritization using code coverage, Int. J. Comput. Sci. Eng. (IJCSE), ISSN: 0975-3397 3 (2011), 1839–1847.Search in Google Scholar

[30] A. Kaur and S. Goyal, A genetic algorithm for fault based regression test case prioritization, Int. J. Comput. Appl. (0975 – 8887) 32 (2011), 30–37.Search in Google Scholar

[31] A. Konak, D. W. Coit and A. E. Smith, Multi-objective optimization using genetic algorithms: a tutorial, Reliab. Eng. Syst. Saf.91 (2006), 992–1007.10.1016/j.ress.2005.11.018Search in Google Scholar

[32] Z. Li, M. Harman and R. M. Hierons, Search algorithms for regression test case prioritization, IEEE T. Softw. Eng.33 (2007), 225–237.10.1109/TSE.2007.38Search in Google Scholar

[33] L. Magdalena, What is soft computing? Revisiting possible answers, Int. J. Comput. Intell. Syst.3 (2010), 148–159.10.1142/9789812799470_0001Search in Google Scholar

[34] R. Malhotra, A. Kaur and Y. Singh, A regression test selection and prioritization technique, J. Inf. Process. Syst.6 (2010), 235–252.10.3745/JIPS.2010.6.2.235Search in Google Scholar

[35] E. H. Mamdani and S. Assilian, An experiment in linguistic synthesis with a fuzzy logic controller, Int. J. Man Mach. Stud.7 (1975), 1–13.10.1016/B978-1-4832-1450-4.50032-8Search in Google Scholar

[36] MATLAB: Fuzzy Logic toolbox user’s guide version 2.2.16 The MathWorks, 2012.Search in Google Scholar

[37] S. Nachiyappan, A. Vimaladevi and C. B. Selva Lakshmi, An evolutionary algorithm for regression test suite reduction, in: Proceedings of the International Conference on Communication and Computational Intelligence – 2010, pp 503–508, 2010.Search in Google Scholar

[38] A. Nanda, S. Mani, S. Sinha, M. J. Harrold and A. Orso, Regression testing in the presence of non-code changes, in: Software Testing, Verification and Validation (ICST), 2011 IEEE Fourth International Conference on, pp. 21–30. IEEE, 2011.10.1109/ICST.2011.60Search in Google Scholar

[39] M. Nashat and W. Statieh, Regression test selection for C# programs, Adv. Softw. Eng.2009 (2009), 1.10.1155/2009/535708Search in Google Scholar

[40] S. Parsa and A. Khalilian, On the optimization approach towards test suite minimization, Int. J. Softw. Eng. Appl.4 (2010), 15–28.Search in Google Scholar

[41] S. Prasad, M. Jain and S. Singh, Regression optimizer a multi coverage criteria test suite minimization technique, Int. J. Appl. Inf. Syst. (IJAIS) – ISSN: 2249-0868, Foundation of Computer Science FCS, New York, USA Volume 1– No.8, April 2012 – www.ijais.org, pp. 5–11.10.5120/ijais12-450215Search in Google Scholar

[42] S. Raju and G. V. Uma, Factors oriented test case prioritization technique in regression testing using genetic algorithm. Eur. J. Sci. Res.74 (2012), 389–402.Search in Google Scholar

[43] S. Sumathi and P. Surekha, Computational intelligence paradigms theory and applications using MATLAB, CRC Press Taylor & Francis Group, Boca Raton, London, New York, ISBN: 978-1-4398-0902-0, 2010.10.1201/9781439809037Search in Google Scholar

[44] C.-A. Sun, A constraint-based test suite reduction method for conservative regression testing, J. Softw.6 (2011), 314–321.10.4304/jsw.6.2.314-321Search in Google Scholar

[45] L. H. Tahat, A. Bader, B. Vaysburg and B. Korel. Requirement-based automated black-box test generation, in: Proceedings of the 25th International Computer Software and Applications Conference on Invigorating Software Development (COMPSAC ’01), pp 489–495, IEEE Computer Society, Washington, DC, USA, 2001.Search in Google Scholar

[46] T. Takagi and M. Sugeno. Fuzzy identification of systems and its applications to modeling and control, IEEE T. Syst. Man Cybern.1 (1985), 116–132.10.1016/B978-1-4832-1450-4.50045-6Search in Google Scholar

[47] J. Vieira, F. Morgado Dias and A. Mota. Neuro-fuzzy systems: a survey, in: 5th WSEAS NNA International Conference on Neural Networks and Applications, Udine, Italia, 2004.Search in Google Scholar

[48] G. Whyte and D. L. Mulder, Mitigating the impact of software test constraints on software testing effectiveness, Electron. J. Inf. Syst. Eval.14 (2011), 254–270.Search in Google Scholar

[49] W. E. Wong, J. R. Horgan, S. London and H. Agrawal, A study of effective regression testing in practice, in: 8th IEEE International Symposium on Software Reliability Engineering (ISSRE’97), pp. 264–274, Albuquerque, NM, November 1997.Search in Google Scholar

[50] Z. Xu, K. Gao, T. M. Khoshgoftaar and N. Seliya, System regression test planning with a fuzzy expert system. Inf. Sci. 259 (2014), 532–543.10.1016/j.ins.2010.09.012Search in Google Scholar

[51] S. Yoo and M. Harman, Regression testing minimisation, selection and prioritisation: a survey, Softw. Test. Verif. Reliab.00:1–7 (2007), 1–60.Search in Google Scholar

[52] S. Yoo and M. Harman, Pareto efficient multi-objective test case selection, in: ISSTA, pp. 140–150, 2007.10.1145/1273463.1273483Search in Google Scholar

[53] X. Zhiwei, K. Gao and T. M. Khoshgoftaar. Application of fuzzy expert system in test case selection for system regression test, in: IRI-2005 IEEE International Conference on Information Reuse and Integration, Conf, 2005., IEEE, 2005.Search in Google Scholar

[54] Q. Zhongsheng, Test case generation and optimization for user session-based web application testing, J. Comput.5 (2010), 1655–1662.10.4304/jcp.5.11.1655-1662Search in Google Scholar

[55] A. Zhou, B.-Y. Qu, H. Li, S.-Z. Zhao, P. N. Suganthan and Q. Zhang, Multiobjective evolutionary algorithms: a survey of the state of the art, Swarm Evol. Comput.1 (2011), 32–49.10.1016/j.swevo.2011.03.001Search in Google Scholar

Received: 2014-10-13
Published Online: 2015-5-5
Published in Print: 2016-4-1

©2016 by De Gruyter

This article is distributed under the terms of the Creative Commons Attribution Non-Commercial License, which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Downloaded on 28.3.2024 from https://www.degruyter.com/document/doi/10.1515/jisys-2014-0152/html
Scroll to top button