Lightweight Static Analysis for GUI Testing

ISSRE 2012 Supporting Material

This page provides supporting material for the paper Lightweight Static Analysis for GUI Testing, presented at the International Symposium on Software Reliability Engineering (ISSRE 2012).

Contents

  1. Example Application
  2. Statistics
  3. EDG-2

1. Example Application

Click here to download the example from our submission. Then, run the application by double-clicking example.jar.

2. Statistics

2.1 Events

JabRefFreeMind
RachotaTerpWord

The light green part of each Figure represents the events which DO NOT have an event handler. The dark green part of each Figure represents the events which DO have an event handler. If an event has no event handlers, then the event is irrelevant. Irrelevant events are considered in the generation of event flow sequences in the black-box approach, e.g., EFG-2. In contrast, irrelevant are discarded in the Static Analysis and, thus, in the generation of event dependency sequences, e.g., EDG-2.

2.2 Fields

FieldsRead

For each application, this Figure shows how many fields are read inside conditions, and how many fields are read outside conditions. We observe that 24% - 32% of the field reads are inside conditions. In order to trigger a certain branch of a condition (that is, increasing the branch coverage), suitable test data for fields is needed. Since we focus on sequence generation, data generation is not in the scope of this paper.

2.3 Invokes

Invokes

The Static Analysis of our approach explores the writes and reads of each event handler, i.e., we collect the PUTFIELD and GETFIELD instructions of the bytecode. Further, the Static Analysis explores the reads and writes of each method invoked in the event handler, i.e., we follow the INVOKE instructions of the bytecode.
This boxplot shows how many method invocations the Static Analysis has to explore. For example, JabRef contains event handlers with at least 3 method invocations. Thus, the Static Analysis explores 3 method invocations at a minimum. Further, the Static Analysis explores 425 method invocations at a maximum. In particular, JabRef contains event handlers that have up to 1200 method invocations that are explored by the Static Analysis.

3. EDG-2

3.1 Detected Bugs

Here we illustrate all bugs found in our experiments for n = 2:

Bug #1: In JabRef, the following test sequence causes an ArrayOutOfBoundsException: (1) In the main window, click Database properties, which opens a new dialog; (2) switch to the main window and choose Close database. Then, (3) switch to the previously opened dialog and click OK.

In the test sequence generation for event Close database, our approach detects the dependency to event OK. When setting the parameter n = 2, the event dependency sequence « Close database, OK » is generated and then converted into the test sequence « Database properties, Close database, Database properties, OK ». In the pure black-box approach, one has to set the parameter n = 3 in order to detect this bug.

EFG EDG

Bug #2: In JabRef, the following test sequence causes an ArrayOutOfBoundsException: (1) In the main window, click Manage content selectors, which opens a new dialog; (2) switch to the main window and choose Close database. Then, (3) switch to the previously opened dialog and click OK.

In the test sequence generation for event Close database, our approach detects the dependency to event OK. When setting the parameter n = 2, the event dependency sequence « Close database, OK » is generated and then converted into the test sequence « Manage content selectors, Close database, Manage content selectors, OK ». In the pure black-box approach, one has to set the parameter n = 3 in order to detect this bug.

EFG EDG

Bug #3: In JabRef, the following test sequence causes a NullPointerException: (1) In the main window, click Manage custom imports, which opens a new dialog; (2) In the new dialog, click Add from folder and press Cancel.

In our approach and in the pure black-box approach, one has to set the parameter n = 2 in order to detect this bug.

EFG EDG

Bug #4: In Rachota, the following test sequence causes a NullPointerException at restart: Click on Menu, System, System Settings. Click on Add task, leave the text fields blank, and click the OK. Click on Menu, System, System Settings. Click on OK.

In the test sequence generation for event OK2, our approach detects the dependency to event OK1. When setting the parameter n = 2, the event dependency sequence « OK2, OK1 » is generated and then converted into the test sequence « Menu, System, System settings, Add task, OK2, Menu, System, System settings, OK1 ». In the pure black-box approach, one has to set the parameter n = 5 in order to detect this bug.

EFG EDG

3.2 Overall Result

Here we report the overall result of our experiments. This includes the comparison to Random-3, and the number of broken test sequences. Broken test sequences could not be executed on the GUI due to the imprecision of the Event Flow Graph. Thus, broken test sequences were discarded.

Application under Test EFG-2 EDG-2 Random-2
JabRef
# test sequences 43,017 5,860 5,860
generation time (m) 93 21 13
generation time per test sequence (s) 0.13 0.22 0.13
execution time (h) 358 49 49
line coverage (%) 54 54 19
branch coverage (%) 26 26 14
# detected bugs bug bug bug bug -
# broken test sequences 258 83 120
 
FreeMind
# test sequences 11,396 9,944 9,944
generation time (m) 25 43 22
generation time per test sequence (s) 0.13 0.21 0.13
execution time (h) 98 88 86
line coverage (%) 53 53 32
branch coverage (%) 37 37 21
# detected bugs - - -
# broken test sequences 57 63 129
 
Rachota
# test sequences 1,310 1,407 1,407
generation time (m) 3 4 3
generation time per test sequence (s) 0.14 0.17 0.13
execution time (h) 6 6 6
line coverage (%) 61 62 49
branch coverage (%) 34 36 24
# detected bugs - bug -
# broken test sequences 0 0 0
 
TerpWord
# test sequences 3,307 2,695 2,695
generation time (m) 7 8 5
generation time per test sequence (s) 0.13 0.18 0.11
execution time (h) 12 10 10
line coverage (%) 55 55 41
branch coverage (%) 36 36 22
# detected bugs - - -
# broken test sequences 0 0 0