Dr.-Ing. Frank Böhr

I studied applied computer science with a focus on embedded systems at the Technical University of Kaiserslautern.
After receiving my diploma (Dipl.-Inf.) in computer science I started to work at the Fraunhofer Institute for Experimental Software Engineering (IESE).
During my time at the IESE I focused on quality assurance of embedded software systems. Supported by a scholarship from the “Klaus Tschira Graduate School on
the Architecture of Highly Reliable Software Systems”, I returned to the Technical University of Kaiserslautern to work on my PhD thesis (The Usage Net Approach).
Here my focus was on Model-based testing of embedded software systems and reliability estimation.
Currently I hold a PostDoc position within the graduate school “Embedded Microsystems” at the Albert-Ludwig-University of Freiburg/Breisgau.

Selected Publications

F. Böhr (PhD Thesis),
Model-Based Statistical Testing of Embedded Real-Time Software with Continuous and Discrete Signals in a Concurrent Environment: The Usage Net Approach,
Technische Universität Kaiserslautern (ISBN-10: 3843903484), 2011

F. Böhr, R. Eschbach,
SIMOTEST: A tool for automated testing of hybrid real-time Simulink models,
16th Conference on Emerging Technologies & Factory Automation, 2011

F. Böhr,
Model-Based Statical Testing of Embedded Systems,
7th Workshop on Advances in Model Based Testing, 2011

F. Böhr,
Model-Based Statistical Testing and Concurrent Streams of Use,
3nd Workshop on Model-based Testing in Practice, 2010

F. Böhr,
Model-Based Statistical Testing and Durations,
7th Workshop on System Testing and Validation, 2010

F. Böhr,
Model-Based Statistical Testing and Time,
2nd Workshop on Model-based Testing in Practice, 2009

T. Bauer, F. Böhr, R. Eschbach,
On MiL, HiL Statistical Testing, Reuse and Effort,
Fraunhofer IESE, 2008

T. Bauer, T. Beletski, F. Böhr, R. Eschbach, D. Landman, J. H. Poore,
From requirements to automated testing of quasar aussenspiegeleinstellung,
Fraunhofer IESE, 2007

T. Bauer, T. Beletski, F. Böhr, R. Eschbach, D. Landmann, J. H.Poore,
From Requirements to Statistical Testing of Embedded Systems,
Proceedings of 4th International ICSE Workshop on Software Engineering for Automotive Systems, 2007

M. Anastasopoulos, F. Böhr, T. Patzke, C. Peper, D. Schneider, M. Sohn,
Ambient intelligence systems. Scenarios and challenges,
Fraunhofer IESE, 2006

F. Böhr (Diploma Thesis),
Anwendung und Evaluierung verhaltensbasierter Ansätze zur Entwicklung verlässlicher Automotivesysteme,
Technische Universität Kaiserslautern, 2006

Teaching

Summer 2012,
Einführung in die Informatik (Lecture),
More information

Invited Talks

SQRL-Week 10,
Model-Based Statistical Testing and Simulink,
University of Tennessee, 2010

SQRL-Week 09,
Model-Based Statistical Testing and Stochastic Petri-Nets,
University of Tennessee, 2009

Research Interests

My main interest is Model-based software testing, especially Model-based statistical testing (MBST).
A list of papers from this field is given below.

A constraint-based approach to the representation of software usage models

Type	Journal Article
Author	J. H. Poore
Author	G.H. Walton
Author	J.A. Whittaker
Abstract	Software usage models are the basis for statistical testing. They derive their structure from specifications and their probabilities from evolving knowledge about the intended use of the software product. The evolving knowledge comes from developers, customers and testers of the software system in the form of relationships that should hold among the parameters of a model. When software usage models are encoded as Markov chains, their structure can be represented by a system of linear constraints, and many of the evolving relationships among model parameters can be represented by convex constraints. Given a Markov chain usage model as a system of convex constraints, mathematical programming can be used to generate the Markov chain transition probabilities that represent a specific software usage model.
Publication	Information and Software Technology
Volume	42
Issue	12
Pages	825-833
Date	2000.9.1
DOI	doi:10.1016/S0950-5849(00)00101-4
URL	http://www.ingentaconnect.com/content/els/09505849/2000/00000042/00000012/art00101
Date Added	Thu 13 Nov 2008 02:28:15 PM CET
Modified	Thu 13 Nov 2008 02:30:30 PM CET

A Cost-Benefit Stopping Criterion for Statistical Testing

Type	Conference Paper
Author	S. J. Prowell
Abstract	Determining when to stop a statistical test is an important management decision. Several stopping criteria have been proposed, including criteria based on statistical similarity, the probability that the system has a desired reliability, and the expected cost of remaining faults. This paper proposes a new stopping criterion based on a cost-benefit analysis using the expected reliability of the system (as opposed to an estimate of the remaining faults). The expected reliability is used, along with other factors such as units deployed and expected use, to anticipate the number of failures in the field and the resulting anticipated cost of failures. Reductions in this number generated by increasing the reliability are balanced against the cost of further testing to determine when testing should be stopped.
Date	2004
Proceedings Title	Proceedings of the Proceedings of the 37th Annual Hawaii International Conference on System Sciences
Conference Name	International Conference on System Sciences
Place	Hawaii
Publisher	IEEE Computer Society
Volume	9
Pages	pp. 6
ISBN	0-7695-2056-1
Date Added	Tue 14 Oct 2008 04:15:42 PM CEST
Modified	Tue 27 Sep 2011 10:19:03 AM CEST

Adapted Statistical Usage Testing: A Case Study

Type	Conference Paper
Author	Perter B. Lakey
Abstract	This paper provides a summary of the experience and lessons learned when applying statistical usage testing (SUT) to the U.S. Army's Crusader program. Even though the program was discontinued in August, 2002, significant results were obtained that are worthy of reporting to the software reliability engineering community. The cancelled Crusader was to be a technically advanced, fully automated artillery system. Due to the high degree of automation and the substantial portion of the system to be controlled by software, the Army's procurement agency required a software reliability assessment capability. United Defense Industries (UDLP), Crusader's prime contractor, formed a subcontract with Cognitive Concepts (CC) to lead the software reliability support effort. Based on knowledge and experience, CC recommended the SUT methodology be used to assess the reliability of the tactical software. CC believes that SUT is the most fundamentally sound method available today for certifying the reliability of systems with safety and mission-critical software [1]. In statistical usage testing, behavioral models of the usage of a software system are constructed. Then a large sample of tests are randomly generated from the models. After the sample is executed and results obtained a project can determine, with a high degree of confidence, how reliable the software is as measured during system test.
Date	1.9.2002
Proceedings Title	Proceedings 13th International Symposium on Software Reliability Engineering
Conference Name	International Symposium on Software Reliability Engineering
Place	Annapolis
ISBN	0-7695-1763-3
Date Added	Tue 14 Oct 2008 04:23:14 PM CEST
Modified	Tue 30 Aug 2011 11:47:15 AM CEST

A Machine Learning Approach for Statistical Software Testing.

Type	Conference Paper
Author	Nicolas Baskiotis
Author	Michèle Sebag
Author	Marie-Claude Gaudel
Author	Sandrine-Dominique Gouraud
Editor	Manuela M. Veloso
Date	2007
Proceedings Title	IJCAI
Pages	2274-2279
URL	http://dblp.uni-trier.de/db/conf/ijcai/ijcai2007.html#BaskiotisSGG07
Accessed	Thu 22 Jan 2009 10:07:13 AM CET
Archive	unAPI
Date Added	Thu 22 Jan 2009 10:07:13 AM CET
Modified	Thu 22 Jan 2009 10:07:13 AM CET

A Markov Chain Model for Statistical Software Testing

Type	Journal Article
Author	James A. Whittaker
Author	Michael G. Thomason
Abstract	Statistical testing of software establishes a basis for statistical inference about a software system's expected field quality. This paper describes a method for statistical testing based on a Markov chain model of software usage. The significance of the Markov chain is twofold. First, it allows test input sequences to be generated from multiple probability distributions, making it more general than many existing techniques. Analytical results associated with Markov chains facilitate informative analysis of the sequences before they are generated, indicating how the test is likely to unfold. Second, the test input sequences generated from the chain and applied to the software are themselves a stochastic model and are used to create a second Markov chain to encapsulate the history of the test, including any observed failure information. The influence of the failures is assessed through analytical computations on this chain. We also derive a stopping criterion for the testing process based on a comparison of the sequence generating properties of the two chains.
Publication	IEEE Transactions on Software Engineering
Volume	20
Issue	10
Pages	812-824
Date	1994.10
URL	http://portal.acm.org/citation.cfm? id=188730&dl=
Accessed	Wed 22 Oct 2008 09:22:33 AM CEST
Library Catalog	ACM
Date Added	Wed 22 Oct 2008 09:22:33 AM CEST
Modified	Mon 14 Dec 2009 01:55:23 PM CET

An API testing method

Type	Conference Paper
Author	Paul C. Jorgenson
Author	James A. Whittaker
Abstract	This paper presents a technique to test APIs that combines aspects of two published software testing methods, namely Markov modeling and category partitioning. Markov modeling provides a basis for model based testing. Markov modeling establishes the context for generating API calls and call sequences within a single test case. For modeling purposes, each combination of parameter values for each function call is a unique “input.” Category partitioning helps select parameter values and effective combinations of multiple parameters for individual API calls. Small examples demonstrate these techniques and two case study summaries illustrate its effectiveness. One case under laboratory conditions established proof-of-concept and the other applicability to a large commercial API. Some aspects of these techniques are manually intensive and suggest a need for automation.
Date	2000.5.1
Proceedings Title	International Conference on Software Testing Analysis and Review
Conference Name	STAREAST
Place	Orlando
Date Added	Thu 13 Nov 2008 03:05:10 PM CET
Modified	Thu 13 Nov 2008 03:13:18 PM CET

An automated testing environment to support operational profiles of software intensive systems

Type	Conference Paper
Author	R. S. Oshana
Abstract	Proceedings of the International Software Quality Week
Date	1999
Place	San Francisco
Publisher	Software Research Institute
Date Added	Thu 15 Jan 2009 03:46:00 PM CET
Modified	Thu 15 Jan 2009 03:47:43 PM CET

Application of statistical science to testing and evaluating software intensive systems

Type	Conference Paper
Author	J.H. Poore
Author	C.J. Trammell
Abstract	Defense systems are becoming increasingly software intensive. While software enhances the effectiveness and flexibility of these systems, it also introduces vulnerabilities related to inadequacies in software design, maintenance, and configuration control. Effective testing of these systems must take into account the special vulnerabilities introduced by software. The software testing problem is complex because of the astronomical number of scenarios and states of use. The domain of testing is large and complex beyond human intuition. Because the software testing problem is so complex, statistical principles must be used to guide testing strategy in order to get the best information for the resources invested in testing. From a statistical point of view, all the topics in this paper follow sound problem solving principles and are direct applications of well established theory and methodology. From a software testing point of view the application of statistical science is relatively new and rapidly evolving, as an increasing range of statistical principles is applied to a growing variety of systems. Statistical testing is used in pockets of industry and agencies of government, including the DoD, on both experimental and routine bases. This paper is a composite of what is in hand and within reasonable reach in the application of statistical science to software testing
Date	1999
Proceedings Title	Science and Engineering for Software Development: A Recognition of Harlan D. Mills' Legacy, 1999. Proceedings
Conference Name	Science and Engineering for Software Development: A Recognition of Harlan D. Mills' Legacy, 1999. Proceedings
Pages	40-57
DOI	10.1109/SESD.1999.781111
Archive	IEEE Xplore
Date Added	Fri 17 Oct 2008 05:53:27 PM CEST
Modified	Wed 22 Oct 2008 12:29:29 PM CEST

Applying models in your testing process

Type	Journal Article
Author	S. Rosaria
Author	H. Robinson
Abstract	Model-based testing allows large numbers of test cases to be generated from a description of the behavior of the system under test. Given the same description and test runner, many types of scenarios can be exercised and large areas of the application under test can be covered, thus leading to a more effective and more efficient testing process.
Publication	Information and Software Technology
Volume	42
Issue	12
Pages	815-824
Date	September 1, 2000
DOI	10.1016/S0950-5849(00)00125-7
URL	http://www.sciencedirect.com/science/article/B6V0B-414N9FF-4/2/59276a620928e543d5caad4e8cc44479
Accessed	Thu 13 Nov 2008 02:22:33 PM CET
Library Catalog	ScienceDirect
Date Added	Thu 13 Nov 2008 02:22:33 PM CET
Modified	Thu 13 Nov 2008 02:22:33 PM CET

A Scenario-Centric Approach for the Definition of the Formal Test Specifications of Reactive Systems

Type	Conference Paper
Author	V. Entin
Author	S. Siegl
Author	K. Meyer-Wegener
Author	A. Kern
Author	M. Reichel
Abstract	Complex modern embedded automotive software systems require different test techniques in each of the development stages. Most common are Model in the Loop, Software in the Loop, and Hardware in the Loop. The majority of these test techniques are automated. Each uses different notations for test data, pass/fail criteria, system-under-test interface definition and test-platform-specific parameterization. This leads to a series of problems such as exchangeability of test-specification notations among different teams working on the same functional module, reusability of test cases and uniformity of test-specification representation. This contribution proposes an approach for the formal and test-platform-independent definition of the test specification of reactive systems. Additionally, the application of the approach in three concrete use-case scenarios elicited in a pre-development department of AUDI AG is shown.
Date	4-6 Sept. 2009
Proceedings Title	Testing: Academic and Industrial Conference - Practice and Research Techniques, 2009. TAIC PART '09.
Conference Name	Testing: Academic and Industrial Conference - Practice and Research Techniques, 2009. TAIC PART '09.
Publisher	IEEE
Pages	179-183
Language	English
DOI	10.1109/TAICPART.2009.21
ISBN	978-0-7695-3820-4
Library Catalog	IEEE Xplore
Date Added	Wed 22 Feb 2012 03:04:36 PM CET
Modified	Wed 22 Feb 2012 03:04:36 PM CET

A Statistics-Based Framework For Automated Software Testing

Type	Journal Article
Author	Huey-Der Chu
Author	John E Dobson
Date	1996
URL	http://citeseerx.ist.psu.edu/viewdoc/summary? doi=10.1.1.52.5308
Accessed	Wed 28 Jul 2010 03:12:27 PM CEST
Date Added	Wed 28 Jul 2010 03:12:27 PM CEST
Modified	Wed 28 Jul 2010 03:30:04 PM CEST

Automated Functional Testing of Simulink Control Models

Type	Conference Paper
Author	J. Carter
Author	L. Lin
Author	J. Poore
Date	2008.6.12
Proceedings Title	1st Workshop on Model-based Testing in Practice - MoTiP
Conference Name	4th European Conference on Model Driven Architecture Foundations and Applications
Place	Berlin
Date Added	Mon 08 Jun 2009 10:19:42 AM CEST
Modified	Mon 08 Jun 2009 10:20:56 AM CEST

Automated Generation of Statistical Test Cases from UML State Diagrams

Type	Conference Paper
Author	Philippe Chevalley
Author	Pascale Thévenod-Fosse
Abstract	The adoption of the object-oriented (OO) technology for the development of critical software raises important testing issues. This paper addresses one of these issues: how to create effective tests from OO specification documents? More precisely, the paper describes a technique that adapts a probabilistic method, called statistical functional testing, to the generation of test cases from UML state diagrams, using transition coverage as the testing criterion. Emphasis is put on defining an automatic way to produce both the input values and the expected outputs. The technique is automated with the aid of the Rational Software Corporation's Rose RealTime tool. An industrial case study from the avionics domain, formally specified and implemented in Java, is used to illustrate the feasibility of the technique at the subsystem level. Results of first test experiments are presented to exemplify the fault revealing power of the created statistical test cases.
Date	2001
Proceedings Title	Proceedings of the 25th International Computer Software and Applications Conference on Invigorating Software Development
Publisher	IEEE Computer Society
Pages	205-214
ISBN	0-7695-1372-7
URL	http://portal.acm.org/citation.cfm? id=645983.675105
Accessed	Thu 13 Nov 2008 04:37:32 PM CET
Archive	ACM
Date Added	Thu 13 Nov 2008 04:37:32 PM CET
Modified	Thu 13 Nov 2008 04:37:32 PM CET

Automated gui testing guided by usage profiles

Type	Conference Paper
Author	Penelope A. Brooks
Author	Atif M. Memon
Abstract	Most software developed in recent years has a graphical userinterface (GUI). The only way for the end-user to interact with the software application is through the GUI. Hence, acceptance and system testing of the software requires GUI testing. This paper presents a new technique for testing of GUI applications. Information on the actual usage of the application, in the form of "usage profiles," is used to ensure that a new version of the application will function correctly. Usage profiles, sequences of events that end-users execute on a GUI, are used to develop a probabilistic usage model of the application. An algorithm uses the model to generate test cases that represent events the user is most likely to execute. Reverse engineering methods are used to extract the underlying structure of the application. An empirical study on four open source GUI applications reveals that test suites generated from the probabilistic model are 0.2-22% of the size of test suites produced directly from usage profiles. Furthermore, the test suites generated from the model detect more faults per test case than those detected directly from the usage profiles, and detect faults not detected by the original profiles
Date	2007
Proceedings Title	Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering
Place	Atlanta, Georgia, USA
Publisher	ACM
Pages	333-342
DOI	10.1145/1321631.1321681
ISBN	978-1-59593-882-4
URL	http://portal.acm.org/citation.cfm? id=1321681
Accessed	Wed 26 Nov 2008 05:53:22 PM CET
Archive	ACM
Date Added	Wed 26 Nov 2008 05:53:22 PM CET
Modified	Wed 26 Nov 2008 05:53:22 PM CET

Automated testing of embedded automotive systems from requirement specification models

Type	Conference Paper
Author	S. Siegl
Author	K. -S Hielscher
Author	R. German
Author	C. Berger
Abstract	Embedded software for modern automotive and avionic systems is increasingly complex. In early design phases, even when there is still uncertainty about the feasibility of the requirements, valuable information can be gained from models that describe the expected usage and the desired system reaction. The generation of test cases from these models indicates the feasibility of the intended solution and helps to identify scenarios for which the realization is hardly feasible or the intended system behavior is not properly defined. In this paper we present the formalization of requirements by models to simulate the expected field usage of a system. These so called usage models can be enriched by information about the desired system reaction. Thus, they are the basis for all subsequent testing activities: First, they can be used to verify the first implementation models and design decisions w.r.t. the fulfillment of requirements and second, test cases can be derived in a random or statistic manner. The generation can be controlled with operational profiles that describe different classes of field usage. We have applied our approach at a large German car manufacturer in the early development phase of active safety functionalities. Test cases were generated from the usage models to assess the implementation models in MATLAB/Simulink. The parametrization of the systems could be optimized and a faulty transition in the implementation models was revealed. These design and implementation faults had not been discovered with the established test method.
Date	27-30 March 2011
Proceedings Title	Test Workshop (LATW), 2011 12th Latin American
Conference Name	Test Workshop (LATW), 2011 12th Latin American
Publisher	IEEE
Pages	1-6
Language	English
DOI	10.1109/LATW.2011.5985928
ISBN	978-1-4577-1489-4
Library Catalog	IEEE Xplore
Date Added	Wed 22 Feb 2012 02:30:31 PM CET
Modified	Wed 22 Feb 2012 03:19:14 PM CET

Automated Testing of Generic Computational Science Libraries

Type	Conference Paper
Author	Kirk Sayre
Author	Jesse Poore
Abstract	The design and maintenance of large computational science systems can be aided greatly by the use of generic computational science libraries. The object-oriented design of software using a generic library presents many advantages over non-object-oriented or non-generic based software design. However, the testing of generic libraries involves some unique issues not usually faced during the testing of non-generic libraries. This paper describes these testing issues as they pertain to libraries designed using C++ templates. A proposed solution to these issues using a usage model-based automated testing framework would be presented
Date	2007
Proceedings Title	40th Annual Hawaii International Conference on System Sciences, 2007. HICSS 2007.
Conference Name	Hawaii International Conference on System Sciences
Place	Waikoloa, HI
Pages	277
DOI	10.1109/HICSS.2007.97
ISBN	1530-1605
Archive	IEEE Xplore
Date Added	Mon 17 Nov 2008 09:31:40 AM CET
Modified	Tue 27 Sep 2011 10:38:20 AM CEST

Certifying the correctness of software

Type	Conference Paper
Author	H.D. Mills
Abstract	Software is either correct or incorrect in design to a specification in contrast to hardware which is reliable to a certain level to a correct design. Software of any size or complexity can only be tested partially, and typically a very small fraction of possible inputs are actually tested. Certifying the correctness of such software requires two conditions, namely (1) statistical testing with inputs characteristic of actual usage, and (2) no failures in the testing. If any failures arise in testing or subsequent usage, the software is incorrect, and the certification invalid. If such failures are corrected, the certification process can be restarted, with no use of previous testing
Date	1992
Proceedings Title	Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences
Conference Name	Twenty-Fifth Hawaii International Conference on System Sciences
Place	Kauai, Hawaii
Publisher	IEEE Computer Society Press
Volume	2
Pages	373-381
DOI	10.1109/HICSS.1992.183251
Archive	IEEE Xplore
Date Added	Wed 22 Oct 2008 12:27:59 PM CEST
Modified	Mon 26 Sep 2011 05:15:47 PM CEST

Cleanroom Software Engineering Reference Model Version 1.0

Type	Report
Author	Richard C. Linger
Author	Carmen J. Trammel
Abstract	Cleanroom software engineering is a theory-based, team-oriented process for development and certification of high-reliability software systems under statistical quality control [Mills 92, Linger 93, Linger 94]. A principal objective of the Cleanroom process is development of software that exhibits zero failures in use. The Cleanroom name is borrowed from hardware Cleanrooms, with their emphasis on rigorous engineering discipline and focus on defect prevention rather than defect removal. Cleanroom combines mathematically-based methods of software specification, design, and correctness verification with statistical, usage-based testing to certify software fitness for use. Cleanroom projects have reported substantial gains in quality and productivity........
Report Number	Technical Report CMU/SEI-96-TR-022 ESC-TR-96-002
Report Type	Technical Report
Place	Pittsburg
Institution	CMU/SEI
Date	1996.11.1
Pages	135
Date Added	Thu 13 Nov 2008 03:33:38 PM CET
Modified	Mon 14 Dec 2009 10:37:59 AM CET

Cleanroom Software Engineering: Technology and Process

Type	Book
Author	Stacy J. Prowell
Author	Carmen J. Trammell
Author	Richard C. Linger
Author	Jesse H. Poore
Publisher	Addison-Wesley Professional
Date	1999-03-19
ISBN	9780201854800
Short Title	Cleanroom Software Engineering
Library Catalog	Amazon.com
Date Added	Mon 04 Apr 2011 12:34:26 PM CEST
Modified	Mon 04 Apr 2011 12:34:26 PM CEST

Combinatorial Test Case Selection with Markovian Usage Models

Type	Conference Paper
Author	Sergiy A. Vilkomir
Author	W. Thomas Swain
Author	Jesse H. Poore
Abstract	A method of using Markov chain techniques for combinatorial test case selection is presented. The method can be used for statistical and coverage testing of many software programs, in particular for scientific computational software. The central point of the approach is modeling of dependencies between input parameters. Several different types of such dependencies are considered and models for each situation are created. Based on these models, test cases can be automatically generated and executed. Results of using the JUMBL tool for analyzing models and generating test cases are described.
Date	2008.4.7
Proceedings Title	Proceedings of the Fifth International Conference on Information Technology: New Generations (ITNG 2008)
Conference Name	ITNG
Place	Las Vegas, Nevada, USA
Publisher	IEEE Computer Society
Pages	3-8
ISBN	978-0-7695-3099-4
URL	http://portal.acm.org/citation.cfm? id=1396808.1397477&…
Accessed	Mon 03 Nov 2008 09:47:54 AM CET
Archive	ACM
Date Added	Mon 03 Nov 2008 09:47:54 AM CET
Modified	Mon 14 Dec 2009 01:15:35 PM CET

Computations for Markov Chain Usage Models

Type	Report
Author	S. J. Prowell
Abstract	This document summarizes the basic computations for Markov chain usage models, presents their derivations, and includes Scilab code to compute each of them. The contents of this document are the result of years of work by many different people, and very few results are original. James Whittaker, Michael Thomason, and Jesse Poore did the original work on Markov chain usage models [17, 18]. Gwen Walton’s research applied mathematical programming techniques to set model probabilities under testing constraints [16]. Jenny Morales [8] and Dave Pearson [9] investigated combining information across tests to improve reliability measurements. Kirk Sayre’s research provided many new and useful analytical results, and provided a framework for simulation and partition testing [13]. Walter Gutjahr demonstrated how a Markov chain could be modified to bias test generation toward low-use critical function, and how the bias could be removed in the results [1]. This document only discusses computations. Carmen Trammell and Jesse Poore have written about experimental control during the testing process [15]; Rob Oshana and Dave Kelly have written about their experience applying Markov chain usage models to very large, embedded, real-time, distributed systems [5, 4].
Report Number	UT-CS-03-505
Report Type	Technical
Place	Knoxville, TN
Institution	The University of Tennessee
Date	2000
Pages	80
Date Added	Wed 15 Oct 2008 12:34:31 PM CEST
Modified	Mon 14 Dec 2009 01:05:28 PM CET

Conditional-event usage testing

Type	Conference Paper
Author	Denise Woit
Abstract	In limited applications, operational profile testing has been shown to increase productivity, improve customer satisfaction, and reduce cost and time to market [4, 1, 5]. Unfortunately, underlying assumptions of this method make it generally inapplicable to the large class of software for which dependencies exist among the software's operations [2]. We propose conditional-event usage testing (CEUT), which will retain the benefits of traditional operational profile testing but which is not limited in the class of systems to which it applies. In CEUT, the expected usage of the software is modeled in such a way as to allow specification of conditions upon expected input event sequences. The model is used to generate test cases which will exercise the software in a fashion typical of actual operation. For systems with dependencies among operations, using CEUT resulted in test sequences that were typical of those experienced in operation, and allowed the benefits of operational profile-type testing to be realized on such systems.
Date	1998
Proceedings Title	Proceedings of the 1998 conference of the Centre for Advanced Studies on Collaborative research
Place	Toronto, Ontario, Canada
Publisher	IBM Press
Pages	23
URL	http://portal.acm.org/citation.cfm? id=783183&dl=GUIDE&…
Accessed	Wed 26 Nov 2008 06:00:32 PM CET
Archive	ACM
Date Added	Wed 26 Nov 2008 06:00:32 PM CET
Modified	Wed 26 Nov 2008 06:00:32 PM CET

Construction Method of a High-Order Markov Chain Usage Model

Type	Conference Paper
Author	Tomohiko Takagi
Author	Zengo Furukawa
Abstract	Statistical testing is a method for evaluating software reliability. Test cases are generated according to a usage model that represents the behavior of users. Accuracy of a usage model is an important factor since effectiveness of statistical testing (i.e., accuracy of software reliability) depends on it. This paper shows the usage model construction method using high-order Markov chains in order to construct accurate usage models. It is illustrated that the usage models constructed by this method are more accurate than traditional ones based on a simple or time-homogeneous Markov chain and can be constructed automatically. Some problems for more effective statistical testing are then found.
Date	2007
Proceedings Title	Proceedings of the 14th Asia-Pacific Software Engineering Conference
Publisher	IEEE Computer Society
Pages	120-126
ISBN	0-7695-3057-5
URL	http://portal.acm.org/citation.cfm? id=1332626
Accessed	Tue 10 Feb 2009 09:33:25 AM CET
Archive	ACM
Date Added	Tue 10 Feb 2009 09:33:25 AM CET
Modified	Tue 10 Feb 2009 09:33:25 AM CET

Design and Validation of Embedded Real-Time Applications

Type	Conference Paper
Author	Sebastian Siegel
Author	Christoph Lauer
Abstract	The design and validation of embedded real-time applications is challenging, especially when legacy sub-systems are involved. To account for the uncertainty in system-development at early design stages we use statistical modelling and discrete event simulation to perform sensitivity analysis. These analysis results provide vital information about the system characteristics and indicate usage scenarios where the behaviour of the system differs significantly from the average case. Based on the simulation results and the initial system requirements a usage model for the application is being set up. The model represents the requirements in an unambiguous and traceably correct manner. For each possible path through the model, considering stimuli and their timing, a unique system reaction is defined. This way the requirements are clarified. The usage model allows the derivation of test cases that can be used in the design phase to validate the model and in the acceptance phase to test the final system. Through the combination of the simulation results and the usage modelling we are able to: •identify critical system conditions. •validate the system design w.r.t. the usage model The proposed methods are currently applied in both the design and validation of safety critical applications.
Date	2010.05.01
Proceedings Title	Proc. of Embedded Real Time Software and Systems
Conference Name	ERTS 2010
Place	Toulouse
Date Added	Wed 22 Feb 2012 02:46:56 PM CET
Modified	Wed 22 Feb 2012 02:50:59 PM CET

Engineering Practices for Statistical Testing

Type	Document
Author	Jesse H. Poore
Author	Carmen J. Trammel
Abstract	This article describes the application of statistical science to the testing and evaluation of software and software-intensive systems. Engineering practices are described for statistical testing based on a usage model, which is an engineering formalism that represents the use of a system in a specific environment or situation, or for a specific customer class. Engineering practices for statistical testing are based on a view of software use as a stochastic process and of software testing as a problem amenable to statistical solution.
Date	1998
Date Added	Wed 15 Oct 2008 12:47:33 PM CEST
Modified	Thu 16 Oct 2008 03:39:55 PM CEST

Engineering software under statistical quality control

Type	Journal Article
Author	R.H. Cobb
Author	H.D. Mills
Abstract	Eight common misconceptions of software quality are examined and refuted. The concept of cleanroom engineering of software is introduced. Cleanroom engineering achieves intellectual control by applying rigorous, mathematics-based engineering practices, establishes an errors-are-unacceptable attitude and a team responsibility for quality, delegates development and testing responsibilities to separate teams, and certifies the software's mean time to failure through the application of statistical quality-control methods. A typical project is used to explain the concepts and procedures
Publication	Software, IEEE
Volume	7
Issue	6
Pages	45-54
Date	1990
Journal Abbr	Software, IEEE
DOI	10.1109/52.60601
ISSN	0740-7459
Library Catalog	IEEE Xplore
Date Added	Mon 10 Nov 2008 03:00:30 PM CET
Modified	Mon 10 Nov 2008 03:00:30 PM CET

Experiences in Applying Statistical Testing to a Real-Time, Embedded Software System

Type	Conference Paper
Author	K. Agrawal
Author	J.A. Whittaker
Date	1.10.1993
Proceedings Title	Proceedings of the Pacific Northwest Software Quality Conference
Conference Name	PNSQC
Place	Portland
Pages	154-170
URL	http://www2.umassd.edu/swpi/ProcessBibliography/bibliography.txt
Accessed	Tue 28 Oct 2008 11:57:21 AM CET
Date Added	Tue 28 Oct 2008 11:57:21 AM CET
Modified	Mon 09 Feb 2009 03:24:18 PM CET

Extending the Reach of Statistical Software Testing

Type	Thesis
Author	Robert J. Weber
Abstract	Statistical software testing is a promising technique for reducing the software testing burden by applying test cases to the software under test according to a model of the expected usage of the system in operation, called an operational profile. However, current statistical testing techniques have not been put into general practice, as they do not scale well as the complexity of the systems under test increases. In particular, as system complexity increases, the matrices required to generate test cases and perform model analysis can grow dramatically, even exponentially, overwhelming the test generation and analysis applications. In this work, a new technique for representing the operational profile is proposed to mitigate the complexity issue for statistical test case generation. The use of a state-based requirements specification model as the basis of the operational profile leverages parallelism to reduce the visible size of the model. The specification model can then be extended to include conditional probabilities of input data occurrence. * This work also proposes a statistical testing framework using this specification-based operational profile to generate and execute test cases. Such a framework allows us to evaluate the effectiveness and efficiency of this statistical testing technique.
Type	Doctoral thesis
University	University of Minnesota Minneapolis
Date	1.2.2004
# of Pages	170
Language	English
URL	http://oai.dtic.mil/oai/oai? &…
Accessed	Tue 28 Oct 2008 03:12:31 PM CET
Loc. in Archive	ADA421743
Date Added	Tue 28 Oct 2008 03:12:31 PM CET
Modified	Tue 28 Oct 2008 03:16:19 PM CET

Extract Test Sequences from a Markov Software Usage Model by ACO

Type Document

Author Karl Doerner

Author Walter J. Gutjahr

Date 1.7.2003

Date Added Wed 15 Oct 2008 09:42:28 AM CEST

Modified Thu 16 Oct 2008 03:57:05 PM CEST

Related
- Certifying the reliability of software
- Certifying the correctness of software
- Experiences in Applying Statistical Testing to a Real-Time, Embedded Software System
- Operational profiles in software-reliability engineering

Finite State Model-Based Testing on a Shoestring

Type	Conference Paper
Author	Robinson Harry
Abstract	Model-based testing is a software test technique that generates tests from an explicit model of software behavior. Modern programmable test tools allow us to use this technique to create useful, flexible and powerful tests at a very reasonable cost.
Date	1999
Proceedings Title	Software Testing Analysis & Review Conference 1999
Conference Name	Software Testing Analysis & Review Conference
Place	San Jose, California
Extra	Intelligent Search Test Group Microsoft Corporation
Date Added	Mon 19 Jan 2009 10:58:38 AM CET
Modified	Mon 26 Sep 2011 03:00:11 PM CEST

Formal specification and systematic model-driven testing of embedded automotive systems

Type	Conference Paper
Author	S. Siegl
Author	K. -S Hielscher
Author	R. German
Author	C. Berger
Abstract	Increasingly intelligent energy-management and safety systems are developed to realize safe and economic automobiles. The realization of these systems is only possible with complex and distributed software. This development poses a challenge for verification and validation. Upcoming standards like ISO 26262 provide requirements for verification and validation during development phases. Advanced test methods are requested for safety critical functions. Formal specification of requirements and appropriate testing strategies in different stages of the development cycle are part of it. In this paper we present our approach to formalize the requirements specification by test models. These models serve as basis for the following testing activities, including the automated derivation of executable test cases from it. Test cases can be derived statistically, randomly on the basis of operational profiles, and deterministically in order to perform different testing strategies. We have applied our approach with a large German OEM in different development stages of active safety and energy management functionalities. The test cases were executed in model-in-the-loop and in hardware-in-the-loop simulation. Errors were identified with our approach both in the requirement specification and in the implementation that were not discovered before.
Date	14-18 March 2011
Proceedings Title	Design, Automation & Test in Europe Conference & Exhibition (DATE), 2011
Conference Name	Design, Automation & Test in Europe Conference & Exhibition (DATE), 2011
Publisher	IEEE
Pages	1-6
Language	English
ISBN	978-1-61284-208-0
Library Catalog	IEEE Xplore
Date Added	Wed 22 Feb 2012 02:36:07 PM CET
Modified	Wed 22 Feb 2012 03:23:12 PM CET

Foundations of sequence-based software specification

Type	Journal Article
Author	S.J. Prowell
Author	J.H. Poore
Abstract	Rigorous specification early in the software development process can greatly reduce the cost of later development and maintenance, as well as provide an explicit means to manage risk and identify and meet safety requirements. Sequence-based software specification is a collection of techniques for implementing rigorous, practical software specification. The primary result of this research is the sequence enumeration method of specification writing. Straightforward, systematic enumeration of all sequences to produce an arguably complete, consistent, and traceably correct specification is made practical by controlling the growth of the process.
Publication	Software Engineering, IEEE Transactions on
Volume	29
Issue	5
Pages	417-429
Date	2003
Journal Abbr	Software Engineering, IEEE Transactions on
DOI	10.1109/TSE.2003.1199071
ISSN	0098-5589
Library Catalog	IEEE Xplore
Date Added	Mon 17 Nov 2008 09:17:01 AM CET
Modified	Mon 17 Nov 2008 09:17:01 AM CET

From random testing of hardware to statistical testing of software

Type	Conference Paper
Author	P. Thevenod-Fosse
Abstract	Random or statistical testing consists in exercising a system by supplying to it valued inputs which are randomly selected according to a defined probability distribution on the input domain. The specific role one can expect from statistical testing in a software validation process is pointed out. The notion of test quality with respect to the experiment goal allows the testing time to be adjusted to a target test quality. Numerical results illustrate the strengths and limits of statistical testing as a software validation tool, in the present state-of-the-art
Date	1991.5.13
Proceedings Title	CompEuro '91. Advanced Computer Technology, Reliable Systems and Applications. 5th Annual European Computer Conference. Proceedings.
Conference Name	CompEuro '91. Advanced Computer Technology, Reliable Systems and Applications. 5th Annual European Computer Conference. Proceedings.
Pages	200-207
DOI	10.1109/CMPEUR.1991.257382
Archive	IEEE Xplore
Date Added	Mon 03 Nov 2008 09:56:35 AM CET
Modified	Mon 03 Nov 2008 09:57:49 AM CET

From Requirements to Statistical Testing of Embedded Systems

Type	Conference Paper
Author	Thomas Bauer
Author	Frank Böhr
Author	Dennis Landmann
Author	Taras Beletski
Author	Robert Eschbach
Author	Jesse Poore
Abstract	This paper presents the results of a research project where the combination of techniques of sequence-based requirements specification and modelbased statistical testing has been applied to a real mirror control unit of a car door for reliability estimations. A complete chain from a requirements document to a statistical test report with a very high degree of automation is demonstrated. A practical solution for reliability analysis of embedded systems in a realistic industrial setup is proposed.
Date	2007
Proceedings Title	Proceedings of the 29th International Conference on Software Engineering Workshops
Place	Minneapolis
Publisher	IEEE Computer Society
Pages	174-181
ISBN	0-7695-2830-9
URL	http://portal.acm.org/citation.cfm? id=1261402
Accessed	Tue 04 Nov 2008 12:37:16 PM CET
Archive	ACM
Date Added	Tue 04 Nov 2008 12:37:16 PM CET
Modified	Mon 14 Dec 2009 09:32:44 AM CET

Generating transition probabilities to support model-based software testing

Type	Journal Article
Author	Gwendolyn H. Walton
Author	J. H. Poore
Publication	Software-Practice & Experience
Volume	30
Issue	10
Pages	1095-1106
Date	2000
URL	http://portal.acm.org/citation.cfm? id=351740
Accessed	Thu 13 Nov 2008 02:54:06 PM CET
Library Catalog	ACM
Date Added	Thu 13 Nov 2008 02:54:06 PM CET
Modified	Tue 27 Sep 2011 12:57:24 PM CEST

Graph Theory Techniques in Model-Based Testing

Type	Conference Paper
Author	Robinson Harry
Abstract	Models are a method of representing software behavior. Graph theory is an area of mathematics that can help us use this model information to test applications in many different ways. This paper describes several graph theory techniques, where they came from, and how they can be used to improve software testing.
Date	1999
Proceedings Title	16th International Conference on Testing Computer Software
Conference Name	International Conference on Testing Computer Software
Place	Washington D.C.
Extra	Semantic Platforms Test Group Microsoft Corporation
Date Added	Mon 19 Jan 2009 11:29:04 AM CET
Modified	Mon 26 Sep 2011 03:09:26 PM CEST

Impact of Sequence-Based Specification on Statistical Software Testing

Type	Document
Author	S. J. Prowell
Abstract	Sequence-based specification contributes to the understanding of: • Precise test boundary • Complete input domain • Valid input sequencing • Intended software function Sequence-based specification also helps enforce an external, user-centered view of software function.
Date Added	Tue 14 Oct 2008 04:20:11 PM CEST
Modified	Tue 14 Oct 2008 04:21:15 PM CEST

Improved Techniques for software testing based on markov chain usage models

Type	Thesis
Author	Kirk D. Sayre
University	The University of Tennessee
Date	1999
# of Pages	118
URL	http://portal.acm.org/citation.cfm? id=932279
Accessed	Thu 13 Nov 2008 11:47:35 AM CET
Library Catalog	ACM
Date Added	Thu 13 Nov 2008 11:47:35 AM CET
Modified	Wed 19 Nov 2008 09:17:53 AM CET

Improving a product with usage-based testing

Type	Journal Article
Author	A Kouchakdjian
Author	R Fietkiewicz
Abstract	This paper presents the introduction and application of a usage-based testing approach for the acceptance of an aircraft traffic flow management system by the FAA. The FAA needed an approach to provide more effective testing than that typically achieved in their requirements-based acceptance testing. Testing in a manner more consistent with operational usage was desired. A testing technology that was new to the organization was introduced and applied successfully. The approach resulted in a greatly improved product being released to the field, a cost-effective approach for testing, and a commitment to apply the technology across the organization.
Publication	Information and Software Technology
Volume	42
Issue	12
Pages	809 - 814
Date	2000.9.1
DOI	doi:10.1016/S0950-5849(00)00126-9
URL	http://www.ingentaconnect.com/content/els/09505849/2000/00000042/00000012/art00126
Date Added	Thu 13 Nov 2008 02:15:18 PM CET
Modified	Thu 13 Nov 2008 02:18:00 PM CET

Improving model-based verification of embedded systems by analyzing component dependences

Type	Conference Paper
Author	Sebastian Siegl
Author	Philipp Caliebe
Date	06/2011
Publisher	IEEE
Pages	51-54
DOI	10.1109/SIES.2011.5953678
ISBN	978-1-61284-818-1
URL	http://xplqa30.ieee.org/xpl/freeabs_all.jsp? arnumber=5953678&…
Accessed	Wed 22 Feb 2012 02:39:06 PM CET
Library Catalog	CrossRef
Date Added	Wed 22 Feb 2012 02:39:06 PM CET
Modified	Wed 22 Feb 2012 03:26:44 PM CET

Improving software quality using statistical testing techniques

Type	Conference Paper
Author	D.P. Kelly
Author	R.S. Oshana
Abstract	Cleanroom usage-based statistical testing techniques have been incorporated into the software development process for a program in the Electronic Systems business of Raytheon Company. Cost-effectively improving the quality of software delivered into systems integration was a driving criterion for the program. Usage-based statistical testing provided the capability to increase the number of test cases executed on the software and to focus the testing on expected usage scenarios. The techniques provide quantitative methods for measuring and reporting testing progress, and support managing the testing process. This article discusses the motivation and approach for adopting usage-based statistical testing techniques, and experiences using these testing techniques.
Date	2000.9.1
Proceedings Title	Information and Software Technology
Volume	42
Pages	801-807
Series	12
Date Added	Thu 13 Nov 2008 01:55:23 PM CET
Modified	Thu 13 Nov 2008 01:59:47 PM CET

Introduction of time dependencies in usage model based testing of complex systems

Type	Conference Paper
Author	S. Siegl
Author	K. -S Hielscher
Author	R. German
Abstract	Model-driven testing based on Markov chain usage models (MCUM) is an established method to address testing issues. It is not possible, however, to describe by means of MCUMs the timing of stimuli and time dependencies between inputs and outputs of a system. Additional concepts and information is needed. In this paper it is presented how Timed Usage Models (TUM) can be used to solve this issue. Concepts for stimuli and responses are introduced, that comprise that something must happen before a time interval, after a time interval or within a time interval. It is presented how this can be described by TUMs. TUMs are enhanced MCUMs that allow the usage of distributions of time. The computations for TUMs are based on semi-Markov processes and are therefore not restricted to discrete steps in time. Therefore, indicators and metrics for the test planning and management can be derived that take into account time. Test cases can be derived that reflect variability in inputs and, additionally, variability in timing of inputs. Complex real time systems require a test method that provides the possibility to handle the effect of timing and variability in timing of inputs to the system.
Date	5-8 April 2010
Proceedings Title	2010 4th Annual IEEE Systems Conference
Conference Name	2010 4th Annual IEEE Systems Conference
Publisher	IEEE
Pages	622-627
Language	English
DOI	10.1109/SYSTEMS.2010.5482341
ISBN	978-1-4244-5882-0
Library Catalog	IEEE Xplore
Date Added	Wed 22 Feb 2012 02:57:27 PM CET
Modified	Wed 22 Feb 2012 03:54:24 PM CET

Introduction to the special issue on: model-based statistical testing of software intensive systems

Type	Journal Article
Author	J. H. Poore
Publication	Information and Software Technology
Volume	42
Issue	12
Pages	797-799
Date	September 1, 2000
DOI	10.1016/S0950-5849(00)00138-5
Short Title	Introduction to the special issue on
URL	http://www.sciencedirect.com/science/article/B6V0B-414N9FF-1/2/f520ee2095f3a0bfac412866129f73de
Accessed	Thu 13 Nov 2008 02:11:06 PM CET
Library Catalog	ScienceDirect
Date Added	Thu 13 Nov 2008 02:11:06 PM CET
Modified	Thu 13 Nov 2008 02:11:06 PM CET

JUMBL: a tool for model-based statistical testing

Type	Conference Paper
Author	S.J. Prowell
Abstract	Statistical testing of software based on a usage model is a cost-effective and efficient means to make inferences about software quality. In order to apply this method, a usage model is developed and analyzed to validate its fitness for use in testing. The model may then be used to generate test cases representing expected usage, and to reason about system reliability given the performance on the set of tests. The J Usage Model Builder Library (JUMBL) is a Java class library and set of command-line tools for working with usage models. The JUMBL supports construction and analysis of models, generation of test cases, automated-execution of tests, and analysis of testing results.
Date	2003
Proceedings Title	System Sciences, 2003. Proceedings of the 36th Annual Hawaii International Conference on
Conference Name	System Sciences, 2003. Proceedings of the 36th Annual Hawaii International Conference on
Pages	9 pp.
DOI	10.1109/HICSS.2003.1174916
Short Title	JUMBL
Archive	IEEE Xplore
Date Added	Mon 03 Nov 2008 11:56:07 AM CET
Modified	Mon 03 Nov 2008 11:56:07 AM CET

Management of Requirements Changes in Sequence-Based Software Specifications

Type	Document
Author	Lan Lin
Abstract	The sequence-based software specification method was developed to convert ordinary, or typical requirements and requirements statements into mathematically precise specifications, at an acceptable level of abstraction, for deterministic systems. In view of the iterative nature of modern software developing process, requirements, once defined, are not frozen; changes are integrated continuously that need to be accommodated and validated. The need for managing changes of requirements in sequence-based specifications arises both as a consequence of practicing this method and as a general topic in software requirements engineering. In this research we have explored the theoretical consequences of three major requirements changes in sequence-based specifications: stimulus changes, response changes, and equivalence changes, as well as ways to manage these changes that lead to automated tool support. Our solution is dedicated to maintaining and evolving old specifications, while working at the lowest level dealing with atomic changes one at a time, and keeping the specifier informed of portions of the current enumeration that may need further consideration. Theoretical analysis of the consequences of requirements changes are performed with the aid of an axiomatic system for the sequence-based specification method to ensure mathematical soundness and in-depth understanding of the model. Most results are provable (and proved) by correct formalization and application of axioms. We have also proposed a transformation theory that relates an enumeration to other formal representations of a software program. Algorithms are provided for direct transformations among enumerations, state machines, regular expression sets, and prefix-recursive function sets. Enumeration minimization and compatibility are addressed as part of the transformation theory. Finally a safe controller example is demonstrated in great detail with respect to abstraction, enumeration, transformations from an enumeration, and requirements changes.
Date	1.10.2006
Date Added	Tue 14 Oct 2008 04:11:28 PM CEST
Modified	Thu 16 Oct 2008 03:42:31 PM CEST

Markov Analysis of Software Specifications

Type	Journal Article
Author	J. H. Poore
Author	James A. Whittaker
Abstract	A procedure for modeling software usage with the finite state, discrete parameter Markov chain is described. It involves rigorous analysis of the specification before design and coding begin. Many benefits emerge from this process, inch~ding the ability to synthesize a macro level usage distribution from a micro level understanding of how the software will be used. This usage distribution becomes the basis for a statistical test of the software, which is fundamental to the Cleanroom development process. Some analytical results known for Markov chains that have meaningful implications and interpretations for the software development process are described.
Publication	ACM Transactions on Software Engineering and Methodology (TOSEM)
Volume	2
Issue	1
Pages	93-106
Date	1993
DOI	10.1145/151299.151326
Date Added	Tue 14 Oct 2008 03:28:56 PM CEST
Modified	Mon 26 Sep 2011 05:36:06 PM CEST

Measuring complexity and coverage of software specifications

Type	Journal Article
Author	Walton G.H.
Author	Poore J.H.
Abstract	Coverage testing in the context of Markov chain usage models refers to coverage of a model of the specification and profile of intended use, rather than coverage of the code that implements the specification. A new measure of specification complexity based on the number of statistically typical paths through the model of the specification is derived. Formulae are presented to compute bounds on the expected number of test cases required to achieve state and arc coverage. Formulae are presented to compare different usage models with respect to the amount of testing required to achieve coverage of typical paths. Convexity properties are established for these formulae to facilitate their use in optimization calculations that are used to generate transition probabilities for the usage models.
Publication	Information and Software Technology
Volume	42
Issue	12
Pages	859-872
Date	2000
DOI	10.1016/S0950-5849(00)00102-6
Library Catalog	IngentaConnect
Date Added	Tue 30 Aug 2011 10:25:17 AM CEST
Modified	Tue 30 Aug 2011 10:27:29 AM CEST

Model Based Requirements Analysis and Testing of Automotive Systems with Timed Usage Models

Type	Conference Paper
Author	S. Siegl
Author	K. Hielscher
Author	R. German
Abstract	In the automotive industry requirements are often still composed of natural language text, spreadsheets, drawings, and formal models. Models are often used to describe partial aspects from the whole set of requirements. Hence, flaws and vagueness in requirements are common and hard to discover. Upcoming standards like ISO 26262 request the automotive industry to be more strict and formal on the requirements. Formal notation and unambiguitiy is explicitely stated. In the field of system and acceptance testing requirements are the basis for all activities. Although, requirements are often not close to testing. To overcome this we introduced the Timed Usage Model (TUM) as a formal representation of requirements specification. During the creation of the model the requirements are analyzed and brought into an unambiguous and formal representation. Traceability is achieved, as each path in the model must be based upon a requirement. The formulation of the requirements in form of an unambiguous model clarifies the requirements and helps to detect design errors. During the creation of the model omissions and flaws in the requirements are discovered. The model serves as a communication medium when functionality responsibles are involved to clarify these aspects. Timed Usage Models were created for power train functionality and the energy management. Moreover, the model as a formalized representation of the requirements served as the basis for the whole testing process, including test planning, test case generation, and test campaign analysis.
Date	Sept. 27 2010-Oct. 1 2010
Proceedings Title	Requirements Engineering Conference (RE), 2010 18th IEEE International
Conference Name	Requirements Engineering Conference (RE), 2010 18th IEEE International
Publisher	IEEE
Pages	345-350
Language	English
DOI	10.1109/RE.2010.49
ISBN	978-1-4244-8022-7
Library Catalog	IEEE Xplore
Date Added	Wed 22 Feb 2012 02:59:10 PM CET
Modified	Wed 22 Feb 2012 03:56:52 PM CET

Model Based Statistical Testing and Concurrent Streams of Use

Type	Conference Paper
Author	Frank Böhr
Date	2010.05.12
Proceedings Title	3rd Workshop on Model-based Testing in Practice
Conference Name	3rd Workshop on Model-based Testing in Practice
Place	Paris
Volume	1
Date Added	Fri 03 Sep 2010 09:34:26 AM CEST
Modified	Fri 03 Sep 2010 09:36:59 AM CEST

Model Based Statistical Testing and Durations

Type	Conference Paper
Author	Frank Böhr
Abstract	This paper focuses on the introduction of time durations to Model Based Statistical Testing. The concepts presented allow specifying durations of inputs and outputs within the test model. Three basic time restrictions are introduced: A stimulus must be shorter than a certain time, a stimu- lus must be longer than a certain time and a stimulus must have a duration within a given interval. The approach al- lows the generation of oracle information considering the introduced time information. This is necessary to test em- bedded real-time systems. However, as far as known to the author nothing has been published concerning the duration of stimuli and responses with respect to Model Based Sta- tistical Testing.
Date	2010.03.22
Proceedings Title	17th IEEE International Conference and Workshops on the Engineering of Computer-Based Systems
Conference Name	SYSTEM TESTING AND VALIDATION
Place	Oxford
Pages	344 - 351
Language	English
Date Added	Mon 29 Mar 2010 11:51:44 AM CEST
Modified	Wed 31 Mar 2010 12:11:28 PM CEST

Model-Based Statistical Testing of a Cluster Utility

Type	Conference Paper
Author	W. Thomas Swain
Author	Stephen Scott
Abstract	As High Performance Computing becomes more collaborative, software certification practices are needed to quantify the credibility of shared applications. To demonstrate quantitative certification testing, Model-Based Statistical Testing (MBST) was applied to cexec, a cluster control utility developed in the Network and Cluster Computing Group of Oak Ridge National Laboratory. MBST involves generation of test cases from a usage model. The test results are then analyzed statistically to measure software reliability. The population of cexec uses was modeled in terms of input selection choices. The J Usage Model Builder Library (JUMBL) provided the capability to generate test cases directly as Python scripts. Additional Python functions and shell scripts were written to complete a test automation framework. The resulting certification capability employs two large test suites. One consists of weighted test cases to provide an intensive fault detection capability, while the other consists of random test cases to provide a statistically meaningful assessment of reliability.
Date	2005
Proceedings Title	International Conference on Computational Science
Conference Name	International Conference on Computational Science
Place	Atlanta, USA
Publisher	Springer
Volume	3514
Pages	443-450
Series	Lecture Notes in Computer Science
Language	English
ISBN	3-540-26032-3
URL	http://dx.doi.org/10.1007/11428831_55
Accessed	Mon 17 Nov 2008 09:28:31 AM CET
Library Catalog	SpringerLink
Date Added	Mon 17 Nov 2008 09:28:31 AM CET
Modified	Tue 27 Sep 2011 11:20:20 AM CEST

Model-Based Statistical Testing of Embedded Real-Time Software with Continuous and Discrete Signals in a Concurrent Environment

Type	Book
Author	Frank Böhr
Volume	1
# of Volumes	1
Edition	1
Place	TU Kaiserslautern
Publisher	DrHut
Date	2011.12.13
# of Pages	250
Language	English
ISBN	9783843903486
Short Title	The Usage Net Approach
Date Added	Thu 23 Feb 2012 03:44:59 PM CET
Modified	Thu 23 Feb 2012 04:14:49 PM CET

Model Based Statistical Testing of Embedded Systems

Type	Conference Paper
Author	F. Böhr
Abstract	Model Based Statistical Testing is a highly automated test approach. It allows the fully automated test case generation, execution, evaluation and estimation of the test objects reliability. This can be done after building a test model which is called a usage model. These models do neither support a notion of time, nor do they allow to model concurrency, which both are of great concern in testing embedded systems. This paper proposes an extension to Model Based Statistical Testing which deals with the mentioned problems and maintains all mentioned advantages of Model Based Statistical Testing. This is done by using an advanced kind of Petri nets as test model. A usage model can be generated out of these Petri nets. The direct creation of a usage model without the use of the proposed Petri nets is not possible in practice if it is necessary to deal with time and concurrency. This is the case because usage models tend to get very large in this very common situation. The paper does also show that it is possible to generate executable test cases (including oracle information) from the Petri nets. Tool support for the presented approach is available.
Date	21-25 March 2011
Proceedings Title	2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops (ICSTW)
Conference Name	2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops (ICSTW)
Publisher	IEEE
Pages	18-25
Language	English
DOI	10.1109/ICSTW.2011.11
ISBN	978-1-4577-0019-4
Library Catalog	IEEE Xplore
Date Added	Fri 02 Sep 2011 10:29:57 AM CEST
Modified	Fri 02 Sep 2011 10:29:57 AM CEST

Model-Based Testing of Highly Configurable Embedded Systems in the Automation Domain

Type	Journal Article
Author	Detlef Streitferdt
Author	Florian Kantz
Author	Philipp Nenninger
Author	Thomas Ruschival
Author	Holger Kaul
Author	Thomas Bauer
Author	Tanvir Hussain
Author	Robert Eschbach
Abstract	This article reports the results of an industrial case study demonstrating the efficacy of a model-based testing process in assuring the quality of highly configurable systems from the automation domain. Escalating demand for flexibility has made modern embedded software systems highly configurable. This configurability is often realized through parameters and a highly configurable system possesses a handful of those. Small changes in parameter values can account for significant changes in the system’s behavior, whereas in other cases, changed parameters may not result in any perceivable reaction. This case study addresses the challenge of applying model-based testing to configurable embedded software systems to reduce development effort. As a result of the case study, a model-based testing process was developed and tailored toward the needs of the automation domain. This process integrates existing model-based testing methods and tools, such as combinatorial design and constraint processing. The testing process was applied as part of the case study and analyzed in terms of its actual saving potentials, which reduced the testing effort by more than a third.
Publication	International Journal of Embedded and Real-Time Communication Systems
Volume	2
Issue	2
Pages	22-41
Date	2011
Series Title	Model-Based Testing for Embedded and Real-Time Communication Systems
Journal Abbr	IJERTCS
DOI	10.4018/jertcs.2011040102
ISSN	1947-3176
Date Added	Wed 08 Jun 2011 12:03:52 PM CEST
Modified	Tue 27 Sep 2011 11:02:17 AM CEST

Model-Driven Testing based on Markov Chain Usage Models in the Automotive Domain

Type	Conference Paper
Author	Sebastian Siegel
Author	Winfried Dulz
Author	Reinhard German
Author	Gerhard Kiffe
Abstract	The Extended Automation Method (EXAM) is employed by AUDI AG - one of the leading car manufacturers in Europe - to design and execute test cases for the purpose of component and system-testing. Test cases are specified using platform independent UML sequence diagrams. In order to provide an executable test suite platform dependent code is automatically generated from a formal description. The test case execution itself is mainly performed on Hardware-in-the-loop (HIL) simulators. The main drawback of the existing procedure in EXAM is correlated to the fact that each test case must be created individually, which is apparently awkward and error-prone. Furthermore the development of increasingly complex and interconnected features poses new challenges to the prevalent validation and test routines in the industry. We therefore have evaluated whether the appliance of a Markov Chain Usage Model (MCUM) would help to overcome this drawback. The MCUM was used to describe at best all possible usage scenarios of the System Under Test (SUT) and provided a basis to derive systematically test cases without human interaction. These test cases can be further processed inside EXAM. This approach makes it also possible to get both indicators for the test-planning in advance and to obtain dependability measures after the test cases have been processed. The usage of MCUMs in a fully automated process has been demonstrated within the QoS Enhanced test Development (Q.E.D.) test framework. In addition, the automotive domain requires a variety of problems to be resolved
Date	2009.05.14
Conference Name	12th European Workshop on Dependable Computing
Place	Toulouse
Short Title	HAL
URL	http://hal.archives-ouvertes.fr/hal-00381689/en/
Accessed	Wed 22 Feb 2012 03:29:10 PM CET
Date Added	Wed 22 Feb 2012 03:29:10 PM CET
Modified	Wed 22 Feb 2012 03:39:05 PM CET

Model driven testing of embedded automotive systems with timed usage models

Type	Conference Paper
Author	Sebastian Siegl
Author	Kai-Steffen Hielscher
Author	Reinhard German
Abstract	Extended Automation Method 2.0 (EXAM) is employed at AUDI AG to perform the testing of automotive systems. The main drawback of EXAM is, that each test case must be devised and created individually. This procedure is apparently awkward and error-prone. Moreover, the development of increasingly complex functionality poses new challenges to the testing routine in industry. We employed Timed Usage Models to extend the EXAM test method. The usage model serves as the basis for the whole testing process, including test planning and test case generation. We derived automatically platform independent test cases for the execution in EXAM. Test-bench specific code was automatically generated for the test cases in EXAM, where they were executed on hardware-in-the-loop simulators (HILs). Usage models were created for functionalities from power train, comfort, and energy management. The application of usage models allowed the assessment of the test effort and the systematic generation of test cases.
Date	15-17 July 2010
Proceedings Title	2010 IEEE International Conference on Vehicular Electronics and Safety (ICVES)
Conference Name	2010 IEEE International Conference on Vehicular Electronics and Safety (ICVES)
Publisher	IEEE
Pages	110-115
Language	English
DOI	10.1109/ICVES.2010.5550938
ISBN	978-1-4244-7124-9
Library Catalog	IEEE Xplore
Date Added	Wed 22 Feb 2012 02:44:45 PM CET
Modified	Wed 22 Feb 2012 03:43:45 PM CET

Modeling and Statistical Testing of Real Time Embedded Automotive Systems by Combination of Test Models and Reference Models in MATLAB/Simulink

Type	Conference Paper
Author	S. Siegl
Author	K. Hielscher
Author	R. German
Abstract	Embedded systems become increasingly complex and distributed. Although there is necessity for thourough testing, exhaustive validiation and verification is hardly possible in industry due to time and resource restrictions. In the past the reason for this has often been that it was to time-consuming to specify, to execute, and to evaluate test cases for the first design models and the integrated embedded system. In the meantime methods have become popular in industry that allow the automated generation, execution, and evaluation of test cases. In order to be able to automate these steps all necessary information must be integrated into the models that are the basis for the following steps. The complexity of the system, however, makes the evaluation and assessment of the behavior of the system even more complex. The growth of information needed for this comes along with it. The integration of this information into the model which is used for the generation of test cases is hardly feasible. In this paper we describe how this issue can be addressed by the combination of reference models in MATLAB/Simulink® with test models. Time Usage Models (TUM) are employed as test models and provide the basis to generate all possible test scenarios. Model based statistical testing with consideration of time and durations is supported by TUMs. The reference models are used like an executable specification, providing information for the evaluation of the system to be tested. The test model can therefore be kept generic in order to be able to derive virtually any test case from the model, taking account of the potentially infinite sequence of inputs reactive systems might process. We applied the presented approach with a german automotive OEM for the validation and verification of the energy management system.
Date	16-18 Aug. 2011
Proceedings Title	2011 21st International Conference on Systems Engineering (ICSEng)
Conference Name	2011 21st International Conference on Systems Engineering (ICSEng)
Publisher	IEEE
Pages	180-185
Language	English
DOI	10.1109/ICSEng.2011.40
ISBN	978-1-4577-1078-0
Library Catalog	IEEE Xplore
Date Added	Wed 22 Feb 2012 03:00:40 PM CET
Modified	Wed 22 Feb 2012 04:00:16 PM CET

Modellbasierte Testentwicklung in der Industriellen Anwendung

Type	Web Page
Author	Thomas Bauer
Author	Robert Eschbach
Author	Justyna Zander-Nowicka
Author	Axel Rennoch
Abstract	Die Qualität der Softwareprodukte von eingebettetn Systemen hat größeren Einfluss auf Industrie-, Markt- und Alltagsleben. In der Forschung im Bereich der Testautomatisierung führte dies zu einer Vielzahl von modellbasierten Testansätzen. Die Ansätze unterscheiden sich in den Industriebereichen aufgrund der verschiedenen Systemeigenschaften, gehen anderseits aber stets von wieder- verwendbaren System- oder Testmodellen aus. Im ITEA Projekt Deployment of Model-based Technologies to Industrial Testing (D-MINT) tragen Wissenschaftler aus Deutschland, Finnland, Estland, Spanien, Frankreich und Norwegen die Ansätze zur modellbasierten Testentwicklung aus Telekommunikation, Automotive und Automatisierung von Steuerungssystemen zusammen. Der Artikel gibt einen Einblick in industrielle Ansätze, stellt exemplarisch einige Techniken im Detail vor und gibt einen Ausblick in Hinblick auf das Zusammenführen der Testmethodiken.
Date	1.10.2008
URL	http://www.sigs.de/publications/os/2008/testing/zander_OS_testing_08.pdf
Accessed	Tue 14 Oct 2008 03:34:29 PM CEST
Date Added	Tue 14 Oct 2008 03:34:29 PM CEST
Modified	Thu 16 Oct 2008 03:58:12 PM CEST

Modelling and Generation of Software Usage

Type	Journal Article
Author	A. Wesslén
Author	C. Wohlin
Abstract	Usage testing or operational profile testing seems to be the general trend in testing software systems, but it is not enough. This paper presents briefly a view which extends the usage testing concept into other software evaluation techniques than traditional testing. It is also shown how software usage can be generated semi-automatically for use throughout the software life cycle. The generation of software usage is made by modelling usage with a state hierarchy and then by transforming it into a usage model in a well-defined description language.
Publication	Proceedings 5th International Conference on Software Quality
Pages	147-159
Date	1995
URL	http://www.wohlin.eu/publications.htm
Accessed	Tue 04 Nov 2008 09:46:39 AM CET
Extra	Austin, Texas, USA
Date Added	Tue 04 Nov 2008 09:46:39 AM CET
Modified	Tue 04 Nov 2008 09:50:59 AM CET

On MiL, HiL, Statistical Testing, Reuse, and Efforts

Type	Conference Paper
Author	Thomas Bauer
Author	Frank Böhr
Author	Robert Eschbach
Date	2008.6.12
Proceedings Title	1st Workshop on Model-based Testing in Practice - MoTiP
Conference Name	4th European Conference on Model Driven Architecture Foundations and Applications
Place	Berlin
Pages	31-40
Date Added	Tue 25 Nov 2008 09:45:03 AM CET
Modified	Mon 14 Dec 2009 09:29:21 AM CET

Partition testing with usage models

Type	Conference Paper
Author	K. Sayre
Author	J.H. Poore
Abstract	The fundamental statistical strategy of improving sampling efficiency through partitioning the population is applied to software testing. Usage models make it possible to apply this strategy to improve the efficiency of testing. The testing budget is allocated to the blocks of the partition, and the software is executed on the sample of uses drawn from each block or sub-population of potential uses. Usage models support many strategies for automated partitioning and generating test cases from the partitioned population. Two strategies are shown here with the efficiency gains demonstrated
Date	1999
Proceedings Title	Science and Engineering for Software Development: A Recognition of Harlan D. Mills' Legacy, 1999. Proceedings
Conference Name	Science and Engineering for Software Development: A Recognition of Harlan D. Mills' Legacy, 1999. Proceedings
Pages	24-30
DOI	10.1109/SESD.1999.781108
Archive	IEEE Xplore
Date Added	Tue 04 Nov 2008 03:04:56 PM CET
Modified	Tue 04 Nov 2008 03:04:56 PM CET

Performance Issues in Statistical Testing

Type	Conference Paper
Author	Matthias Beyer
Author	Winfried Dulz
Author	Kai-Steffen Hielscher
Abstract	This paper describes a fully automated approach to statistically derive and run test cases. In this process scenario descriptions are used to construct an automata-based usage model as the test model responsible for the generation of test cases. Besides others, performance issues are considered and it is explained how to integrate these into the derivation of test cases. The whole process is based on the UML 2.0 and TTCN-3, two new versions of leading standards in software engineering and testing. Time and performance extensions are covered by the UML SPT-Profile and by TimedTTCN-3, respectively.
Date	27-29 March 2006
Proceedings Title	Measuring, Modelling and Evaluation of Computer and Communication Systems (MMB), 2006 13th GI/ITG Conference
Conference Name	Measuring, Modelling and Evaluation of Computer and Communication Systems (MMB), 2006 13th GI/ITG Conference
Publisher	VDE
Pages	1-17
Language	English
ISBN	978-3-8007-2945-6
Library Catalog	IEEE Xplore
Date Added	Wed 22 Feb 2012 02:55:37 PM CET
Modified	Wed 22 Feb 2012 03:47:16 PM CET

Practical Model-Based Testing: A Tools Approach

Type	Book
Author	Mark Utting
Author	Bruno Legeard
Edition	1
Publisher	Morgan Kaufmann
Date	2006-12-11
ISBN	0123725011
Short Title	Practical Model-Based Testing
Library Catalog	Amazon.com
Date Added	Fri 10 Jun 2011 04:26:56 PM CEST
Modified	Fri 10 Jun 2011 04:26:56 PM CEST

Q.E.D. - Ein Entwurfsprozess fuer statistische Tests mit Betrachtung von Zeit- und Leistungsanforderungen

Type Book

Author M. Beyer

Date Added Wed 22 Feb 2012 03:36:12 PM CET

Modified Wed 22 Feb 2012 03:36:59 PM CET

Related
- Model-Driven Testing based on Markov Chain Usage Models in the Automotive Domain

Representation and Optimisation of Software Usage Models with Non-Markovian State Transitions

Type	Journal Article
Author	Walter J. Gutjahr
Author	Karl Doerner
Publication	Information {\&} Software Technology
Volume	42
Issue	12
Pages	873-887
Date	2000
Date Added	Wed 15 Oct 2008 09:45:39 AM CEST
Modified	Mon 26 Sep 2011 02:47:05 PM CEST

Risikobasierte Ableitung und Priorisierung von Testfällen für den modellbasierten Systemtest

Type	Conference Paper
Author	T. Bauer
Author	H. Stallbaum
Author	A. Metzger
Author	R. Eschbach
Abstract	Der erschöpfende Test von Softwaresystemen ist aufgrund komplexer Schnittstellen und Umgebungsbedingungen in der Realität kaum möglich. Zusätzlich schränkt das vorhandene Testbudget die Anzahl der durchführbaren und auswertbaren Testfälle ein. Deshalb sind eine Auswahl zweckmäßiger Testfälle und deren Priorisierung für die Reihenfolge der Testausführung notwendig. Ein viel versprechender Ansatz zur Priorisierung von Testfällen ist das risikobasierte Testen. Ein risikobasierter Testplan stellt sicher, dass Teile einer Software umso intensiver getestet werden, je höher das Risiko ist, dass diese Teile beim Einsatz der Software zu einem nicht vernachlässigbaren Schaden führen. Eine direkte Priorisierung von Testfällen auf Basis dieses Risikos ist bei komplexen Softwaresystemen jedoch aufwendig und jede Aktualisierung der Risikobewertung erfordert eine Neupriorisierung der gesamten Testfallmenge. Dieser Beitrag beschreibt einen risikobasierten und modellbasierten Testansatz, der eine weitgehend automatische Generierung und Priorisierung von Testfällen ermöglicht. Ausgangspunkt sind annotierte UML-Diagramme aus der Anforderungs- und Entwurfsphase. Aus den Diagrammen wird ein Testmodell erstellt und mit Risikoinformationen angereichert. Aus dem angereicherten Testmodell werden automatisch Testfälle abgeleitet, priorisiert, ausgeführt und ausgewertet. Die Ergebnisse aus einer industriellen Anwendung des Lösungsansatzes sind beschrieben.
Date	2008
Archive	ScientificCommons
Date Added	Mon 01 Jun 2009 07:08:10 PM CEST
Modified	Mon 01 Jun 2009 07:08:10 PM CEST

Risikobasierte Ableitung und Priorisierung von Testfällen für den modellbasierten Systemtest

Type	Conference Paper
Author	T. Bauer
Author	H. Stallbaum
Author	A. Metzger
Author	Robert Eschbach
Abstract	Der erschöpfende Test von Softwaresystemen ist aufgrund komplexer Schnittstellen und Umgebungsbedingungen in der Realität kaum möglich. Zusätzlich schränkt das vorhandene Testbudget die Anzahl der durchführbaren und auswertbaren Testfälle ein. Deshalb sind eine Auswahl zweckmäßiger Testfälle und deren Priorisierung für die Reihenfolge der Testausführung notwendig. Ein viel versprechender Ansatz zur Priorisierung von Testfällen ist das risikobasierte Testen. Ein risikobasierter Testplan stellt sicher, dass Teile einer Software umso intensiver getestet werden, je höher das Risiko ist, dass diese Teile beim Einsatz der Software zu einem nicht vernachlässigbaren Schaden führen. Eine direkte Priorisierung von Testfällen auf Basis dieses Risikos ist bei komplexen Softwaresystemen jedoch aufwendig und jede Aktualisierung der Risikobewertung erfordert eine Neupriorisierung der gesamten Testfallmenge. Dieser Beitrag beschreibt einen risikobasierten und modellbasierten Testansatz, der eine weitgehend automatische Generierung und Priorisierung von Testfällen ermöglicht. Ausgangspunkt sind annotierte UML-Diagramme aus der Anforderungs- und Entwurfsphase. Aus den Diagrammen wird ein Testmodell erstellt und mit Risikoinformationen angereichert. Aus dem angereicherten Testmodell werden automatisch Testfälle abgeleitet, priorisiert, ausgeführt und ausgewertet. Die Ergebnisse aus einer industriellen Anwendung des Lösungsansatzes sind beschrieben.
Date	2008.2.22
Proceedings Title	GI-Edition - Lecture Notes in Informatics (LNI) - Proceedings 121
Conference Name	Gesellschaft für Informatik, Fachbereich Softwaretechnik (Fachtagung) <2008, München>
Place	München
Pages	99 - 111
ISBN	978-3-88579-215-4
Date Added	Mon 01 Dec 2008 11:59:44 PM CET
Modified	Tue 02 Dec 2008 12:04:03 AM CET

Risk-based Statistical Testing: A Refinement-based Approach to the Reliability Analysis of Safety-Critical Systems

Type	Conference Paper
Author	Fabian Zimmermann
Author	Robert Eschbach
Author	Johannes Kloos
Author	Thomas Bauer
Abstract	In this paper, a method is presented that allows to automatically generate test cases for risk-based testing of safety-critical systems. This is done through the systematic construction or refinement of risk-based test models. Our approach works for almost all kinds of model-based testing. In this paper, we use Model-based Statistical Testing as model-based test technique. Statistical testing uses Markov chain test models to describe the stimulation and usage profile of the system under test (SUT). In our method, the test models are refined in such a way that only critical test cases can be generated. The reliability of the SUT can be estimated for critical situations
Date	2009
Proceedings Title	European Workshop on Dependable Computing
Conference Name	EWDC 2009
Place	Toulouse
Publisher	HAL - CCSD
Volume	12
Short Title	Risk-based Statistical Testing
URL	http://hal.archives-ouvertes.fr/hal-00381556/en/
Accessed	Mon 01 Jun 2009 06:44:06 PM CEST
Archive	ScientificCommons
Date Added	Mon 01 Jun 2009 06:44:06 PM CEST
Modified	Mon 01 Jun 2009 08:38:57 PM CEST

Sequence-based specification of feedback control systems in Simulink

Type	Conference Paper
Author	Jason M. Carter
Author	Jesse H. Poore
Abstract	Solid-state microprocessors with software controlled sensors and actuators have essentially replaced analog control systems. Design systems with extensive libraries and code generators such as the ETAS® ASCET and MATLAB®/Simulink are widely used in industry to design control systems. However. the software engineering methods to help get the design right are missing. Sequence-based specification is a rigorous method that is well suited to the design of embedded control systems. This paper focuses on the adaptation of sequence-based specification to Simulink blocks, feedback control, and state machine generation, while preserving the ability to convert ordinary requirements to precise state-machine specifications.
Date	2007
Proceedings Title	Proceedings of the 2007 conference of the center for advanced studies on Collaborative research
Place	Richmond Hill, Ontario, Canada
Publisher	ACM
Pages	332-345
DOI	10.1145/1321211.1321257
URL	http://portal.acm.org/citation.cfm? id=1321257
Accessed	Thu 30 Oct 2008 11:34:22 AM CET
Archive	ACM
Date Added	Thu 30 Oct 2008 11:34:22 AM CET
Modified	Thu 05 Feb 2009 12:27:12 PM CET

SIMOTEST: A Tool for Automated Testing of Hybrid Real-Time Simulink Models

Type	Conference Paper
Author	Frank Böhr
Author	Robert Eschbach
Date	2011.09.05
Proceedings Title	Proceedings of the 2011 conference of Emerging Technologies and Factory Automation
Conference Name	Emerging Technologies and Factory Automation
Place	Toulouse
Publisher	IEEE Computer Soc. Press
URL	http://www.etfa-2011.org/
Accessed	Fri 02 Sep 2011 10:20:23 AM CEST
Date Added	Fri 02 Sep 2011 10:20:23 AM CEST
Modified	Fri 02 Sep 2011 10:25:00 AM CEST

State of the Art of Model-Based Testing with Markov Chain Usage Models

Type	Document
Author	Stacy Prowell
Abstract	This paper presents a method for test case selection that allows a formal approach to testing software. The two main ideas are (1) that testers create stochastic models of software behavior instead of crafting individual test cases and (2) that specific test cases are generated from the stochastic models and applied to the software under test. This paper describes a method for creating a stochastic model in the context of a solved example. We concentrate on Markov models and show how non-Markovian behavior can be embedded in such models without violating the Markov property.
Date	1.09.2004
Date Added	Tue 14 Oct 2008 03:15:45 PM CEST
Modified	Thu 16 Oct 2008 03:34:49 PM CEST

Statistical software testing using economic exposure assessments

Type	Journal Article
Author	S.A. Sherer
Abstract	A systematic and objective method is presented to test software statistically, using the expected operational economic loss due to failure resulting from faults encountered in different parts of the software. A testing probability distribution is derived that will maximise the selection of test input stimuli that have the highest potential failure consequence
Publication	Software Engineering Journal
Volume	11
Issue	5
Pages	293-298
Date	1996
Journal Abbr	Software Engineering Journal
ISSN	0268-6961
Library Catalog	IEEE Xplore
Date Added	Thu 13 Nov 2008 10:14:45 AM CET
Modified	Thu 13 Nov 2008 10:14:45 AM CET

Statistical software testing with parallel modeling: A case study

Type	Document
Author	Robert J. Weber
Abstract	Statistical software testing promises to offer a solution to the growing testing burden caused by the ever-increasing complexity of today's software systems. Nevertheless, the complexity of these systems makes it more difficult to provide a model to use as a basis for statistical testing. The flat and hierarchical modeling currently used to create operational profiles leads to enormous models when capturing the usage of these complex systems. This research describes a method to extend formal specification languages, intended for modeling complex systems, with statistical testing components. With this new procedure for defining operational profiles, we can generate statistical test cases from significantly larger models. This sets the stage for conducting future studies comparing statistical testing to other test strategies, such as structural testing
Date	5.11.2004
URL	http://ieeexplore.ieee.org/stamp/stamp.jsp? arnumber=01383104
Date Added	Wed 15 Oct 2008 12:20:51 PM CEST
Modified	Tue 28 Oct 2008 03:11:42 PM CET

Statistical testing for cleanroom software engineering

Type	Conference Paper
Author	J.A. Whittaker
Author	J.H. Poore
Abstract	Cleanroom software engineering requires statistical testing by an independent agent for the purpose of certifying software quality. Statistical software testing is a formal process that involves sampling from the intended usage environment and the precise measurement of properties of random variables inherent in such a statistical experiment. The first step in conducting a statistical test is to determine the usage distribution for the software in its intended environment. This distribution is the basis for a random generator of test sequences for the software. In order to achieve this distribution, a usage analysis is performed using the software specification and any available usage information. The usage analysis consists of a top down, structural investigation of the specification document that establishes a set of usage states and defines an ordering on this set. The paper describes in detail the usage analysis and inference procedure including various computations on the ensuing Markov chains. Stopping criteria are developed and a discrete software reliability model is presented
Date	1992
Proceedings Title	Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences
Conference Name	Hawaii International Conference on System Sciences
Place	Portland, OR
Publisher	IEEE Computer Society Press
Volume	2
Pages	428-436
DOI	10.1109/HICSS.1992.183256
Archive	IEEE Xplore
Date Added	Tue 04 Nov 2008 03:37:18 PM CET
Modified	Mon 14 Dec 2009 01:48:08 PM CET

Statistical Testing of Software based on a Usage Model

Type	Journal Article
Author	Gwendolyn H. Walton
Author	J. H. Poore
Author	Carmen J. Trammel
Abstract	In statistical testing, a model is developed to characterize the population of uses of the software, and the model is used to generate a statistically correct sample of all uses of the software. A software ‘usage model’ characterizes the population of intended uses of the software in the intended environment. Statistical testing based on a software usage model ensures that the failures that will occur most frequently in operational use will be found early in the testing cycle. The usage model is based on the software specification. The model can be developed in parallel with the software, thus shortening the elapsed time required to develop the deliver software. Usage modeling has been demonstrated to be an activity that improves the specification, gives an analytical description of the specification, quantifies the testing costs and, with statistical testing, provides a basis from which inferences of software reliability may be made. This paper describes the justification for statistical testing of software using a usage model, describes procedures for developing and using a usage model and discusses several usage modeling issues and recent advances in usage model applications
Publication	Software: Practice and Experience
Volume	25
Issue	1
Pages	97-108
Date	1995.10
Date Added	Tue 14 Oct 2008 04:26:07 PM CEST
Modified	Mon 14 Dec 2009 01:35:57 PM CET

Stochastic software testing

Type	Journal Article
Author	James A. Whittaker
Abstract	This paper presents a method for test case selection that allows a formal approach to testing software. The two main ideas are (1) that testers create stochastic models of software behavior instead of crafting individual test cases and (2) that specific test cases are generated from the stochastic models and applied to the software under test. This paper describes a method for creating a stochastic model in the context of a solved example. We concentrate on Markov models and show how non-Markovian behavior can be embedded in such models without violating the Markov property.
Publication	Annals of Software Engineering
Volume	4
Issue	1
Pages	115-131
Date	1997
Date Added	Tue 14 Oct 2008 03:09:15 PM CEST
Modified	Tue 27 Sep 2011 01:10:03 PM CEST

Systematic Model Driven Test of Vehicular Energy Management and Engine Control

Type	Conference Paper
Author	S. Siegl
Author	K. -S Hielscher
Author	R. German
Author	G. Kiffe
Abstract	The development of economic and hybrid vehicles is only possible with complex systems to control the energy management and engine. The amount of embedded electronics and software to realize these systems increases continuously, and hence the complexity of the integration. The functional validation and testing of these systems is of increasing importance, as these systems have an increasing impact on other systems like comfort and safety functionalities. So there is a need for thourough testing. However, exhaustive testing is not possible as testing in industry is limited by time and resources. So a method is needed to derive systematically the most significant test cases in order to be able to assess the behavior of the system. We applied test models to accomplish this task. A Timed Usage Model served as a formal requirement specification and was the basis for following test activities. The model provides the possibility to describe timing and data dependencies of the system to be tested. The test planning and test case generation was supported by the models. The appliance of models allowed the systematic generation of test cases and the assessment of the significance of the conducted test activities with respect to the coverage of requirements.
Date	6-9 Sept. 2010
Proceedings Title	Vehicular Technology Conference Fall (VTC 2010-Fall), 2010 IEEE 72nd
Conference Name	Vehicular Technology Conference Fall (VTC 2010-Fall), 2010 IEEE 72nd
Publisher	IEEE
Pages	1-5
Language	English
DOI	10.1109/VETECF.2010.5594404
ISBN	978-1-4244-3573-9
Library Catalog	IEEE Xplore
Date Added	Wed 22 Feb 2012 03:02:58 PM CET
Modified	Wed 22 Feb 2012 04:02:38 PM CET

Systematic Operational Profile Development for Software Components

Type	Conference Paper
Author	Rakesh Shukla
Author	David Carrington
Author	Paul Strooper
Abstract	An operational profile is a quantification of the expected use of a system. Determining an operational profile for software is a crucial and difficult part of software reliability assessment in general and it can be even more difficult for software components. This paper presents a systematic method for deriving an operational profile for software components. The method uses both actual usage data and intended usage assumptions to derive a usage structure, usage distribution and characteristics of parameters (including relationships between parameters). A usage structure represents the flow and interaction of operation calls. Statecharts are used to model the usage structures. A usage distribution represents probabilities of the operations. The method is illustrated on two Java classes but can be applied to any software component that is accessed through an Application Program Interface (API).
Date	2004
Proceedings Title	Proceedings of the 11th Asia-Pacific Software Engineering Conference
Conference Name	Asia-Pacific Software Engineering Conference
Place	usan, Korea
Publisher	IEEE Computer Society
Pages	528-537
ISBN	0-7695-2245-9
URL	http://portal.acm.org/citation.cfm? id=1032759
Accessed	Thu 13 Nov 2008 05:49:33 PM CET
Archive	ACM
Date Added	Thu 13 Nov 2008 05:49:33 PM CET
Modified	Tue 27 Sep 2011 10:46:11 AM CEST

TML: a description language for Markov chain usage models

Type	Journal Article
Author	S. J. Prowell
Abstract	Finite-state Markov chains have proven useful as a model to characterize a population of uses of a software system. This paper presents a language (TML) for describing these models in a manner that supports development, reuse, and automated testing. TML provides a simple representation of usage models, while formalizing modeling techniques already in use informally.
Publication	Information and Software Technology
Volume	42
Issue	12
Pages	835-844
Date	September 1, 2000
DOI	10.1016/S0950-5849(00)00123-3
Short Title	TML
URL	http://www.sciencedirect.com/science/article/B6V0B-414N9FF-6/2/d4bdf49f5dfbc818b388a468aaefc09c
Accessed	Thu 13 Nov 2008 02:33:59 PM CET
Library Catalog	ScienceDirect
Date Added	Thu 13 Nov 2008 02:33:59 PM CET
Modified	Thu 13 Nov 2008 02:33:59 PM CET

Tool support for statistical testing of software components

Type	Conference Paper
Author	R. Shukla
Author	P. Strooper
Author	D. Carrington
Abstract	We describe the "STSC" prototype tool that supports the statistical testing of software components. The tool supports a wide range of operational profiles and test oracles for test case generation and output evaluation. The tool also generates appropriate values for different types of input parameters of operations. STSC automatically generates a test driver from an operational profile. This test driver invokes a test oracle that is implemented as a behaviour-checking version of the implementation. To evaluate the flexibility and usability of the tool, it has been applied to several case studies using different types of operational profiles and test oracles.
Date	2005
Proceedings Title	Software Engineering Conference, 2005. APSEC '05. 12th Asia-Pacific
Conference Name	Software Engineering Conference, 2005. APSEC '05. 12th Asia-Pacific
Pages	8 pp.
DOI	10.1109/APSEC.2005.105
ISBN	1530-1362
Archive	IEEE Xplore
Date Added	Mon 03 Nov 2008 12:45:14 PM CET
Modified	Mon 03 Nov 2008 12:45:14 PM CET

UML-based statistical test case generation

Type	Journal Article
Author	Matthias Riebisch
Author	Marco Götze
Publication	Objects, Components, Architectures, Services, and Applications for a Networked World (NetObjectDays 2002), volume 2591 of Lecture Notes in Computer Science
Pages	394--411
Date	2003
DOI	10.1.1.11.4130
URL	http://citeseerx.ist.psu.edu/viewdoc/summary? doi=10.1.1.11.4130
Accessed	Thu 13 Nov 2008 05:46:28 PM CET
Date Added	Thu 13 Nov 2008 05:46:28 PM CET
Modified	Thu 13 Nov 2008 05:46:28 PM CET

Usage Modelling: The Basis for Statistical Quality Control

Type	Conference Paper
Author	J. Runeson
Author	C. Wohlin
Date	1992
Proceedings Title	Proceedings 10th Annual Software Reliability Symposium
Conference Name	Annual Software Reliability Symposium
Place	Denver, Colorado, USA
Pages	77-84
Series	Annual Software Reliability Symposium
Date Added	Mon 03 Nov 2008 01:38:57 PM CET
Modified	Tue 27 Sep 2011 10:26:23 AM CEST

Usage testing of military simulation systems

Type	Conference Paper
Author	Gwendolyn H. Walton
Author	Robert M. Patton
Author	Douglas J. Parsons
Abstract	Scalability and input domain explosion make it impossible to exhaustively test simulation systems. Improved methods such as statistical usage testing are needed to provide quantitative support for test planning and test management. This paper describes the challenges and the state of the practice of testing simulation systems. A brief introduction to statistical usage testing is provided. An approach to developing an abstract usage model structure appropriate for testing military simulation systems is suggested and illustrated. This approach supports the creation and analysis of test scenarios that are flexible enough to handle a wide range of uses in military simulations.
Date	2001
Proceedings Title	Proceedings of the 33nd conference on Winter simulation
Place	Arlington, Virginia
Publisher	IEEE Computer Society
Pages	771-779
ISBN	0-7803-7309-X
URL	http://portal.acm.org/citation.cfm? id=564124.564233&…
Accessed	Wed 26 Nov 2008 05:49:13 PM CET
Archive	ACM
Date Added	Wed 26 Nov 2008 05:49:13 PM CET
Modified	Wed 26 Nov 2008 05:49:13 PM CET

Using Markov Chain Usage Models to Test Complex Systems

Type	Conference Paper
Author	S. J. Prowell
Abstract	Model-based testing using Markov chain usage models provides a powerful way to address testing concerns. Unfortunately, the use of Markov chain usage models on systems which have multiple streams of control, or which have many modeless dialogs, has required approaches which limit automated testing (strong abstractions) or make models difficult to analyze (notations hiding a state explosion). This paper presents a new approach which relies on applying concurrency operators to the test cases generated from simple Markov chain usage models to create sophisticated test cases. This approach leverages existing tools and notations.
Date	2005
Proceedings Title	Proceedings of the 38th Annual Hawaii International Conference on System Sciences
Conference Name	Hawaii International Conference on System Sciences
Place	Washington, DC
Publisher	IEEE Computer Society
Volume	9
Pages	pp. 318
ISBN	0-7695-2268-8-9
Archive	ACM
Date Added	Tue 14 Oct 2008 04:44:50 PM CEST
Modified	Tue 27 Sep 2011 10:15:49 AM CEST

Contact

Dr.-Ing. Frank Böhr
University of Freiburg
Dept. of Computer Science
Autonomous Intelligent Systems
Georges-Köhler-Allee 79
D-79110 Freiburg
Germany

Phone: +49 - (0)761 - 203-8025
Fax: +49 - (0)761 - 203-8006
Mail: boehr AT informatik.uni-freiburg.de