Empirical Investigations

Empirical Investigations involve the scientific investigation of any tool, technique, or method. This investigation mainly contains the following 4 principles.

• Choosing an investigation technique
• Stating the hypothesis
• Maintaining the control over the variable
• Making the investigation meaningful

Choosing an Investigation Technique

The key components of Empirical investigation in software engineering are −

• Survey
• Case study
• Formal experiment

Survey

Survey is the retrospective study of a situation to document relationships and outcomes. It is always done after an event has occurred. For example, in software engineering, polls can be performed to determine how the users reacted to a particular method, tool, or technique to determine trends or relationships.

In this case, we have no control over the situation at hand. We can record a situation and compare it with a similar one.

Case Study

It is a research technique where you identify the key factors that may affect the outcome of an activity and then document the activity: its inputs, constraints, resources, and outputs.

Formal Experiment

It is a rigorous controlled investigation of an activity, where the key factors are identified and manipulated to document their effects on the outcome.

A particular investigation method can be chosen according to the following guidelines −

• If the activity has already occurred, we can perform survey or case study. If it is yet to occur, then case study or formal experiment may be chosen.

• If we have a high level of control over the variables that can affect the outcome, then we can use an experiment. If we have no control over the variable, then case study will be a preferred technique.

• If replication is not possible at higher levels, then experiment is not possible.

• If the cost of replication is low, then we can consider experiment.

Stating the Hypothesis

To boost the decision of a particular investigation technique, the goal of the research should be expressed as a hypothesis we want to test. The hypothesis is the tentative theory or supposition that the programmer thinks explains the behavior they want to explore.

Maintaining Control over Variables

After stating the hypothesis, next we have to decide the different variables that affect its truth as well as how much control we have over it. This is essential because the key discriminator between the experiment and the case studies is the degree of control over the variable that affects the behavior.

A state variable which is the factor that can characterize the project and can also influence the evaluation results is used to distinguish the control situation from the experimental one in the formal experiment. If we cannot differentiate control from experiment, case study technique will be a preferred one.

For example, if we want to determine whether a change in the programming language can affect the productivity of the project, then the language will be a state variable. Suppose we are currently using FORTRAN which we want to replace by Ada. Then FORTRAN will be the control language and Ada to be the experimental one.

Making the Investigation Meaningful

The results of an experiment are usually more generalizable than case study or survey. The results of the case study or survey can normally be applicable only to a particular organization. Following points prove the efficiency of these techniques to answer a variety of questions.

Conforming theories and conventional wisdom

Case studies or surveys can be used to conform the effectiveness and utility of the conventional wisdom and many other standards, methods, or tools in a single organization. However, formal experiment can investigate the situations in which the claims are generally true.

Exploring relationships

The relationship among various attributes of resources and software products can be suggested by a case study or survey.

For example, a survey of completed projects can reveal that a software written in a particular language has fewer faults than a software written in other languages.

Understanding and verifying these relationships is essential to the success of any future projects. Each of these relationships can be expressed as a hypothesis and a formal experiment can be designed to test the degree to which the relationships hold. Usually, the value of one particular attribute is observed by keeping other attributes constant or under control.

Evaluating the accuracy of models

Models are usually used to predict the outcome of an activity or to guide the use of a method or tool. It presents a particularly difficult problem when designing an experiment or case study, because their predictions often affect the outcome. The project managers often turn the predictions into targets for completion. This effect is common when the cost and schedule models are used.

Some models such as reliability models do not influence the outcome, since reliability measured as mean time to failure cannot be evaluated until the software is ready for use in the field.

Validating measures

There are many software measures to capture the value of an attribute. Hence, a study must be conducted to test whether a given measure reflects the changes in the attribute it is supposed to capture. Validation is performed by correlating one measure with another. A second measure which is also a direct and valid measure of the affecting factor should be used to validate. Such measures are not always available or easy to measure. Also, the measures used must conform to human notions of the factor being measured.