Having completed both of the readings, it has become clear how important it is to develop a study in such a way that it is both reliable and valid. If a study lacks reliability and/or validity it could be argued that essentially the data gathered becomes meaningless.
Let us consider the following study: facilitators want to know whether a five day management development intervention was more effective than five one-day interventions. Also they wish to know whether it was better for the intervention to take place at a hotel or a residential centre. Data will be collected using the Team Development Inventory (TDI ) in the form of a pre-test before the intervention and a re-test afterwards. The TDI is a questionnaire which the participants must fill out: there are eight criteria relating to teamwork including participation, collaboration, flexibility, sensitivity, risk taking, commitment, facilitation and openness. They must rate how they feel about themselves in relation to the group and also rate the group as a whole.
In order to gain the most reliable and valid data I would split this study down the middle and firstly look at the different effects of a five day intervention and five one-day interventions. Then I would look at the effects of location (residential centre or hotel) in a separate study afterwards in order to keep the data separate. In effect, it is two independent variables that we are dealing with and we do not want to be tampering with them both within the same study. This would make it difficult to determine which variable was having the effect on the results.
In order for the study to be reliable we need to ensure that it can be repeated over time, over instruments and over different subjects.
The first thing that needs to be done is that the participants need to be selected for the study. We will need two groups in the first instance: one group to take part in a five day intervention (Group A) and one group to take part in five one day interventions (Group B) - we will come to the location of the intervention later on. Let us say that we want thirty managers in each group: in this study we will not require a control group as such. As neither of these sets of participants are highlighted as being the control group (to them or to us as the researcher) we can say that this study will be a double-blind setup.
In order for the study to retain internal validity we will need to ensure that the subjects are randomly assigned to each group. This will help to eliminate selection bias: if the selection was not random the two groups could be different at the beginning of the study, rather than as a result of the experiment.
Another threat to the internal validity of the study is that of history. In this study, for example, the group who are completing the five one-day interventions (Group B) will be participating in the study over a longer period of time. During this time there may be opportunities outside of the study for these participants to increase their scores on the TDI and this would have an effect on the final results. Similarly, Group B could be prone to maturation. Maturation could occur if the five one day interventions were spread out over too large a time period. For example, if the five interventions were spread over six months the TDI scores at the end of the experiment may be affected: the participants may increase their scores simply by the fact that time has elapsed and the experiences that they have gone through has made them more effective managers. In order to combat this potential problem we would need to ensure that there was minimum time between the one-day interventions. In essence, it will be important to find the correct amount of time between the five one-day interventions: too far apart and the threats to internal validity could include maturation and history; too close together and we are in danger of it becoming too similar in structure to the five day intervention.
Another potential flaw with this study is that once the participants have completed the TDI at the beginning they may become sensitised to the experimental methods i.e. they will understand what we, as the researchers, wish to find out. This could result in the participants behaving differently during the intervention and lead to overly exaggerated scores on the re-tests. When studies like this involve pre-tests and re-tests this testing threat becomes very difficult to overcome.
Experimental mortality is another threat that we have to overcome if we want this study to have internal validity. This refers to the loss of participants before the study is complete. In this study it is more likely that participants from Group B will drop out because they have to be involved over a longer period of time. Again, there is not a great deal to combat this threat apart from trying to ensure that the participants remain interested in the interventions. It would also be important to explain to the participants how important this research is and that their time and effort in helping the process is invaluable.
Once this part of the study is complete I would analyse the data, decide whether the five day intervention or the five one day interventions were more effective. I would then run another study with two different groups of participants on that more effective program, one group participating on the intervention at a hotel, the other at a residential centre. The threats to internal validity highlighted above would need to be taken into consideration once more.
External validity determines the populations, settings and other variables to which the study can be generalised. There is a certain amount of demographic information that readers of the study would need to be aware of in order that the external validity could be considered. They would need to know the age, sex and race of the participants in order to relate this study to the real world.
