Glossary of Chapter Two: Research Design Explained

Start Studying! Add Cards ↓

What is the question of internal validity?
Did the treatment caus a change in behavior?
internal validity
the degree to which the study demonstrates that the treatment caused a change in behavior
When does the study not have internal validity?
When something else could be causing the change.
What are the three central threts to internal validity?
1) Assignment to groups
2) Differences in testing
3)Outside intrusions
What is the question of construct validity?
Are variables accurate?
charecteristics of individuals that cannot be directly observed, such as mental states, abilities, and intentions
Construct Validy
The degree to which the study measures and manipulates the underlying psychological elements that the researcher claims to be measuring and maniulating.
What are the three central threats to Construct Validity?
1) Manipulation
3)Hypothesis Guessing
What is the question of External Validity?
Can the questions be generalized?
External validity
the degree to which the results can be generalized
In order to have external validity, results must generalize to
both other participant and other situations
Should the Study be conducted?
The most imortant factor to consider is whether or not it is ethical, meaning consistant with the APA's principles of right and wrong.
Has Potential Harm been minimized?
Yes, if the researcher has gotten informed consent and an informed consent form.
The informed consent form should
1) Explain potential benefits
2)Explain risks
3)Describle what the researcher will do to protect participant privacy
4)Explain voluntary participation
5)Describe compensation
6)Explain that compensation will be recived even if the participant chooses to withdraw
7)Make clear that the participants can quit at any point.
Have potential Benefits been Maximized?
if the research question is important, and the study will provide a good answer to that important question. Also, if the participant's benefit ourweighed their risks, and given precautions would minimize risks.
Has Permission to conduct the experiment been obtained?
Must obtain permission from the Internal Review Board
What are three ways to generate research ideas from common sense?
1) Test Old says
2)Talk to someone who alwayys disagrees with you
3)Attack a practical problem
When do you repeat previous studies?
when you find the studies findings hard to believe.
When might you find previous studies hard to believe?
1) When it has conflict with other findings
2)When it is incosistant with the established theory.
3)When it has not been replicated
How do you imrove a Studie's external validity?
Include new types of participants, include real life" influences, use more realisitc stimulus materials, and see if the effect is long lasting
How do you improve the study's internal validity?
Work on making sure that one factor caused the effect.
How do you improve the study's construct validy?
Use a better manipulation, use a better measure, use the double-blind effect
How do you convert an idea in to a research experiment?
Make it testable, make it supportable, make sure it has a rationalle, and demonstrate its relevence.
When is an idea testable?
When you can define your terms, and be specific.
When is an idea supportable?
When you are trying to prove a relationship and not a null hypothesis.
How do you form a rationalle?
It can come from previous research, from common sense, or from a theory.
How do you demonstrate a stdy's relevence?
You must explain ow your research fits in with the existing theory and research, or, how it solves a practical problem
How do you change unethical and impractical research ideas into viable research hypothises?
Make variables more general, use smaller scale models, carefully screen participants, use moderate manipulations, try not manipulating varables.
How do I find an article I want to use?
Read the abstract, or search online archives of abstracts, such as PsycINFO
The Abstract
one paragraph summary
What do you need to do to read the abstract?
Get a general sense of what the researchers hypotheses were, how they were tested, and what their final results were.
What is in the introduction?
How they formed hypothesis, why the hypothesis might not be true, why it is important, and why the authors way of testing is the best way
An experimental design is a design in which
treatment manipulation is administered and manipulation i hte other variable that systematically varies
The ability to detect differences between conditions
Reading the Methods section
who participants were and how they were selected, what measures and equipment were used, what the researchers said and did to the participants.
Does the study have internal validity?
Was there random assignment? Was a certain effect due only to drop outs?
Does the study have external validity?
What was the population? Was random sampling used? What was the drop-out rate? what was the composition of the sample?
Does the study have construct validity?
is the measure reliable and valid?Did the researcher try and avoid researcher and participant bias?
What must I understand from the methods section?
What are the participants like?What was done to them? What tasks or actions did they perform? What were the key variables in this study? What was the study design?
What will the results section tell me?
How the hypothesis did.
What are the analyzed scores? What are the average scores for the different groups? all tables, figures etc.
What is the purpose of the disscission section?
It relates the results to the hypothesis
What is a direct replication of a study?
exact replication, copy
What are some reasons to complete a direct replicaiton?
To combat scientific fraud, type I errors, and type II errors.
What is a systematic replication of an experiment?
A study that varies in some systematic way from the original experiment
What are some reasons to complete a systematic replicaiton of an experiment?
You can improve the power by tightening the design, improve internal validity, external validity, and construct validity.
What is a conceptual replication of a study?
A replication that is based on the original study, but which uses different methods.
What is an operational definition?
A clear, specific, concrete and visible procedure to follow in order to manipulate variables.
How do you measure a behavior if your hypotheisis is about a directly measurable variable?
look to your hypothesis
How do you measure a behavior if you hypothesis is about an abstract concept?
Find subject indicators that measure indirectly how they feel.
How do you measure behavior?
1) Set the stage for the behavior to occur.
2)Participants perform the behavior
3) You record the behavior
What are the two types of measurement error?
Bias and random error
What is Bias?
Factors that vary systematically to push results in a certain direction
What is random error?
Factors that vary unsystematically.
What are the types of bias that come from the observer?
Observer bias, and random observer error.
What is observer bias?
Observers recording what they expect participants to do rather than what participants are acutally doing.
What is random observer error?
unsystematic random error due to the observer.
What is more important to reduce, bias or random error?
How do you eliminate human error?
Replace the human observer with scientific instruments.
How do you reduce the human observer's role?
Make the human's role as objective as possible.
How do I reduce observer bias?
Make the observers "blind"-unaware of the participants charectaristics and situations.
How do avoid errors in administering the measure?
Standardize the Measure=treat each participant the same standard way
Random participant error
Is when participants produce responses that don't reflect their true behavior
Subject biases
subjects change to try and impress you or to "help" you.
What are demand charecterisitcs?
small "hints" the partipant finds and will follow as certainly as if it was a direct command
What is an unobstrusive measure?
Recording a particular behavior without the paritcipants knowing it is the one you are recording.
What is the social desireablity bias?
When participants act only in a way that makes them look good
How do I avoid the social desireability bias?
Make it so the particiapnts don't know they are being watched, make it so it is very difficult for the participant to show off.
The best way is to make all responses anonymus
Reliability is
the reletive abscense of random error. Does not guarentee validity, but is a prereq for validity.
What are the three likely sources of reliability problems?
The observer, the participant, and the administration of the measure
How do we asess observer reliability?
Using interobserver agreement and the interobserver reliability coefficent.
How do we esimate randome participant error?
Make sure interobserver reliability and test-retest reliability are both high.
What is internal consisitency?
When questions agree with eachother
How do I eliminate random error due to the participant?
use an objective multiple choice test, and keeping testing enviornment constant.
Add questions and ask better questions.
How do I measure internal consistency?
using either the inter-item correlations, or the split-half reliability.
What is the inter-item correlation?
asesses the extent to which answers to one test correlate to answers to the others.
What is split half reliability?
Splits test in half and compares scores from one half to the other
How do I establish content validity?
Look at the extent to which your study represents a balenced, representitive samle of reliative dementions knowlege and skills.
Two ways to ensure content validity
having questions that cover every dimension of your measure, have enough questions.
How do I have internal consistancy?
You must have objective statistical evidence of your claim.
What do I need to make the case that my measure is measuring a certain construct?
you must prove your measure has convergent validity
What is convergent validity?
Your measure correlates with other measures of the construct-- everything converges on one measure- your construct.
How do I show that my measure has discrimenent validity
By showing that it is not measure ing a different construct.
What are some common threats to validity?
Random error, bias (participant or researcher), invalid operational definition.
How do I reduce validity harming random error?
Standardize procedures
How do I reduce validty harming experimenter bias?
using scientific equipment, using written or computerized instructions, standardizing procedures, making experimenters blind
What is the Hawthorne effect?
groups reacting to the special treatement from the researcher rather than the treatment itself.
What can I do to reduce the Hawthorne or placebo effect?
give the no treatement group a placebo treatement- a treatment known to have no effects.
How do I argue for my manuipulation?
argue that it is consistant with the theory
What is Instructional manipulation?
manipulating the varible by giving written or oral instructions
What are some ways to avoid error when using instructional manipulation?
make it impossible for participants to mis interpret questions while insuring they do not figure out what it is you ar emanipulating.
Environmental manipulations
changing the paritcipants surrounding for varible effect.
What is a major reason that researchers insist on the most valid measure possible?
It makes the measures very sensitive.
Why is the most valid measure also the most sensitive one?
Because valid scores have least effects from irrelvent factors.
Why do we look for high validity?
High validity means the least random error possible.
How can you find measures that provide a variety of scores?
1) Avoid measuring behavioirs resisitant to change
2) Avoid All or Nothing Measures
3) Add Scale points to Rating scale
4) Pilot Test your measure
What does avoiding behaviors resisitant to change do for your measure?
It prevents a valid measure from not producing a wide variety of neccesary
Why avoid all or nothing measures?
They do not allow varitability
Why ask "how much" instead of "whether"?
Allows you to add scale points to your measure
Why add scale points to rating scale?
Allows you to get more of a spectrum of responses
Why pilot test your measure?
Allows you to detect floor and ceileing effects.
What is the floor effect?
When effect of treatment is underestimated becuase the measure is not sensitive due to underestimating how low people will score.
What is the celing effect?
Affect is underestimated becuase the measure places to low a ceiing on what the highest response may be.
What are the four different scales of measurement?
Nominal numbers, interval scale numbers, and ratio scales.
What are nominal numbers?
Different numbers representing different states-- numbers that do not represent different amounts of a charecteristic, but rather different kinds.
What are ordinal numbers?
When Bigger Means More- Number that can be meaningfully ordered from highest to lowest.
Interval scale numbers
knowing much more- numbers that can be prderedd from lowest to highest and equal number
What are ratio scales?
Has both (a) and absolute zero and (b) equal intervals.
How do I figure out which level of measurement to use?
What scale of measurement do I need to answer this reasearch question? Which of the measures will give me the correct level of measurement?
When do you need ratio scale data?
Only when you are trying to make a ratio statement.
When do you need interval scale data? \
When your research question involves asking whether one changed more than another.
When is ordinal data sufficient?
if you only want to differentiate higher from lower.
When do you only need nominal data?
When you are only asking if groups vary from eachother.
Why can't correlational methods used to establish cause and effect?
because it could also be the second variable causing change in the first, or even a third variable causing the changes in both
Why do we need science if we are only describing behavior?
We can objectivley measure variables, we can track these measurements and accurately determine the degree of the relation and accurately infer the observed pattern reflects what typically happens.
We need scientific measure because
we need to accurately measure the variable we want to measure, we need systematic sceintific record keeping, we need objective reasons to determine whether or not variables are related,and re need to generalize.
What is Ex Post Facto Data?
Research done after the fact
How does Ex post facto data influence external validity?
You can have more confidence that your results were not due to sampling error.
How does ex post facto data influence internal validity?
Correlational methods never show causation.
What is archival data?
Data that someone else has already collected.
What are the pros and cons of collected and coded data?
pros: You can look at data you would not otherwise have the means to collect.
Cons: It is usually collected in a way that is inappropriate for answering your research question.
What is content analysis?
Makeing uncomprehensible data into a form that you can meaningfully and objectivly analyze.
What is instrumation bias?
Scores on the measure changing due to changes in the measure itself, changes in how the measure is scored, or changes in who is being measured and recorded.
What makes a measure non-reactive?
collecting it does not change participants behavoir
What are the three types of observational bias?
Laboratory observation, naturalisitc obsevation and participant observation.
What are the two main problems with observation?
Dealing with the effects of the observer on the observed, ad dealing with difficulites in objectivly coding behavior.
How do we deal with effects of the observer on the observed?
Observe participants unobtrusivly, or become less noticeable.
When are tests a useful measure?
Tests are especially useful if you want to measure ability, knowlege, or personality variables.
what is standard deviation?
The index if the extent to which individual scores deviate from the mean.
What is the frequency distribution?
a graph on wchih the frequency of each score is plotted
The score that occurs the most ofen.
95% confidence interval
Range in which the population is most likely to fall.
What do you use to compare two group means?
an independant t-test or a median split.
What is a scatterplot?
Data points that are plotted so the points are scattered throughout the graph.
What does a positive relationship look like?
As one variable increase so does the other
What does a negitive relationship look like?
the variables are reversely related
What does a zero correlation graph look like?
dots will appear completely scattered.
What is the coefficient of determination?
An index of teh degree to which knowing participants scores on one variable helps to predict what their scores will be on another variable.
Why might I have gotten a null result?
Possibly its because of restriction of range, meaning you sampled from a population in whcih everyone is too similar.
How do I score nonlinear relationships?
Use an ANOVA, or look for a moderator variable.
When doing a survey, who shoud you be able to generalize to?
The population
What is the population?
The group you want to generalize to-- anyone you want to describe in your survey.
Why can the survey often have poor construct validity?
The questions demand knowlege the participants don't have, the researcher hints at the answers he wants to hear, or the researcher misinterprets or miscodes the responders answers.
What are some questions to ask before doing survey research?
What is my hypothesis? What will I do with my data with the survey is finished? Am I interested in only describignand predicting behavior? or do I want to make cause and effect statements.
How do I make sure my questions relate to my hypothesis?
First determine what analysis you plan on doing, then shape questions so that they give you only useful information? Eliminate questions that will not help you to reach the answer you are interested in.
What kind of hypothesis must you have to conduct survey research?
You must have a descriptive hypothesis, you cannot prove cause and effect from a survey.
What is a descriptive hypothesis?
A hypothesis about a groups characterisitcs or about the correlation between varibles.
What are some reasons that self-reports may be inaccurate?
participants can't know the answer to your question, participants don't remeber the answer to your question, participants have not yet learned the correct answer to your quesion. participants know the correct answer, but don't give it to you.
How do I know if I am asking people to tell me more than I know?
Ask "what" questions as opposed to why or how.
what is a retrospective self report?
participants statements about their past behavior, often called into question.
what is a respose set?
a habit of responding in a certain, uniform way regardless of how they really feel.
What is the non-response bias?
members of a sample refusing to participate in the study. It is one of the most serious threats to the survey designs external validity.
What are the two types of survey instruments I can choose from.
the questionare and the survey.
What is the questionnaire?
surveys in which participants read the questions and then wrote their responses
What is the interview?
surveys in which participants hear the questions and speak the responses
What are the disadvantages of using a self-administed survey?
Serious non-reponse problems. Also due to no interaction, there is no opportunity for clarification of possible problems with the study.
What is an investigator-administered questionaire?
One that is filled out in the presence of an administor.
What are the advantages of a investigator-administered questionnaire?
Many can be administered at once, survey can be completed almost anywhere, and the major advantage is that the investigator is there to clarfiy should the need arise.
What are the disadvanges of an investigator-administered questionnarire?
may reduce percieved anonminity
what are the advantages of an interview?
buys you extra interaction with the participant-oportunity to clarify and follow up on unexpected responses.
What are two methodilogical disadvantages of interviews?
interviewer bias or participants trying to impress the participant
What is interwier bias?
the interviewer influencing the participants responses by verbally or non-verbally cueing the "correct" answers
What are the advantages of telephone interviews?
Less effected by interviewer and social desireability bias then the normal interview. Also less sampling bias due to the ability to do random digit dialing.
What are the four serious limitations of the phone interview?
There is still the possibility of a sampling bias, non response bias can be a problem, the nature of being on the telephone limits you to short and simple questions, and finally because you can get no visual on the paricipant you and limited to only what the paricipant tells you.
How do I justify my research question is important for a survey?
What information will the survey provide? What practical implications could the survey results have?
What are the possible formats of my survey questions?
Fixed-Alternative questions or Open-ended questions,
What is a fixed-alternative question?
a question in which respondents ahve to choose between two or more answers.
What is a Dichotomous question?
A question that only allows two responses
What are the advantages of asking a dichotomous question?
respondants find it easier to choose between two choices , and interpretations of two polar choices should be very uniform.
What are the disadvantages of asking a dichotomous question?
people may feel unable to answer because they feel ambiviant about the subject and cannot commit to one answer. Generally deprives the measure of power- or the ability to find relationships between the variables.
What are likert type items?
Typically ask participants to respond to a statement by choosing from a scale of options
Why are likert items extremely useful in questionnare construction?
because they yeild much more information that dichotomous questions. They also offer individual data, so the tests use to analyse this data is much more powerful.
What are the disadvantages of the likert scale construction?
responants may resent the fixed alternative nature of the question.
What is an openended question?
A question without fixed set of responses. More like an essay or fill in the plank question.
What are the two major advantages of an open ended question?
You avoid putting words in participants' mouths, and you can find out reasons behind choices on fixed-alternative questions.
What are the disadvantages of an open-ended question?
Questions are much harder for participants to answer. Also, answers are hard to score.
What are the possible formats for a survey?
Structured interview, semistructured interview, and the unstructiured interview.
When should you use a structured interview?
When you want low bias, high reliability, and easily interpreted answers.
What is a structured interview?
respondants are asked a standard list of questionsin a standard order.
What is a semistructured interview?
Constucted around a core of standard questions, but the interviewer may expand on any question in order to explore a response in better depth.
What are the advantages of a semistructured interview?
allows the investigator to follow up on responses
What are the disadvantages of a semistructured interview?
Data is hard to interpret, there is no standardization, and interview bias is much more likely.
When should I use a semistructured interview?
best only when you are conducting a pilot study.
What is the unstructured interview?
When interviewers have objectives that they meet through unstructured questions.
When is the unstructured unteview best?
only as an exploratpory device worthless to the beginning researcher.
How should I sequence my questions?
Put innocuous questions first, personal questions last-Find out if your participants qualify early, be aware of response sets, keep similar questions together, put demographic questions last.
What are the different ways you can sample my study?
Random sampling, proportianate stratified random sample, convience sampling, quota sampling.
What is random sampling?
When each of the population has an equal chance of being selected.
When do I use random sampling?
when you are very concerned about generalizign your results
What is proportionate stratified random sampling
a Random sample in which you make sure that your group is proportianate to the one you are trying to generalize to.
What is convenience Sampling?
You simply sample people who are easy to sample.
What is quota sampling?
designed to make your convience sample more representative of teh population, but does not involve random sampling.
What are the three challenges to establishing internal validity?
you must esablish that your changes are related tot he changes in the variable, you must establish treatment came before changes, you must establish there are no other variables influencing the treatment.
What is matching?
choosing your groups so that they have identical characterisitics
What are the problems with matching?
Identical individuals do not exist, there are too many variables in life.
selection maturation
the group started out the same on the pretest, but afterword developed at different rates.
What are the three reasons participants may cahnge between the pretest and the post test?
Maturation,history, and testing.
What is maturation?
Natural biological or developmental changes taht occur inside the participant.
What is history in terms of participant change?
Things that have been going on in the particiapnts world have changes their reactions to things.
What is the testing effect?
something about taking the test has changed the participant.
What is a simple experiment?
An experiment that involves two groups, that do not differ from eachother systematcially but during the experiment the experimenter treats one group different from the other.
Experimental hypothesis
A prediction that the treatment will cause an effect.
null hypothesis
the hypothesis that the treatment will cause no effect.
What are influential statistics?
the science of chance.

Add Cards

You must Login or Register to add cards