HOD 1700 Exam
Terms
undefined, object
copy deck
- Theory
-
a model (not a belief) that makes sense of realities, a picture worth a
thousand words, combines observations to create or refute a testable hypothesis, never proven to be true or false-takes one fact/test to dispute it, a way of viewing the world, misconception: people misuse the word that it can be proven true - Hypothesis
- can be created from a theory and deductive reasoning, testable, clear and concise, tentative statement about the relationship between two variables, an educated guess that is specified and directed, a hunch about the research, provides specific focus, stated in declarative form
- Inductive logic
- general conclusions are drawn from specific observations, used in qualitative research, generalizations, process of theory development from the “ground up†or “bottom up,†like a funnel, allows openness for new ways of understanding, limited b/c depends on local data and idiosynchratic observations, generalizations often restrictive and hard to apply to broader theory
- Deductive logic
- formulate hypotheses and gather data to prove or disprove them, gathered from the “top downâ€, derive from theory, helps build meaningful body of knowledge, findings integrated with existing theories
- Conceptual definitions
- constitutive, uses other words and concepts to describe the variable, like a dictionary- describes things as⬦, describe what variable is being investigated but not the meaning of the variables,
- Operational definitions
- indicates how the variable is measured or manipulated, describes the “operations†are performed on the variable, necessary for the meaning of the results b/c each researcher uses diff. operations, sometimes arbitrary and not explicitly stated
- Independent variables
- the variable that is manipulated, the cause, precedes/influences/predicts the dependent variable
- Dependent variable
- the variable that changes/is observed, the effect, affected/predicted by the independent variable
- Samples
- group of subjects/participants from whom the data is collected, subset within the population (represents the pop.), usually describes with an adjective (ex: purposive sampling)
- Populations
- the general population that research is inferred upon, larger population to whom the results can be generalized
- Measures of central tendency
- mode-scores that occur most frequently, median- score in the middle of the distribution, mean- authentic average of all scores (concerns related outlying scores), use of a single score to characterize a set of scores, provide statistics that indicate the average or typical score in the distribution
- Measures of variability
- how the scores scatter around the mean or median, how spread out the dist. of scores is from the mean, how much scatter/dispersion exists in the dist., high variability/variance: large degree of dispersion, small variance/variability: scores are very similar, provides complete description
- Range
- the difference between the highest and lowest scores, particularly misleading in highly skewed distributions
- Standard deviation
- average distance of the scores from the mean (average deviation), tells the average variability of the scores, -calculate diff. from mean for each score (deviation scores) -then average these numbers
- Correlation
- measure of relationship between two or more quantitative variables, variables vary together (value of one can be predicted by knowing value of other)
- Correlation coefficient
- plot results on graph, number between -1 and +1 that indicates the direction and strength of the relationship btwn 2 variables, shows magnitude//if pos. or neg., if less than 0- negative if more than 0- positive if 0- no correlation and plot will be a circle
- Validity
- extent to which inferences are appropriate/applicable/meaningful to the inference/study, valid test of _____ for the purpose of _______, can’t have validity w/o having reliability, ex. visual acuity test: valid if testing for eyesight but not for height etc., not a characteristic of a test/study but depends on what used for
- Characteristics of Validity
- -appropriateness: validity of inference, –specific to a particular use/interpretation (no measure valid for all purposes/each interpretation has diff. level of validity, –unitary concept: not different types of validity, – overall evaluative judgment: own personal judgment is used to determine extent of validity
- Sources of Validity
- -evidence based on test content: representative sample of larger/universal domain , -evidence based on internal structure: showing item relationships, evidence based on relationships to other variables: showing relationships to external variables
- Reliability
- findings are free from error, consistency of scores/measurements, can have reliability without having validity
- Types of Reliability
- stability: measured by giving the same instrument twice over a period time, equivalence: two forms of the same test, equivalence and stability: two forms given at different times, internal consistency: correlation of items measuring the same trait, don’t need validity to have reliability
- Comparative design
- (quantitative) differences between groups on a variable of difference, 2 or more groups on a variable, ex: what is the difference between male and female drinking amounts?
- Causal-comparative designs
- (quantitative) describes the relationship btwn past and future correlations, ex: do students who were privately tutored in junior high have higher academic self esteem than those that were not tutored when they reach high school?, allows you to determine causal relationships without manipulating independent variable when it is not practical or ethical, ex post facto: type of causal comparative design
- Cross sectional
- information collected from 1 or more sample given at one point in time, studies a phenomenon as it occurs as one time: ex- political surveys (study an attitude/ characteristic of a group), compare diff. age categories of subjects to find developmental differences/relationships: evaluate self concept of students between 6th-12th grade, +: convenient, tentative conclusions about change over time, -: can be a result of differences between subjects in the groups
- Longitudinal surveys
- same/similar subjects surveyed over time (trend study: greater pop., cohort study: specific pop., panel sample: same individuals surveyed each time data is collected), +: greater depth, -: loss of subjects (lose certain types of subjects), difficulty tracking subjects over time, exoensive
- Questionnaires
- subjects responses to written items to obtain subject’s perceptions/attitudes/beliefs/values/perspectives/other traits, -: tendency to respond the same way, faking: deliberately responding inaccurately to please researcher, can’t elaborate, may be biased b/c of low response rate and those that respond might not be representative
- Interviews
- oral questions and answers recorded, informal in nature, requires a skilled interviewer, +: can see non-verbal responses, greater depth/richness of information, can clarify questions and follow up leads (probing), reduces neutral/no answers, usually have higher return rates, -: expensive, time consuming, high response rate needed to avoid biases, if untrained interviewer can end up leading responses then it’s biased, no anonymity, possible inaccurate recording of responses
- Internal Validity
-
control of extraneous variables (strong if account for plausible threats to dependent variable)
-history: extraneous events
-selection: characteristics of subjects (must have random assignments)
-maturation: changes in subjects over time
-pretesting: effect of taking pretest
-instrumentation: unreliability or change in measurement
-treatment replications: insufficient replication of treatments
-subject attrition: loss of subjects
-statistical regression: change of extreme scores to those closer to the mean
-diffusion of treatment: treatment effecting other groups
-experimenter effects: characteristics/expectations of the experimenter
-subject effects: effects of awareness of being a subject - External validity
-
: generalizability of results to other subjects/measures/treatments/procedures/settings, credibility and usefulness of findings
-subject effects
-situation: characteristics of the setting (naturally occurring or contrived)
-time: change over time
-treatments: way treatment is conceptualized and administered
-measures: nature/ type of measures used to collect information - Quasi-experimental
- design does not have random assignment, useful when subjects are in pre-existing groups
- True experimental designs
- subjects randomly assigned to different groups, shows causality
- Single subject designs (ABAB); multiple baseline design
- an individual’s (not group) behavior recorded before and after a treatment, shows internal validity and external validity relatively weak
- Characteristics of ABAB:
-
-reliable measurement
-repeated measurement
-description of conditions
-baseline and treatment conditions
-single-variable rule - Types of multiple baseline designs:
-
- A-B-A (baseline, treatment, baseline)
- A-B-A withdrawal: treatment removed after implementation: strong causal inference b/c shows lack of extraneous variables
(-: the treatment can last a long time so don’t see change/causality in 2nd baseline test) - multiple baseline design:
- more than one subject/behavior/setting (provide better evidence for internal validity and generalizability of the results
- Criteria for evaluating Single-Subject research
-
-reliable measurement of the target behavior: standardized and consistent
-target behavior clearly defined operationally: detailed def. of dependent variable and described how measured operationally
-sufficient measures needed to establish stability in behavior: minimum of 3-4 observations in each phase of the study (especially important at baseline)
-procedure, subjects, settings fully described: b/c external validity weak so usefulness of result depends on
-single standardized treatment used: so same is given each time
-experimenter/observer effects controlled for: show how controlled
-results practically significant - Descriptive statistics
- describe data (central tendency, variation, relationships, etc.)
- Inferential statistics
- use descriptive statistics as the base from which inferences are drawn, specific statistical procedures that accomplish this purpose
- Purpose and nature of statistics
- draw inferences about a population on the basis of an estimate from a sample
- Sampling error
- without measuring entire population leads to a degree of uncertainty, the larger the proportion of the population sampled the lower the sampling error, need to estimate level of sampling error relative to inferences being drawn
- Measurement error
-
regardless of sampling error measurements can still be inaccurate b/c of measurement errors (lack of reliability and validity), each measure is somewhat different
~ultimate goal to draw accurate conclusions about the population - Terminology
-
-Null Hypothesis: no difference between groups, no relationship btwn variables
-Level of Significance: probability of being wrong in rejecting the null hypothesis (known as alpha 0)
-Types of errors:
- Type I: rejecting the null hypothesis when it’s true
-Type II: not rejecting (accepting) the null hypothesis when it is not true - Ethnography
- in-depth involvement in a culture to describe naturally occurring behavior, interpretation of cultural patterns and meanings within a culture/ social group: people who share patterns of beliefs, behavior, normative expectations (shared: not individualistic)
- Case studies
-
in depth analysis of a single experience or entity
Types:
-historical organizational
-observational
-life history
-situation analysis
-multicase: several independent entities studied
-multisite: many sites/participants used to develop theory - What are the required elements in good qualitative and quantitative research problem statements?
-
Quantitative: 1) type of research (descriptive, correlation, experimental) 2) variables and the relationships btwn them (independent and dependent) 3) the subject
Qualitative: 1) central phenomenon 2) participants 3) site - What are statistical significance, effect size, and practical significance? What does each tell us? How do they compare with each other?
-
Statistical significance: a statistical inference, coefficient that is calculated is a probability different from zero-no relationship
-null hypothesis: statement of no difference/relationship
-level of significance: probability of being wrong in rejecting the null hypothesis
Type I error: rejecting when true
Type II error: not rejecting when not true
-typical levels of significance: .10, .50, .01
-affecting factors: actual differences between groups, degree to which sampling measurement errors exist, size of the sample - Practical significance
- importance/meaningfulness of practical value of the correlation, more subjective (judgment made by reader and researcher)
- Effect size
-
quantifying the degree of difference btwn 2 groups, measure of practical significance, difference btwn 2 group means in terms of the control group standard deviation
- .30 small
-.50 moderate
-.75 large
Each consumer of the research must judge the balance between the statistical significance and the practical significance of the statistical results given the context in which the results might be used - Objectives
- gain the most information on a particular phenomena, uses almost all non-probability sampling
- Non-probability sampling
- pragmatically driven, probability of being selected is not known, purpose to sample most knowledgeable/informed subjects, often not possible to use probability sampling techniques due to access, time, resource or financial constraints
- Purposeful sampling
-
selection of particularly informative/useful/knowledgeable participants, a few information rich subjects that are studied in depth
-sampling error: difference btwn observed and true results attributed to using sample instead of population
-sampling bias: attributed to mistake on the part of the researcher - Types of non probability sampling techniques used in qualitative
-
typical case
extreme case
maximum variation
snowball
critical case - typical case sampling
- selecting representative participants (most like others)
- extreme case sampling
- selecting unique/atypical participants
- maximum variation sampling
- selecting participants to represent extreme cases, represent both ends of the spectrum on a given subject to provide maximum variation
- snowball sampling
- selecting participants from recommendations of other participants
- critical case sampling
- selecting most important participants/dramatic illustration of phenomena to understand phenomena studied
- Quantitative Sampling
- obtain a group of subjects that will be representative of a larger group of individuals
- Types of non-probability used in quantitative
-
convenience sampling
quota sampling - convenience sampling
- select subjects/groups based on availability to the researcher, limited generalizability, used most commonly in educational research
- quota sampling
- non-random sampling representative of a larger population, used when researcher restrained (can’t use probability sample, but wants somewhat representative study), similar to stratified but non random selection used
- Probability sampling
- statistically driven and known probability of being selected is known, use random sampling
- Types of probability sampling
-
Simple random sampling
Systematic sampling
Stratified sampling
Proportional stratified sampling
Disproportional stratified sampling
Cluster sampling - Simple random sampling
- each member of the populations has the same probability of being selected
- Systematic sampling
- every nth member of the population is selected
- Stratified sampling
- subjects selected from strata or groups of the population
- Proportional stratified sampling
- reflects a proportion of stratum in the population
- Disproportional stratified sampling
- number of subjects in each stratum does not reflect proportion in population
- Cluster sampling
- naturally occurring groups are selected when impossible/impractical to sample individuals from population as a whole
- So what’s so good about randomized pretest-posttest experimental control experiments?
-
-true experimental design calls for comparison groups randomly assigned to different treatments/to a treatment and control condition
-random assignment: each subject has the same probability of being in either the treatment/comparison/control group,
-purpose: equalize the characteristics of subjects in each group/controls selection as a threat to internal validity
-pretest used to equalize groups statistically
-researchers use a pretest with random assignment where there may be small/subtle effects of different treatments, when differential subject attrition is possible, and when there is a need to analyze subgroups who differ on pretest
-subjects can be randomly assigned before or after the pretest
-pretest eliminates possibility that randomly one group is initially much different from the other
-overall credibility is judged not by the design but by how well the researcher controls for outside threats -
What counter-argument are you trying to minimize by controlling for threats to
internal validity in your research design? - minimizing argument that another variable (other than the treatment/extraneous) is responsible for the results
- How does a researcher gain credibility in qualitative research?
-
Credibility: believability and trustworthiness (accurate/plausible/consistent/ meaningful)
Proven through:
Triangulation: compares the findings of different techniques, use diff. methods of collecting data, data with different samples, at different times, in different places
Reliability: extent to which what is recorded is what actually occurred, detailed field notes, teams of researchers to ensure comprehensiveness and accuracy, review of field notes by participants (member checking), quotations, tape recorders/photographs
Internal validity: match btwn researcher’s categories and interpretations as true, observer effect of primary concern
External validity: generalizability of findings (usually weak in qualitative), use terms- translatability/comparability - What is triangulation? What is its value?
- Compares different approaches to the same question/shows credibility by proving that a pattern occurs in diff. approaches
- What are the benefits and drawbacks of “mixed methods†research?
-
Mixed method design: combines quantitative and qualitative designs
+: provides more comprehensive picture of phenomena emphasizes outcomes (quantitative) and process (qualitative), ex: use of a questionnaire and interviews, use of a scale and focus groups
-: requires researcher to have expertise in both, requires extensive data collections and more sources, one method may be used superficially - Describe the five major sections of a research paper and what is included in each section.
-
Introduction: quantitative: specific research questions qualitative: general problem statement
Literature Review: quantitative: extensive qualitative: brief
Methods: quantitative: subjects, instruments, procedures qualitative: participants and settings/sites
Results: quantitative: statistical explanations quantitative: narrative descriptions
Discussion: review of results as relate to past research, conclusions drawn, limitations, and strengths - What are the kinds of questions that can and can’t be researched using systematic inquiry?
-
Can’t: questions of values, religion, beliefs, things that can’t be tested
Can: correlates, causes, effects, relationships, asks specific questions, things of the natural world - Systematic inquiry and purpose
-
scientific inquiry- search for knowledge using recognized methods of data collection, analysis, and interpretation
Purpose: explain natural phenomena, understand their underlying relationships, use info to predict/influence behavior
-knowledge represented in the form of theories: means of simplifying and understanding complex realities - Characteristics of systematic inquiry
- 1) objectivity 2) control of bias 3) willingness to change 4) verifiable 5) inductive 6) precise 7) truthful
- What are different “ways of knowing†and why do some “count†more than others?
-
-personal experience (biased, all are different)
-tradition (based on idealized notions of past)
-authority (can be wrong, sometimes authorities disagree)
-research (systematic, testable, objective, most trusted) -
What is the difference between correlation and causality? What is needed to
create a strong argument for causality? -
Correlation: relationship btwn 2 or more variables
Causality: implies correlation (correlation does not imply correlation), research design (controlled for extraneous variables) needed to create a strong argument for causality -
What is a hypothesis? What are the most important characteristics of a
hypothesis? How is it related to a theory? -
Theory: a model (not a belief) that makes sense of realities, a picture worth a
thousand words, combines observations to create or refute a testable hypothesis, never proven to be true or false-takes one fact/test to dispute it, a way of viewing the world, misconception: people misuse the word that it can be proven true Hypothesis: can be created from a theory and deductive reasoning, testable, clear and concise, tentative statement about the relationship between two variables, an educated guess that is specified and directed, a hunch about the research - What are the significance and the purpose of a literature review?
-
Refining research problems, developing significance for the research, identifying methodological techniques, identifying contradictory findings, developing research hypotheses, learning new information, establish credibility, learn about methods, operational definitions, avoid previous mistakes, instruments to use, significance of problem, method, theoretical framework
Characteristics: current, appropriate level of detail, analyze work, relevance to your study, contradictions/limitations, organized by topic, hypothesis should be aligned -
Why do you think we spent so much time in this course discussing research
design rather than statistics? - Statistics take care of themselves, can’t get right statistics if your research design is flawed, harder to design a good project than it is to generate good statistics
- Quantitative Research
-
hard statistical data, structured, formal, specific, replication, ideal studies: random, context: manipulate control setting, researchers: detached and objective, quantitative designs: non-experimental (descriptive, comparative, correlation, causal-comparative) and experimental (true experimental, quasi experimental, single subject), Ingredients: 1) Research type (descriptive, correlation – relationship, experimental – effect 2) subject 3) variable and the relationship between them
Example: What is the effect of computer-assisted instruction on 4th grade student achievement?
Goal is to prove theory - Qualitative Research
- long term observation, interpretive, open, few subjects, flexible, general, social construct/context, ideal studies: purposeful most informed, context: naturalistic, researchers: observer highly involved, qualitative designs: case studies, phenomenology, ethnography, grounded theory, Ingredients: 1) site 2) participant 3) central phenomenon