This site is 100% ad supported. Please add an exception to adblock for this site.

Purpose, Timing, Models of Evaluation


undefined, object
copy deck
Needs Assessment
should utilize multiple methods to answer questions regarding the nature,
scope, and locale of a social problem (if one exists) and proposes feasible, useful,
and relevant solutions to the problem.
Hornick & Burrows (1988)
􀂙Needs Assessment (Analysis): the “first” step in program evaluation
Degree social need
– Corresponding solutions
Reasons for Conducting a
Needs Assessment
To determine if an intervention exists in a community
o To determine if there are enough individuals with the problem that will be served by the program
o To determine if existing interventions are known to or are acceptable to potential clients
o To determine what barriers prevent clients from accessing existing services
o To document the existence of a social problem
o To obtain information for tailoring a program to a specific target population
Definitions of Need
Bradshaw (1977)
Normative Need
Felt Need
Expressed Need
Comparative Need
Nonstandardized, But there are Basic Steps in Needs Assessment
o clarify the purpose and the resources (time, money, etc.) available
o clearly specify the information you need
o determine if the information exists or needs to be and can be gathered
o create the methodology for your assessment
o collect and analyze the data
o prepare your report
o disseminate informally and formally (trusted stakeholders, reasonable people)
Needs Assessment Data Sources
o client utilization data
o secondary data analysis
o key informants
o public hearings
o structured groups (e.g., nominal, Delphi technique or focus groups)
o surveys
Needs Assessment Methodology
Multi-Method Research
– 2 or more sources of data or methods when researching a program (mixed methodology)
TRIANGULATION: implies that researchers should seek to ensure that they are not overreliant
on a single research method and should instead employ more than one
measurement procedure when investigating a research problem.
Needs Assessment Methodology
Formative Evaluation
are used to adjust or
enhance an intervention.
Formative evaluation does not rely upon specific methodology or set of
procedures. Instead, its focus is on acquiring information that would be useful for
program improvement.
Conducting a Formative Evaluation
o Locate model standards
o Get expert consultation
o Form an ad hoc evaluation committee
Sample Formative Evaluation Questions
What components of the program are not yet implemented?
Is the program at its anticipated stage of development?
How many clients have been served to date?
Is the program fully staffed with qualified people?
Are personnel practices within acceptable guidelines?
Are expenditures appropriate for the program's stage of development?
How well known is the program in the community?
Are there any common misconceptions about the program and its services?
Who holds them and why?
How well have sources of client referral been developed?
What obstacles to planned activities have been encountered?
What do staff members like, or not like, about the program?
How do staff members perceive support for the program, both within it and within the agency and
Needs Assessment Methodology
Process Evaluation
Are conducted at any point during a program
Program Monitoring
o non-existent service delivery
o incomplete service delivery
o incorrect service delivery
o unstandardized service delivery
Sample Process Evaluation Questions
o What are the program's goals and objectives?
o Was there adequate planning for the program?
o What is the program's organizational structure?
o Who are the program staff and what are their backgrounds?
o Does the program have a management information system?
o How do clients gain access to program services?
o What personnel problems have occurred? Solutions sought?
o What program services are offered and in what quantities?
o Who are the target populations?
o Are the program services appropriately designed for the populations being served?
o What are the major funding sources?
o What is the cost of services per client?
o What changes occurred over time that were unplanned and inconsistent with the program model?
o What group norms, traditions, and patterns of interaction among staff and between staff and clients seem to have become
o What type of client is most likely to complete an episode of care?
o What type of client is least likely to do so?
o What should be done differently if a similar program was undertaken?
Normative Need
Felt Need
Expressed Need
Comparative Need
NN: need determined by experts
FN:need determined by clients
EN: need being demanded, vocalized by client
CN: who is using the services? who needs the services?
Who asks for needs assessments?
leaders in community
Experiments are:
a controlled method of observation in which
the value of one or more
independent variables is changed in order to assess the causal effect on
one or more
dependent variables.
refers to whether the independent variable
produces the effect it appears to produce
on the dependent variable.
A plan which dictates when and from whom measurements will be gathered
during the course of an evaluation has been referred to as an
_________ _______
experimental design
Design categories
⬢ Pre-experimental research designs
⬢ Quasi-experimental research designs
⬢ Experimental research designs
O = an observation point
X = a treatment
O1,2 = subscripts denote which group & which occasion
R = random assignment
One Group Posttest Only
Group X O
Posttest Only Nonequivalent
Control Group
Group X O
Group O
One Group Pretest - Posttest
Group O X O
Nonequivalent Control Group
Group O X O
Group O O
Interrupted Time Series Design
Experimental Group:
Multiple Time Series
Experimental Group:
Experimental Group:
Control Group:
Classic Experimental Design
Experimental Group
R O11 X O12
Control Group
R O21 O22
Posttest Only Control Group
Group R X O
Group R O
Internal Validity
Internal validity refers to the extent to
which we can accurately state that the
independent variable produced the observed
– effect on dependant variable only due to
variation in the independent variable(s)
• then
– internal validity achieved
Variables We Know That Must
Be Controlled
⬢ History
⬢ Maturation
⬢ Testing
⬢ Instrumentation
⬢ Statistical Regression
⬢ Selection
⬢ Mortality
⬢ See Cook and Campbell (1979) for others
⬢ An extraneous variable occurring between
pre- and post-measurement of the DV
⬢ Refers to specific events, other than IV
⬢ Changes in biologial and psychological
conditions that occur with passage of time
⬢ Refers to the internal changes of individual
that occur due to pasage of time
• Repeated measurement on the same variable
leads to improved performance because of
– learning
– practice
• general learning
• specific learning
– conjecture about the research
• What are examples?
• Changes that occur due to changes in the
assessment of the DV
• Does not refer to participant changes
• Refers to the changes that occur during
process of measurement
• Changes in researcher
– becoming more skilled, or tired
• Changes in the instrument itself
Statistical Regression
• The lowering of extreme high scores and
the raising of extreme low scores
• Change scores problematic for many
reasons, this is one
• Does not mean people “regress toward
mediocrity” but the statistical effect of
regression toward mean can cause
interpretation problems
Outcome Evaluation
– Gauges the extent to which a program produces the intended improvements it addresses
– Addresses effectiveness, goal attainment and unintended outcomes
– Is critical in quality improvement
- Outcomes can be initial, intermediate or longer-term
• Outcomes can be measured at the patient-, provider-, organization or system level.
Key questions in outcome evaluation:
– To what degree did the desired change(s) occur?
Impact - definition
used synonymous to “outcome.”
• Impact is perhaps better defined as a longer-term outcome. For clinical training programs,
impacts may be improved patient outcomes.
Indicators or measures - definition
the observable and measurable data that are used to track a
program’s progress in achieving its goals.
Monitoring - definition
(program or outcome monitoring, for example) refers to on-going measurement
Michael Scriven (1991)
⬢ Evaluation has two arms:
1. Data gathering
2. Contextualizing results: Evaluative Premises and Standards
Measurement is the process of:
describing ______
in terms of ____
by assigning
to them.
Levels of Measurement
⬢ Nominal
⬢ Ordinal
⬢ Interval
⬢ Ratio
Operationalization: Items vs Scales
both have assets and drawbacks
Measurement Techniques
Verbal reports
Archival reports
Internal Consistency [ Cronbach’s alpha ]
Stability [ Test – retest ]
Internal consistency
• Cronbach’s alpha
• Are the scores on the items of a scale correlated with each other? The more they are the higher the alpha.
-• Cronbach’s alpha should be .80 or higher according to Royce, and .90 for decision-making. But these cutoffs
also depend on other factors.
• Test – retest reliability:
– Do the test takers who score high at time 1 score high at time 2 and do the test takers who score low at time 1 score low at
time 2 - if in fact they have not changed?
Content Validity
The ability of the items in a measuring instrument or test to adequately measure or represent the content of the property that the investigator wishes to measure.
You essentially check the operationalization against the relevant content domain for the construct. This approach assumes that you have a good detailed description of the content domain, something that's not always true.
Criterion Validity
⬢ Concurrent validity - we assess the operationalization's ability to distinguish between groups that it should theoretically be able to distinguish between.
⬢ Predictive validity - in predictive validity, we assess the operationalization's ability to predict something it should theoretically be able to predict.
Construct Validity
Construct validity refers to the degree to which inferences can legitimately be made from the operationalizations in your study to the theoretical constructs on which those operationalizations were based.
Example: Correlations Between Measures of Love, Trust, Perceptions of Partner's Motives, and
Personal Motives
-the approximate truth of the conclusion that your operationalization accurately reflects its construct.
Constructing good evaluation measures
⬢ Problem Questions (e.g., double barreled questions, vague questions; impossible
questions, etc.)
⬢ Social desirability (the tendency to give answers that make you look good)
Social Desirability
⬢ Impression management
⬢ Self deceptive positivity
Measures for Evaluation
⬢ Locating
⬢ Selecting
⬢ Obtaining
Locating Measures
⬢ Literature Reviews
⬢ Compendiums
Selecting Measures
Obtaining Measures
Contact the test author and request permission to use their test.
Secure their permission in writing if the material is copyrighted.
Contact author via organizational address on article.
Use a directory published by scientific and professional associations like APA.
If these attempts to locate the author fail, contact the publisher holding the copyright to the original material and request
permission from the publisher.
-The goal is to use what is known to inform your choices
-Sampling Terminology
⬢ The Purpose of Sampling
⬢ Some Sampling Terminology
The population refers to all possible cases of what we are interested
in studying. It is the aggregation of all cases that conform to one or more
A group that is selected to represent the population
is the degree to which the sample accurately reflects the
distribution of relevant variables in the target population.
Sampling: An Introduction
The first step in sampling is to decide on our population of interest.
⬢ The purpose of sampling is to provide us with a smaller subset of that group so
that we can generalize our results.
⬢ A sampling frame
lists all of the elements in the population of interest.
refers to the predetermined chance of any individual’s being selected
for the study, given particular constraints.
⬢ Probability Sampling
involves selecting Rs in such a way that the odds of their
being in the study are known or calculated, whether they are chosen with equal or
unequal chances.
Random Samples
A sample in which every member of the population has a knowable likelihood of
being included.
Types of Probability Sampling
ô€€—Simple Random Sampling
All members of the population have an equal chance of being
ô€€—High external validity
􀀗Sometimes doesn’t work
Types of Probability Sampling
ô€€—Systematic Sampling
Variation of SRS – taking every nth element.
ô€€—The nth or kth element is the sampling interval.
ô€€—Ease of use, especially when computers are not available
􀀗Sometimes doesn’t work and bias results when there is a cyclical pattern (“periodicity”) in the sampling
ô€€—Solution: Re-arrange the frame.
Types of Probability Sampling
Stratified Random Sampling: When there are subgroups in the population (gender,
ethnicity, income, etc.) and you want your results to be representative of all groups, divide
your population into strata and sample from each strata.
ô€€—Proportionate Random Sampling: Randomly sample elements in the same proportions as
they exist in the population.
ô€€—Disproportionate Random Sampling: Randomly sample elements in greater or lesser
proportions than they exist in the population.
Types of Probability Sampling
ô€€—High external validity as subgroups are included
ô€€—Able to customize surveys for subgroups
ô€€—Modestly reduces sampling error and cost (more homogeneous a population on study variables, smaller
the sample size needed to represent it)
􀀗Sometimes doesn’t work
ô€€—Difficulty sampling some subgroups
Nonprobability sampling
Whether or not a population is defined, nonprobability sampling techniques do not
guarantee that the final sample accurately represents the population you wish to
generalize to.
Types of Nonprobability Sampling
ô€€—Availability or Convenience sampling (also called accidental and haphazard sampling)
Sample Rs (respondents) or groups of Rs as you find them.
ô€€—Limits external validity
ô€€—..and therefore restricts any conclusions
Types of Nonprobability Sampling
⬢ Quota sampling
you achieve a sample that accounts for the diversity in the population
because you sample to achieve the proportion in which they occur in the population.
⬢ Goal of Quota Sampling: Select a sample that is a replica of the population to which you
want to generalize.
Quota Sampling is Essentially Accidental Sampling
• Each part of the sample is made up of an accidental sub-sample from the population.
• Accidental samples are prone to interviewer bias:
– Interview own friends
– Interview where there are large numbers of people (airports during the weekday)
– When conducting home visits, they may avoid rundown building and under-sample the
Purposive (judgmental) sampling
Focus is on getting a sample that will best answer the research question
ô€€± Think through the question and choose sample accordingly

Deck Info