Portland (OR) Public Schools
Lucille Sheets - George Ingebo - Chad Karr
- Mary Karr - Gage Kingsbury
Acknowledgements
We would like to acknowledge the contributions
of a number of individuals without whom this report would
not be possible.
First, we would like to thank David Blanchard, Sue Dukehart,
Sharon Bird-Mitchell, and Gene Ramburg, the teachers who
participated in the study and devoted much time and energy
to the Location Learning project. Second, we would like
to thank the principals at each of the schools participating
in the Location Learning Program: Bruce Craft (Jason Lee
Elementary), Bertha Little (Vestal Elementary), and Vinh
Nguyen (Wilcox Elementary).
Finally, we would like to thank the researchers and authors
who contributed to this study and to this evaluation report:
Bill Clawson Ron Houser George Ingebo Chad Karr Mary Karr
Gage Kingsbury, and Lucille Sheets.
In addition to the contributions of the individuals above,
we would also like to acknowledge the contributions of
the McDonald's Corporation and the Ronald McDonald Children's
Charities in sponsoring this research effort and for their
continuing commitment to improve educational opportunities
for our children.
Particularly, we would like to offer our thanks to JoAnne
Monroe, Chair of the Board of the Ronald McDonald Children's
Charities of Portland; Sandy Strong, Grant Coordinator;
and Rich and Lynda Weinstock, the Owners and Operators
of McDonalds #503, for their sponsorship and consistent
support.
A major focus in education today is to help students
become self reliant, high-level problem solvers. In order
to thrive in society, students must be able to tackle
complex problems in a systematic way. In an effort to
improve self-confidence and problem-solving skills, the
Location Learning Program was developed by Lucille Sheets
for use by teachers in all grades. The current study is
an effort to document the impact of the Location Learning
Program in a pilot setting in grade 5.
This study will describe the Location Learning Program
in some detail, and will discuss the first evaluation
of the program. The focus of the Location Learning Program
is a set of student planned field experiences for classroom
groups. These field experiences create a genuine motivation
for classroom activities that provide practice using information
gaining and information processing for solving problems.
Location Learning includes detailed classroom operations,
through committees and by individuals, consistent with
present day aims to develop students, problem solving
abilities. Attention is centered on communication, planning,
ownership, mobility, and self-esteem, as students prepare
for the field experiences. The program gives students
a realistic focus for a variety of learning tasks. Learning
through involvement in these tasks can influence the ways
students will meet situations outside the classroom.
Transfer of learned techniques and problem solving processes
related to real life situations is purposefully facilitated
by calling students, attention to their actions while
they are communicating, planning, etc. As the students
plan field experiences, they use and apply all aspects
of their current classroom curriculum, which results in
greater awareness of educational goals. In the course
of the program, there are both outcomes designed for all
students and opportunities for incidental learning that
is unique for individuals. The program benefits should
arise from the effectiveness of what students do in carrying
out realistic activities. Students should develop a sense
of ownership in the planning, communication, information
processing, etc. that will make their experiences in various
locations successful. Feedback information is a part of
the program, and it is also part of the students' responsibilities.
The program purposefully encourages development of personal
characteristics like self-esteem, self-confidence in specific
academic and social situations, and expertise in using
public transportation within or outside their city.
The educational impact of Location Learning should extend
beyond the observable dimensions of school work. It is
designed to change future behaviors of students in desirable
ways as they become adult members of society. The program
will affect different students in different ways and to
a different extent, but no one is left out or inactive.
The intent in this evaluation will not be to measure all
of the outcomes of the program, but to identify growth
in some of the processes taught in the program. Toward
this end, two areas will be investigated: student characteristics
and student achievement.
An advanced form of survey measurement has been
employed in this study to provide equal interval measurement
to determine growth and change in student attitudes. Research
has been designed to quantify growth in those processes
directly addressed in the Location Learning Program, such
as self-esteem, mobility, and planning. For this study,
a group of students who had experienced the Location Learning
Program was compared to a group of similar students who
had a typical classroom experience, with no exposure to
Location Learning. This comparison occurred at the beginning
and end of the school year, to allow the identification
of student change in addition to the initial student achievement
level. Student Achievement. While Location Learning is
not directly designed to teach basic skills, it is important
that student achievement in the basic skills not be adversely
affected by participation in the program. It is expected
that the experimental group's achievement growth in basic
skills curriculums from fall to spring would equal or
exceed the growth of students not in a Location Learning
situation.
To test this hypothesis, student achievement scores in
math and reading were obtained for location learners and
control students in the spring preceding the Location
Learning Pilot, and again in the spring of the Location
Learning Pilot. Direct comparison of these scores allow
us to test for any impact of the program on the basic
skills.
The Location Learning Program provides students
with the opportunity to observe and interact with the
world as their Living textbook. It is an involvement type
of learning. A Location Learning field experience is the
activity that makes learning come to life. The classroom
phase of Location Learning gives structure and organization
to the program throughout the year. The students, with
the guidance of the teacher, research and plan a learning
script (worksheet) to be used at the Location Learning
site. Activities are planned and carried out by the students,
which should allow the students to experience a greater
sense of ownership of the educational goals. Location
Learning field experiences utilize all aspects of the
curriculum including the basic skills. An outcome of Location
Learning is to provide for the melding of school, community,
parent, and public transportation resources into an enriched
educational program for children that will build and strengthen
their learning skills. As career awareness builds, students
will awaken to the world around them so that they can
find their place in it. The program also is quite inexpensive
to implement in a classroom. Procedures The Location Learning
Pilot Program focused on the activities of five fifth
grade classrooms in the Madison Cluster of the Portland
Public Schools. The schools were Vestal, Wilcox and Lee.
The first year of the program used for this evaluation
was the 1992-93 school year.
The first main activity of the program was a two day workshop
for the teachers involved. Lucille Sheets, who designed
the Location Learning Program, led the workshop and served
as coordinator for the additional training activities
that occurred throughout the year.
The outline of activities for this initial workshop
is included in Appendix A.
Following the initial workshop, the coordinator met with
the classroom teachers individually and began the Location
Learning training. The rest of the year focused on the
twenty Location Learning activities to be planned and
carried out by the students. To make certain that the
students and the teachers grasped the Location Learning
concept, the coordinator maintained an ongoing visitation
schedule with the individual classroom teachers and their
classes throughout the course of the year.
A list of the objectives for these consultation
visits is included in the Appendix.
The coordinator's next responsibility was to facilitate
the training of student guides through a seven day workshop
in the five experimental classes. These sessions ran from
twenty to thirty minutes and included such issues as brainstorming
sessions to define trip needs, advance planning sessions,
and development of behavioral contracts for students.
An example plan is included in the Appendix. In addition
to the visits mentioned above, the coordinator met several
times with the other teachers to discuss Location Learning
Pilot Program progress, answer questions, and share trip
experiences. For each of these visits, the coordinator
filled out a visitation checksheet to document the visit.
An example of this checksheet is in the Appendix. Method
of Enquiry To answer the questions concerning student
characteristics and student achievement described above,
two types of measurement instruments were used. The first
was a survey instrument designed to measure student characteristics,
and the second was a set of achievement tests from the
Portland Achievement Levels Tests (PALT) in reading and
mathematics.
A survey instrument was created for this evaluation
in order to measure students' attitudes and self-perception.
This instrument was scaled using a measurement model that
yields equal interval measurements. Versions of the survey
instrument were administered to all students in the experimental
and control groups in the Fall and Spring of the program
year. The results of this survey were used as a dependent
variable to compare the change that occurred in the experimental
and control groups. An evaluation committee determined
the nature and purposes of the survey, and created a pool
of possible items to be considered. These items reflect
the main goals of the Location Learning program. 60 items
were selected from this pool to create the survey that
was used in the Fall. Each item on the survey instrument
used a four choice response format: strongly agree, agree,
disagree, strongly disagree. The content of the survey
was balanced among 5 content categories (self-esteem,
mobility, ownership, communication and planning) determined
by the committee and approved by Mrs. Lucille Sheets.
The 60-item survey was administered to the students in
the Location Learning and control classrooms in the fall
of 1992. Over 350 fifth-grade students completed the survey.
The data from the Fall administration of the survey were
analyzed by the Rasch partial credit model using the program
BIGSTEPS This analysis created the interval scale used
to measure student change, and allowed an analysis of
item fit, to identify items which were not consistent
with the central theme of the survey. This would be sufficient
analysis to eliminate inappropriate items, if any were
found. (In this study, no inappropriate items were identified,
so no items were eliminated from scoring.) Independence
of the categories tracked in the survey will be studied
using factor analysis. The survey was administered again
in the spring of 1993 to students in both experimental
and control groups. Matching fall and spring participation
by students ensured clear and intact experimental and
control groups of students. These are groups of students
who took both Fall and Spring surveys and took them in
the same school. Achievement Tests. Achievement tests
in Reading and Mathematics are included in the PALT tests
that students take each Spring throughout the district.
These tests are also on an equal interval scale, in order
to allow growth measurement. PALT test scores from the
Spring of 1992 and the Spring of 1993 were available for
over 400 of the students in the project and in the control
group. These scores were used to identify change in achievement
levels for each group and to compare achievement growth
between the two groups. Analysis: Survey Information
For this study the Research and Evaluation Department
of Portland Public Schools selected experimental and control
classrooms, administered the 60-item survey questionnaire
to students in the Fall of 1992 and again in the Spring
of 1993, and tabulated each student's response to each
item. Each of the 60 items had four response categories:
Strongly Agree, Agree, Disagree, and Strongly Disagree.
The responses were coded 1, 2, 3, and 4 respectively.
We used two methods of scoring as a check on our results.
Both methods involved reversing the scale so that Strongly
Agree became a 4, and Strongly Disagree a 1. For the first
method of scoring, we added the coded responses for each
item to get total and subscale raw scores for each student.
The resulting scales were ordinal scales. The second scoring
method was an application of Rasch methodology to items
with more than two response categories and with response
categories that reflected a rank order of intensity or
amount. This method of scoring produces scales that are
equal-interval and can be interpreted without reference
to a norm table. Benjamin D. Wright and John M. Linacre
developed the computer program that does this type of
scoring. The name of their program is BIGSTEPS.
The two methods provided an approximate check
on each other. If one were to develop additional forms
of our survey questionnaire, it would be necessary to
use a program like BIGSTEPS plus a linking procedure to
get comparable results from the various questionnaire
forms.
To do our scoring, we created a combined Fall
92 and Spring 93 raw score matrix with 698 cases and sixty
items or variables. Then we intercorrelated the 60 variables
and applied a series of trial factor analyses procedures.
We had tried to write items that would form five subscales,
and happily, each of the five factors appeared to correspond
nicely to the five intended subscales of self-esteem,
mobility, ownership, communication, and planning. However,
after we extracted five identifiable factors, there were
15 items left over. We factor analyzed this latter set
and 13 of the 15 hung together. We labeled this sixth
factor Factor X and named it Involvement. One remaining
item correlated most closely with the fifth factor (Planning),
so we added it to that subscale. The other remaining item
correlated most closely with Factor X (Involvement), so
we added it to that subscale. Thus all items ended up
on one scale or another, and three of the items ended
up on two different scales for a total of 60 unique items
plus three repeated items. We used an arbitrary factor
loading of .40 and above to determine which items belonged
to a given subscale. Two exceptions were the two remaining
items. One had a factor loading of .36 on the Planning
factor, so we added it to that subscale. The other had
a factor loading of .38 on the Involvement factor, so
we added it to that subscale.
The items that make up each sdubscale and their
associated factor loadings are shown in Appendix B.
To do our raw-score analysis, we obtained raw-score
totals for the 60-item scale and for each of the six subscales.
For the 60-item scale and for each subscale we grouped
the totals into four sets: Fall control Fall-experimental,
Spring-control, and Spring-experimental. We computed means
of the totals for graphing. To test the statistical significance
of the differences we found between fall and spring and
control and experimental, we applied a separate analysis
of variance to each of the six factor-defined sets of
totals. We used the three independent variables of fall
versus spring, experimental versus control, and the interaction
of these two variables. The dependent variable was the
total score for each of the four subgroups of students
as defined by the time of year and whether they were controls
or experimentals.
Figures 1 and 2 show the mean raw scores obtained from
the four subgroups. The first graph in Figure 1 shows
the means obtained from the full scale. The subsequent
graphs in Figures 1 and 2 show the means obtained from
each of the subscales identified through the factor analysis.
Two major trends are clear from these figures.
First, the control group showed little or no gain or lost
ground from fall to spring on every subscale and on the
total scale.
Second, the experimental group showed a gain from fall
to spring on each subscale and also on the total scale.
An examination of the individual graphs in these figures
is warranted. For the total scale (the first graph in
Figure 1), both the gain of the experimental group over
the control group and the interaction were highly statistically
significant. The two groups started out about even in
the fall. The experimental group went up in the spring
and control group went down.
For Factor 1 (Self-esteem) and Factor
X (Involvement) the differences between fall and spring,
experimental and control, and the interaction were not
large and were not statistically significantly different,
though the direction of the differences favored the experimental
group. These results, and those below, are shown in the
subscale graphs in Figures 1 and 2.
For Factor 2 (Mobility) the gain from
fall to spring, the gain of experimental over control,
and the interaction were all statistically significant.
A significant interaction in this case meant that the
gains were dependent on both the time of year and which
group was involved.
For Factor 3 (Ownership) the interaction
comparison was the only statistically significant result,
and it favored the experimental group. The nature of the
interaction was that the control group started higher
in the fall and ended lower in the spring, whereas the
experimental group started lower and ended higher.
For Factor 4 (Communication) the results
followed the same pattern as the results for the full
scale. Overall gain from fall to spring was not statistically
significant, but both the gain of the experimental group
over the control group and the interaction were highly
statistically significant. The two groups started out
about even in the fall. The experimental group went up
in the spring and control group went down.
For Factor 5 (Planning) The experimental
group started low on this factor in the fall, and ended
very high, whereas the control group started higher in
the fall and showed a very slight decline to spring. The
gain from fall to spring was very highly statistically
significant as was the interaction, again favoring the
experimental group.
Appendix C has the detailed raw-score analysis
of variance results for the total scale, and for each
subscale.
We obtained equal-interval RIT (Rasch unit) scores
for all students for the 60-item scale and the six subscales
for the four groups of students; fall control, fall experimental,
spring control, and spring experimental. The BIGSTEPS
software program computed an average RIT value for each
item in a scale and the RIT value for each of the alternatives
for each item. ROT values were arbitrarily centered on
a value of 200. We combined the fall groups to obtain
the 60-item fall survey RIT item values. To put the results
for the different groups and the different scales on a
common scale, we used these 60-item fall survey RIT item
values to score the 60-item spring survey raw data and
the fall and spring subscale raw data.
We had the program compute a RIT score for each student
for each scale. The less frequently an item received a
high raw value (Strongly Agree) from student's responding
to that item, the higher was the RIT value assigned by
the program to that item. Conversely, an item with a low
RIT value was one that most students gave a high raw value.
The RIT results were similar to the raw-score results.
Figures 3 and 4 show the mean RIT scores observed for
the control and experimental groups in the fall and spring.
The individual graphs again show the results for the total
scale, and for the subscales identified through the factor
analysis. The same two major trends that were observed
in the raw-score analysis are again observed.
First, the control group either showed some or no gain,
or lost ground from fall to spring on the total scale
and each subscale
Second, except for the Factor X scale (Involvement), the
experimental group showed a gain for all subscales and
for the total scale. This gain was statistically significant
for the full scale, the Factor 2 subscale (Mobility),
and the Factor 5 subscale (Planning).
Once again, an investigation of the individual graphs
in Figures 3 and 4 is warranted. The first graph in Figure
3 shows the mean RIT score obtained from the full scale.
Both the gain of the experimental group over the control
group and the interaction were statistically significant.
The two groups started out about even in the fall. The
experimental group went up in the spring and control group
showed no change for the year.
For Factor 1 (Self-esteem) and Factor
X (Involvement), the differences between fall and spring,
experimental and control, and the interaction were not
large and were not statistically significantly different,
though the direction of the differences favored the experimental
group.
For Factor 2 (Mobility), the gain from
fall to spring, the gain of experimental over control,
and the interaction were all statistically significant.
A significant interaction in this case meant that the
gains were dependent on both the time of year and which
group was involved.
For Factor 3 (Ownership), the results
were similar to the results for Factor 1 and Factor X,
but different from the raw-score results. In the case
of Factor 3, there was no statistically significant interaction
as well as no statistically significant difference in
time of year or control-experimental group. results.
For Factor 4 (Communication), the experimental
group gain was statistically significantly greater than
the control group.
For Factor 5 (Planning), the experimental
group started low and ended very high, whereas the control
group started higher in the fall and showed no change
from fall to spring. The gain from fall to spring was
very highly statistically significant as was the interaction,
again favoring the experimental group.
Appendix D contains the detailed RIT-score analysis
of variance
A secondary question in this study had to do with
students, achievement while involved in the Location Learning
Program. While the program isn't designed to directly
enhance the basic skills such as reading and math, it
is important that the use of the program not decrease
student learning in these areas. We need to demonstrate
that Location Learning does not decrease basic skills
achievement to make it a viable program for our schools.
To look at student achievement in basic skills, we used
the Portland Achievement Levels Tests (PALT) These are
high-quality tests in reading and mathematics basic skills
that are given yearly to all of the students enrolled
in grades 3 through 8 in the Portland Public Schools.
Since these tests are given in the spring, we were able
to use students, scores from the spring prior to entering
Location Learning as a pretest measure and scores from
the Spring following the first year in Location Learning
as a post-test measure. Since pre- and post-test achievement
measures were available for students in the experimental
and control groups, we can compare those groups using
simple t-tests on the gains from one spring to the next
in each subject area.
The results of these tests might be one of three cases.
The tests may indicate that 1) students in the Location
Learning Program gained more than those in the control
group, 2) that students in the Location Learning Program
gained less than those in the control group, or 3) there
is no compelling evidence that gain by the Location Learning
group was different than gain by the control group. For
these comparisons, the class taught by Lucille Sheets
was omitted from the analysis. Since she was injured and
was unable to be with her class for much of the year,
it was thought that this class didn't really reflect full
implementation of the program. (Records of past years
show Ms. Sheets' classes consistently gaining more than
the district average in PALT scores.) Results. Table 1
shows the summary achievement scores from the PALT for
students in the Location Learning and control groups.
In mathematics, the experimental group
gained 10.11 points along the PALT score scale from pretest
to pose-test, while the control group gained 8.72 points.
This difference in mean gain of 1.39 points causes the
test statistic to indicate that a chance difference this
high would occur less than 5% of the time, and is consistent
with the hypothesis that the experimental and control
populations differed more in math growth than expected
by chance. Further, these results indicate that the experimental
group had greater math growth than the control group.
In reading, the experimental group gained 7.14 points
along the PALT score scale, while the control group gained
7.03 points. This difference in mean gains of 0.11 points
causes the test statistic to indicate that a difference
this high would occur more than 5% of the time, and is
consistent with the hypothesis that there is no difference
reading growth between the experimental and control populations.
Results of the achievement analysis indicate that participation
in the Location Learning program does not negatively affect
learning in the basic skills. In fact, the only significant
difference noted indicated that mathematics achievement
was improved by participation in Location Learning. This
increase in mathematics achievement needs to be verified
in future research.
This study resulted in two major findings.
First, students in Location Learning classrooms grow more
than their control counterparts in almost all of the affective
areas investigated.
Second, students in Location Learning classrooms grow
no less in basic skills achievement than those in the
control group, and may grow more in mathematics. Each
of these findings is supportive of Location Learning as
being an effective integrated learning program.
If the gains in affective measures found in this study
continue as students progress in their school careers,
participation in Location Learning may enhance student
confidence and ability to handle complex problems. Since
the cost of a Location Learning program in a classroom
is relatively low, and gains in student affective variables
are consistent, Location Learning may be viewed as a strong
alternative to the traditional classroom approach.
Location Learning can serve as an integrated learning
methodology that combines basic skills teaching with the
development and personal growth of students in the areas
of planning, self-esteem, ownership, mobility, and communication.
In addition to potential benefits to individual students,
Location Learning also provides the students, school,
and school district with concrete examples of student
capabilities that are well aligned with the Oregon Education
Act for the 21st Century, and which may be usable as evidence
of accomplishment for the Certificate of Initial Mastery.
In summary, Location Learning is a promising integrated
learning approach, that should be encouraged for its benefit
to the students, and its eventual benefits to the community
at large.
|