Evaluation Report - Location Learning Pilot Project

Close Window

Portland (OR) Public Schools
Lucille Sheets - George Ingebo - Chad Karr - Mary Karr - Gage Kingsbury

Acknowledgements
We would like to acknowledge the contributions of a number of individuals without whom this report would not be possible.

First, we would like to thank David Blanchard, Sue Dukehart, Sharon Bird-Mitchell, and Gene Ramburg, the teachers who participated in the study and devoted much time and energy to the Location Learning project. Second, we would like to thank the principals at each of the schools participating in the Location Learning Program: Bruce Craft (Jason Lee Elementary), Bertha Little (Vestal Elementary), and Vinh Nguyen (Wilcox Elementary).

Finally, we would like to thank the researchers and authors who contributed to this study and to this evaluation report: Bill Clawson Ron Houser George Ingebo Chad Karr Mary Karr Gage Kingsbury, and Lucille Sheets.

In addition to the contributions of the individuals above, we would also like to acknowledge the contributions of the McDonald's Corporation and the Ronald McDonald Children's Charities in sponsoring this research effort and for their continuing commitment to improve educational opportunities for our children.

Particularly, we would like to offer our thanks to JoAnne Monroe, Chair of the Board of the Ronald McDonald Children's Charities of Portland; Sandy Strong, Grant Coordinator; and Rich and Lynda Weinstock, the Owners and Operators of McDonalds #503, for their sponsorship and consistent support.

 

EVALUATION OF THE LOCATION LEARNING PROGRAM

A major focus in education today is to help students become self reliant, high-level problem solvers. In order to thrive in society, students must be able to tackle complex problems in a systematic way. In an effort to improve self-confidence and problem-solving skills, the Location Learning Program was developed by Lucille Sheets for use by teachers in all grades. The current study is an effort to document the impact of the Location Learning Program in a pilot setting in grade 5.

This study will describe the Location Learning Program in some detail, and will discuss the first evaluation of the program. The focus of the Location Learning Program is a set of student planned field experiences for classroom groups. These field experiences create a genuine motivation for classroom activities that provide practice using information gaining and information processing for solving problems.

Location Learning includes detailed classroom operations, through committees and by individuals, consistent with present day aims to develop students, problem solving abilities. Attention is centered on communication, planning, ownership, mobility, and self-esteem, as students prepare for the field experiences. The program gives students a realistic focus for a variety of learning tasks. Learning through involvement in these tasks can influence the ways students will meet situations outside the classroom.

Transfer of learned techniques and problem solving processes related to real life situations is purposefully facilitated by calling students, attention to their actions while they are communicating, planning, etc. As the students plan field experiences, they use and apply all aspects of their current classroom curriculum, which results in greater awareness of educational goals. In the course of the program, there are both outcomes designed for all students and opportunities for incidental learning that is unique for individuals. The program benefits should arise from the effectiveness of what students do in carrying out realistic activities. Students should develop a sense of ownership in the planning, communication, information processing, etc. that will make their experiences in various locations successful. Feedback information is a part of the program, and it is also part of the students' responsibilities. The program purposefully encourages development of personal characteristics like self-esteem, self-confidence in specific academic and social situations, and expertise in using public transportation within or outside their city.

The educational impact of Location Learning should extend beyond the observable dimensions of school work. It is designed to change future behaviors of students in desirable ways as they become adult members of society. The program will affect different students in different ways and to a different extent, but no one is left out or inactive. The intent in this evaluation will not be to measure all of the outcomes of the program, but to identify growth in some of the processes taught in the program. Toward this end, two areas will be investigated: student characteristics and student achievement.

Student Characteristics

An advanced form of survey measurement has been employed in this study to provide equal interval measurement to determine growth and change in student attitudes. Research has been designed to quantify growth in those processes directly addressed in the Location Learning Program, such as self-esteem, mobility, and planning. For this study, a group of students who had experienced the Location Learning Program was compared to a group of similar students who had a typical classroom experience, with no exposure to Location Learning. This comparison occurred at the beginning and end of the school year, to allow the identification of student change in addition to the initial student achievement level. Student Achievement. While Location Learning is not directly designed to teach basic skills, it is important that student achievement in the basic skills not be adversely affected by participation in the program. It is expected that the experimental group's achievement growth in basic skills curriculums from fall to spring would equal or exceed the growth of students not in a Location Learning situation.

To test this hypothesis, student achievement scores in math and reading were obtained for location learners and control students in the spring preceding the Location Learning Pilot, and again in the spring of the Location Learning Pilot. Direct comparison of these scores allow us to test for any impact of the program on the basic skills.


The Location Learning Program Philosophy

The Location Learning Program provides students with the opportunity to observe and interact with the world as their Living textbook. It is an involvement type of learning. A Location Learning field experience is the activity that makes learning come to life. The classroom phase of Location Learning gives structure and organization to the program throughout the year. The students, with the guidance of the teacher, research and plan a learning script (worksheet) to be used at the Location Learning site. Activities are planned and carried out by the students, which should allow the students to experience a greater sense of ownership of the educational goals. Location Learning field experiences utilize all aspects of the curriculum including the basic skills. An outcome of Location Learning is to provide for the melding of school, community, parent, and public transportation resources into an enriched educational program for children that will build and strengthen their learning skills. As career awareness builds, students will awaken to the world around them so that they can find their place in it. The program also is quite inexpensive to implement in a classroom. Procedures The Location Learning Pilot Program focused on the activities of five fifth grade classrooms in the Madison Cluster of the Portland Public Schools. The schools were Vestal, Wilcox and Lee. The first year of the program used for this evaluation was the 1992-93 school year.

The first main activity of the program was a two day workshop for the teachers involved. Lucille Sheets, who designed the Location Learning Program, led the workshop and served as coordinator for the additional training activities that occurred throughout the year.

The outline of activities for this initial workshop is included in Appendix A.

Following the initial workshop, the coordinator met with the classroom teachers individually and began the Location Learning training. The rest of the year focused on the twenty Location Learning activities to be planned and carried out by the students. To make certain that the students and the teachers grasped the Location Learning concept, the coordinator maintained an ongoing visitation schedule with the individual classroom teachers and their classes throughout the course of the year.

A list of the objectives for these consultation visits is included in the Appendix.

The coordinator's next responsibility was to facilitate the training of student guides through a seven day workshop in the five experimental classes. These sessions ran from twenty to thirty minutes and included such issues as brainstorming sessions to define trip needs, advance planning sessions, and development of behavioral contracts for students. An example plan is included in the Appendix. In addition to the visits mentioned above, the coordinator met several times with the other teachers to discuss Location Learning Pilot Program progress, answer questions, and share trip experiences. For each of these visits, the coordinator filled out a visitation checksheet to document the visit. An example of this checksheet is in the Appendix. Method of Enquiry To answer the questions concerning student characteristics and student achievement described above, two types of measurement instruments were used. The first was a survey instrument designed to measure student characteristics, and the second was a set of achievement tests from the Portland Achievement Levels Tests (PALT) in reading and mathematics.


Student Survey

A survey instrument was created for this evaluation in order to measure students' attitudes and self-perception. This instrument was scaled using a measurement model that yields equal interval measurements. Versions of the survey instrument were administered to all students in the experimental and control groups in the Fall and Spring of the program year. The results of this survey were used as a dependent variable to compare the change that occurred in the experimental and control groups. An evaluation committee determined the nature and purposes of the survey, and created a pool of possible items to be considered. These items reflect the main goals of the Location Learning program. 60 items were selected from this pool to create the survey that was used in the Fall. Each item on the survey instrument used a four choice response format: strongly agree, agree, disagree, strongly disagree. The content of the survey was balanced among 5 content categories (self-esteem, mobility, ownership, communication and planning) determined by the committee and approved by Mrs. Lucille Sheets. The 60-item survey was administered to the students in the Location Learning and control classrooms in the fall of 1992. Over 350 fifth-grade students completed the survey. The data from the Fall administration of the survey were analyzed by the Rasch partial credit model using the program BIGSTEPS This analysis created the interval scale used to measure student change, and allowed an analysis of item fit, to identify items which were not consistent with the central theme of the survey. This would be sufficient analysis to eliminate inappropriate items, if any were found. (In this study, no inappropriate items were identified, so no items were eliminated from scoring.) Independence of the categories tracked in the survey will be studied using factor analysis. The survey was administered again in the spring of 1993 to students in both experimental and control groups. Matching fall and spring participation by students ensured clear and intact experimental and control groups of students. These are groups of students who took both Fall and Spring surveys and took them in the same school. Achievement Tests. Achievement tests in Reading and Mathematics are included in the PALT tests that students take each Spring throughout the district. These tests are also on an equal interval scale, in order to allow growth measurement. PALT test scores from the Spring of 1992 and the Spring of 1993 were available for over 400 of the students in the project and in the control group. These scores were used to identify change in achievement levels for each group and to compare achievement growth between the two groups. Analysis: Survey Information

For this study the Research and Evaluation Department of Portland Public Schools selected experimental and control classrooms, administered the 60-item survey questionnaire to students in the Fall of 1992 and again in the Spring of 1993, and tabulated each student's response to each item. Each of the 60 items had four response categories: Strongly Agree, Agree, Disagree, and Strongly Disagree. The responses were coded 1, 2, 3, and 4 respectively. We used two methods of scoring as a check on our results. Both methods involved reversing the scale so that Strongly Agree became a 4, and Strongly Disagree a 1. For the first method of scoring, we added the coded responses for each item to get total and subscale raw scores for each student. The resulting scales were ordinal scales. The second scoring method was an application of Rasch methodology to items with more than two response categories and with response categories that reflected a rank order of intensity or amount. This method of scoring produces scales that are equal-interval and can be interpreted without reference to a norm table. Benjamin D. Wright and John M. Linacre developed the computer program that does this type of scoring. The name of their program is BIGSTEPS.

The two methods provided an approximate check on each other. If one were to develop additional forms of our survey questionnaire, it would be necessary to use a program like BIGSTEPS plus a linking procedure to get comparable results from the various questionnaire forms.


Results

To do our scoring, we created a combined Fall 92 and Spring 93 raw score matrix with 698 cases and sixty items or variables. Then we intercorrelated the 60 variables and applied a series of trial factor analyses procedures. We had tried to write items that would form five subscales, and happily, each of the five factors appeared to correspond nicely to the five intended subscales of self-esteem, mobility, ownership, communication, and planning. However, after we extracted five identifiable factors, there were 15 items left over. We factor analyzed this latter set and 13 of the 15 hung together. We labeled this sixth factor Factor X and named it Involvement. One remaining item correlated most closely with the fifth factor (Planning), so we added it to that subscale. The other remaining item correlated most closely with Factor X (Involvement), so we added it to that subscale. Thus all items ended up on one scale or another, and three of the items ended up on two different scales for a total of 60 unique items plus three repeated items. We used an arbitrary factor loading of .40 and above to determine which items belonged to a given subscale. Two exceptions were the two remaining items. One had a factor loading of .36 on the Planning factor, so we added it to that subscale. The other had a factor loading of .38 on the Involvement factor, so we added it to that subscale.

The items that make up each sdubscale and their associated factor loadings are shown in Appendix B.

Raw-Score Analysis
To do our raw-score analysis, we obtained raw-score totals for the 60-item scale and for each of the six subscales. For the 60-item scale and for each subscale we grouped the totals into four sets: Fall control Fall-experimental, Spring-control, and Spring-experimental. We computed means of the totals for graphing. To test the statistical significance of the differences we found between fall and spring and control and experimental, we applied a separate analysis of variance to each of the six factor-defined sets of totals. We used the three independent variables of fall versus spring, experimental versus control, and the interaction of these two variables. The dependent variable was the total score for each of the four subgroups of students as defined by the time of year and whether they were controls or experimentals.

Figures 1 and 2 show the mean raw scores obtained from the four subgroups. The first graph in Figure 1 shows the means obtained from the full scale. The subsequent graphs in Figures 1 and 2 show the means obtained from each of the subscales identified through the factor analysis. Two major trends are clear from these figures.

First, the control group showed little or no gain or lost ground from fall to spring on every subscale and on the total scale.

Second, the experimental group showed a gain from fall to spring on each subscale and also on the total scale. An examination of the individual graphs in these figures is warranted. For the total scale (the first graph in Figure 1), both the gain of the experimental group over the control group and the interaction were highly statistically significant. The two groups started out about even in the fall. The experimental group went up in the spring and control group went down.

For Factor 1 (Self-esteem) and Factor X (Involvement) the differences between fall and spring, experimental and control, and the interaction were not large and were not statistically significantly different, though the direction of the differences favored the experimental group. These results, and those below, are shown in the subscale graphs in Figures 1 and 2.

For Factor 2 (Mobility) the gain from fall to spring, the gain of experimental over control, and the interaction were all statistically significant. A significant interaction in this case meant that the gains were dependent on both the time of year and which group was involved.

For Factor 3 (Ownership) the interaction comparison was the only statistically significant result, and it favored the experimental group. The nature of the interaction was that the control group started higher in the fall and ended lower in the spring, whereas the experimental group started lower and ended higher.

For Factor 4 (Communication) the results followed the same pattern as the results for the full scale. Overall gain from fall to spring was not statistically significant, but both the gain of the experimental group over the control group and the interaction were highly statistically significant. The two groups started out about even in the fall. The experimental group went up in the spring and control group went down.

For Factor 5 (Planning) The experimental group started low on this factor in the fall, and ended very high, whereas the control group started higher in the fall and showed a very slight decline to spring. The gain from fall to spring was very highly statistically significant as was the interaction, again favoring the experimental group.

Appendix C has the detailed raw-score analysis of variance results for the total scale, and for each subscale.


RIT-Score Analysis

We obtained equal-interval RIT (Rasch unit) scores for all students for the 60-item scale and the six subscales for the four groups of students; fall control, fall experimental, spring control, and spring experimental. The BIGSTEPS software program computed an average RIT value for each item in a scale and the RIT value for each of the alternatives for each item. ROT values were arbitrarily centered on a value of 200. We combined the fall groups to obtain the 60-item fall survey RIT item values. To put the results for the different groups and the different scales on a common scale, we used these 60-item fall survey RIT item values to score the 60-item spring survey raw data and the fall and spring subscale raw data.

We had the program compute a RIT score for each student for each scale. The less frequently an item received a high raw value (Strongly Agree) from student's responding to that item, the higher was the RIT value assigned by the program to that item. Conversely, an item with a low RIT value was one that most students gave a high raw value. The RIT results were similar to the raw-score results. Figures 3 and 4 show the mean RIT scores observed for the control and experimental groups in the fall and spring. The individual graphs again show the results for the total scale, and for the subscales identified through the factor analysis. The same two major trends that were observed in the raw-score analysis are again observed.

First, the control group either showed some or no gain, or lost ground from fall to spring on the total scale and each subscale

Second, except for the Factor X scale (Involvement), the experimental group showed a gain for all subscales and for the total scale. This gain was statistically significant for the full scale, the Factor 2 subscale (Mobility), and the Factor 5 subscale (Planning).

Once again, an investigation of the individual graphs in Figures 3 and 4 is warranted. The first graph in Figure 3 shows the mean RIT score obtained from the full scale. Both the gain of the experimental group over the control group and the interaction were statistically significant. The two groups started out about even in the fall. The experimental group went up in the spring and control group showed no change for the year.

For Factor 1 (Self-esteem) and Factor X (Involvement), the differences between fall and spring, experimental and control, and the interaction were not large and were not statistically significantly different, though the direction of the differences favored the experimental group.

For Factor 2 (Mobility), the gain from fall to spring, the gain of experimental over control, and the interaction were all statistically significant. A significant interaction in this case meant that the gains were dependent on both the time of year and which group was involved.

For Factor 3 (Ownership), the results were similar to the results for Factor 1 and Factor X, but different from the raw-score results. In the case of Factor 3, there was no statistically significant interaction as well as no statistically significant difference in time of year or control-experimental group. results.

For Factor 4 (Communication), the experimental group gain was statistically significantly greater than the control group.

For Factor 5 (Planning), the experimental group started low and ended very high, whereas the control group started higher in the fall and showed no change from fall to spring. The gain from fall to spring was very highly statistically significant as was the interaction, again favoring the experimental group.

Appendix D contains the detailed RIT-score analysis of variance


Analysis Achievement Scores

A secondary question in this study had to do with students, achievement while involved in the Location Learning Program. While the program isn't designed to directly enhance the basic skills such as reading and math, it is important that the use of the program not decrease student learning in these areas. We need to demonstrate that Location Learning does not decrease basic skills achievement to make it a viable program for our schools.

To look at student achievement in basic skills, we used the Portland Achievement Levels Tests (PALT) These are high-quality tests in reading and mathematics basic skills that are given yearly to all of the students enrolled in grades 3 through 8 in the Portland Public Schools. Since these tests are given in the spring, we were able to use students, scores from the spring prior to entering Location Learning as a pretest measure and scores from the Spring following the first year in Location Learning as a post-test measure. Since pre- and post-test achievement measures were available for students in the experimental and control groups, we can compare those groups using simple t-tests on the gains from one spring to the next in each subject area.

The results of these tests might be one of three cases. The tests may indicate that 1) students in the Location Learning Program gained more than those in the control group, 2) that students in the Location Learning Program gained less than those in the control group, or 3) there is no compelling evidence that gain by the Location Learning group was different than gain by the control group. For these comparisons, the class taught by Lucille Sheets was omitted from the analysis. Since she was injured and was unable to be with her class for much of the year, it was thought that this class didn't really reflect full implementation of the program. (Records of past years show Ms. Sheets' classes consistently gaining more than the district average in PALT scores.) Results. Table 1 shows the summary achievement scores from the PALT for students in the Location Learning and control groups.


In mathematics, the experimental group gained 10.11 points along the PALT score scale from pretest to pose-test, while the control group gained 8.72 points. This difference in mean gain of 1.39 points causes the test statistic to indicate that a chance difference this high would occur less than 5% of the time, and is consistent with the hypothesis that the experimental and control populations differed more in math growth than expected by chance. Further, these results indicate that the experimental group had greater math growth than the control group.

In reading, the experimental group gained 7.14 points along the PALT score scale, while the control group gained 7.03 points. This difference in mean gains of 0.11 points causes the test statistic to indicate that a difference this high would occur more than 5% of the time, and is consistent with the hypothesis that there is no difference reading growth between the experimental and control populations.

Results of the achievement analysis indicate that participation in the Location Learning program does not negatively affect learning in the basic skills. In fact, the only significant difference noted indicated that mathematics achievement was improved by participation in Location Learning. This increase in mathematics achievement needs to be verified in future research.


Conclusions and Discussion

This study resulted in two major findings.

First, students in Location Learning classrooms grow more than their control counterparts in almost all of the affective areas investigated.

Second, students in Location Learning classrooms grow no less in basic skills achievement than those in the control group, and may grow more in mathematics. Each of these findings is supportive of Location Learning as being an effective integrated learning program.

If the gains in affective measures found in this study continue as students progress in their school careers, participation in Location Learning may enhance student confidence and ability to handle complex problems. Since the cost of a Location Learning program in a classroom is relatively low, and gains in student affective variables are consistent, Location Learning may be viewed as a strong alternative to the traditional classroom approach.

Location Learning can serve as an integrated learning methodology that combines basic skills teaching with the development and personal growth of students in the areas of planning, self-esteem, ownership, mobility, and communication. In addition to potential benefits to individual students, Location Learning also provides the students, school, and school district with concrete examples of student capabilities that are well aligned with the Oregon Education Act for the 21st Century, and which may be usable as evidence of accomplishment for the Certificate of Initial Mastery.

In summary, Location Learning is a promising integrated learning approach, that should be encouraged for its benefit to the students, and its eventual benefits to the community at large.

Close Window