Anytown Public Schools

Year Two Educational Technology Plan

Professional Development Evaluation

Final Report

August, 1998

Table of Contents

Executive Summary

Overview and Methodology

Team Leader Training

Data Collected

Process and Format of Team Leader Training

Content and Benefits of Team Leader Training

Team Leader Survey Data

Learning Team Experiences

Data Collected

The Learning Team Process

General Comments and Findings

Scoring the Rubrics

Evaluation Question One

Evaluation Question Two

Summary

Recommendations

Recommendations for Further Research

Executive Summary

The focus of Anytown Public Schools’ Year Two Educational Technology plan implementation was staff development. The plan specified that all staff were to attain a standard degree of proficiency in using basic computer technologies. To give focus to this effort, the basic proficiencies were given a common name – Kansas – and thus the Year Two staff development effort was known as "reaching Kansas." The journey to Kansas made use of a staff development approach – known as Learning Teams – which emphasized collaborative learning and constructivism amongst the participating staff.

Early in the 1997/98 school year, 31 Learning Team leaders were identified at the Elementary and Middle School levels. A decision was made to delay High School technology staff development until the 1998/99 school year. Elementary and Middle School team leaders were trained over a 13 week period starting in October. Even before this training was complete, the team leaders began to meet with their respective teams. Learning Teams – assembled from all Elementary and Middle School staff – began to meet in November and continued their work until May, 1998.

The Learning Team approach emphasized both the process and products of technology staff development. While each team was expected to develop the technology skills of its members (i.e., the members were to "reach Kansas"), they were also supposed to form an on-going collaborative and cooperative learning environment. This environment was designed to be utilized well beyond Kansas. The Learning Team was to be a vibrant and supportive structure for Anytown Public Schools’ staff.

In light of this design, the purpose of our evaluation was to assess the success of the Learning Team approach in assisting Elementary and Middle School staff in their journey to Kansas. Specifically, the evaluation effort developed a set of indicators of success and then proceeded to collect a variety of data to determine how the Learning Team approach performed on these indicators. The evaluation effort was facilitated by Sun Associates and involved the work and guidance of Anytown’s District Technology Committee (DTC), the Coordinator of Educational Technology, the Coordinator of Curriculum and Instruction, and other district-level staff. The District Technology Committee developed criteria for success and rubrics through which the success of both the Learning Team product and process were assessed. In the course of the evaluation, Elementary and Middle School team leaders were interviewed and surveyed, team members were interviewed and surveyed, and learning team meetings were observed.

In summary, this evaluation found that the vast majority of Anytown’s learning team members "reached Kansas." Overall, Elementary and Middle School teachers had mastered the Kansas skills by the end of the 1997/98 school year. At the same time, the learning team process itself was determined – using the DTC rubric criteria -- to be less successful than had been hoped.

Since the Year Two evaluation is at heart a formative evaluation, we have developed several recommendations for continuing the success of Learning Teams in ensuing years. In summary, these are:

  1. Learning teams should be built around groups of individuals with a clearly identifed common purpose.
  2. Provide remedial, or very basic, technology skills training independent of the learning team process.
  3. Provide team leaders with a longer time to train and otherwise prepare for their responsibilities.
  4. Provide both team leaders and team members with more concrete, step-by-step, training materials.
  5. Create a "review loop" in the learning team process.
  6. Work to improve the technical competence of the team leaders.
  7. Clarify participant expectations and concerns prior to participation in the Learning Team process.

Overview and Methodology

Overview

In October, 1998, Sun Associates was contracted to conduct an evaluation of Anytown Public School's Year Two Educational Technology Plan implementation. The Year Two implementation focused on providing staff development to all Anytown Public Schools’ staff in basic technology skills, termed "Kansas."

Kansas training consisted of three primary components. These were:

The Year Two evaluation primarily focuses on the effectiveness of the Learning Team approach -- and its Team Leader and Learning Team meeting components -- in helping Anytown's elementary and middle school staff reach Kansas.

Methodology

The evaluation methodology developed by Sun Associates consists of two primary approaches. These are:

  1. The development of rubrics that facilitate assessment of the performance of Anytown Public Schools staff on two key evaluation questions. Paraphrased, these questions are:
    1. Did Anytown Public Schools staff reach Kansas?
    2. Was the Learning Team approach successful in developing Kansas skills?

    The rubrics, as well as the evaluation questions that underlie the rubrics, were developed by Anytown's District Technology Committee (DTC) through a process facilitated by Sun Associates. Copies of these rubrics follow.

  2. Data gathering directly from Learning Team Leaders and team members. Leaders and members were surveyed, interviewed in focus groups, and learning team meetings were observed. The objective of this approach was to collect information on how learning teams and their members performed. The ultimate use of this data was to use it for scoring the rubrics (above).

Evaluation Rubrics

Evaluation Question One

How effective has the learning team approach been in helping Anytown Public School professional staff reach KS?

Indicator Statement:

The learning team approach has been effective in helping APS professional staff collaborate to achieve the core technology competencies identified as "Kansas."

Level 1 (just starting to fulfill -- work towards -- the goal)

A few teachers (less than 20%) report active involvement in their learning teams and use one KS skill for teacher productivity and curriculum activity. While some teachers may exhibit proficiency in the KS skills, these proficiencies have been developed outside of the learning team and not through the learning team process.

Evidence:

  • Most participants only get help and assistance from team leaders
  • Team members rely upon team leaders for direction
  • Team members work on developing KS skills only during team meeting time and when directed

Data Collected to Support Evidence (all levels):

  • Attendance
  • Observational Data
  • Surveys
  • Interviews
  • Message archives/electronic postings

Level 2 (somewhat fulfilling the goal)

Some teachers (50%) are actively involved in learning teams and use several KS skills for teacher productivity and curriculum activities.

Evidence:

  1. Most assistance to learning team members occurs during learning team time
  • Learning team members spend some time seeking and/or providing assistance in Kansas skills to other team members during learning team time
  • Learning team members spend some time seeking and providing assistance in Kansas skills outside of learning team time

Level 3 (near fulfillment of the goal)

Most (80%) staff members are actively involved learning teams and generally using most KS skills for productivity and curriculum activities.

Evidence:

  • Learning team members spend considerable time seeking and providing assistance in Kansas skills during learning team time
  • Learning team members spend considerable time seeking and providing assistance in Kansas skills outside of learning team time
  • Teachers are involved in mentoring relationships with other team members outside of learning team time

Level 4 (complete fulfillment of the goal)

Throughout the system, all (95% or more) teachers ask for and give help to other teachers in the core competencies skills. Teachers routinely use all KS skills for productivity and curriculum unit integration.

Evidence:

  • (95% or more of all )Teachers share techniques in Mac Basics, using network, ClarisWorks, searching the web, email
  • Teachers develop ad hoc learning teams to explore integration of technology into curriculum units and activities and to work with teachers other than learning team members
  • Teachers collaborate to design curriculum units that integrate technology using KS skills
  • Team projects clearly show unique input from each team member
  • Lesson plans routinely integrate technology

Level 5 (beyond complete fulfillment)

Throughout the system, the learning team approach has become an institutionalized, replicable, model for professional development

Evidence:

  • Within buildings, teachers develop learning teams (affinity groups)
  • Across buildings, teachers develop learning teams (affinity groups)
  • Outside of the system, teachers develop learning teams (affinity groups)

 

 

Evaluation Question Two

How effective have KS skills been in helping the professional staff do things that they could not have done otherwise?

Indicator Statement:

The achievement of "Kansas" skills has helped the professional staff increase efficiency, productivity and collaboration.

Level 1:

Anytown teachers have been exposed to KS skills and use some of them occasionally for personal/professional productivity. Teachers seldom communicate using technology.

Evidence:

  • Teachers have written an occasional document with Clarisworks
  • Teachers have accessed their email mailbox less than once a month
  • Periodic use of Netscape to begin WWW exploration

Data Collected to Support Evidence (all levels):

  • Readmister self-assessment (survey)
  • Additional survey on usage
  • Interviews
  • Samples of student work
  • Samples of teacher work
  • Observations
  • Tracking/recording of electronic data on server use, logins, etc.

Level 2:

Anytown teachers have begun to use some KS skills regularly for personal or professional productivity. Teachers occasionally communicate using technology.

Evidence:

  • As in level 1, but use of any technology occurs more frequently -- at least once a week.
  • Prepares unit materials and stores files in folders.

-worksheets, homework, tests, class lists, etc.

  • Incorporate the use of word processing for lessons.

-brainstorming, writing, editing, etc.

 

Level 3:

Anytown teachers have become proficient using KS skills for personal/professional productivity. Teachers frequently communicate with other professionals within their building.

Evidence:

  • More than one KS skill is used regularly for teacher productivity -- at least once a week
  • Can download and copy selections from the Internet.
  • Has familiarity with a list of useful web sites.

Level 4:

Anytown teachers use KS skills for their own personal/professional productivity, but have not applied their skills to direct student instruction. Teachers routinely communicate for professional purposes using technology with other professionals within the school district .

Evidence:

  • Use KS skills to collaborate with teachers within their district
  • Uses relevant curriculum material found on the Internet.
  • Creates and uses mailing lists (nicknames) for use in regular communications.

Level 5:

Anytown Teachers have moved beyond using KS skills as personal productivity tools and are extending the benefits of these skills toward lesson delivery/student instruction. Teachers routinely communicate using technology with other professionals outside the school district to expand their classroom curriculum projects.

Evidence:

  • Uses Kansas skills to collaborate with teachers outside of the school system. -- share materials, lesson plans, research; send attachments; access and store files in the file server; and participates in Listserves
  • Help other teachers and students with Kansas skills.
  • Bring students to the writing lab or bring writing lab into the classroom (AlphaSmarts).
  • Create Claris slide shows for instructional purposes.
  • Uses relevant curriculum material found on the Internet for instructional purposes with students.

Team Leader Training

Data Collected

Number of on-line surveys completed:

Number of leaders attending focus group:

Process and Format of Team Leader Training

Our on-line surveys and focus group interviews provided considerable insight into the team leader training process. These observations have been organized in themes and are detailed below.

Issues Related to Grouping

Team leader feedback indicated a strong preference for homogeneous grouping when composing the learning team. Many team leaders felt that considerable time was lost in bringing some team members up to speed, and this could have been avoided through homogeneous grouping. Furthermore, some team leaders felt that they were less technically prepared to lead their teams than many of their team members.

Nevertheless, the on-line survey indicated that most respondents — at least by the conclusion of team leader training — felt that they had mastered the core Kansas skills and that they in fact generally used these skills both for personal productivity and in their classrooms. This being the case, it should be noted that Team Leaders are technically prepared to lead their teams.

Those team leaders who reported that their teams were ability grouped expressed more satisfaction in terms of how their training/team meetings progressed. This seems to indicate that the issue is less one of true technical "competence" but rather one of comfort in dealing with ones peers (or not) around a new subject.

Team Leader Training and Preparation

Most team leaders interviewed had more than two years experience teaching in Anytown Public Schools. While the majority of these team leaders stated that they had not ever previously participated in a train-the-trainers experience such as Team Leader Training, they also felt reasonably well-prepared to be team leaders and stated that they were reasonably aware of what they were "getting into" when agreeing to be a team leader.

Nevertheless, after the team leader training had been completed, a number of those interviewed felt that they had not been fully aware of the time commitment involved in being a team leader. In particular, leaders felt that they needed much more time to develop materials (agendas, lesson plans, adaptations of the Kansas Handbook) for their learning team members.

Through our evaluation, we discovered that team leaders were not chosen until after the start of the 1997/98 school year. This meant that most team leaders did not anticipate having to include the duties of team leader in their year's work until after the year had already started. Naturally, this lead to some leaders being less prepared and more over-worked than would be desired. Further, the timing of leader selection did not allow leaders to reconsider their decision to serve as leaders until it was practically too late.

Training Session Format and Related Issues

Trainer trainers were credited with being extremely technically knowledgeable and very personable. Trainer received many positive comments in the focus group interviews and 52% of the respondents in the Team Leader on-line survey felt that the "consultants" (i.e., Trainer) provided adequate training.

There was feeling that Trainer over estimated the technical expertise of both the team leaders and ultimately the team members. In our survey, several leaders echoed a statement made in the focus group interview that "People don’t want to be enabled, they just want to be shown how." While not all team leaders would appear to agree with this statement, the data shows that by-and-large, team leaders wanted more training in the specifics of using Kansas software and less training in the facilitation aspects of technology staff development.

Several comments were received (via survey and interview) citing problems with the time of day in which team leader training was scheduled. Team Leaders noted the difficulty with which anything is scheduled in the busy Anytown Public Schools day, but asked that in the future training such as Team Leader Training be scheduled for longer blocks of time, earlier in the day.

Another issue related to the availability of working technology for the training sessions. Several comments were received (via both survey and interview) citing problems with having a sufficient number of working computers for learning team members. One team leader noted that several of the team leader training sessions, as well as team meetings, would have been "impossible" if competent Anytown Public Schools’ technology staff had not been present to troubleshoot and technically facilitate their team leader training sessions.

Content and Benefits of Team Leader Training

Team leader training was generally perceived by its participants as a valuable experience. The leaders felt that it was a "good start" in assisting them in supporting/training their team members.

A general, and often-repeated, concern was that the team leader training did not sufficiently prepare team leaders to be "experts" in technology use. Rather, the training seemed to support the development of "facilitators." There was much discussion in the focus group about which is most needed (by team members) -- experts or facilitators. The final consensus was experts. This is in keeping with many other comments about the training that basically came down to the training and preparation (including aspects such as the KS Handbook) was interesting and well-conducted, but it came up short in terms of providing team leaders with concrete resources and "expertise" that they feel is required of the team leader. How much of an issue this is is likely related to the (pre)existing technical expertise of the individual team leader and the composition of her/his learning team. It will be necessary to verify this team leader observation by seeking corresponding data from team members (i.e., did they feel that their team leader was sufficiently prepared to "technically" support the team).

As noted above, a number of team leaders felt ill-equipped to "prepare lessons" and create "3 to 4 agenda items for training time" for their learning teams. They felt that there was not sufficient time to do this, and that it went beyond what they had expected in their roles. A number of team leaders expressed the desire to have had more guidance in creating these agendas, or perhaps to have had more sample agendas provided.

 

The Kansas Handbook was felt to be a useful general reference tool, but largely inadequate for a large number of learning team members. A number of team leaders cited the fact that they felt compelled to "re-write" the Handbook sections for use by their team members. It was noted that the Handbook would be useful eventually for all Anytown Public Schools teachers, but was not adequate for the rank beginners who composed many learning teams.

Interviewed team leaders noted several times that they had spent considerable time — time which they had not anticipated having to spend — rewriting, digesting, and otherwise adapting the Handbook for use with team members.

 

Team Leader Survey Data

The following table summarizes responses to the Learning Team Leader on-line survey, conducted in late February, 1998. Out of a possible 31 Elementary and Middle School team leaders, we had 25 responses to the survey — an 80% return rate.

The survey was designed to assess respondent’s attitudes, beliefs, and (self-assessed) skills in mastery and use of Kansas skills. It is important to view the on-line survey results against the backdrop of the more expansive comments provided by focus group interviews of Team Leaders. Our analysis of this data follows.

TLT helped me master skills I would not have mastered otherwise 68% (n=17)
TLT was a valuable use of my time 76% (n=19)
Sufficient time was allotted for TLT 72% (n=18)
Team Leaders were adequately trained by Anytown Public Schools staff and consultants 52% (n=13)
Believe that the Learning Team Approach was the best way to develop Anytown Public Schools staff technology skills 84% (n=21)
Believe that mastery of KS skills will help Anytown Public Schools staff make technology a valuable part of teaching and learning 96% (n=24)
Confident in Mac Basics 100%
Confident using Eudora 100%
Confident using Netscape 92% (n=23)
Confident using ClarisWorks 100%
Confident using the file server and network operations 96% (n=24)
Have not mastered Mac Basics 0%
Have not mastered Eudora 0%
Have not mastered Netscape 12% (n=3)
Have not mastered ClarisWorks 8% (n=2)
Have not mastered the file server and network operations 12% (n=3)
Have written documents using Clarisworks 100% (n=25)
Use ClarisWorks for professional tasks (memos, tests, worksheets) 84% (n=21)
Use ClarisWorks with students 56% (n=14)
Check email more than once a week 100%
Send at least one email message each week 96% (n=24)
Know how to set up Eudora mailing lists 80% (n=20)
Send email to people outside of the Anytown Public Schools system 76% (n=19)
Can use Netscape to locate WWW sites 96% (n=24)
Use the WWW in the classroom at least once a week 56% (n=14)
Have developed a set of WWW bookmarks 88% (n=22)
Can store and access materials on the school file server 68% (n=17)

Attitudes and Beliefs

Respondents overwhelmingly believe in the Kansas approach. Further, there is a strong (84%) belief that the learning team approach was/is the best way to develop the Kansas skills amongst Anytown Public Schools’ staff.

Specific to Team Leader Training, a majority (76%) of respondents felt that the training was a valuable use of their time. A somewhat fewer number of respondents (68%) felt that the training helped them develop skills that they would not have developed otherwise, and this — combined with the fact that very high numbers of respondents indicated that they were confident in using the Kansas applications — may indicate that a number of team leaders were already "in Kansas" prior to the team leader training.

Confidence and Mastery

One survey respondent accurately noted (in a written comment) that "mastery" is a subjective term; and indeed, this is true. Nevertheless, the goal of this part of the survey was to have respondents self-assess their own mastery and confidence in using the core Kansas applications.

Virtually 100% of all respondents indicated that they were comfortable using the five core Kansas applications/skills. Correspondingly, very few respondents felt that they were still weak in using any of these applications.

Actual Use of Kansas Applications

While the previous set of questions simply sought a subjective response to whether a skill had been mastered, the final set of survey questions set out to discover what exactly Team Leaders are doing with Kansas skills.

By and large, it would appear that most respondents are using Kansas skills both for personal productivity and to a lesser degree in the classroom. It should be noted that this is exactly the expected/desired outcome of Anytown Public Schools’ Year Two staff development! For example, 100% of respondents indicated that they had used ClarisWorks to create a document (any document) and have checked their email "more than once a week." Only 56% indicated that they had ever used ClarisWorks with their students or used the WWW in their classrooms more than once a week. Once again, it is important to view these results in context of other information about Anytown Public Schools technology as it likely that lower use of classroom technology has as much to do with the availability of such technology as it does teacher training in its use.

Learning Team Experiences

Data Collected

  1. Number of on-line surveys completed:

    87 Elementary Learning Team Members

    29 Middle School Learning Team Members

    116 Total

    Note that this represents a 34% sample of all Anytown Public Schools teachers participating in learning teams during the 1997/98 school year. In terms of breakdown by grade level, 42% of all participating elementary teachers completed the on-line survey versus only 22% of middle school participating teachers.

    A large number of respondents to the on-line survey offered detailed, textual, comments. Some of these comments have been excerpted to illustrate particular points in this evaluation and are shown as block quotes in the text below.

  2. Anytown Public Schools district technology staff administered an on-line "mid-year learning team review" survey in February, 1998. 73% (n=154) of all elementary team members and 59% (n=79) completed this survey which asked questions primarily related to participation in learning team meetings and surveyed staff on needs/desires for 1998/99 school year staff development offerings.
  3. Number of team members attending focus group:

    9 Elementary

    1 Middle School

    10 Total

  4. Middle School Learning Teams were observed during their next-to-last meeting on April 27, 1998.
  5. All elementary and middle school staff completed the Anytown Public Schools-produced on-line "Kansas Self Assessment" of technical abilities in September, 1997 and again in May, 1998. Return rates for the September survey were nearly 100%, but were less for the May survey. As of this writing, data is not available from the May survey. Similar data, in somewhat less fine detail, was collected in Sun Associates' May on-line team member survey.

The Learning Team Process

Elementary and middle school learning teams met nine times between November and May. Learning team meeting time overlapped significantly with team leader training sessions (which ran through March), so in most cases, teams were meeting with leaders who were still involved in leader training.

Elementary teams were mostly composed of teachers from within a single school. Middle school teams were composed of teachers from several different schools grouped by a common curriculum interest. In both elementary and middle schools, teams were largely heterogeneous in terms of the team members’ technology skills.

The general format of learning team meetings was for the team leader to assess team interests and abilities and then to facilitate activities which addressed both existing interests and the skills that the leader deemed necessary for team members to develop. This naturally resulted in a wide variety of different activities and format for each learning team. In theory, this should have also resulted in very individualized and relevant training. In practice, just how individualized and relevant the training was depended upon several factors. We found these factors to include:

Most learning teams met during the latter portion of the day on designated learning team meeting days.

General Comments and Findings

The above comments sum up the majority of the findings collected on the elementary and middle school learning team experience. In general, most participants enjoyed the experience and felt that the learning team model for staff development was sound. Nevertheless, many participants felt that their experience would have been more positive if they had been grouped with others of like technical ability.

Overall Effectiveness

The main question related to the learning team process is: How effective was the learning team approach in developing Kansas skills amongst participants? In answer to this, our data indicates that the learning team approach assisted Anytown Public Schools elementary and middle school teachers develop and improve their Kansas skills.

Truthfully, we cannot state that another professional development approach would not have been just as successful; but based on data collected via survey, we can say that 75% of the participants in Anytown's learning team process felt that the learning team approach "..is the best way of developing the technology skills of Anytown Public Schools staff." Furthermore, the vast majority (more than 90%) of surveyed staff feel "confident" in their mastery of the core Kansas skills. The only skill where there appears to be any problem with mastery is in the area of File Server/Network usage where only 55% of surveyed staff feel "confident."

In short, our data shows that a majority of Anytown Public Schools elementary and middle school staff are now confidently "in Kansas."

 

Grouping

The preference for homogenous grouping – by technology skill level -- applied to both participants of high as well as low technical skill. Highly skilled participants felt slowed down by their groups whereas novice technical users felt responsible for causing frustration for their more advanced team members. Problems related to grouping were highlighted repeatedly by team members in all of our data collection areas. Nevertheless, technology skill level was only one variable in terms of heterogeneous grouping. While differences in individual skill level within the learning team was indeed frustrating to many of the members, it appears that even more frustrating was the need for some middle school members to move outside of their own buildings. In other words, the learning teams were heterogeneous in several variables, and this is important to understand when investigating the frustrations of individual team members.

It should also be noted that a few team members reported that they favored heterogeneous groups. Both in the on-line survey and in the focus groups the comment was made that heterogeneous groups allowed the participant to select from a range of people to ask for assistance and thereby lessened dependence upon the team leader. This would seem to be a strong advantage and clearly is one of the designed purposes for heterogeneous groups and for the learning team approach in general. Nevertheless, it is an advantage that seemed lost on most participants who once again strongly preferred homogenous groups.

Format

Most participants felt that sufficient time was allocated for team meetings. Several times, we heard the comment that meetings could have been scheduled more conveniently earlier in the day rather than at the end of what for most teachers is a very long day.

Content

As noted above, most on-line survey respondents felt that the content of the team meetings led them to Kansas. The feedback received from in-person interviews on this subject was less clear. Focus group interviewees felt that for the most part the content of the team meetings was sufficient, but that they were not terribly sure that they had indeed "reached Kansas." Once again, the sentiment was that considerably more content could have been covered if the teams had been homogeneously grouped.

Comments received on the survey and in the focus group indicated that many team members felt that the content of the team meetings should have been more step-by-step and "how-to" in nature. Echoing a sentiment from the team leader training, there was a feeling that while the learning team process was enjoyable -- particularly the opportunity it afforded for discussion and reflection -- the meeting time could have been more productive if more attention was paid to simple details of "how-to" use the various Kansas tools.

Process and Product

In both the on-line survey and the focus group we probed to determine whether team members understood that one of the primary benefits of the learning team approach was the creation of collaborative groups of colleagues to support each other in their on-going exploration of technology and its application to teaching and learning.

Survey respondents were asked two specific questions related to this issue:

  1. "My learning team experience has enabled me to share knowledge, skills, and resources with my fellow learning team members." (yes or no)

    To this question, 96% (n=104) responded "yes"

  2. "At least once a week, I provide technical assistance to some of my fellow learning team members." (yes or no)

    To this question, 26% (n=31) responded "yes"

Focus group participants were asked to choose one of the following statements as being "more important" than the other. The two statements were:

The focus group participants felt that the first statement should be their priority; that is, that the ultimate point of team meetings should be to learn the Kansas skills. While the group recognized the design for collaboration, they felt that this should be a lower priority than technical skills development

Finally, in a "mid-year" survey issued to learning teams by Anytown Public Schools’ technology staff, team members were asked if their expectations for "skill development" and "collegial interaction" were being met through team meetings. In both the elementary and middle school groups, the majority of respondents indicated that "most" of their expectations were being met for skill development where as the majority noted that "all" of their expectations were being met for collegial interaction.

It appears that the objective of collaborative and constructivist learning was clear to most team members. On the other hand, it also appears that they have not in all cases realized the particular benefits of this type of learning nor do they necessarily feel that the creation of such a learning environment should be the primary objective of technology staff development.

Learning Team Resources

Team members were asked to evaluate the quality of the resources provided to learning teams. In particular, they were asked about the quality of team leaders and the Kansas and Beyond handbook.

In terms of team leaders, some survey respondents enthusiastically praised their team leaders but an equal number specifically criticized their team leaders for being insufficiently prepared and lacking in technical knowledge. This observation is confirmed when one considers that a number of team leaders (see section on team leaders) felt unprepared and in general less skilled than some of their team members. Overall, 78% of all survey respondents felt that their team leaders had been "adequately trained."

Focus group interviews provided a number of suggestions for possible improvements in team leaders. It was noted that team leaders seemed to be quite varied in their level of preparedness and in their "enthusiasm" for their work as team leaders. Furthermore, it was noted that it would have been helpful if all team leaders had prepared materials -- lessons -- to use in their learning teams. Some leaders apparently did this, and it was perceived of as being quite helpful. Other teams wished that their leaders had done something similar.

This degree of difference between team leaders -- and subsequently how they worked with their teams -- was noted when we observed teams in action on 4/27. While some teams seemed quite directed, with the leader clearly facilitating a "designed" lesson, other teams seemed much more open-ended and exploratory with the team leader taking an almost invisible role. Once again, if team members expected and appreciated a "guide on the side" style leader, they were no-doubt pleased when their leader worked in this way. On the other hand, it seems that many team members (and team leaders as well!) expected leaders to be visible "experts" who orchestrated their sessions. Members with this expectation were often frustrated by the lack of technical know-how held by some team leaders. As for the handbook, teachers interviewed in the focus group felt that the handbook was a good resource, but needed to be considerably more "step-by-step" if it was to be of practical use to most teachers.

Scoring the Rubrics

A key set of questions that can be used to score both of the Year 2 evaluation rubrics was included in the on-line survey administered to all learning team members in May, 1998. These questions, and the percent of respondents responding affirmatively to each question is shown in the table below.

Have written documents using Clarisworks 96% (n=112)
Use Clarisworks for professional tasks (memos, tests, etc.) 82% (n=96)
Use Clarisworks with students 54% (n=63)
Check email more than once a week 98% (n=114)
Know how to set up Eudora mailing lists 49% (n=57)
Send email to people outside of the Anytown Public Schools system 71% (n=82)
Can use Netscape to locate WWW sites 91% (n=106)
Use the WWW in the classroom at least once a week 37% (n=43)
Have developed a set of WWW bookmarks 60% (n=70)
Can store and access materials on the school file server 46% (n=53)

It should be noted that this data is roughly equivalent to that collected from learning team leaders responding to the same questions in their on-line survey (see team leader section).

Evaluation Question One

Using the criteria developed by the evaluation committee, Anytown Public Schools elementary and middle school teachers are at level 2 "somewhat fulfilling the goal" in the indicator rubric for this evaluation question.

While the teachers' involvement in their learning teams and their "use" of the Kansas skills would place them in level 3 (or above) of the rubric, our data shows that Anytown Public Schools teachers fall short in the significant requirement that that spend "considerable" time seeking and providing technical assistance outside of learning team time. Our data shows that only 26% of surveyed teachers provided or received assistance "at least once a week." Combined with data that shows that most (96%) of teachers do share experiences with team members, it appears that most of this assistance must occur during learning team time.

In short, while the majority of elementary and middle school teachers are involved in learning teams and use Kansas skills for personal productivity and (to a somewhat lesser extent) curriculum activities, a high degree of self-directed sharing and collaboration around Kansas skills does not yet exist.

 

Evaluation Question Two

Using the criteria developed by the evaluation committee, Anytown Public Schools elementary and middle school teachers are at level 4 in the indicator rubric for this evaluation question.

Level 4 in this rubric is defined by a majority of teachers using Kansas skills for personal productivity and communication with their peers within the district. Our data shows that a majority of teachers have clearly attained this level of performance. 82% report using Clarisworks (a key Kansas skill) for "professional work" tasks. Nearly all (98%) check their email more than once a week and a large number (71%) use email to communicate with individuals outside of the district. This latter fact begins to point to level 5 degrees of performance. Nevertheless, Anytown Public Schools teachers are not yet at level 5 as they are still somewhat limited in their use of Kansas skills within the curriculum. It must be noted that the curricular use of technology was not an objective of Anytown Public Schools in year two of its technology plan implementation.

Summary

Evaluation question one and its associated rubric is designed to assess the effectiveness of the Anytown's Year Two Technology Professional Development process. Question two focuses on the product of that process. Breaking our evaluation into these two process and product components, we can say that the district has currently realized greater success in reaching its product goals than in its process goals.

To many, this distinction might be inconsequential. After all, if the primary goal was to reach Kansas, then this seems to have been achieved to a rather high degree (level 4, specifically). The process by which Kansas was reached may not be important. On the other hand, the fact that most teachers are now in Kansas (so to speak) may simply be an indication of the fact that a good number were already in Kansas when the learning team process began. If this is the case, then it is perhaps important to understand the learning team approach as a process for reaching other professional development goals. The fact that it may (and we emphasize the word "may" as it would be irresponsible to draw definitive conclusions without more research and data collection) not have been particularly effective in building collaborative technical assistance environments is important to investigate the process further before replicating it for other professional development purposes.

Specifically, if the learning team approach is to support Year Three and Year Four professional development goals, Anytown Public Schools would in our opinion be advised to fine-tune and revise certain aspects of the process. Suggestions for these revisions are detailed in the following section of the report.

Recommendations

Sun Associates has developed seven key recommendations for Anytown Public Schools for it to consider when re-employing the learning team approach as a process for reaching Kansas or in the achievement of other technology professional development goals. These are:

  1. Learning teams should be built around homogenous groups of individuals with similar degrees of experience in using the Kansas technologies and a common purpose for being. While an overwhelming majority of Anytown teachers say that they would have preferred to have participated in homogenous groups, it is important to understand that their frustrations with their groups may have had as much to do with heterogeneous skills as with the lack of common purpose for being (outside of learning technology).
  2. Provide remedial, or very basic, technology skills training independent of the learning team process. Learning teams should be about refining and applying skills. A more step-by-step, how-to, approach is needed by many teachers. This did occur within the context of some learning teams, but this then lead to frustration on the part of those members who felt slowed down as well as those who felt guilty of being responsible for slowing down their group. Once again, homogenous grouping by technical ability level would have mitigated this concern somewhat.
  3. Choose team leaders earlier in the school year and thereby provide them with a longer time to prepare for their responsibilities. This preparation should include training on both technology skills and peer-group facilitation. By their own admission as well as through assessment by team members, relatively few team leaders seemed to be equally skilled in both of these areas.
  4. Provide both team leaders and team members with more concrete, step-by-step, training materials. Require that all team leaders develop and provide such materials to their teams. Team leaders who developed such material were generally praised by their teams. Furthermore, many team leaders felt that one short-coming of the team leader training process was that such materials were not provided to them for their later use.
  5. Create a "review loop" in the learning team process -- i.e., spread the experience out over a somewhat longer time so that a) team leaders could be fully trained before they were expected to meet with their teams; b) team leaders and their performance could be evaluated -- and subsequently adjusted -- before the completion of the learning team experience. The effect of both of these suggestions would be that the process could be adjusted and fine-tuned on a formative basis rather than waiting until the end of the experience to assess whether it worked or failed. In order to accomplish this interim adjustment, the overall pace of team leader training and learning team meetings would need to be moderated somewhat.
  6. Work to improve the technical competence of the team leaders. Both team leaders and team members indicated concerns that the leaders be greater "experts" in technical areas and not just good facilitators. The team leader training and its support materials could be strengthened to develop and support this greater degree of expertise.
  7. Clarify participant expectations and concerns prior to participation in the process. Clearly a number of participants were disappointed to discover that the learning team approach seemed to be as much about process (collaboration and constructivism) as product (learning basic technology skills). It is not that these disappointed individuals were against process, but rather that they simply did not expect "technology training" to be about something in addition to technology. It is highly likely that relatively inexperienced technology users expected more product whereas more experienced users were ready to consider process. So once again, grouping by experience level and commonality of purpose could likely have moderated this concern.

Recommendations for Further Research

The Year Two Educational Technology Plan evaluation is designed to be a formative assessment. In particular, the rubrics developed by the District Technology Committee are not intended for "one time" use. Rather, they are to be applied – in some form – each of several years. This way, the Learning Team approach, its success as a professional development model, and the technology skills of Anytown Public Schools’ staff can be continually monitored and improved.

With this in mind, we believe that there are several areas where continuing research and evaluation could occur. In essence, the following are recommendations for further use of the evaluation process.

  1. Review the rubrics in light of the evaluation data. Now that Anytown has been through one cycle of the Learning Team process, it is possible that the DTC may have somewhat different expectations of the process and its products. These expectations should be reflected in the rubrics.
  2. Collect more objective data on Learning Team member skills than the largely self-reported data which currently exists. Now that teachers supposedly possess the Kansas skills, we need to observe their use of these skills.
  3. Cross-tabulate measures of team member satisfaction with data on the construction of particular learning teams. For example, we should further investigate the correlation or lack of correlation between satisfaction and homogenous groups. Furthermore, we should delineate between groups that are heterogeneous by skill, common curriculum interest, grade level, and/or other factors. As noted several times in this report, there are a number of variables to heterogeneous grouping. Since this seemed to be such a big issue with Year Two participants, these variables should be investigated.
  4. Simply reapply the rubrics – whether modified (per recommendation number 1 above) or not – to learning teams in following years. This becomes particularly important as the recommendations of the Year Two evaluation are put into place.