Data Collection for Program Evaluation

Evaluation should be a data-driven process. When considering how teachers use and integrate technology or how students use technology to inspire and expand learning, it is important to have data which documents this activity. In this article, we will discuss several different data collection strategies which can support a technology evaluation effort.

Before we dive into a discussion of particular strategies, it makes sense to place data collection into the context of the evaluation and assessment initiatives it supports. It is our contention that meaningful program evaluation - for instance in the realm of digital learning -- is about examining the impact that digital learning tools have on teaching and student achievement. This impact cannot be measured simply by "counting devices" nor for that matter entirely by counting teachers or students who simply "use" their devices. Rather, we believe that impact is measured and documented through observation of a complex picture composed of teacher behavior, student work, attitudes, curriculum, and technology.

We have worked with school districts, educational organizations, and higher education institutions to develop authentic assessments of program impact that focus on teaching and learning. In virtually all cases, these assessments are rooted in the goals of a strategic plan such as a digital learning plan. Assuming that the plan goes beyond an outline for technical infrastructure and has a vision (and related goals) for programmatic impact in the classroom, it is possible to create descriptive indicators for the desired outcomes of the plan's goals. These indicators form a framework for data collection. In this way, data collected is intended to confirm the achievement (or lack thereof) of specific indicators.

A full description of our evaluation process is available online. Sample focus group and interview questions can be found here. We also have sample surveys and classroom observation protocols. A list of other online resources related to technology evaluation is found at the end of this article.

Tying Data to Indicators

Within the context of an evaluation effort, data collection makes use of tools and techniques such as surveys, observations, interviews/focus groups, review of teacher/student work, and public meetings. The point is to collect data which relates to the developed indicators. Data collection is designed in response to assessment rubrics developed by a district from their goals and objectives for technology. The point of data collection is to gather information which will enable the district to "answer" the evaluation questions and "score" their performance on their rubrics. Therefore, it is not possible to predict a data collection strategy without knowing something about a district's evaluation rubrics.

For example, if an indicator of high achievement in teacher use of technology is that teachers will use technology to communicate with peers outside of the district, then data is needed which shows the amount as well as qualitative substance of teacher online communications. This might include technical logs (e.g., how often do teachers access their email accounts); teacher surveys to determine how often online communication is used and for what; and teacher interviews to determine the value placed upon virtual communication strategies. All of this quantitative and qualitative data is used to determine a level of overall achievement in the indicator rubric. A similar logic would be used to measure achievement with any set of indicators.

We have found a variety of tools useful for technology evaluation data collection.

Here, it is worth mentioning that while data collection might take place at the individual level of performance, individual data it should never be reported. The mission of a district-wide evaluation is to determine the progress of the district as a group of individuals in meeting its goals. Nothing will undermine an evaluation project faster than the perception that it is measuring or ranking individuals. If individual assessments are important, these should be developed and administered separate from your district technology evaluation.

Other data collection strategies and mechanisms can be deployed. For example, some districts have found success with a "public meeting" format where Sun Associates facilitates (and documents) a discussion about a community's goals, aspirations, and concerns. As a cost-saving strategy -- particularly in larger districts -- we can work with in-district evaluators to adapt and deploy existing data collection mechanisms and/or to identify statistically relevant sample populations so as create a more manageable data collection effort.

Triangulation

A meaningful evaluation will never rely on a single data source (e.g., surveys). Rather, you need to design and implement a data collection strategy which has the optimum chance of capturing the big picture of technology's use and impact your district. This is why we generally employ most, if not all, of the above-mentioned data collection tools in any given evaluation project. We suggest that districts "triangulate" their data findings. By this we mean that a survey can be used to make a broad sweep of data. Then, an analysis of the survey data can highlight particular issues which can be explored during focus group interviews. Finally, themes and findings arising from the survey and focus group data can be explored through "walk arounds" and classroom observations. This represents the basic "triangle" of data collection. Artifact analysis (examining student and/or teacher work product) adds yet another valuable dimension to the whole data picture.

Another reason for implementing multiple data collection strategies is that while surveys can generally reach a large percentage of your district staff/students/administrators, they are rather limited in the amount of detail captured. In other words, you can get a large sample with surveys but relatively little depth as compared to labor-intensive yet data rich interviews and observations.

Data Analysis and Reporting

Clearly the point of data collection is to support the development of evaluative findings. This means scoring of rubrics and the creation of some sort of report which provides a textual overview. Nevertheless, it is also useful and important to share the summarized results of the data collection itself. We have found that survey data and summarized focus group data can be powerful tools for building and district decision-makers. In one example, a district technology coordinator asked us to provide survey data -- from a district-wide survey aggregated to the building level -- to site-based management teams. Teams were then able to use this data to make decisions related to technology support, resource, and professional development allocation in their individual schools.

Finally, it's important that the people who provide you with data -- e.g., teachers, administrators, students, parents -- receive some tangible product from their efforts. All too often teachers complete surveys which basically "vanish" into some black hole of data collection. Over time, this serves to create a large disincentive to cooperating with district data collection efforts. It's a rather simple matter to provide respondents with a timely data report that indicates how much you value their input.

Information on this site that has been produced by Sun Associates is Copyright 1997 - 2018 Sun Associates and is available for individual, one-time, use by educators. Duplication is prohibited without permission. All other material is the property of its authors and Sun Associates makes no warranty for its use or accuracy.

Last updated, May 10. 2018