Imagens das páginas
PDF
ePub

Chapter 11

PILOT STUDIES, EVALUATION, AND TRANSFERABILITY

by

Bryce Hudgins

Washington University

This chapter is concerned with the procedures for implementing programs of school reform. Several fundamental arguments and recommendations are advanced. Principal among them is the argument that proposals for school reform should be tried out on a pilot basis, their effects analyzed, and the factors that weigh for and against their adoption assessed before large amounts of Federal money are authorized for their support on a wide basis. This recommendation issues from two chief points: Massive funding of programs to solve particular social problems has not always led to the reduction of the difficulty for which the remedy was proposed; and educational phenomena, including innovations that have received Federal support, are frequently little analyzed and poorly understood. The recommendation of this chapter for a program of pilot studies is not a plea for money for research. It is an argument that the process of school reform can move more surely and more effectively if the total resources available for them are directed first at the tryout of proposals for reform, the study and analysis of those proposals as they occur in the real world of the schools, the recommendation of planned variants stemming from careful initial study of the pilot, and further tryout and evaluation of these variants on a somewhat larger but still demonstration scale in what are styled "second-round pilot studies." The results of such studies should then lead to recommendations about large-scale funding of a given program on a nationwide basis. The belief is that this process of pilots and evaluation procedures will, on the average, strengthen the Nation's efforts to reform its public school system.

Many issues and problems must be addressed in the development of a program of pilot efforts. We will foreshadow the problems in this introductory section, and deal with each of them in more detail in the body of the chapter. Some assumptions must first be explicated. Pilot studies will be based upon proposals for changing or reforming the school. Proposals, however well-conceived and detailed they may be in exposition, cannot begin to foresee or describe the exigencies with which any pilot project will be confronted, once it is placed in operation. Sociologists refer to the unanticipated consequences and the unintended outcomes of programs that entail social change. Thus the essence of a pilot study is that it is exploratory, as opposed to experimental. Questions of interest are as much of the order, "How do the participants modify or redefine elements of the program to survive in the environment?" as are questions concerning the extent to which variables detailed in the plan of operation do in fact operate.

551-373 074 - 19

Indeed, we regard these latter issues of the degree of installation of a project to be more telling for second-round pilot studies than for its introduction. These assumptions have important implications for the concept of pilot studies developed here, and for the concept of their assessment or evaluation.

Another set of assumptions concerns the mechanisms by which pilot studies are sanctioned and installed. We have assumed that a pilot program has a greater likelihood of success if it creates a need for as little new administrative machinery as possible, and if it passes through the decisionmaking apparatus and channels of control of those agencies and offices that have been created for larger purposes that would subsume activities such as the installation, evaluation, and diffusion of pilot projects. This assumption will have the consequence of lessening direct Federal bureaucratic control over details of such items as pilot site selection, project personnel, and related issues. But it will result in a much broader basis of support for, participation in, and ultimate success of the pilot program, than if direct control were maintained between the Federal funding agency and individual pilot project directors.

A third and critical assumption is that effective school reform can probably be achieved more quickly through the gradualist, reflective strategy of a pilot program than through the continuation of frontal assaults upon school change. Later in this chapter, the year 1985 is identified as the target year for the achievement of major reforms of the school. If one accepts such a target date, and also accepts the strategy of piloting as a vehicle for trying, assessing, and building tested models of school reform, then we think the outcomes for the Nation's schools by that time can be far superior to what they would be if resources continue to be placed into widespread installation of plausible-sounding ideas, about which very little is known in advance of their implementation. The strategy for pilot programs that we recommend is a gradualist one. An idea about reform would first be tried out, perhaps in only one or a few settings. It would be studied and analyzed in great detail. Several years after its inception, variations of it--capitalizing upon its most workable components, and adding others that seem to have sufficient merit to be tested--would be implemented on a somewhat larger, but still pilot basis. Those second-round pilots that demonstrate an effective capacity to bring about reform would then be validated procedures, strategies for change that, if installed on a broad scale, could be expected to work very well. If this strategy is adopted, it should be with the full recognition that first-round pilot studies that are introduced as early as 1974 may not generate fully tested and validated procedures for reform before 1980. The strategy will obviously have little appeal for the reformer who is in a hurry to bring about dramatic change in the schools. If, as we think, a decade is not too long a time to invest in the process of seeking out and validating effective proposals for reform, the gradualist strategy of pilot studies seems to hold promise for the future.

Within the framework of these assumptions, then, we shall attempt to work out a concept of pilot studies and of factors that enter into judgments about their adequacy. To achieve this, we must confront questions that deal with the resources to be assigned to pilot studies, criteria for selecting proposals for first-round pilot support, and general exploration of the responsibilities and obligations that accrue to the funding agency and to the recipients of those funds. A backlog of experience with the development and operation pilot studies will no doubt reveal the inadequacies of this first effort to raise questions and propose courses of action.

School Reform as Inquiry Rather than Assertion

The practice of education has traditionally been constructed upon educators' views of the nature of man--what he should be, and what he is capable of becoming. To some extent, scientific knowledge about children and adolescents has been translated crudely into approaches to education. Basically, educational practice--the conduct of schooling--has been a highly empirical process. As such, its practitioners have relied more heavily upon knowledge generated at the level of a craft than that which stems from the theories, laws, and even the data of the disciplines that hypothetically undergird the practice of education; that is, the social sciences and philosophy. Formulations about how education should be conducted are thus more likely to involve assertions based upon past personal experience with the institution of schooling than they are to reflect the tentative, hypothetical mode of inquiry represented by the behavioral sciences. We would suggest that an attitude of inquiry is fundamental to the successful conduct of a pilot strategy of school reform.

Campbell1 constructs an argument close to what we intend in his discussion of "trapped" versus "experimental" administrators. The trapped administrator is one who places himself in a position where his tenure in office (or at least the confidence of his constituency) hinges upon successful outcomes emanating from the program of reforms that he has advocated. If the position of a leader or administrator depends upon the success of the particular reform he has advocated, he cannot afford to call for open and thorough evaluation processes, especially if there is any likelihood that such an evaluation would reflect failure of his programs.

On the other hand, the "experimental administrator" begins with an inquiry attitude. "Program A will be tried for x period of time. If at the end of that time, it has not proved itself, we will shift to Program B." Thus, Campbell suggests, the emphasis can be placed upon approaches to the solution of problems rather than targeting administrators as villains in the event that reform does not live up to

[merged small][ocr errors][merged small]

expectations. Notice,, also, that the experimental stance should free the administrator (or other advocate) from "over-selling" his program in advance, as is likely to happen when alternatives between programs are settled on the basis of who has the greatest net quantity of control.

There is an important corollary to be considered in connection with the discussion of personal advocacy of programs versus inquiry models of examining alternatives. If the decisionmaker starts from the position that the program he recommends is "good" and will produce "benefits" for the population with which he is concerned, his tendency will be to extend the program to as large a proportion of the population as his resources can be made to accommodate. There are several difficulties attendant upon such a practice. First, large amounts of money may be wasted if the outcomes are unsuccessful, or harmful-though the latter is rarely a serious hazard in educational practice. Second, stretching the resources to make it possible for many agencies or individuals to participate can doom the entire program to failure by spreading resources too thinly. Finally, since one was confident that the program would succeed, evaluations tend to be sketchy or nonexistent, and in the end, the program's initiator is in a sorry position. He has a filing program on which large amounts of money, interest, and publicity have been lavished. He may have few or no good reasons for the failure of the program. We must recognize, however, that if the program generally succeeds, the decisionmaker has delivered important benefits to a vast number of people in a short period of time. For the entrepreneurial decisionmaker who is inclined to have great confidence in his own judgment and ability, the chance to deliver effectively, perhaps even at the national level, is extremely attractive. The inquiry-oriented model of pilot studies that we recommend is slower, and much less glamorous, although in the end the payoffs for society can be many times larger.

Results of Large-Scale Efforts to Change Schooling2

The RAND analysis of large-scale intervention programs undertaken has been restricted largely to two of the major efforts to provide compensatory education for children from minority groups, Title I of the Elementary and Secondary Education Act of 1965, and Project Head Start. Although the weight of the evidence reviewed by RAND points to disappointing outcomes of these two national programs, we are aware that the issues are extremely complex, and that simple generalizations about the value of either of the programs would be misleading. For

2This discussion is based upon the recent report, How Effective

Is Schooling?, prepared for the President's Commission on School Finance, by RAND Corporation, March, 1972. Page references in the text are keyed to quotations from that document.

« AnteriorContinuar »