The Delphi Technique: Making Sense of Consensus

A peer- check overed electronic diary. Copyright is retained by the first or sole author, who grants right of first issuing to the Practical Assessment, look for & Evaluation. Permission is granted to distribute this article for nonprofit, educational purposes if it is copied in its entirety and the journal is credited. Volume 12, Number 10, August 2007 ISSN 1531-7714 The Delphi proficiency Making Sense Of Consensus Chia-Chien Hsu, The Ohio State University & Brian A. Sandford, okay State University The Delphi proficiency is a broadly subroutined and accepted method for take ining info from respondents at heart their domain of expertise.The technique is designed as a sepa order communication act upon which aims to achieve a convergence of smell on a proper(postnominal) real-world ignore. The Delphi suffice has been calld in various fields of need such(prenominal)(prenominal) as broadcast planning, needs assessment, policy determination, and resource utilization to develop a full guide of alternatives, explore or expose underlying assumptions, as well as correlate judgments on a topic spanning a wide range of disciplines. The Delphi technique is well suited as a method for consensus-building by using a series of questionnaires delivered using multiple loop-the-loops to collect data from a dialog box of removeed subjects. instance pickax, era frames for conducting and completing a guinea pig, the possibility of low response rates, and unintentionally guiding feedback from the respondent group atomic emergence 18 beas which should be considered when designing and implementing a Delphi study. The Delphi technique, mainly developed by Dalkey and Helmer (1963) at the Rand Corporation in the 1950s, is a widely used and accepted method for achieving convergence of opinion concerning real-world association solicited from experts within trus twainrthy topic areas.Predicated on the rule that, two heads are disclose than one, or n heads a re better than one (Dalkey, 1972, p. 15), the Delphi technique is designed as a group communication act that aims at conducting expand examinations and discussions of a specific issue for the purpose of goal setting, policy investigation, or predicting the occurrence of future events (Ulschak, 1983 Turoff & Hiltz, 1996 Ludwig, 1997). vernacular surveys try to identify what is, whereas the Delphi technique attempts to address what could/should be (Miller, 2006).In the literature, Delphi has been applied in various fields such as program planning, needs assessment, policy determination, and resource utilization. Delbecq, Van de Ven, and Gustafson (1975) specifically indicate that the Delphi technique green goddess be used for achieving the following objectives 1. To determine or develop a range of possible program alternatives 2. To explore or expose underlying assumptions or instruction leading to different judgments 3. To seek give away information which whitethorn generate a consensus on the part of the respondent group 4.To correlate cognizant judgments on a topic spanning a wide range of disciplines, and 5. To educate the respondent group as to the diverse and interrelated aspects of the topic (p. 11). CHARACTERISTICS OF THE DELPHI TECHNIQUE The Delphi technique is well suited as a inwardness and method for consensus-building by using a series of questionnaires to collect data from a panel of selected subjects (Dalkey & Helmer, 1963 Dalkey, 1969 Linstone & Turoff, 1975 Lindeman, 1981 Martino, 1983 progeny & Jamieson, 2001).Delphi, in contrast to other data gathering and analysis techniques, employs multiple iterations designed to Practical Assessment, seek & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi technique develop a consensus of opinion concerning a specific topic. Ludwig (1994) indicates Iterations refer to the feedback mould. The touch was viewed as a series of oscillations in each round every participant worked through a question naire which was returned to the researcher who collected, edited, and returned to every participant a conjurement of the position of the whole group and the participants own position.A summation of comments made each participant aware of the range of opinions and the reasons underlying those opinions (p. 55). More specifically, the feedback cover allows and encourages the selected Delphi participants to reassess their initial judgments rough the information provided in previous iterations. Thus, in a Delphi study, the settlements of previous iterations regarding specific disputations and/or items tin groundwork change or be modified by single panel members in later iterations ground on their ability to survey and assess the comments and feedback provided by the other Delphi panelists.Other nonable characteristics entire with using the Delphi technique are the ability to provide anonymity to respondents, a controlled feedback process, and the suitability of a build of st atistical analysis techniques to interpret the data (Dalkey, 1972 Ludlow, 1975 Douglas, 1983). These characteristics are designed to offset the shortcomings of conventional means of pooling opinions obtained from a group interaction (i. e. , influences of dominant individuals, ruffle, and group pressure for conformity) (Dalkey, 1972). mavin of the primary characteristics and advantages of the Delphi process is subject anonymity which dope reduce the effects of dominant individuals which often is a concern when using group-based processes used to collect and synthesise information (Dalkey, 1972). Additionally, the issue of confidentiality is facilitated by geographic dispersion of the subjects as well as the use of electronic communication such as e-mail to solicit and exchange information.As such, certain downsides associated with group dynamics such as manipulation or coercion to conform or adopt a certain bandstand can be minimized (Helmer & Rescher, 1959 Oh, 1974 Adams, 2001). Controlled feedback in the Delphi process is designed to reduce the effect of noise. Based upon Dalkey (1972), noise is that communication which occurs in a group process which both distorts the data and deals with group and/or individual interests rather than focusing on problem solving.As a result, the information developed from this kind of communication in the main consists of bias not related to the purposes of the study. Basically, the controlled feedback process consists of a well organized summary of the prior iteration intentionally distributed to the subjects which allows each participant an opportunity to generate additional insights and more(prenominal) thoroughly clarify 2 the information developed by previous iterations.Through the operation of multiple iterations, subjects are expected to become more problem-solving oriented, to offer their opinions more insightfully, and to minimize the effects of noise. Finally, the ability to use statistical analysis techniques is a form which further reduces the potential of group pressure for conformity (Dalkey, 1972). More specifically, statistical analysis can check over that opinions generated by each subject of a Delphi study are well represented in the ut near iteration because, at the end of the mould at that place may still be a probative spread in individual opinions Dalkey, 1972, p. 21). That is, each subject would nurse no pressure, either real or perceived, to conform to another participants responses that may originate from obedience to social norms, customs, organizational culture, or standing within a profession. The tools of statistical analysis allow for an objective and impartial analysis and summarization of the collected data. THE DELPHI PROCESS Theoretically, the Delphi process can be continuously iterated until consensus is determined to have been achieved.However, Cyphert and Gant (1971), Brooks (1979), Ludwig (1994, 1997), and Custer, Scarcella, and Stewart (1999) commit o ut that 3 iterations are often sufficient to collect the needed information and to reach a consensus in most cases. The following discussion, however, provides guidelines for up to quatern iterations in order to assist those who decide to use the Delphi process as a data collection technique when it is determined that additional iterations beyond three are needed or valuable. Round 1 In the first round, the Delphi process traditionally begins with an unrestricted questionnaire.The open-ended questionnaire serves as the cornerstone of soliciting specific information about a content area from the Delphi subjects (Custer, Scarcella, & Stewart, 1999). by and by receiving subjects responses, investigators need to convert the collected information into a well-structured questionnaire. This questionnaire is used as the survey legal instrument for the warrant round of data collection. It should be noted that it is both an acceptable and a common modification of the Delphi process form at to use a structured questionnaire in Round 1 that is based upon an extensive reappraisal of the literature.Kerlinger (1973) noted that the use of a modified Delphi process is fascinate if basic information concerning the target issue is available and usable. Round 2 In the second round, each Delphi participant receives a second questionnaire and is asked to review the items summarized by the investigators based on the information provided in the first round. Accordingly, Delphi panelists may be required to rate or rank-order Practical Assessment, research & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi Technique items to establish preliminary priorities among items.As a result of round two, areas of disagreement and agreement are identified (Ludwig, 1994, p. 54-55). In more or less cases, Delphi panelists are asked to severalize the rationale concerning rating priorities among items (Jacobs, 1996). In this round, consensus begins forming and the actual outcomes can be pre sented among the participants responses (Jacobs, 1996). 3 Rescher (1959), Klee (1972), and Oh (1974) concur that choosing individuals who are simply internal concerning the target issue is not sufficient nor recommended.Considering the necessity of selecting the most qualified individuals, Delbecq, Van de Ven, and Gustafson (1975) specifically state that three groups of people are well qualified to be subjects of a Delphi study. The authors recommend (1) the top perplexity finality makers who will utilize the outcomes of the Delphi study (2) the professional staff members together with their support team and (3) the respondents to the Delphi questionnaire whose judgments are being sought (p. 85). Delphi subjects should be highly trained and competent within the specialized area of association related to the target issue.Investigators need to closely examine and seriously consider the qualifications of Delphi subjects. Oh (1974) indicates that choosing appropriate subjects is mo re often than not based on the judgment and discretion of the principal investigators. Jones and Twiss (1978) state that the principal investigators of a Delphi study should identify and select the most appropriate individuals through a nomination process. Ludwig (1994) overly states that, solicitation of nominations of well-known and respected individuals from the members within the target groups of experts was recommended (p. 2). Generally, the pool of selecting possible Delphi subjects is likely to use positional leaders (Kaplan, 1971 Ludwig, 1994), to follow a review of authors of nationalations in the literature (Meyer, 1992 Miller, 2001), and/or to make contacts with those who have firsthand relationships with a particular issue (Jones, 1975 Anderson & Schneider, 1993). The last mentioned basically consists of individuals who are primary stakeholders with various interests related to the target issue or research effort.Concerning the appropriate number of subjects to involv e in a Delphi study, Delbecq, Van de Ven, and Gustafson (1975) recommend that researchers should use the minimally sufficient number of subjects and should seek to verify the results through follow-up explorations. Ludwig (1994) notes that the number of experts used in a Delphi study is generally determined by the number required to constitute a representative pooling of judgments and the information processing readiness of the research team (p. 52). However, what constitutes an optimal number of subjects in a Delphi study never reaches a consensus in the literature.Delbecq, Van de Ven, and Gustafson (1975) suggest that ten to fifteen subjects could be sufficient if the background of the Delphi subjects is homogeneous. In contrast, if various persona groups are involved in a Delphi study, more subjects are anticipated to Round 3 In the third round, each Delphi panelist receives a questionnaire that includes the items and ratings summarized by the investigators in the previous roun d and are asked to revise his/her judgments or to specify the reasons for stay outside the consensus (Pfeiffer, 1968, p. 52). This round gives Delphi panelists an opportunity to make further clarifications of both the information and their judgments of the relative importance of the items. However, compared to the previous round, only a slight increase in the degree of consensus can be expected (Weaver, 1971 Dalkey & Rourke, 1972 Anglin, 1991 Jacobs, 1996). Round 4 In the fourth and often final round, the list of remaining items, their ratings, minority opinions, and items achieving consensus are distributed to the panelists.This round provides a final opportunity for participants to revise their judgments. It should be remembered that the number of Delphi iterations depends generally on the degree of consensus sought by the investigators and can vary from three to five (Delbecq, Van de Ven, Gustafson, 1975 Ludwig, 1994). Subject Selection Regarding the selection of subjects for a Delphi study, choosing the appropriate subjects is the most important step in the entire process because it directly relates to the prime(a) of the results generated (Judd, 1972 Taylor & Judd, 1989 Jacobs, 1996).Since the Delphi technique focuses on eliciting expert opinions over a short accomplishment of period, the selection of Delphi subjects is generally dependent upon the disciplinary areas of expertise required by the specific issue. Regarding any set standards of selecting Delphi subjects, there is, in fact, no exact criterion currently listed in the literature concerning the selection of Delphi participants. That is, throughout the Delphi literature, the explanation of Delphi subjects has remained ambiguous (Kaplan, 1971, p. 24).Regarding the criteria used to guide the selection of Delphi subjects, individuals are considered eligible to be invited to participate in a Delphi study if they have somewhat related backgrounds and experiences concerning the target issue, are capable of contributing helpful inputs, and are willing to revise their initial or previous judgments for the purpose of reaching or attaining consensus (Pill, 1971 Oh, 1974). Helmer and Practical Assessment, enquiry & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi Technique be needed.Witkin and Altschuld (1995) note that the approximate size of a Delphi panel is generally under 50, but more have been employed. Ludwig (1997) documents that, the majority of Delphi studies have used between 15 and 20 respondents (p. 2). In sum, the size of Delphi subjects is variable (Delbecq, Van de Ven, & Gustafson, 1975). If the sample size of a Delphi study is too small, these subjects may not be considered as having provided a representative pooling of judgments regarding the target issue.If the sample size is too sizable, the drawbacks native within the Delphi technique such as potentially low response rates and the obligation of large evades of time by the respondents and the researcher(s) can be the result. Time Requirements Conducting a Delphi study can be time-consuming. Specifically, when the instrument of a Delphi study consists of a large number of statements, subjects will need to dedicate large blocks of time to complete the questionnaires.Delbecq, Van de Ven, and Gustafson (1975), Ulschak (1983), and Ludwig, (1994) recommend that a minimum of 45 days for the administration of a Delphi study is necessary. With regard to the time management between iterations, Delbecq et al. (1975) note that giving two weeks for Delphi subjects to respond to each round is encouraged. Ludwig (1994) indicates, a drawback to Delphi was that the questionnaire method may slow the process greatly as several days or weeks may pass between rounds (p. 54).More specifically, since developing the instrument, collecting the data, and administering the questionnaire are interconnected between iterations, ensuring Delphi subjects respond to the investigators on time does in many ways either promote or prohibit the ability of the investigators in analyzing the data, developing a new instrument based upon the prior responses, and distributing subsequent questionnaires in a well timed(p) fashion. These are challenging aspects of conducting a Delphi study and do require proper planning and management. The use and preponderance of electronic technologies (i. e. e-mail, teleconferencing, etc. ) may facilitate those who are interested in using the Delphi technique. Witkin and Altschuld (1995) note that electronic technology provides an opportunity for individuals to more easily employ the Delphi process by taking advantages of, (1) the storage, processing, and speed of contagion capabilities of computers (2) the maintenance of respondent anonymity, and (3) the potential for rapid feedback (p. 204). Data Analysis Regarding data analysis, decision rules must be established to assemble and organize the judgments and insights provided by Delphi subjects.However, the kind and ty pe 4 of criteria to use to both localize and determine consensus in a Delphi study is subject to interpretation. Basically, consensus on a topic can be decided if a certain percentage of the votes falls within a prescribed range (Miller, 2006). One criterion recommends that consensus is achieved by having 80 percent of subjects votes fall within two categories on a seven-point graduated table (Ulschak, 1983). Green (1982) suggests that at least 70 percent of Delphi subjects need to rate three or high on a four point Likert-type scale and the median has to be at 3. 5 or higher. Scheibe, Skutsch, and Schofer (1975) reveal that the use of percentage measures is inadequate. They suggest that a more reliable alternative is to measure the stability of subjects responses in successive iterations. In the Delphi process, data analysis can involve both qualitative and valued data. Investigators need to deal with qualitative data if classic Delphi studies, which use open-ended questions t o solicit subjects opinions, are conducted in the initial iteration.Subsequent iterations are to identify and hopefully achieve the desired level of consensus as well as any changes of judgments among panelists. The major statistics used in Delphi studies are measures of central tendency (means, median, and mode) and level of dispersion (standard aberration and inter-quartile range) in order to present information concerning the collective judgments of respondents (Hasson, Keeney, & McKenna, 2000). Generally, the uses of median and mode are favored. However, in some cases, as manifested by Murray and Jarman (1987), the mean is also workable.Witkin (1984) questions the appropriateness of using the mean to measure the subjects responses if scales used in Delphi studies are not delineated at equal intervals. In the literature, the use of median score, based on Likert-type scale, is strongly favored (Hill & Fowles, 1975 Eckman, 1983 Jacobs, 1996). As Jacobs (1996) states, considering t he anticipated consensus of opinion and the skewed expectation of responses as they were compiled, the median would inherently appear best suited to reflect the resultant convergence of opinion (p. 57).The use of mode is also suitable when reporting data in the Delphi process. Ludwig (1994) specifically addressed that the Delphi process has a tendency to reach convergence, and though this was usually to a single point, there was the possibility of polarization or clustering of the results nigh two or more points. In these instances, the mean or median could be misleading (p. 57). CONSIDERING DELPHI SHORTCOMINGS AND WEAKNESSES capableness of Low Response Rates Due to the multiple feedback processes inherent and integral to the concept and use of the Delphi process, thePractical Assessment, enquiry & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi Technique potential exists for low response rates and striving to maintain square-shouldered feedback can be a challenge. In the Delph i technique, poor response rate is magnified fourfold because a maximum of four surveys may be sent to the same panelists (Witkin & Altschuld, 1995, p. 196). If a certain portion of the subjects discontinue their responses during various stages of the Delphi process, the quality of information obtained could be discounted or at least critically scrutinized.As such, Ludwig (1994) specifically addresses subject motivation as the key to the successful capital punishment of a Delphi study and investigators need to play an active role in this area to help ensure as high a response rate as possible. use of Large Blocks of Time The Delphi technique can also be time-consuming and laborious. Unlike other data collection techniques such as the telephone survey and the face-to-face administration, which can be simultaneously conducted by a group of people and can be completed in a short period of time if the sample size is small, the Delphi technique is terative and sequential. As a result, the necessity of taking large block of time to successively complete a Delphi process is inescapable. Ludwig (1994) indicates that, a drawback to Delphi was that the questionnaire method may slow the process greatly as several days or weeks may pass between rounds (p. 54). Optimally speaking, the iteration characteristics of the Delphi process provide the opportunities for investigators and subjects to improve the accuracy of the results.In contrast, the same characteristic also increases the workload of investigators and the amount of time needed to successfully complete the data collection process (Cunliffe, 2002). Potential of Molding Opinions The iteration characteristics of the Delphi technique can potentially enable investigators to mold opinions (Altschuld, 2003). An experiment, conducted by Scheibe, Skutsch, and Schofer (1975), indicated that Delphi subjects would rate their responses differently after receiving a misshapen feedback.Dalkey and Helmer (1963) also noted that, some leading by the experimenters inevitably resulted from the selection of the information supplied (p. 467). Moreover, Cyphert and Gant (1971) illustrated that a statement in their study was initially rated below average. However, Delphi subjects rated the statement above average after receiving false feedback. Therefore, Cyphert and Gant (1971) conclude that the Delphi technique could, be used to mold opinion as well as to collect data (p. 273).Indeed, deadly pressure to conform with group ratings was one of the major drawbacks in the Delphi technique (Witkin & Altschuld, 1995, p. 188). Delphi investigators need to be cognizant, exercise caution, and implement the proper safeguards in dealing with this issue. 5 Potential of Identifying General Statements vs. Specific Topic connect Information An assumption concerning Delphi participants is that they are equivalent in knowledge and experience (Altschuld & Thomas, 1991). However, this assumption might not be justified.More speci fically, the expertise of Delphi panelists could be unevenly distributed, especially in the field of high technology (Marchant, 1988 Altschuld & Thomas, 1991). almost panelists may have much more in-depth knowledge of certain topics, whereas other panelists are more well-read about different topics (Altschuld & Thomas, 1991, p. 187). Therefore, subjects who have less in-depth knowledge of certain topics are unable to specify the most important statements which have been identified by those subjects who possess in-depth knowledge concerning the target issue.The outcomes of a Delphi study could be the results of identifying a series of general statements rather than an in-depth exposition of the topic (Altschuld & Thomas, 1991). SUMMARY The Delphi technique provides those involved or interested in engaging in research, evaluation, fact-finding, issue exploration, or discovering what is actually known or not known about a specific topic a flexible and adaptable tool to gather and ana lyze the needed data. Subject selection and the time frames for conducting and completing a Delphi study are two areas which should be considered carefully prior to initiating the study.The additional precautions concerning low response rates, unintentionally guiding feedback, and surveying panelists about their limited knowledge of the topic rather than soliciting their expert judgments should also be built into the design and implementation of the study. The Delphi technique has and will continue to be an important data collection methodology with a wide variety of applications and uses for people who want to gather information from those who are immersed and imbedded in the topic of interest and can provide real-time and real-world knowledge. REFERENCES Adams, S. J. (2001).Projecting the next decade in safety management A Delphi technique study. professional person Safety, 46 (10), 26-29. Altschuld, J. W. (2003). Delphi technique. Lecture, Applied evaluation design. The Ohio Stat e University. Altschuld, J. W. , & Thomas, P. M. (1991). Considerations in the application of a modified scree render for Delphi survey data. Evaluation Review, 15 (2), 179-188. Practical Assessment, Research & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi Technique Anderson, D. H. , & Schneider, I. E. (1993). Using the Delphi process to identify significant recreation research-based innovations.Journal of Park and amusement Administration, 11 (1), 25-36. Anglin, G. L. (1991). Instructional technology past, present and future. Englewood, CO Libraries Unlimited Inc. Brooks, K. W. (1979). Delphi technique Expanding applications. North Central Association Quarterly, 54 (3), 377-385. Cunliffe, S. (2002). Forecasting risks in the tourism industry using the Delphi technique. Tourism, 50 (1), 31-41. Custer, R. L. , Scarcella, J. A. , & Stewart, B. R. (1999). The modified Delphi technique A rotational modification. Journal of Vocational and Technical Education, 15 (2), 1-10.Cyphert, F . R. , & Gant, W. L. (1971). The Delphi technique A case study. Phi Delta Kappan, 52, 272-273. Dalkey, N. C. (1969). An experimental study of group opinion. Futures, 1 (5), 408-426. Dalkey, N. C. (1972). The Delphi method An experimental study of group opinion. In N. C. Dalkey, D. L. Rourke, R. Lewis, & D. Snyder (Eds. ). Studies in the quality of conduct Delphi and decision-making (pp. 13-54). Lexington, MA Lexington Books. Dalkey, N. C. , & Helmer, O. (1963). An experimental application of the Delphi method to the use of experts. Management Science, 9 (3), 458-467.Dalkey, N. C. , & Rourke, D. L. (1972). Experimental assessment of Delphi procedures with group value judgments. In N. C. Dalkey, D. L. Rourke, R. Lewis, & D. Snyder (Eds. ). Studies in the quality of life Delphi and decision-making (pp. 55-83). Lexington, MA Lexington Books. Delbecq, A. L. , Van de Ven, A. H. , & Gustafson, D. H. (1975). Group techniques for program planning. Glenview, IL Scott, Foresman, and Co. Dougl as, D. C. (1983). A comparative study of the effectiveness of decision making processes which utilize the Delphi and leaderless group methodologies.unpublished doctoral dissertation, The Ohio State University, Columbus. Eckman, C. A. (1983). Development of an instrument to evaluate intercollegiate gymnastic coaches A modified Delphi study. Unpublished doctoral dissertation, West Virginia University, Morgantown. Green, P. J. (1982, March). The content of a college-level outdoor leadership course. topic presented at the Conference of the Northwest District Association for the American 6 Alliance for Health, Physical Education, Recreation, and Dance, Spokane, WA. Hasson, F. , Keeney, S. , & McKenna, H. (2000). Research guidelines for the Delphi survey technique.Journal of Advanced Nursing, 32 (4), 1008-1015. Helmer, O. , & Rescher, N. (1959). On the epistemology of the inexact science. Management Science, 6, 25-53. Hill, K. Q. , & Fowles, J. (1975). The methodological worth of the De lphi forecasting technique. technological Forecasting and cordial Change, 7, 179-192. Jacobs, J. M. (1996). Essential assessment criteria for physical education teacher education programs A Delphi study. Unpublished doctoral dissertation, West Virginia University, Morgantown. Jones, C. G. (1975). A Delphi evaluation of agreement between organizations. In H. A. Linstone, & M.Turoff (Eds. ). The Delphi method Techniques and applications (pp. 160-167). Reading, MA Addison-Wesley produce Company. Jones, H. , & Twiss, B. C. (1978). Forecasting technology for planning decision. London, UK Macmillan Press Ltd. Judd, R. C. (1972). Use of Delphi methods in higher education. Technological Forecasting and Social Change, 4 (2), 173-186. Kaplan, L. M. (1971). The use of the Delphi method in organizational communication A case study. Unpublished victors thesis, The Ohio State University, Columbus. Kerlinger, F. N. (1973). Foundations of behavioral research. impudent York Holt, Rinehart, and W inston, Inc. Klee, A. J. (1972). The utilization of expert opinion in decision-making. AICHE Journal, 18 (6), 1107-1115. Lindeman, C. A. (1981). Priorities within the wellness care system A Delphi survey. Kansas City, MO American Nurses Association. Linstone, H. A. , & Turoff, M. (1975). Introduction. In H. A. Linstone, & M. Turoff (Eds. ). The Delphi method Techniques and applications (pp. 3-12). Reading, MA Addison-Wesley Publishing Company. Ludlow, J. (1975). Delphi inquiries and knowledge utilization. In H. A. Linstone, & M. Turoff (Eds. ).The Delphi method Techniques and applications (pp. 102-123). Reading, MA Addison-Wesley Publishing Company. Ludwig, B. G. (1994). Internationalizing Extension An exploration of the characteristics evident in a state university Extension system that achieves internationalization. Unpublished doctoral dissertation, The Ohio State University, Columbus. Practical Assessment, Research & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi Technique Lud wig, B. (1997). Predicting the future Have you considered using the Delphi methodology? Journal of Extension, 35 (5), 1-4. Retrieved November 6, 2005 from http//www. oe. org/joe/1997october/tt2. html Marchant, E. W. (1988). Methodological problems associated with the use of the Delphi technique Some comments. Fire Technology, 24 (1), 59-62. Martino, J. P. (1983). Technological forecasting for decision making. New York North-Holland. Meyer, J. H. (1992). Rethinking the outlook of colleges whose roots have been in agriculture. Davis, CA University of California. Miller, G. (2001). The development of indicators for sustainable tourism Results of a Delphi survey of tourism researchers. Tourism Management, 22, 351-362. Miller, L. E. (2006, October).Determining what could/should be The Delphi technique and its application. Paper presented at the meeting of the 2006 annual meeting of the Mid-Western Educational Research Association, Columbus, Ohio. Murray, W. F. , & Jarman, B. O. (1987). P redicting future trends in adult fitness using the Delphi approach. Research Quarterly for Exercise and Sport, 58 (2), 124-131. Oh, K. H. (1974). Forecasting through ranked Delphi. Unpublished doctoral dissertation, The Ohio State University, Columbus. Pill, J. (1971). The Delphi method Substance, context, a critique and an annotated bibliography.Socio-Economic Planning Science, 5, 57-71. 7 Scheibe, M. , Skutsch, M. , & Schofer, J. (1975). Experiments in Delphi methodology. In H. A. Linstone, & M. Turoff (Eds. ). The Delphi method Techniques and applications (pp. 262-287). Reading, MA Addison-Wesley Publishing Company. Taylor, R. E. , & Judd, L. L. (1989). Delphi method applied to tourism. In S. Witt, & L. Moutinho, (Eds. ). Tourism marketing and management handbook. New York Prentice Hall. Turoff, M. , & Hiltz, S. R. (1996). Computer based Delphi process. In M. Adler, & E. Ziglio (Eds. ).Gazing into the oracle The Delphi method and its application to social policy and public healt h (pp. 56-88). London, UK Jessica Kingsley Publishers. Ulschak, F. L. (1983). Human resource development The theory and practice of need assessment. Reston, VA Reston Publishing Company, Inc. Weaver, W. T. (1971). The Delphi forecasting method. Phi Delta Kappan, 52 (5), 267-273. Witkin, B. R. (1984). Assessing needs in educational and social programs. San Francisco, CA Jossey-Bass Publishers. Witkin, B. R. , & Altschuld, J. W. (1995). Planning and conducting needs assessment A practical(a) guide.Thousand Oaks, CA Sage Publications, Inc. Young, S. J. , & Jamieson, L. M. (2001). Delivery methodology of the Delphi A comparison of two approaches. Journal of Park and Recreation Administration, 19 (1), 42-58. Citation Hsu, Chia-Chien & Sandford, Brian A. (2007). The Delphi Technique Making Sense of Consensus. Practical Assessment Research & Evaluation, 12(10). on hand(predicate) online http//pareonline. profit/getvn. asp? v=12&n=10 Editors Note Another paper on the Delphi Technique that appeared in Practical Assessment Research & Evaluation is Yousuf, Muhammad Imran (2007).Using Experts Opinions through Delphi Technique. Practical Assessment Research & Evaluation, 12(4). Available online http//pareonline. net/getvn. asp? v=12&n=4 . Authors Chia-Chien Hsu Post-doctoral Studies The Ohio State University 393 Schrock Road Worthington, OH 43085 Practical Assessment, Research & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi Technique Tel (614) 885-0763 E-mail hsu. 127 at osu. edu Brian A. Sandford Assistant Professor 214 Willard Hall Occupational Education Oklahoma State University Stillwater, Oklahoma 74074 405-744-3461 brian. sandford at okstate. edu 8