Reference and Instruction Publications and Papers Reference and Instruction 4-2005 Web Citation by Graduate Students: A Comparison of Print and Electronic Theses Jeffrey D. Kushkowski Iowa State University, kushkows@iastate.edu Follow this and additional works at: http://lib.dr.iastate.edu/refinst_pubs Part of the Library and Information Science Commons The complete bibliographic information for this item can be found at http://lib.dr.iastate.edu/ refinst_pubs/60. For information on how to cite this item, please visit http://lib.dr.iastate.edu/ howtocite.html. This Article is brought to you for free and open access by the Reference and Instruction at Iowa State University Digital Repository. It has been accepted for inclusion in Reference and Instruction Publications and Papers by an authorized administrator of Iowa State University Digital Repository. For more information, please contact digirep@iastate.edu.
Web Citation by Graduate Students: A Comparison of Print and Electronic Theses Abstract This article reports on the Web citation behavior of print and electronic thesis authors at Iowa State and Virginia Tech from 1997 to 2003. Citations from print theses were compared with those submitted as an electronic thesis or dissertation (ETD). This study suggests that students who are required to publish their theses digitally exhibit citation behavior that is no different from students who produce their theses in print. Web citations accounted for 2.2 percent of citations in print theses and 5.4 percent of citations in ETDs. Persistence of Web citations was uniformly poor. The implications for library services and future research directions are discussed. Disciplines Library and Information Science Comments This article is from portal: Libraries and the Academy 5 (2005): 259 276, doi:10.1353/pla.2005.0028. Posted with permission. This article is available at Iowa State University Digital Repository: http://lib.dr.iastate.edu/refinst_pubs/60
b t t n b r d t t d nt : p r n f Pr nt nd l tr n Th J ffr D. h portal: Libraries and the Academy, Volume 5, Number 2, April 2005, pp. 259-276 (Article) P bl h d b J hn H p n n v r t Pr DOI: 10.1353/pla.2005.0028 For additional information about this article http://muse.jhu.edu/journals/pla/summary/v005/5.2kushkowski.html Access provided by Iowa State University (3 Oct 2015 06:07 GMT)
Jeffrey D. Kushkowski 259 Web Citation by Graduate Students: A Comparison of Print and Electronic Theses Jeffrey D. Kushkowski abstract: This article reports on the Web citation behavior of print and electronic thesis authors at Iowa State and Virginia Tech from 1997 to 2003. Citations from print theses were compared with those submitted as an electronic thesis or dissertation (ETD). This study suggests that students who are required to publish their theses digitally exhibit citation behavior that is no different from students who produce their theses in print. Web citations accounted for 2.2 percent of citations in print theses and 5.4 percent of citations in ETDs. Persistence of Web citations was uniformly poor. The implications for library services and future research directions are discussed. Introduction The Internet s ubiquity and the ease of finding information on it have changed the way that scholarship is documented. Papers at the undergraduate and graduate levels and articles in peer-reviewed scholarly literature now routinely contain citations to Web resources. Changes are also occurring in the publication and distribution of journals as publishers move from print to electronic formats. This move to electronic formats is having an impact on academe as institutions either enable or require the submission of an electronic thesis or dissertation (ETD). One question that the move to electronic formats raises is whether citation patterns of ETD and print thesis authors are different. Do citation patterns of graduate students vary based on the format of their theses or are they format neutral? This article investigates the Web citation behavior of print and electronic thesis authors. It examines the rates of Web citation in economics theses and dissertations between 1997 and 2003 at Virginia Polytechnic Institute and State University (Virginia Tech) and Iowa State University (Iowa State). The following four questions are addressed: portal: Libraries and the Academy, Vol. 5, No. 2 (2005), pp. 259 276. Copyright 2005 by The Johns Hopkins University Press, Baltimore, MD 21218.
260 Web Citation by Graduate Students: A Comparison of Print and Electronic Theses 1. What are the characteristics of the Web citations? 2. Are there any identifiable trends in Web citations? 3. What are the implications for libraries? 4. What additional research does this study suggest? Literature Review Dissemination of ETDs Virginia Tech, which began requiring ETDs of its students in January 1997, has played a key role in their development. 1 Larry Thompson discusses Virginia Tech s ETD program and its impact on traditional access to theses and dissertations. 2 The Virginia Tech Library Web site maintains extensive links to available ETDs, technical information for ETD authors, and links to other ETD resources. 3 The National Digital Library of Theses and Dissertations, founded in 1997 to coordinate ETD efforts among different institutions, quickly changed its name to the Networked Digital Library of Theses and Dissertations (NDLTD) as international interest in ETDs increased. 4 As of January 2005, 189 universities and 28 institutions are members of the NDLTD, with over 60 of them requiring ETDs of their students. 5 Edward Fox published an update on NDLTD activities in 2000. 6 There are a number of articles detailing ETD efforts in different countries including Australia, the United Kingdom, South Africa, Korea, and Brazil. 7 Other ETD initiatives, including diplomica.com and dissertation.com, are part of the NDLTD universe, while the Konstanzer Online-Publikations-System, Theological Research Exchange Network, and ProQuest Digital Dissertations are independent initiatives. 8 With all the interest in establishing ETD programs, quite a bit has been written about the production and implications of ETDs. NTLTD maintains an archive of software, technical help, and a bibliography of relevant articles for institutions starting their own ETD projects. 9 Yale Fineman provided a comprehensive overview of ETD history, technology, and methodology. 10 Gail McMillan s 2003 presentation at the American Library Association described the benefits of ETD programs to universities. 11 Hussein Suleman et al. have written two articles about the NTLTD, one focusing on the history of NTLTD and the other focusing on the services it offers to members. 12 Ursula Goldsmith surveyed graduate faculty about their perceptions of ETDs as part of her graduate thesis. 13 The implications for libraries extend beyond using ETDs to disseminate theses to larger constituencies. Thomas Teper and Beth Kraemer argue that libraries need to consider the preservation implications of ETD programs, given that ETDs are entirely electronic documents. 14 Electronic Resource Bibliometrics Citation analysis or other bibliometric studies of electronic resources are rare relative to the corpus of literature on electronic resources. Version 54 of Charles Bailey s Scholarly Electronic Publishing Bibliography lists 108 items under Electronic Serials: Research out of more than 2,150 references in a recent edition of the bibliography. 15 Of those 108, bibliometric studies of electronic serials account for fewer than 20 entries. The majority
Jeffrey D. Kushkowski 261 of articles focus on questions about the provision of access to e-journals, scholars views on e-journals relative to print titles, and the role of promotion and tenure decisions on the decision to publish electronically. Initial bibliometric studies involving electronic resources considered whether e- resources had an impact on citation trends of scholars. Stephen Harter and Yin Zhang concluded that although researchers cited e-journals, they had little measurable impact on citation patterns of library and information science researchers. 16 Yin Zhang s longitudinal study of e-citation by scholars showed that, even though more electronic resources are being cited, they are still much less frequently cited than print materials. 17 A more recent study by Susan Davis Herring provided evidence that electronic citations accounted for 16 percent of the citations in a select set of e-journals. 18 Burnette Green found that mathematics scholars who publish in electronic journals are more highly cited than a control group but that citations for these scholars are to their print publications. 19 Researchers are also looking at the electronic citation habits of students, although the majority of studies involve undergraduate citations. Debbie Malone and Carol Videon examined 219 student bibliographies, finding that only 7 percent cite electronic resources defined as full texts from CD-ROM databases or the Internet. 20 Philip Davis three articles on the electronic citation habits of undergraduates examined the impact of instructor expectations on trends in student citation of Web resources. 21 Paul Jenkins analyzed citations from 116 student bibliographies and found that faculty expectations for use of Web resources had an impact on student citations. 22 In the only study examining graduate citation behavior, Nicholas Okrent examined the print citations of philosophy graduate students. Okrent s study focused on the availability of electronic versions of the print materials not their actual use. He searched the print citations in 12 graduate theses and discovered that 21 percent of citations in these theses were available in electronic formats. He concluded that electronic availability of materials was low because of the heavy use of monographs and anthologies by philosophy graduate students. 23 Limitations of Bibliometric Analysis Bibliometric analysis of Web citations can be used as a means for describing how scholars use Web citations in their work. It is useful for demonstrating the incidence of Web citation and explaining the characteristics of how Web citations are used. It is less helpful in explaining the motivations of authors who cite Web resources. Research investigating the norms and mores that define how scholars act in their respective disciplines may better explain the motivations for using and citing Web resources. Considerable research devoted to information seeking tries to explain how groups find information that meets their specific needs. 24 An example of this methodology is Lisa Covi s examination of the electronic communication practices of doctoral students, in which she found Research investigating the norms and mores that define how scholars act in their respective disciplines may better explain the motivations for using and citing Web resources.
262 Web Citation by Graduate Students: A Comparison of Print and Electronic Theses that the students work practices reinforced existing patterns of work and resource use in their disciplines. 25 Her article investigates how information-seeking behavior is influenced by the organizational structure of disciplinary groups. Two other methodological approaches are important for explaining the organizational influence of information-seeking behavior. Elfreda Chatman s work, termed small-world theory, examines information seeking of such diverse groups as janitors and single mothers. 26 Chatman observed the interpersonal relationships within groups and found that groups tend to seek information within their unique social environment. Brenda Dervin s sense-making methodology contends knowledge is a verb, always an activity, embedded in space and time. 27 Sense-making examines information as a process that when studied from the perspective of the user can provide valuable observations about both the means users employ in looking for information and the mechanisms providers use to convey information. 28 Extensions of Chatman s and Dervin s work may provide further insights into the role of organizational influence on individual information-seeking behavior. Trends and Gaps in the Literature Current trends in the literature include a heavy emphasis on research about Open Access Initiatives (OAI). OAI provides free and open access to electronic content of journals, monographs, and other such materials as sound and video. 29 There are significant economic issues related to providing content through OAI not the least of which is development of pricing models that will allow publishers to recoup costs from lost subscription revenue. Debate about both the mechanics and economic issues related to OAI continues. 30 The focus of bibliometric research has also shifted from the print to electronic arena. The field of Webometrics, the study of Web phenomena pioneered by Tomas Almind and Peter Ingwersen, borrows heavily from bibliometric methods to explore the quantitative attributes of the Web. 31 Exuberance about OAI and Webometrics is crowding out more traditional research topics. Much of the research involves quantitative analysis of linking patterns that purport to show how the Web is contributing to scholarly communication. Researchers are past the point of asking whether Web resources are being used; the focus is shifting to how they are used. As discussed above, further research may result in methods for analyzing Webometric data that take into account the author s motivations for citing a particular resource. At present, the question that is not being asked is why scholars are using Web resources. Research into the behavioral process of citation is an area that needs more study. By examining specific user groups and their motivations for citing or not citing electronic materials, researchers may gain insights into the creation and dissemination of knowledge that better explain both traditional and Web citation patterns. Methodology Sample Selection To examine the differences in Web citation rates between print theses and ETDs, two sets of theses were selected one from Virginia Tech and the other from Iowa State. As
Jeffrey D. Kushkowski 263 stated earlier, Virginia Tech the first institution to require ETDs began requiring them in 1997. Iowa State does not accept ETDs and currently has no plans to do so. 32 Both institutions are land-grant schools with established economics programs that award both master s and doctoral degrees. The institutional characteristics of the two schools are shown in table 1. The sampling frame for this study is all available economics theses and dissertations from Virginia Tech and Iowa State University produced between 1997 and 2002. The starting date coincides with the date that Virginia Tech began requiring ETDs; the ending date is the latest year for which a complete set of theses and dissertations from both institutions was available. Students at Virginia Tech submit their electronic theses or dissertations to a central location. There are four options for making their ETD available: (1) immediate worldwide access, (2) access to Virginia Tech only for a period of up to three years, (3) mixed access (part available worldwide, part available only to Virginia Tech), or (4) delay release because of proprietary information such as information included in patent applications. 33 As of January 2005, the Virginia Tech site included over 6,300 ETDs. 34 ETDs selected for this project included all globally available economics ETDs from Virginia Tech published between 1997 and 2002 a total of 73 titles. An additional 33 titles were identified but not used, because their use was restricted to the Virginia Tech community. The Iowa State sample included 78 theses and dissertations submitted to the Department of Economics between 1997 and 2002. The data for this study include all the citations contained in bibliographies from the selected theses and dissertations at each institution. Thesis and Citation Analysis Theses were classified by subject to validate the similarity between the two institutions using the Journal of Economic Literature (JEL) classification system, the standard subject classification used for economics literature. 35 Descriptors from titles and abstracts were used to classify each thesis. The results of this classification are described in a later section. Citations were analyzed twice, once using all citations and a second time using only Web citations. In both cases, citations were classified using JOBET, a system developed for this project. JOBET is an acronym for Journal, Other, Book, Electronic, and Thesis, which refers to the coding order used at the bottom of bibliography pages to total the citation counts. Table 2 describes the elements of the JOBET classification. Using JOBET allows for quick classification of citations without an unduly long number of categories. Standard categories such as journals, books, theses, and electronic formats are supplemented by several subcategories of other documents. These other subcategories allow citations to be rapidly classified into categories by type of material (e.g., working paper or government document) rather than by a traditional bibliographic description (e.g., book or journal) of the citation. Since the focus of this research is electronic citation behavior, this is a reasonable approach to sorting the citations into categories. Citations to electronic resources were classified with the JOBET typology used to describe print citations and also underwent further analysis. URLs for the electronic
264 Web Citation by Graduate Students: A Comparison of Print and Electronic Theses Table 1 Institutional Comparison Institutional Comparison Iowa State Virginia Tech Faculty (2003 04) 69 48 Graduate Students (2003 04) 119 75 Degrees Granted (1997 2002) M.A./M.S./Ph.D. M.A./M.S./Ph.D. M.A./M.S. 18 52 Ph.D. 60 21 Total Degrees Granted 78 73 Table 2 JOBET Classification System Category Journal Other O 1 O 2 Description Standard serial publications Working papers and conference proceedings Agricultural reports and extension publications,including publications from FAO, World Bank,and government sources O 3 Government documents not included in O 2 Truly other materials including personal correspondence,e-mail, and other O 4 unpublished materials Book Complete monographs Book Chapter Chapters from monographs Electronic Materials cited as being available on the Internet Thesis Theses from academic institutions citations were placed in a separate file, and the validity of URLs was checked using the method described by Steve Lawrence et al. 36 This method involved a three-step verification process: (1) searching for the URL, (2) searching for alternate URLs, and (3) browsing the Web. A successful match for an electronic citation meant either that the URL provided in the citation was correct and included the precise content from the URL or that the exact content mentioned in the citation was found after a search of the Web. When searching the Web, care was taken to ensure that the content included the exact information listed in the citation. In contrast to the recent paper by Mary Casserly
Jeffrey D. Kushkowski 265 and James Bird, searches of the Web did not include a search of the historical Web as found in the Wayback Machine (http://www.archive.org), but similar methods were used to decide whether the content of a Web page matched what the original authors used in their theses or dissertations. 37 The URL and content searching of electronic citations was straightforward, but there were instances when judgments were made about whether the content of a Web site matched the thesis author s electronic citation. In addition to checking the persistence of URLs, data were collected about the domains of electronic resources in order to show the distribution of different domain types. This study does not make the distinction between original and implied domains described by Wallace Koehler, because the number of citations with geographic designation (e.g., country codes such as.ca for Canada or.nl for the Netherlands) was fewer than 10 percent of the total electronic citations. 38 Results Thesis Characteristics Before examining citation content and the trends in electronic citation use at the two institutions, it is important to know something about the theses samples. Table 1 shows that Iowa State and Virginia Tech graduate similar numbers of students. The differences in the distribution of types of degrees granted are a reflection of the size of the Economics Department at each institution. The subject classification of the theses shows that more than one-third of theses from each school, 37 percent at both Iowa State and Virginia Tech, were classified in agricultural and natural resource economics. This is not a surprising result, given that both institutions are land-grant schools with long traditions of supporting agriculturerelated academic programs. Results of the subject analysis of the theses using the JEL classification are shown in table 3. Citation Characteristics The total number of citations at each institution by type of citation is shown in table 3. Iowa State s theses contained a total of 5,117 citations, and Virginia Tech s theses included 3,696 citations. The percentages of each citation type that contributed to total citations are also shown. Journals accounted for the largest percentage of citations at both schools (Iowa State: 59 percent; VA Tech: 40 percent), followed by books (Iowa State: 15 percent; Virginia Tech: 19 percent). With the exception of the category labeled O 2 : Agricultural Reports from Virginia Tech (10 percent), no other category accounted for more than 9 percent of the total citations. Citations to electronic resources totaled 2.2 percent and 5.4 percent of total citations at Iowa State and Virginia Tech, respectively. Citation totals provide an incomplete comparison of how citations are distributed because of the disparity in the numbers of master s and PhD theses at the two institutions. The box and whisker plot in figure 1 displays the number of citations for each level of thesis at Iowa State and Virginia Tech. A box and whisker plot displays the mean number of citations surrounded by 25th and 75th percentiles (the box). The minimum and maximum values (whiskers) are shown as extensions off the box. The table shows that at both institutions PhD students cite more material than master s students.
266 Web Citation by Graduate Students: A Comparison of Print and Electronic Theses Table 3 Thesis and Citation Characteristics Thesis Subject Area: Iowa State Virginia Tech Reported by JEL Subject Heading no. (%) (N=78) (N=73) C: Mathematical and Quantitative Analysis D: Microeconomics E: Macroeconomics and Monetary Policy F: International Economics G: Financial Economics H: Public Economics I: Health, Education and Welfare J: Labor and Demographic Economics L: Industrial Organization Q: Agricultural and Natural Resource Economics R: Urban, Rural and Regional Economics 13 (16.7) 3 (3.8) 4 (5.1) 6 (7.7) 5 (6.4) 4 (5.1) 7 (9.0) 4 (5.1) 2 (2.6) 29 (37.2) 1 (1.3) 7 (9.6) 2 (2.7) 1 (1.4) 3 (4.1) 7 (9.6) 2 (2.7) 8 (11.0) 7 (9.6) 8 (11.0) 27 (37.0) 1 (1.4) Citation Characteristics: no. (%) Iowa State Virginia Tech (N=5117) (N=3696) Journal Other Total (O 1+ O 2+ O 3+ O 4 ) O 1 : Working Papers O 2 : Agricultural Reports O 3 : Government Documents O 4 : Other Book Book Chapter Electronic Thesis 3010 (58.8)* 829 (16.2) 400 (7.8) 284 (5.6) 123 (2.4) 22 (0.4) 789 (15.4) 325 (6.4) 113 (2.2) 51 (1.0) 1461 (39.5)* 944 (25.5) 285 (7.7) 374 (10.1) 214 (5.8) 71 (1.9) 718 (19.5) 332 (9.0) 199 (5.4) 44 (1.2) Other Citation Characteristics: no. (%) Iowa State Virginia Tech (N=829) (N=944) O 1 : Working Papers O 2 : Agricultural Reports O 3 : Government Documents O 4 : Other *significant difference in means P<0.001, Mann-Whitney Test 400 (48.3) 284 (34.3) 123 (14.8) 22 (2.6) 285 (30.2) 374 (39.6) 214 (22.8) 71 (7.5)
Jeffrey D. Kushkowski 267 Figure 1. Number of Citations Per Thesis
268 Web Citation by Graduate Students: A Comparison of Print and Electronic Theses Web Citation Characteristics Web citations make up a small part of the total citations in this sample. Web citations in the theses from Iowa State and Virginia Tech were 2.2 percent (n=113) and 5.4 percent (n=199), respectively. Figure 2 shows the number of electronic citations by year. Trend lines show a general increase in the number of electronic citations between 1997 and 2002. In 1997, the combined number of Web citations for the two schools was 11; in 2001 the total reached 103. Outliers that skewed annual results explain annual variation in Web citations. A total of 28 percent (n=22) of the theses at Iowa State and 41 percent (n=30) of theses at Virginia Tech included citations to Web resources. Table 4 describes the major electronic citation characteristics. Electronic citations were classified using the same JOBET schema used for print items. It is interesting to note that the categories O 1 Working Papers and O 2 Agricultural Reports accounted for 55 percent of Web citations at Iowa State and 52 percent of Web citations at Virginia Tech. The top four domains (.com,.edu,.gov, and.org) accounted for 89 percent (n=101) of Web citations at Iowa State and 86 percent (n=171) of Web citations at Virginia Tech. Accuracy and Persistence of Web Citations In this study, there was poor retrieval of electronic citations and no difference in the retrieval rates between ETD and print theses. Table 4 shows information about the persistence of Web citations for each institution. At Iowa State In this study, there was poor retrieval of electronic citations and no difference in the retrieval rates between ETD and print theses. Discussion Thesis and Citation Characteristics and Virginia Tech, 55 percent of Web citations led directly to the documents. At Iowa State, 3 percent of Web citations and 6 percent of those at Virginia Tech were found at different URLs. Overall accuracy of the citations that were still valid was high, although retrieval of citations was lower than expected. These results are similar to a recent longitudinal study of Web site validity by Judit Bar-Ilan and Bluma Peritz, which found that nearly 40 percent of the documents disappeared and about half the remaining pages were modified between consecutive data-monitoring points. 39 One of the central questions of this study is whether there is a difference in the way that authors of ETDs cite materials compared to the authors of traditional print theses. The theses represented in this study are remarkably similar. ETD authors did include more Web citations than authors of print theses, but the results were not statistically significant. The only significant difference in citation behavior found between Iowa State and Virginia Tech was journal citations (.P<0.001, Mann-Whitney Test). The difference in the number of journal citations can be attributed to the larger number of PhD theses at Iowa State and the disparity in the average numbers of citations by master s and PhD authors (figure 1). These results confirm earlier research by Jeffrey Kushkowski, Kathy
Jeffrey D. Kushkowski 269 Figure 2. Web Citations by Year
270 Web Citation by Graduate Students: A Comparison of Print and Electronic Theses Table 4 Electronic Citation Characteristics Type of citation no. (%) Journal Book O 1 : Working Papers O 2 : Ag Reports O 3 : Government Docs O 4 : Other No URL Organization Website Iowa State (N=113) 9 (8.0) 3 (2.7) 17 (15.0) 45 (39.9) 22 (19.5) 8 (7.1) 9 (8.0) Virginia Tech (N=199) 33 (16.6) 1 (0.5) 54 (27.1) 50 (25.1) 32 (16.1) 4 (2.0) 3 (1.5) 22 (11.1) Domains no. (%).com.edu.gov.org Geographic No URL Other 16 (14.2) 41 (36.3) 26 (23.0) 18 (15.9) 9 (8.0) 3 (2.7) 16 (8.0) 27 (13.6) 62 (31.3) 66 (33.3) 17 (8.6) 3 (1.5) 7 (3.5) Citation Persistence no. (%) Citation Found Found at Different URL Not Found Missing or No URL 62 (54.9) 3 (2.7) 48 (42.5) 109 (54.8) 12 (6.0) 75 (37.7) 3 (1.5) Parsons, and William Wiese revealing that authors of PhD theses use subject literature differently than their master s-writing counterparts. 40 Web Citations One of the most compelling results of this study is confirmation that Web citations are relatively rare in academic theses. In the sample used in this study, Web citations comprised only 312 of a total 8,813 citations (3.53 percent). ETD authors included more Web citations in their theses than authors of print theses, but the results were not statistically significant. The retrieval rate of 60 percent for Web sites at both institutions after less than five years is problematic for future researchers seeking to replicate an author s
Jeffrey D. Kushkowski 271 findings. This study confirms other research about the impermanence of Web citations. 41 Poor persistence of Web citations in theses and dissertations at both institutions raises issues about the reliability of research based on transient Web resources. The persistence of Web citations is an issue that authors need to address when including Web resources in their theses and dissertations. Web Citation and Graduate Scholarship The low numbers of Web citations in these groups of theses are not a surprising result. Several factors may be responsible, including the use of monographs and other materials that are not available in electronic formats. This study did not look at the number of print citations that have electronic equivalents, as did Okrent s study. Had it done so, the number of items available electronically would doubtless have been higher. The increased use of ETDs has not changed the way that graduate students write a thesis. Most of the available ETDs are simply digital equivalents of a traditional print thesis. There are exceptions that use audio, video, and other multimedia techniques; but the bulk of ETDs follow the same conventions as their print counterparts. The Virginia Tech ETD Web site has a list of sample ETDs selected because of their use of technology and creative attempts to create a truly unique work. 42 A question that has not been addressed is whether the widespread dissemination of ETDs is changing the way that research is conducted. The proportion of Web citations by thesis writers has increased over time, but they are still only a small percentage of total citations. This study examined a small subset of ETDs from Virginia Tech. In this particular instance ETD authors cited more electronic resources than their print-thesis peers, but further research is needed to examine whether this phenomenon is specific to economics ETD authors or if it is a characteristic of ETD authors in general. Limitations of this Study One of the most compelling results of this study is confirmation that Web citations are relatively rare in academic theses. This study has some limitations that need to be discussed. First, the binary designation of items into the JOBET classification may result in the undercounting of journal citations. Some citations in the other categories in the classification might otherwise be classified as journals if the other categories did not exist. Another limitation is the narrow subject range of the study that gives, at best, a point estimate of citation behavior among a discrete group of users. Generalizing these results to other disciplines should be done with care, because further research is needed to confirm these findings with different disciplinary groups. Implications for Libraries This research suggests that Web citations are a small but increasing percentage of citations in graduate student theses. It also found that the Web citations in graduate
272 Web Citation by Graduate Students: A Comparison of Print and Electronic Theses theses are almost exclusively to Web resources not purchased by libraries. Most of the Web citations were to information that is freely available. 43 These findings have several implications for libraries in the areas of technical services, library instruction, and archival functions. Technical Services The cost of electronic resources to libraries includes acquisition costs for products and staff time for processing and cataloging. Phil Davis has suggested a method for estimating online journal use, but there is no reliable way to tell from journal citation counts, for example, how many journals were viewed online and cited as a print resource. 44 Should technical services departments continue to spend staff time including URLs in public catalogs given scant evidence that the materials are used? What types of measures can be developed to reliably track usage of electronic library resources? Library Instruction As was noted, few of the Web citations mentioned in this study were to resources purchased by libraries (e.g., full-text journals or electronic databases). Assessment of library instruction traditionally has been concerned with evaluation of the instruction session itself; only recently has there been an attempt to measure student-centered outcomes. 45 How can library instruction programs be adapted to increase awareness of electronic resources purchased by the library? How can scholars and students be encouraged both to use and cite electronic resources? What impact if any will commercial enterprises such as Google Scholar have on library instruction efforts? 46 What evaluative measures can incorporate these issues so that best practices are clearly articulated? Libraries as Electronic Archives One long-established role for libraries is to serve as repositories of information, both in print and electronic formats. The LOCKSS Project (Lots of Copies Keep Stuff Safe) provides libraries a solution for long-term archival storage of electronic materials. 47 As described in an article by Vicky Reich and David S. H. Rosenthal, LOCKSS allows libraries to maintain permanent copies of electronic materials in a distributed environment. 48 The LOCKSS architecture is being used by a number of libraries around the country to preserve digital content. Further refinement of LOCKSS and development of alternate methods for preserving digital content should be an area of continued research. Future Research Priorities and Conclusion This research raises some important questions that need attention. First, what methodologies will be developed in the future to accurately reflect the use of electronic sources? Quantitative methods for studying the use of Web resources have evolved from traditional bibliometric methods. Refinement of existing quantitative methods and development of more robust methods for analyzing Web resources may provide more reliable information about their use.
Jeffrey D. Kushkowski 273 Another area for future research is that of user motivation. Research has shown that Web citations are appearing with increasing frequency in both scholarly and student research. There is little research, however, examining why scholars cite (or not) Web resources. Why are there disparities in the incidence of Web citation among different disciplines? Are scholars, for example, reading a journal article online and citing the print version? Does it matter that they do this? By examining the incidence of citation and the motivations for citation, researchers may gain a clearer picture of how the Web is changing scholarly communication. Bibliometric studies of Web citation behavior are valuable, but they only provide a partial picture of how scholars use their respective subject literatures. As discussed in the literature review, exploration of the behavioral and social aspects of electronic citation is an emerging area but not one without precedent. What is clear from this study is that there is no single answer to the question of how scholars use subject literatures. Research that seeks to integrate methodologies from information retrieval, bibliometrics, information literacy, and information seeking behavior may provide a clearer picture of how scholars use electronic resources and the impact of that use on libraries. Jeffrey D. Kushkowski is business and economics bibliographer, Iowa State University, Ames, IA; he may be contacted via e-mail at: kushkows@gwgate.lib.iastate.edu. Notes 1. Gail McMillan, director, Digital Library and Archives, Virginia Tech, personal e-mail, March 26, 2003. 2. Larry A. Thompson, Electronic Theses and Dissertations at Virginia Tech, Science & Technology Libraries 20, 1 (2001): 87 101. 3. Digital Library and Archives, Virginia Tech, http://scholar.lib.vt.edu/theses (accessed January 11, 2005). 4. Networked Digital Library of Theses and Dissertations, http://www.ndltd.org (accessed January 11, 2005). 5., NDLTD Membership List, http://tennessee.cc.vt.edu/~lming/cgi-bin/odl/nmui/members/index.htm (accessed January 25, 2005). 6. Edward A. Fox, Update on Networked Digital Library of Theses and Dissertations (NTLTD), in Successes and Failures of Digital Libraries (Graduate School of Library and Information Science, University of Illinois at Urbana-Champaign: Urbana, IL, 2000): 12 20. 7. Tony Carneglutti, The Australian Digital Theses (ADT) Project: A Unique Collaborative Experiment in Developing a Model for a Distributed Database of Digital Theses, LASIE 30, 4 (1999): 88 93; John MacColl, Electronic Theses and Dissertations: A Strategy for the UK, Ariadne 32 (July 8, 2002), http://www.ariadne.ac.uk/issue32/theses-dissertations/ (accessed January 11, 2005); Felix N. Ubogu, Spreading the ETD Gospel: A South African Perspective, International Information & Library Review 33, 2/3 (2000): 249 59; Yin Zhang, Kyiho Lee, and Bum-Jong You, Usage Patterns of an Electronic Theses and Dissertations System, Online Information Review 25, 6 (2001): 370 7; Yin Zhang and Kyiho Lee, Features and Uses of a Multilingual Full-Text Electronic Theses and Dissertations (ETDS) System (at the Korea Institute of Science and Technology Information), in National Online Meeting 2001, Proceedings of the 22nd National Online Meeting, New York, May 15 17, 2001 (Medford, NJ: Information Today, 2001), 555 66; and Carlos Henrique Marcondes and Luis Fernando Sayão, Brazilian Digital Library of Theses and Dissertations, The International Information & Library Review 35, 2 4 (2003): 265 74.
274 Web Citation by Graduate Students: A Comparison of Print and Electronic Theses 8. Diplom.de, http://www.diplomica.com (accessed January 11, 2005); Dissertation Publishers, http://www.dissertation.com (accessed January 11, 2005); Bibliothek der Universitat Konstanz, http://www.ub.uni-konstanz.de/kops/ (accessed January 11, 2005); Theological Research Exchange Network, http://www.tren.com (accessed January 11, 2005); and ProQuest Digital Dissertations, http://wwwlib.umi.com/dissertations (accessed January 11, 2005). A more complete list of available ETD sites can be found at the ETD Digital Library, Networked Digital Library of Theses and Dissertations, http:// www.theses.org (accessed January 11, 2005). 9. Networked Digital Library of Theses and Dissertations, http://www.ndltd.org/ help.en.html (accessed January 11, 2005). 10. Yale Fineman, Electronic Theses and Dissertations, portal: Libraries and the Academy 3, 2 (2003): 219 27. 11. Gail McMillan, Library Issues and Responsibilities: Theses Come Off the Shelf to Become Institutional Assets (presentation, American Library Association, midwinter conference, Philadelphia, PA, January 23, 2003). 12. Hussein Suleman, et al., Networked Digital Library of Theses and Dissertations, Bridging the Gaps for Global Access Part I: Mission and Progress, D-Lib Magazine 7, 9 (September 2001), http://www.dlib.org/dlib/september01/09contents.html (accessed January 11, 2005);, Networked Digital Library of Theses and Dissertations, Bridging the Gaps for Global Access Part 2: Services and Research, D-Lib Magazine 7, 9 (September 2001), http://www.dlib.org/dlib/september01/suleman/09suleman-pt2.html (accessed January 11, 2005). 13. Ursula I. A. Goldsmith, Perceptions of Active Graduate Faculty at a Research Extensive University Regarding Electronic Submission of Theses and Dissertations (ETDS) (PhD thesis, Louisiana State University, 2002), http://etd02.lnx390.lsu.edu/docs/available/etd- 0709102-111850/ (accessed January 11, 2005). 14. Thomas H. Teper and Beth Kraemer, Long-Term Retention of Electronic Theses and Dissertations, College and Research Libraries 63, 1 (2002): 61 72. 15. Charles W. Bailey, Jr., Scholarly Electronic Publishing Bibliography (Version 54, 07/13/04) (Houston: University of Houston Libraries, 1996 2003), http://info.lib.uh.edu/sepb/ archive/54/sepb.html (accessed January 11, 2005);, assistant dean for digital library planning and development, University of Houston, personal e-mail, August 3, 2004. 16. Stephen P. Harter, Scholarly Communication and Electronic Journals: An Impact Study, Journal of the American Society for Information Science 49, 6 (1998): 507 16; Yin Zhang, The Impact of Internet-Based Electronic Resources on Formal Scholarly Communication in the Area of Library and Information Science: A Citation Study, Journal of Information Science 24, 4 (1998): 241 54. 17. Yin Zhang, Scholarly Use of Internet-Based Electronic Resources, Journal of the American Society for Information Science and Technology 52, 8 (2001): 628 54. 18. Susan Davis Herring, Use of Scholarly Resources in Scholarly Electronic Journals: A Citation Analysis, College & Research Libraries 63, 4 (2002): 334 40. 19. Burnett L. Green, Electronic Journal Authors in the Field of Mathematics: A Citation Study (master s thesis, University of North Carolina, 1997). 20. Debbie Malone and Carol Videon, Assessing Undergraduates Use of Electronic Resources: A Quantitative Analysis of Works Cited, Research Strategies 15, 3 (1997): 151 8. 21. Philip M. Davis and Suzanne A. Cohen, The Effect of the Web on Undergraduate Citation Behavior 1996 1999, Journal of the American Society for Information Science and Technology 52, 4 (2001): 309 14; Philip M. Davis, The Effect of the Web on Undergraduate Citation Behavior: A 2000 Update, College & Research Libraries 63, 1 (2002): 53 60;, Effect of the Web on Undergraduate Citation Behavior: Guiding Student Scholarship in a Networked Age, portal: Libraries and the Academy 3, 1 (2003): 41 51. 22. Paul O. Jenkins, They re Not Just Using Web Sites: A Citation Study of 116 Student Papers, College & Research Libraries News 63, 3 (2002): 164.
Jeffrey D. Kushkowski 275 23. Nicholas Okrent, Use of Full-Text Electronic Resources by Philosophy Students at UNC- Chapel Hill: A Citation Analysis (master s thesis, University of North Carolina, 2001), http://ils.unc.edu/mspapers/2685.pdf (accessed January 11, 2005). 24. The Journal of the American Society for Information Science and Technology devoted two recent issues to information-seeking research: June 2004 (55, 8) and July 2004 (55, 9). 25. Lisa M. Covi, Debunking the Myth of the Nintendo Generations: How Doctoral Students Introduce New Electronic Communication Practices into University Research, Journal of the American Society for Information Science 51, 14 (2000): 1284. 26. Maija-Leena Huotari and Elfreda Chatman, Using Everyday Life Information Seeking to Explain Organizational Behavior, Library & Information Science Research 23, 4 (2001): 351 66; Elfreda A. Chatman, The Impoverished Life-World of Outsiders, Journal of the American Society of Information Science 47, 3 (1996): 193 206. 27. Brenda Dervin, Sense-Making Theory and Practice: An Overview of User Interests in Knowledge Seeking and Use, Journal of Knowledge Management 2, 2 (1998): 36 46. 28. For more information about sense-making see: Brenda Dervin, From the Mind s Eye of the User: The Sense-Making Qualitative-Quantitative Methodology, in Qualitative Research in Information Management, ed. Jack D. Glazier and Ronald R. Powell (Englewood, CO: Libraries Unlimited, 1992), 61 84; Brenda Dervin, Lois Foreman-Wernet, and Eric Lauterbach, eds., Sense-Making Methodology Reader: Selected Writings of Brenda Dervin (Cresskill, NJ: Hampton Press, 2003); and Sense-Making Methodology Site, http:// communication.sbs.ohio-state.edu/sense-making/default.html (accessed January 11, 2005). 29. Examples of OAI projects include: University of Michigan s OAIster, http:// www.oaister.org (accessed January 11, 2005); Cornell s A Distributed Digital Library of Mathematical Monographs, http://www.library.cornell.edu/mathbooks (accessed January 11, 2005), and BioMed Central, http://www.biomedcentral.com/ (accessed January 25, 2005). 30. The literature on OAI is voluminous. See Charles W. Bailey, Scholarly Electronic Publishing Bibliography, especially Section 9, Repositories, E-Prints, and OAI, for an introduction to OAI issues. For information about the economic aspects of OAI, see: Access to the Literature: The Debate Continues, Nature: Web Focus, http://www.nature.com/nature/ focus/accessdebate/index.html (accessed January 11, 2005). 31. Tomas C. Almind and Peter Ingwersen, Infometric Analyses on the World Wide Web: Methodological Approaches to Webometrics, Journal of Documentation 53, 4 (1997): 404 26. Webometrics is the focus of the Journal of the American Society for Information Science and Technology 55, 14 (November 2004). 32. Sue Jarnagin, thesis specialist, Iowa State University, personal e-mail, August 16, 2004. Students at Iowa State do have the option of including a CD or DVD appendix with the multimedia portions of their theses. 33. Questions and Answers [about ETDs], Virginia Tech, http://etd.vt.edu/background/ QandA.html (accessed January 27, 2005). 34. Digital Library and Archives, Virginia Tech, http://scholar.lib.vt.edu/theses/ (accessed January 25, 2005). 35. Journal of Economic Literature Classification Menu, http://www.econlit.org / subject_descriptors.html (accessed January 11, 2005). 36. Steve Lawrence, et al., Persistence of Web References in Scientific Research, Computer 34 (February 2001): 26 31. 37. Mary F. Casserly and James E. Bird, Web Citation Availability: Analysis and Implications for Scholarship, College & Research Libraries 64, 4 (2003): 300-17. 38. Wallace Koehler, Web Page Change and Persistence: A Four-Year Longitudinal Study, Journal of the American Society for Information Science and Technology 53, 2 (2002): 162 71. 39. Judit Bar-Ilan and Bluma C. Peritz, Evolution, Continuity, and Disappearance of Documents on a Specific Topic on the Web: A Longitudinal Study of Infometrics, Journal of the American Society for Information Science and Technology 55, 11 (2004): 980 90.
276 Web Citation by Graduate Students: A Comparison of Print and Electronic Theses 40. Jeffrey D. Kushkowski, Kathy A. Parsons, and William H. Wiese, Master s and Doctoral Thesis Citations: Analysis and Trends of a Longitudinal Study, portal: Libraries and the Academy 3, 3 (2003): 459 79. 41. For a discussion of persistence in the library literature, see: Mary F. Casserly and James E. Bird; David C. Tyler and Beth McNeil, Librarians and Link Rot: A Comparative Analysis with Some Methodological Considerations, portal: Libraries and the Academy 3, 4 (2003): 615 32. For other discussions of persistence see: Steve Lawrence, et al.; Diomidis Spinellis, The Decay and Failures of Web References, Communications of the ACM 46, 1 (2003): 71 7. 42. Graduate School Guidelines, Virginia Tech, http://etd.vt.edu/guidelines/samples.html (accessed January 11, 2005). 43. Examples of the free resources for the different types of domains:.com: Flue- Cured Tobacco Cooperative Stabilization Page, http://www.ustobaccofarmer.com (accessed January 11, 2005);.edu: Research Institute on Livestock Pricing, http://www.aaec.vt.edu/ rilp (accessed January 11, 2005);.gov: National Agricultural Statistics Service, http:// www.usda.gov/nass/ (accessed January 11, 2005);.org: Federal Reserve Bank of New York, http://www.ny.frb.org/ (accessed January 11, 2005). 44. Philip M. Davis, For Electronic Journals, Total Downloads Can Predict Number of Users, portal: Libraries and the Academy 4, 3 (2004): 379 92. 45. For discussion of student-centered outcomes assessment, see: Information Literacy Competency Standards for Higher Education (Chicago: Association of College and Research Libraries, 2000); Elizabeth Choinski, Amy E. Mark, and Missy Murphey, Assessment with Rubrics: An Efficient and Objective Means of Assessing Student Outcomes in an Information Resources Class, portal: Libraries and the Academy 3, 4 (2003): 563 75. 46. Google Scholar, http://scholar.google.com (accessed January 11, 2005), is a new search engine designed for academic markets. The product is still in beta-test as of November 2004, and its impact on students library research is as yet unknown. For an early review, see: Shirl Kennedy and Gary Price, Big News: Google Scholar is Born, Resource Shelf, http://www.resourceshelf.com/2004/11/wow-its-google-scholar.html (accessed January 11, 2005). 47. LOCKSS, http://lockss.stanford.edu (accessed January 11, 2005). 48. Vicky Reich and David S.H. Rosenthal, LOCKSS: A Permanent Web Publishing and Access System, D-Lib Magazine 7, 6 (2001), http://www.dlib.org/dlib/june01/reich/ 06reich.html (accessed January 11, 2005). Other publications about the LOCKSS Project may be found at: http://lockss.stanford.edu/news/news.htm (accessed January 11, 2005).