The accuracy of references in PhD theses: a case study

The accuracy of references in PhD theses: a case study Fereydoon Azadeh* & Reyhaneh Vaez *Tehran University of Medical Sciences (TUMS), Faculty of Allied Medicine, Deptartment of Medical library & Information Sciences, Tehran, Iran, and Medical Central Library, Tabriz University of Medical Sciences, Tabriz, Iran Abstract Background: Inaccurate references and citations cause confusion, distrust in the accuracy of a report, waste of time and unnecessary financial charges for libraries, information centres and researchers. Objectives: The aim of the study was to establish the accuracy of article references in PhD theses from the Tehran and Tabriz Universities of Medical Sciences and their compliance with the Vancouver style. Methods: We analysed 357 article references in the Tehran and 347 in the Tabriz. Six bibliographic elements were assessed: authors names, article title, journal title, publication year, volume and page range. Referencing errors were divided into major and minor. Results: Sixty two percent of references in the Tehran and 53% of those in the Tabriz were erroneous. In total, 164 references in the Tehran and 136 in the Tabriz were complete without error. Of 357 reference articles in the Tehran, 34 (9.8%) were in complete accordance with the Vancouver style, compared with none in the Tabriz. Accuracy of referencing did not differ significantly between the two groups, but compliance with the Vancouver style was significantly better in the Tehran. Conclusions: The accuracy of referencing was not satisfactory in both groups, and students need to gain adequate instruction in appropriate referencing methods. Keywords: case studies; PhD theses; referencing; Asia; West DOI: 10.1111/hir.12026 Key Messages Analysis of the accuracy of references in PhD theses of the Tabriz and Tehran Universities of Medical Sciences revealed many errors and inaccuracies. The rate of compliance with Vancouver style was also very low. To improve accuracy, part of the thesis score should be allocated to the accuracy of references and referencing methods. In addition, supervisors and students should pay close attention to the rules, referencing methods and the writing of theses needs. Introduction Scientific studies are inspired by previous research. Consequently, no one can say that his/her research has been accomplished independently, without reference to other researchers work. Researchers must cite the sources of information that they have used in a reference list or bibliography. References Correspondence: Fereydoon Azadeh, Fazlollah Noori and Chamran cross -central office of Tehran University of Medical Sciences, Tehran, 1459953581, Iran. E-mail: azadehtafreshi@yahoo.com are one of the main elements of research reports: they serve as a link between different types of information source and subject field 1 ; they enable readers to identify gaps in knowledge and increase their understanding of a given topic 2 ; and they can be used to establish the originality of statements that are presented in scientific papers. 3 Consequently, every researcher must use references to validate their work. Incorrect or incomplete references are often found in lists of references, despite the emphasis that the editorial boards of reliable scientific 232

The accuracy of references, Fereydoon Azadeh 233 journals place on the importance of referring to original sources and presenting these references correctly. Such errors break the link with the originally intended reference and lead to the transmission of incorrect information, which can result in confusion and a loss of trust among researchers. In addition, libraries and researchers have to pay high costs in terms of time and money to identify these references. A survey has shown that the average cost for the identification of a single reference by the staff of the US National Library of Medicine was 3.37 US dollars in 1980, and this amount had increased to 9.94 US dollars by 1996. 1 Consequently, the accuracy of references is essential to maintain the correct flow of information and the acceptance of published work in the scientific community, as well as to decrease document access charges. To help readers to access documents, standard formats for references, such as the Vancouver style, should be followed to present the list of sources in a uniform manner 4 All scientific works and research reports, including theses, which are also important contributions to scientific knowledge, 5 should have accurate and uniformly presented references. Many studies have assessed the accuracy of references, and sometimes that of quotations, in the journals of different fields of medicine, and all have found reference errors. 6 22 These errors betray inattention to detail by the authors in this regard. 15,16 Other explanations might be that the native language of the author is not the same as that of the references, or the author might refer to a source without seeing the original article. 22 Reference errors may be reduced using the copy and paste method or reference manager software to compile reference lists, rather than noting reference details manually. In addition, references should be reviewed and edited before submission to a journal. 2,21 Librarians might also have a role to play in reducing reference errors. 20 The Vancouver style is the method of referencing that is used by most biomedical journals and many scientific journals. It was developed by editors of medical journals in 1978 at a meeting in Vancouver, BC. Several studies have assessed conformity with the Vancouver style of citation in research papers and theses, and rates of compliance have varied from 70.8% down to as low as 1.59%. 23 25 There are approximately 50 medical universities in Iran which publish about 230 scientific journals. All medical journals that are published in Iran utilise the Vancouver style, and it is also promoted in the theses guidelines of the universities. Given the high rate of error in references and the often low level of compliance with the Vancouver style, the aim of the present study was to review the accuracy of references, and their compliance with the Vancouver style, in PhD theses accepted by the Tabriz and Tehran Universities of Medical Sciences. Methods We analysed references of articles in PhD theses in Farsi that were defended at the Tabriz and Tehran Universities of Medical Sciences in 2007 2008. Other types of reference, such as books, theses, and websites, were omitted because of limited access to these sources. In the theses from each university, all references of articles were identified and numbered. In total, there were 4293 references in the theses from Tehran University of Medical Sciences and 1390 in the theses from Tabriz University. For each group, the appropriate sample size was determined using 26 systematic sampling based on Morgan s Table. 27 In the Tehran group, a sample of 357 article references was chosen and in the Tabriz group, a sample of 347. We analysed the following bibliographic elements of each selected reference: (i) authors names; (ii) article title; (iii) journal title; (iv) publication year; (v) volume number; and (vi) page range. The different types of reference error were divided into two groups on the basis of the study of Taylor. 9 (i) Minor errors: these had no effect on retrieval of the reference; for example, displacement of authors names; addition or elimination of details such as indefinite/definite articles in the title; minor spelling errors in the journal title; or omission of the final page number. (ii) Major errors: these caused serious problems for document retrieval or even made it impossible; for example, incorrect or missing authors names; major spelling errors in the article title; misspelt abbreviation of the journal title; incorrect or missing publication year or volume number; or incorrect first page number.

234 The accuracy of references, Fereydoon Azadeh For any given reference element, all relevant errors in a single reference were grouped as one. For example, two errors in the title, such as misspelling of the article title and omission of part of the title such that it could not be identified, were considered to be one single error. The data was gathered using direct observation. First, we searched for all the selected references in the PubMed database to confirm their existence. Second, the accuracy of each chosen reference was ascertained by comparison with the original article in printed and electronic forms (which were available in the Medical Library of Tehran University and the Central Library of Tabriz University). If access to the full text was not possible, we searched for the abstract or bibliographic information in reliable databases to which the two universities subscribe, such as: Ebsco, Science Direct, Ovid, Web of Science, Scopus, PubMed, Magiran and Iran Medex. After the accuracy of the references had been checked, we noted any errors relative to the original article, as well as whether the reference followed the Vancouver style as a referencing method. The error checklist that was completed for each reference consisted of four parts: (i) information about the thesis, such as the title, the number of article references, and the name of the advisor; (ii) sources used to check the references; (iii) different types of reference error present; and (iv) compliance with the Vancouver style. In the study, qualitative statistics were used to analyse the frequency of reference errors and use of the Vancouver style. Student s t-test was used to analyse the relationships between different reference articles. SPSS version 11 was used for the statistical analysis (Appendix). Results In the Tabriz group, there were 341 reference errors in the 347 references and in the Tehran group; there were 398 errors in the 357 references (Table 1). In both groups, the highest and lowest error rates were identified for author names and publication year, respectively. There were 74 (21%) minor errors of 347 in the Tabriz group and 71 (20%) of 357 in the Tehran group. In both groups, the highest rate of minor errors was identified for page numbers and the lowest for journal title (Table 2). There were 302 major errors in the Tabriz group and 370 in the Tehran group. In both groups, the highest and lowest rates of major errors were for authors names and publication year, respectively (Table 3). In the Tabriz group, 164 references of 347 had no errors, whereas in the Tehran group, 136 references of 357 had no errors. Thus, in the Tabriz group, 183 references (53%) and in Tehran group 221 references (62%) were erroneous. A t-test was used to determine whether the difference in the frequency of errors between the two universities was significant; no significant difference was found between the two groups (P > 0.05). Upon reviewing the article references in the PubMed database, eight major errors identified in the Tabriz group and 11 major errors in the Tehran group were found to occur in the PubMed database itself. Table 4 shows the results for compliance with the Vancouver style for the six bibliographic elements. The highest rates of compliance were for article title (281 items, 81%) in the Tabriz group and for volume number in the Tehran group (297 Table 1 Rate of errors for each reference element Tabriz Tehran Reference elements Frequency Percentage Frequency Percentage Authors names 133 39 150 38 Article title 82 24 97 24 Journal title 44 13 61 15 Publication year 18 5 16 4 Volume umber 21 6 22 6 Page numbers 43 13 52 13 341 100 398 100

The accuracy of references, Fereydoon Azadeh 235 Table 2 Rate of minor errors Tabriz Tehran Reference elements Frequency Percentage Frequency Percentage Authors names 5 7 3 4 Article title 23 31 21 30 Journal title 3 4 Errorless Errorless Page numbers 43 58 47 66 74 100 71 100 Table 3 Rate of major errors Tabriz Tehran Reference elements Frequency Percentage Frequency Percentage Authors names 133 44 150 41 Article title 68 22 97 26 Journal title 42 14 61 17 Publication year 18 6 16 4 Volume number 21 7 22 6 Page numbers 20 7 24 6 302 100 370 100 Table 4 Compliance with Vancouver style for bibliographic reference elements Tabriz Tehran Reference elements Frequency Percentage Frequency Percentage Authors names 219 63.1 152 42.6 Article title 281 81 221 61.9 Journal title 1 0.3 181 50.7 Publication year Errorless Errorless 267 74.8 Volume number 16 4.6 297 83.2 Page numbers 129 37.2 159 44.5 items, 83.2%). The lowest rates of compliance were for publication year in the Tabriz group and authors names in the Tehran group. In the Tabriz group, none of the 347 references were completely compliant with the Vancouver style, and in the Tehran group, only 35 of the 357 references (9.8%) were completely compliant. A t-test was used to test whether the difference in the rate of compliance with Vancouver style between the two universities was significant. A significant difference was found between the two groups (P < 0.05). The references in the Tehran group showed better compliance with the Vancouver style than those in the Tabriz group. Discussion In the Tabriz group, the error rate in the references analysed was 53%, and in the Tehran group, it was 62%. There is wide variation when our results are compared with those of previous studies. Asano et al. and NganKee et al. found similar citation error rates of 48% and 56%, respectively. 6,7 However, several studies had better error rates, which ranged from 11.1% to 45.8%, 6,8 11,14 18,20,21 whereas Azizi et al. found a higher rate of 76%. 22 In the present study, authors names were the most frequently erroneous reference element, which was similar to the findings of many previous

236 The accuracy of references, Fereydoon Azadeh studies. 2,7 9,12,14,16,18,22 Taylor thinks that such errors act as an obstacle to information retrieval, but also that they impede the main goal of referencing, that is acknowledging and valuing the work of a specific author(s). Such errors affect the analysis of references in the earliest stages. 9 However, in several other studies, article title showed the highest error rate. 6,11,13,17,19,20 We found that publication year was the least frequently erroneous reference element, which was similar to previous studies. 8,9,13,14,22 Errors in the publication year or the start of pagination and errors in the title can affect recognition of the reference and the retrieval of information. In the present study, the rate of major errors was higher than the rate of minor errors, which was similar to the findings of the majority of previous studies. 2,9,11,14 However, Azizi et al. 22 found the reverse, with a higher rate of minor errors. Despite the emphasis in the guides to writing theses of Tabriz and Tehran Universities of Medical sciences that the Vancouver style should be followed as a referencing method, the rates of compliance with this referencing method were very low, namely 9.8% in the Tehran group and 0% in the Tabriz group, for the references selected for the study. The difference between the Tehran and Tabriz groups was significant. The rate of complete compliance with the Vancouver style at Tehran University of Medical Sciences (9.8%) was higher than that reported by Rakhsh et al. 24 (1.59%) but much lower than that reported by Aliramazani (70.8%). 23 In the study of Tafreshi and Tajbakhsh of the theses of the Dentistry Faculty of Shahid Beheshti University of Medical Sciences, the rate was lower than 2, namely in the Likert scales of 4, it is lower than average. It indicates that the students and professors do not have any good knowledge in referencing methods. 25 Conclusion The results of the present study with regard to the accuracy of references in theses from Tehran and Tabriz Universities of Medical Sciences indicate that there is scope for improvement in this regard. At present, many workshops are held annually by these two universities with the aim of improving students and professors skills in relation to referencing methods and the searching of electronic databases. In addition, the curricula of the medical universities include courses on research methodology, which provide additional opportunities for students to acquire the skills that are necessary for thesis writing and scientific writing. However, we make the following additional suggestions for improvement: 1 Part of the thesis score should be allocated to the accuracy of references and referencing methods; 2 Experienced librarians and information professionals should provide advice to the students to help them writing their reference lists; 3 Accurate, comprehensive and up-to-date instructions for writing of theses should be provided; 4 Supervisors and advisors should pay attention to the process of writing of theses, in particular with respect to the accuracy of references and referencing methods. References 1 Powley, B. & Dale, R. eds. Evidence-based information extraction for high accuracy citation and author name identification. RIAO Large-Scale Semantic Access to Content; 2007 30 May-1 June; Pittsburgh, Pennsylvania: Robert Dale s Publications. 2 Oermann, M. H. & Ziolkowski, L. D. Accuracy of references in three critical care nursing journals. Journal of Perianesthesia Nursing 2002, 17, 78 83. 3 Hagigi, M. The usage of the citation in scientific writings. Journal of Education and Psychology, 2002, 32, 215 232. (Article in persian) 4 Tondivar, A. A review of the use of bibliographic rules in Master theses of library and information science of Tehran University from 1993 to 1994. Journal of Information Science [Internet]. 1998; 13(3-4). Accessible at: http://jist.irandoc.ac.ir/browse.php?a_code=a-10-2-163&sid=1&slc_lang=fa. (Article in persian) 5 Horri, A. Informative importance of theses review of information and Inform. Tehran: Board of trustees of public libraries, 1993. (Book in persian). 6 Asano, M., Mikawa, K., Nishina, K., Maekawa, N. & Obara, H. Improvement of the accuracy of references in the Canadian Journal of Anaesthesia. Canadian Journal of Anaesthesia 1995, 42(5 Pt 1), 370 372. 7 Ngan Kee, W. D., Roach, V. J. & Lau, T. K. The accuracy of references in the Hong Kong Medical Journal. Hong Kong Medical Journal, 1997, 3, 377 380.

The accuracy of references, Fereydoon Azadeh 237 8 Nishina, K., Asano, M., Mikawa, K., Maekawa, N. & Obara, H. Accuracy of references in Anesthesia & Analgesia does not improve. Anesthesia & Analgesia 1995, 80, 641 642. 9 Taylor, M. K. The practical effects of errors in reference lists in Nursing Research Journals. Nursing Research 1998, 47, 300 303. 10 Fenton, J. E., Brazier, H., De Souza, A., Hughes, J. P. & McShane, D. P. The accuracy of citation and quotation in otolaryngology/head and neck surgery journals. Clinical Otolaryngology Allied Sciences, 2000, 25, 40 44. 11 Oermann, M. H., Cummings, S. L. & Wilmes, N. A. Accuracy of references in four pediatric nursing journals. Journal of Pediatric Nursing, 2001, 16, 263 268. 12 Siebers, R. How accurate are references in clinical chemistry? Clinical Chemistry 2001, 47, 606 607. 13 O Connor, A. E. A review of the accuracy of references in the journal Emergency Medicine. Emergency Medicine 2002, 14, 139 141. 14 Gosling, C. M., Cameron, M. & Gibbons, P. F. Referencing and quotation accuracy in four manual therapy journals. Manual Therapy, 2004, 9, 36 40. 15 Lukic, I. K., Lukic, A., Gluncic, V., Katavic, V., Vucenik, V. & Marusic, A. Citation and quotation accuracy in three anatomy journals. Clinical Anatomy, 2004, 17, 534 539. 16 Aronsky, D. A., Ransom, J. & Robinson, K. Accuracy of references in five biomedical informatics journals (vol 12, pg 225, 2005). Journal of the American Medical Informatics Association, 2005, 12, 587. 17 Buchan, J. C., Norris, J. & Kuper, H. Accuracy of referencing in the ophthalmic literature. American Journal of Ophthalmology, 2005, 140, 1146 1148. 18 Reddy, M. S., Srinivas, S., Sabanayagam, N. & Balasubramanian, S. P. Accuracy of references in general surgical journals an old problem revisited. Surgeon, 2008, 6,71 75. 19 Mohammad, A. E. & Laskin, D. M. Citation accuracy in the oral and maxillofacial surgery literature. Journal of Maxillofacial & Oral Surgery, 2008, 66, 3 6. 20 Oren, G. & Watson, M. Accuracy of references in the ophthalmic literature. Journal of the Medical Library Association 2009, 97, 142 145. 21 Al-Benna, S., Rajgarhia, P., Ahmed, S. & Sheikh, Z. Accuracy of references in burns journals. Burns 2009, 35, 677 680. 22 Azizi, F., Raieszadeh, F., Zareie, S., Tarigateh Saber, G. & Garzan, R. Review of sources citation quality in research articles of scientific- research journals in Islamic Republic of Iran. Journal of Research in Medical Sciences, Shahid Beheshti University of Medical Sciences 2001, 25, 225 249. (Article in persian) 23 Aliramazani, H. Comparative review of Vancouver style and ISI standards use in PhD theses of Kerman University of Medicinal Sciences. Tehran: Iran University of Medical Sciences, 2000. (Thesis in persian). 24 Rakhsh, F., Alibeygy, M. & Hoseyni, F. The study of citing materials from online sources in journal of medical sciences Universities from 1379-1381. Health Information Management, 2006, 3, 25 31. (Article in persian) 25 Tafreshi, S. & Tajbakhsh, A. A survey on rate of using Vancouver style as bibliographies presented in doctoral theses and post graduate theses of dentistry in Shahid Beheshti University of Medical Science between 2004 2009. Journal of Dental School, Shahid Beheshti University of Medical Sciences, 2010, 28, 47 55. (Article in persian) 26 Delavar, A. Probability and applied statistics in psychology and training sciences. Tehran: Roshd, 2006. (Book in persian). 27 Dayyani, M. Sample size in survey researches. Journal of library and Information Science, 1999, 2, 59 70. (Article in persian) Received 16 August 2011; Accepted 2 March 2013

238 The accuracy of references, Fereydoon Azadeh Appendix Form of checking accuracy of referenced articles of the PhD Theses of medical universities of Tabriz and Tehran Thesis information 1. Serial number 2. Field 3. Name of author 4. Advisor professor 6. Title of thesis 7. of chosen references Rate of accordance of chosen references to Vancouver style: 1 Name of authors, and symbols: 2 Title and subtitle of article and symbols: 3 Abbreviation of journal title and symbols: 4 Publication year and symbols: 5 Volume (, if necessary) and symbols: 6 Page numbers and symbols: ----------------------------------------------------------------------------------------------------------------------------------- 1 Name of authors, and symbols: 2 Title and subtitle of article and symbols: 3 Abbreviation of journal title and symbols: 4 Publication year and symbols: 5 Volume (, if necessary) and symbols: 6 Page numbers and symbols Sequence number of reference and checked sources Table of minor errors and their number per reference Author(s) name Misplacing name(s) of author(s) Adding information and other symbols to first name in an identifiable way Adding information or other symbols to last name in an identifiable way Title of article Dictation error Eliminating words or a part of title in an identifiable way Adding information or other symbols to the title in an identifiable way

The accuracy of references, Fereydoon Azadeh 239 Title of journal Dictation error (small) of pages The end page is not mentioned Error of the last page Table of major errors and their number per reference Author(s) name Misspelling of last name Wrong first name The first name or initial is not mentioned The initial is not mentioned completely Wrong last name The last name is not mentioned Eliminating the total name Article Title Title of article is not mentioned Wrong title Misspelling of title Eliminating sub title Eliminating words or a part of title in not identifiable way Adding other information to the title in not identifiable way Title of journal Name of journal is not mentioned Wrong title Misspelling (large) Eliminating part of journal in not identifiable way Adding information to the title in not identifiable Publication year Wrong publication year The publication year is not mentioned

240 The accuracy of references, Fereydoon Azadeh Volume number Wrong volume number The volume number is not mentioned Page numbers Wrong number of the first page The number of the first page is not mentioned Adding parts of page numbers as Eliminating parts of page numbers as