By Jon R. Davids, MD, Daniel M. Weigl, MD, Joye P. Edmonds, MLIS, AHIP, and Dawn W. Blackhurst, DrPH

Transcription

1 1155 COPYRIGHT Ó 2010 BY THE JOURNAL OF BONE AND JOINT SURGERY, INCORPORATED Reference Accuracy in Peer-Reviewe Peiatric Orthopaeic Literature By Jon R. Davis, MD, Daniel M. Weigl, MD, Joye P. Emons, MLIS, AHIP, an Dawn W. Blackhurst, DrPH Investigation performe at the Shriners Hospital for Chilren, Greenville, South Carolina; Schneier Chilren s Meical Center, Petah Tikva, Israel; an Greenville Hospital System University Meical Center, Greenville, South Carolina Backgroun: Reference accuracy of articles publishe in the biomeical literature is etermine by the presence of citation an quotation errors. A recent review emonstrate that the meian citation error rate per biomeical journal was 39%, an the meian quotation error rate per journal was 20%. Reference accuracy in peiatric orthopaeic articles has not been previously reporte, to our knowlege. Methos: Two hunre references from twenty articles publishe in four peer-reviewe orthopaeic journals were ranomly selecte for assessment of citation an quotation accuracy. Full-text copies of all original references were obtaine by interlibrary loan methos an revieweirectly to establish citation accuracy. The presence of citation errors was etermine by a single investigator. The relevance of citation errors was etermine by assessing the ease of reference retrieval through PubMe. Quotation accuracy was etermine by two examiners who reviewe each of the twenty articles an 200 references to compare the claims mae for the references in the article against the ata an opinions expresse in the actual reference. Results: The total citation error rate across all of the journals was 26% (fifty-one of 200 references) with a 95% confience interval of 16.5% to 37.3%. The meian citation error rate per journal was 27% (range, 10% to 38%). Although citation errors were common, most were of minimal significance, as 196 of the 200 references coul be retrieve with ease from PubMe. The total quotation error rate across all of the articles was 38% (152 of 398 reference citations) with a 95% confience interval of 30.1% to 47.0%. The meian quotation error rate per journal was 38% (range, 28% to 46%). Conclusions: Citation an quotation errors are common in the peiatric orthopaeic literature. Reference accuracy continues to be a substantial problem in the biomeical literature espite recent technological avances such as online atabases, easily accessible search engines, an wiely available bibliographic software. It is generally assume that the references provie in articles in peer-reviewe biomeical journals are accurate. These references are provie to corroborate the authors statements, to enable reaers to access relevant literature, an to give creit to the work of previous investigators 1. In aition, the number of times that a publishe article is reference in the literature is tracke to help to establish the impact factor of biomeical journals, which is commonly seen as an inication of the quality of the journal 2-4. Reference accuracy is etermine on the basis of the citation an quotation errors 1,2,5-8. Citation errors involve an incorrect bibliographic escriptor in the reference 1,2,7.Such errors may make retrieval of the reference ifficult or impossible. Quotation errors are more insiious an involve inconsistency between the authors assertions an the actual ata or conclusions foun in the cite reference 1,2,7. Such errors may propagate misinformation 9,10. A recent Cochrane Collaboration review of technical eiting of research reports in biomeical journals emonstrate a meian citation error rate per journal of 39%, with a range of 4% to 67%, an a meian quotation error rate per journal of 20%, with a range of 0% to 44% 11. To our knowlege,assessmentofcitationaccuracyinorthopaeicpeerreviewe journals has been limite to two stuies 12,13, both of which are ate, as they ealt with publications publishe from 1975 to 1995, an are of questionable current relevance, as bibliographic software is now frequently utilize by authors, eitors, an journal staff. We i not ientify any previous stuies assessing quotation accuracy in orthopaeic peerreviewe journals. Disclosure: The authors i not receive any outsie funing or grants in support of their research for or preparation of this work. Neither they nor a member of their immeiate families receive payments or other benefits or a commitment or agreement to provie such benefits from a commercial entity. J Bone Joint Surg Am. 2010;92: oi: /jbjs.i.00063

2 1156 The goal of the current stuy was to evaluate reference accuracy i.e., citation an quotation error rates in peiatric orthopaeic stuies publishe in four peer-reviewe orthopaeic journals. Materials an Methos Four peer-reviewe orthopaeic journals, the American an British eitions of The Journal of Bone an Joint Surgery (JBJS-A an JBJS-B) an the American an European eitions of the Journal of Peiatric Orthopaeics (JPO-A an JPO-B), were selecte for review. The first five articles, beginning in the January issue an proceeing towar the December issue, on original research topics in peiatric orthopaeics were collecte from the JPO-A an JPO-B volumes publishe in 2008 an from the JBJS-A an JBJS-B volumes publishe in 2007 an Review articles an case reports were not inclue. Five articles were selecte from each journal, for a total of twenty articles inclue in the stuy. These twenty articles TABLE I Article Demographics Journal Impact Factor Stuy Type Level of Evience No. of Authors Wor Count No. of Cite References JBJS-A JBJS-B JPO-A JPO-B Ranomize controlle clinical trial Ranomize controlle clinical trial Retrospective case Ranomize controlle clinical trial Retrospective case Therapeutic I Therapeutic II Prognostic IV Therapeutic I Therapeutic III Prospective case Diagnostic IV Retrospective case Prognostic III Prospective case Diagnostic II Retrospective case Therapeutic IV Retrospective case Therapeutic IV Retrospective case Therapeutic IV Retrospective cohort Therapeutic III stuy Retrospective case Therapeutic IV Retrospective case Therapeutic IV Prospective case Therapeutic IV Retrospective case Therapeutic IV Retrospective case Therapeutic IV Retrospective case Therapeutic IV Prospective case Therapeutic IV Prospective case Therapeutic IV

3 1157 were evaluate to etermine the stuy esign, level of evience, number of authors, wor count, an number of cite references. A ranom-sequence generator (a software program that creates a ranom sequence of numbers that fall between two selecte numbers) was use to select ten references from each article 14. Personal communications, letters, textbook chapters, theses, presentations, abstracts, an review articles were exclue from the selection. The final result was fifty cite references from each of the four journals, 200 references in total, which were stuie for citation an quotation accuracy. Full-text copies of all original references were obtaine by interlibrary loan methos an revieweirectly to establish citation accuracy. Citation errors were classifie as major, intermeiate, or minor, on the basis of the ata elements utilize to classify references in the Science Citation Inex, as establishe an utilize in previous stuies 2,15. Major citation errors were relate to the first author s name, the year of publication, the journal title, the journal volume number, an the initial page number. Intermeiate citation errors inclue inaccuracies in the ening page number an misspellings or wrong or missing wors in the article title. Minor citation errors inclue errors of punctuation, misspelling of coauthors names, an other spelling errors. The importance of the citation errors that were ientifie was etermine by one of us (J.P.E.) assessing the ease of reference retrieval with use of PubMe, an electronic search engine 16. Retrieval was classifie as easy when a single search with the Single Citation Matcher (utilizing the first author s name an the year of publication) was successful, ifficult when more than one search (requiring entry of aitional ata fiels) was require for successful retrieval, an impossible when the reference coul not be retrieve through PubMe 1. Quotation accuracy was etermine by comparing the claims mae for the references in the article against the ata anopinionsexpresseintheactualreference.twoofus (J.R.D. an D.M.W.), both fellowship-traine peiatric orthopaeic surgeons, reviewe each of the twenty articles an 200 references to establish quotation accuracy. Quotation errors were classifie as major or minor, accoring to efinitions establishe an utilize in previous stuies, each time the reference was cite in the article 1,2,5,6. Major quotation errors were etermine to occur when the reference contraicte, faile to substantiate, or was unrelate to the authors assertions mae in the article. Minor quotation errors inclue incorrect patient or measurement numbers or percentages, oversimplification, conclusions not reache by the authors of the cite reference, an inirect referencing (when the cite reference i not contain the relevant ata but itself cite another publication containing, presumably, the original work) 8. Disagreements about the assessment of quotation accuracy between the two reviewers were resolve by repeat review until a consensus was reache. Error rates an 95% confience intervals were calculate with use of repeate-measures ichotomous logistic regression analysis (PROC GENMOD; SAS Statistical Software, Cary, North Carolina). This analysis provies the probability of error (yes/no), ajusting for the neste stuy esign of article within journal. Interjournal ifferences in citation an quotation error rates were assesse on the basis of the 95% confience intervals obtaine from the logistic regression. The kappa statistic was use to etermine interobserver agreement on the presence of any type of quotation error. Possible associations between error rates an article emographic factors were assesse with use of the logistic moel escribe above (for possible associations with the type of stuy an level of evience) or with the Pearson correlation (for possible associations with the number of authors, wor count, an number of cite references). Significance was set at p < Source of Funing No outsie funing was receive in support of this research or for the preparation of this work. Results Article Demographics Twenty articles were selecte for assessment of citation an quotationaccuracy(tablei).therewereelevenretrospective case, five prospective case, three ranomize controlle clinical trials, an one retrospective cohort stuy. Twelve of the articles were retrospective, an eight were prospective. Thirteen of the articles were consiere to provie level-iv evience; three, level-iii; two, level-ii; an two, level-i. The mean number of authors was 4.95 (range, two to nine). TABLE II Citation Error Rates No. of Citation Errors No. of Citations Error Rate (95% Confience Interval) (%) Journal Major Intermeiate Minor with Any Error Major Intermeiate Minor Total* JBJS-A ( ) JBJS-B ( ) JPO-A ( ) JPO-B ( ) Total ( ) 9.5 ( ) 15.5 ( ) 25.5 ( ) *N = 50 references per journal; n = 200 references in total.

4 1158 TABLE III Citation Errors by Type Author Name Article Title Journal Title Year of Publication Volume Number Page Number No. of citation errors JBJS-A JBJS-B JPO-A JPO-B Total Error rate (95% confience interval)* (%) 2.5 ( ) 8.0 ( ) 1.0 ( ) 1.0 ( ) 3.5 ( ) 2.0 ( ) *N = 200 references. The mean wor count was 4013 (range, 2119 to 7093). The mean number of cite references was 26.5 (range, fifteen to forty-one). Journal impact factors range from (JPO-B) to (JBJS-A) 17. Citation Errors The total citation error rate across all of the journals was 26% (fifty-one of 200 references; 95% confience interval, 16.5% to 37.3%) (Table II). The meian citation error rate per journal was 27% (range, 10% for JBJS-A to 38% for JPO-A). The total rate of major citation errors was 7% (fourteen major citation errors per 200 references), with a 95% confience interval of 3.3% to 14.3%. The meian rate of major citation errors per journal was 3% (range, 2% for JBJS-A an JBJS-B to 20% for JPO-A). Errors in the article title were the most common type of citation error foun (Table III). There was a strong negative correlation, which was significant, between the journal impact factor an the percentage of total citation errors (r = 20.95, p = 0.055). JBJS-A iffere significantly from JPO-A with respect to the rate of total citation errors (p < 0.05); however, no other significant ifferences between journals were appreciate. With the numbers stuie, there was no ientifiable association between the citation error rate an the stuy esign (prospective or retrospective, p = 0.37) or the level of evience (levels I an II compare with levels III an IV, p = 0.15), an citation accuracy i not appear to be associate with the number of authors (r = , p = 0.70), article wor count (r = 20.19, p = 0.42), or number of references in each article (r = 0.05, p = 0.82). One hunre an ninety-six (98%; 95% confience interval, 95.0% to 99.5%) of the 200 references coul be retrieve with ease from PubMe. Two references were ifficult to retrieve from PubMe, because of errors in the spelling of the lea author s name in both cases. Two references were impossible to retrieve from PubMe: one because the article authors an title i not coincie with the journal title, year, volume, an page numbers an the other because the year of publication was incorrect. Quotation Errors The 200 selecte references were cite 398 times in the twenty articles. Quotation accuracy was assesse inepenently by two of us, who agree with regar to their initial assessment of 350 (87.9%) of the 398 citations. Correlation between the two observers was strong (kappa = 0.70; 95% confience TABLE IV Quotation Error Rates No. of Cite Number of Quotation Errors No. of Citations Error Rate (95% Confience Interval) (%) Journal References Major Minor with Any Error Major Minor Overall JBJS-A ( ) JBJS-B ( ) JPO-A ( ) JPO-B ( ) Total ( ) 14.1 ( ) 38.2 ( )

5 1159 TABLE V Quotation Errors by Type No. of Cite References Reference Contraicte Assertions in Article Reference Faile to Support Assertions in Article Reference Unrelate to Assertions in Article Incorrect Patient Numbers or Percentages Incorrect Measurement Numbers or Percentages Inirect Referencing* No. of citation errors JBJS-A JBJS-B JPO-A JPO-B Total Error rate (95% confience interval) (%) 6.8 ( ) 15.8 ( ) 5.0 ( ) 3.0 ( ) 3.5 ( ) 7.5 ( ) *Cite reference i not contain the relevant ata but itself cite another publication containing, presumably, the original work. interval, 0.62 to 0.78) an significant (p < 0.001). The total quotation error rate across all of the journals was 38% (152 quotation errors per 398 reference citations; 95% confience interval, 30.1% to 47.0%) (Table IV). The meian quotation error rate per journal was 38% (range, 28% for JBJS-B to 46% for JBJS-A). The total rate of major quotation errors was 28% (110 major quotation errors per 398 references; 95% confience interval, 20.8% to 35.7%). The meian major quotation error rate per journal was 27% (range, 23% for JBJS-B to 34% for JPO-B). Failure of the reference to support the claims of the authors who cite the reference was the most common type of quotation error foun in the journals (Table V). With the numbers stuie, we foun little to no correlation between the journal impact factor an the percentage of total quotation errors (r = 0.06, p = 0.94) an no significant ifferences between any of the journals with respect to the total quotation error rate (all p values > 0.05). We coul not ientify an association between stuy esign (prospective or retrospective, p = 0.79) or level of evience (levels I an II compare with levels III an IV, p = 0.34) an the quotation error rate, an the quotation error rate i not appear to be associate with the number of authors (r = 0.16, p = 0.50), article wor count (r = 0.18, p = 0.44), or number of references in each article (r = 0.02, p = 0.92). Discussion Concerns about the accuracy of references in peer-reviewe meical journals have been expresse in the meical literature ating back to the early 1880s 18. A recent Cochrane Collaboration review of technical eiting of research reports in biomeical journals ientifie thirty-five relevant stuies between 1975 an 1999 anetermine that over 15,000 references have been checke in accuracy stuies in the biomeical literature, with 4812 citation errors foun in 13,588 references (a meian citation error rate per journal of 39%) an 431 quotation errors foun in 1899 references (a meian quotation error rate per journal of 20%) 11. The analysis of citation error rates in the orthopaeic literature has been limite. JBJS-A was etermine to have a citation error rate of 13% in 1975, 6% in 1985, an 12% in ,13. Citation error rates in JBJS-B (1985 an 1995), Clinical Orthopaeics an Relate Research (1985 an 1995), an Acta Orthopaeica Scaninavica (1985 an 1994) have range from 3% to 37% 13. The meian citation error rate per orthopaeic journal was 13%, which, while a cause for concern, compares favorably with the citation error rate in the wier meical literature over the same time perio 11. The current stuy emonstrate a meian citation error rate per orthopaeic journal of 27% for the years 2007 an The lowest citation error rate, 10%, was seen in JBJS-A an is similar to the rates establishe for this journal in a previous stuy 12. Single-author articles, a large number of references, an publication in a journal with a low impact factor have all been ientifie as article-relate risk factors for citation errors 1,2,19. The impact factor is a measure of the number of citations of a journal s material ivie by the number of citable materials in the journal over a stanar perio of time 4. The original intent for the use of the impact factor was to allow comparison between the citation rates of journals 20. Although controversial, the impact factor is frequently use as a measure of the importance of a particular journal to its fiel 2-4. The current stuy establishe a significant inverse correlation between a journal s impact factor an the rate of citation errors, suggesting greater attention to citation accuracy by some combination of authors, eitors, an staff of those orthopaeic journals with higher impact factors. No other significant article-relate risk factors for the rate of citation errors were ientifie in the current stuy. The classic efinition of citation accuracy utilize in the literature is base on the bibliographic elements require for manual retrieval of a particular reference from a paper-base journal format. The relevance of this efinition is questionable in the current era with the wiesprea use of electronic literature search engines such as PubMe. Despite a total citation error rate of 26%, 98% (196) of the 200 references in the

6 1160 current stuy were easily retrievable, an only 1% (two) of the 200 references were impossible to retrieve through PubMe. Ease of retrieval with electronic search engines in no way justifies acceptance of citation errors in the biomeical literature, as not everyone has irect access to such search engines an manual retrieval of articles from oler or more obscure journals is still frequently require. Assessment of quotation accuracy in the meical literature has been relatively limite because quotation errors are more ifficult to efine an their assessment can be subjective 1,7. We were able to ientify only a single report in the orthopaeic literature evaluating the misquotation of a commonly reference han surgery stuy 9. Otherwise, there has been no systematic assessment of the quotation accuracy of the peer-reviewe orthopaeic literature, to our knowlege. The reliability of the assessment of quotation errors was optimize in the current stuy by the fact that two inepenent examiners performe the initial assessment an all isagreements were resolve by repeat review until a consensus was reache. The current stuy emonstrate a meian quotation error rate per orthopaeic journal of 38% for the years 2007 an 2008, a value that falls in the upper range of quotation error reporte for the biomeical literature 11. Although other investigators have ientifie an association between the number of references an the rate of quotation errors, we i not fin any significant correlations between the journal s impact factor or other article-relate risk factors an the rate of quotation errors 1. Quotation errors, which may propagate misinformation, are a serious problem that appears to transcen all bounaries within the biomeical literature. Reference accuracy is wiely recognize across the biomeical literature to be the primary responsibility of the author(s) 21,22. Because reference errors reflect poorly on the journals as well, eitorial processes have been evelope to promote reference accuracy 23. Prepublication eiting (also known as technical eiting) specifically, a review of references by eitorial staff has been shown to improve citation an quotation accuracy between submitte an publishe versions of the manuscript On the other han, supplying authors with etaile instructions concerning abstract preparation i not improve abstract quality 28. Each of the orthopaeic journals that we stuie provies etaile instructions concerning manuscript preparation an submission to its authors an is explicit in stating that reference accuracy is the responsibility of the author(s) Two of the journals (JBJS-A an JBJS-B) perform technical eiting for citation accuracy (consisting of irect verification of all references by eitorial staff). In the current stuy, only one of these journals (JBJS-A) showe significantly lower citation error rates than another journal (JPO-A) that eite for grammar an journal style but i not perform technical eiting for citation accuracy. In aition, it seems reasonable to believe that the wiesprea availability of, an presume increase use of, bibliographic software by authors, eitors, an journal staff woul improve reference accuracy in the biomeical literature 1,2,7. Comparable rates of citation error in JBJS-A over time (i.e., the rates in 1975, 1985, an 1995, as previously reporte in the literature, relative to the rates foun in the current stuy of 2007 an 2008 articles) suggest that the potential impact of such software has not been realize 12,13. Direct comparison of journal submissions in which the authors i or i not use bibliographic software woul be require to etermine the actual utility of the software. Improving quotation accuracy presents a more challenging problem. Technical eiting for quotation accuracy cannot generally be performe by non-meical-professional eitorial staff. Routine, selective screening of a portion of the references in a manuscript as a part of the peer-review process has been propose 2. This responsibility woul most likely fall on the unpai professional peer reviewers an woul greatly increase the time require for article review. Others have propose limiting the number an type of references (to primary sources only, with no inirect references or references to review articles) allowe in a manuscript 1,2,7. At this time, there is little available evience to support such suggestions. This stuy showe that citation an quotation errors are common in the peiatric orthopaeic literature. Reference accuracy continues to be a substantial problem in the biomeical literature espite recent technological avances such as online atabases, easily accessible search engines, an wiely available bibliographic software. Reference accuracy remains the primary responsibility of the author, for reasons of professionalism an personal prie 22,33. Authors an reaers alike woul be wise to hee these wors publishe over ninety years ago: Take no reference for grante. Verify the reference that your best frien gives you. Verify the reference that your revere chief gives you. Verify, most of all, the reference that you yourself foun an jotteown. To err is human, to verify is necessary. 34 n Jon R. Davis, MD Joye P. Emons, MLIS, AHIP Shriners Hospital for Chilren, 950 West Faris Roa, Greenville, SC aress for J.R. Davis: javis@shrinenet.org Daniel M. Weigl, MD Schneier Chilren s Meical Center, 14 Kaplan Street, Petah Tikva 49202, Israel Dawn W. Blackhurst, DrPH Greenville Hospital System University Meical Center, 701 Grove Roa, Greenville, SC References 1. Rey MS, Srinivas S, Sabanayagam N, Balasubramanian SP. Accuracy of references in general surgical journals an ol problem revisite. Surgeon. 2008;6: Fenton JE, Brazier H, De Souza A, Hughes JP, McShane DP. The accuracy of citation an quotation in otolaryngology/hea an neck surgery journals. Clin Otolaryngol Allie Sci. 2000;25:40-4.

7 Hecht F, Hecht BK, Sanberg AA. The journal "impact factor": a misname, misleaing, misuse measure. Cancer Genet Cytogenet. 1998;104: Kurmis AP. Unerstaning the limitations of the journal impact factor. J Bone Joint Surg Am. 2003;85: Buchan JC, Norris J, Kuper H. Accuracy of referencing in the ophthalmic literature. Am J Ophthalmol. 2005;140: Evans JT, Najari HI, Burchell SA. Quotational an reference accuracy in surgical journals. A continuing peer review problem. JAMA. 1990;263: Gosling CM, Cameron M, Gibbons PF. Referencing an quotation accuracy in four manual therapy journals. Man Ther. 2004;9: Lukić IK, Lukić A, Gluncić V, Katavić V, Vucenik V, Marusić A. Citation an quotation accuracy in three anatomy journals. Clin Anat. 2004;17: Porrino JA Jr, Tan V, Daluiski A. Misquotation of a commonly reference han surgery stuy. J Han Surg Am. 2008;33: Rastegar DA, Wolfe L. Experience, expertise, or specialty? Uses an misuses of a reference. J Fam Pract. 2002;51: Wager E, Mileton P. Technical eiting of research reports in biomeical journals. Cochrane Database Syst Rev. 2007:MR Goorich JE, Rolan CG. Accuracy of publishe meical reference citations. J Tech Writing an Comm. 1977;7: Sutherlan AG, Craig N, Maffulli N, Brooksbank A, Moir JS. Accuracy of references in the orthopaeic literature. J Bone Joint Surg Br. 2000;82: Ranom sequence generator. Accesse 2008 Aug Institute for Scientific Information Web of Knowlege. com/. Accesse 2008 Aug PubMe. Accesse 2008 Sep Journal Citation Reports. Journal impact factor. com/proucts/jcr/. Accesse 2008 Nov Billings J. International Meical Congress, Lonon, 1881: an aress on our meical literature. Boston Me Surg J. 1881;10: Lok CK, Chan MT, Martinson IM. Risk factors for citation errors in peer-reviewe nursing journals. J Av Nurs. 2001;34: Saha S, Saint S, Christakis DA. Impact factor: a vali measure of journal quality? J Me Libr Assoc. 2003;91: International Committee of Meical Journal. Accesse 2008 Dec Cowell HR. Ethics of meical authorship. J Bone Joint Surg Am. 1998;80: Wager E, Mileton P. Effects of technical eiting in biomeical journals: a systematic review. JAMA. 2002;287: George PM, Robbins K. Reference accuracy in the ermatologic literature. J Am Aca Dermatol. 1994;31: Hobma SO, Overbeke AJ. [Errors in literature references in the Neerlans Tijschrift voor Geneeskune]. Ne Tijschr Geneesk. 1992;136: Dutch. 26. Lowry SR. How accurate are quotations an references in meical journals? BJM. 1985;291: Asano M, Mikawa K, Nishina K, Maekawa N, Obara H. The accuracy of references in Anaesthesia. Anaesthesia. 1995;50: Pitkin RM, Branagan MA, Burmeister LF. Effectiveness of a journal intervention to improve abstract quality. JAMA. 2000;283: Journal of Bone an Joint Surgery (American Volume). Instructions to authors. Accesse 2008 Dec Journal of Bone an Joint Surgery (British Volume). Instructions for authors. Accesse 2008 Dec Journal of Peiatric Orthopaeics. Instructions for authors. ovi.com/jpo/accounts/ifauth.htm. Accesse 2008 Dec Journal of Peiatric Orthopaeics, Part B. Guiance for authors. ovi.com/jpob/accounts/ifauth.htm. Accesse 2008 Dec Cruess RL, Cruess SR, Johnston SE. Professionalism an meicine s social contract. J Bone Joint Surg Am. 2000;82: Place FJ. Verify your references. A wor to meical writers. N Y Me J. 1916;104:697-9.