All the News That's Fit to Digitize

The Serials Librarian ISSN: 0361-526X (Print) 1541-1095 (Online) Journal homepage: http://www.tandfonline.com/loi/wser20 All the News That's Fit to Digitize Brenda Bailey-Hainer & Sarah Sutton To cite this article: Brenda Bailey-Hainer & Sarah Sutton (2007) All the News That's Fit to Digitize, The Serials Librarian, 52:1-2, 67-78, DOI: 10.1300/J123v52n01_07 To link to this article: http://dx.doi.org/10.1300/j123v52n01_07 Published online: 17 Oct 2008. Submit your article to this journal Article views: 63 View related articles Citing articles: 1 View citing articles Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalinformation?journalcode=wser20 Download by: [46.3.192.95] Date: 19 November 2017, At: 22:08

All the News That s Fit to Digitize: Creating Colorado s Historic Newspaper Collection Brenda Bailey-Hainer Presenter Sarah Sutton Recorder SUMMARY. In the second Vision Session of the 2006 NASIG Annual Conference, Brenda Bailey-Hainer gave a presentation on the creation and success of Colorado s Historic Newspaper Collection. The presentation began with the details of the project itself, including a demonstration, and then continued with an account of the overwhelmingly positive user reaction to the project. She also described the funding model used to support ongoing work on the project and concluded with future plans. doi:10.1300/j123v52n01_07 [Article copies available for a fee from The Haworth Document Delivery Service: 1-800-HAWORTH. E-mail address: <docdelivery@ haworthpress.com> Website: <http://www.haworthpress.com>] KEYWORDS. Historic newspapers, digitization, Colorado Historical Society, resource sharing, Colorado s Historic Newspaper Collection 2007 by the North American Serials Interest Group, Inc. All rights reserved. [Haworth co-indexing entry note]: All the News That s Fit to Digitize: Creating Colorado s Historic Newspaper Collection. Sutton, Sarah. Co-published simultaneously in The Serials Librarian (The Haworth Information Press, an imprint of The Haworth Press, Inc.) Vol. 52, No. 1/2, 2007, pp. 67-78; and: Mile-High Views: Surveying the Serials Vista: NASIG 2006 (ed: Carol Ann Borchert, and Gary Ives) The Haworth Information Press, an imprint of The Haworth Press, Inc., 2007, pp. 67-78. Single or multiple copies of this article are available for a fee from The Haworth Document Delivery Service [1-800-HAWORTH, 9:00 a.m. - 5:00 p.m. (EST). E-mail address: getinfo@haworthpress.com]. Available online at http://ser.haworthpress.com doi:10.1300/j123v52n01_07 67

68 MILE-HIGH VIEWS: SURVEYING THE SERIALS VISTA: NASIG 2006 Brenda Bailey-Hainer, Director of Networking and Resource Sharing Colorado State Library Association gave a presentation on Colorado s Historic Newspaper Collection to a full house on Saturday morning, May 6 as part of the 2006 NASIG annual conference. Bailey-Hainer is well-prepared for her work with Colorado s Historic Newspaper collection, having worked for nine years in academic libraries and twelve years in library automation before taking her current position with the Colorado State Library Association in 1999. Bailey-Hainer has earned an MLS and an MN and is currently working on a PhD in Public Affairs. She also has extensive experience managing statewide technology projects that include collaboration between many types of libraries and cultural heritage institutions. She has worked on projects such as AskColorado (http://www.askcolorado.org/), a statewide virtual reference service and SWIFT (StateWide Interlibrary loan Fast Track, the Colorado Virtual Library s (http://www.aclin.org/) interlibrary loan service) as well as Colorado s Historic Newspaper Collection (http://www.coloradohistoricnewspapers.org). Bailey-Hainer received the Colorado Association of Libraries Librarian of the Year Award in 2001, was featured in Library Journal s March 2002 special issue on movers and shakers, and received the Colorado Association of Libraries Outstanding Technology Project of the Year Award in 2004 for her work with Colorado s Historic Newspaper Collection, the subject of this Vision Session. The Colorado s Historic Newspaper Collection project was originally funded in large part by grants obtained by the Collaborative Digitization Project in Denver, CO (http://www.cdpheritage.org/). Now that this funding has ended, the project is transitioning to a permanent home in the Colorado State Library (http://www.cde.state.co.us/ index_library.htm) which, it was felt, is an appropriate home for a state resource and where there is a full time staff to continue the project and support the project in the long term. Bailey-Hainer began her presentation by outlining the topics she intended to cover. She presented the details of the project itself, the user reaction, the funding model used to support ongoing work on the project, and future plans for the collection. PROJECT DETAILS Colorado s Historic Newspaper Collection is a single source of historic newspapers with statewide coverage. The goal of the project is to

Vision Sessions 69 digitize newspapers from each of the sixty-four counties in the state and make them available online, starting with the earliest published newspapers. Their long-term goal is to make the collection as comprehensive as possible. The earliest published newspapers to which they have access were published in 1859. Based on their surveys, they believe there are about 275 or 300 newspapers for which microfilm is available and that were published prior to 1923. Because of copyright concerns, the decision was made to initially limit the focus of the project to digitizing materials that were published before 1923 and are thus clearly in the public domain. However they are not opposed to doing longer runs provided they can be sure that the copyright is cleared. There were originally three partners on the project. The Collaborative Digitization Program, a nonprofit membership organization for libraries and other cultural heritage institutions, was the lead on the LSTA and IMLS grants with which the project was begun. The other two project partners were the Colorado State Library and the Colorado Historical Society. As of April 2006, the Colorado s Historic Newspaper Collection contains ninety-one newspapers published in forty-nine different cities around the state representing thirty-six different counties. A little over 315,000 pages of newspapers have been digitized and made available beginning with the oldest (1859) and continuing through 1928. Many of the newspapers from the earlier years were short lived, having been published in mining towns around the state that no longer exist. The newspaper that runs through 1928 was published in Swedish (because for a while Swedish was the state language of Colorado). They digitized the full run of this particular newspaper because it went out of business in 1928 and they were certain that the copyright had cleared. FEATURES Colorado s Historic Newspaper Collection uses Olive Software products for processing, the searching interface, and the database infrastructure. Olive s software allows users to select newspapers from a title list or from a customized clickable map of the state which Olive created specifically for Colorado s Historic Newspaper Collection. The creation of the clickable map was spurred by a need they recognized among users for a less cumbersome method for selecting titles from a list that was anticipated to grow to 275 titles. They found that this feature also supports users who are looking for information by region.

70 MILE-HIGH VIEWS: SURVEYING THE SERIALS VISTA: NASIG 2006 Once a list of regional titles is selected, the software allows the user to browse through an entire issue of the newspaper or to search through the text of the newspaper. This is possible because Olive had used a process called pipex, which is a sophisticated OCR process also called distillation. Olive indexed the full text of the newspapers so keyword searches on topics, names of individuals, businesses or corporations are possible. One of the best things about the Olive software, and part of the original vision for the project, is that it makes possible searches in an individual newspaper or in all ninety-one newspapers. At the same time that Olive created the clickable map for the project; they also embedded the capability of creating customized subsets of newspapers in which the user can then search. This ability has allowed the staff of the Colorado State Library to create collections of featured articles on specific topics. Once a search returns an individual article, that article can be viewed separately from the page on which it appears or as it originally appeared in the paper in its historical context. This feature is especially popular with historians. Because the digitization involves full imaging of each newspaper, they contain articles, advertisements, letters to the editor, and everything that appeared in the original including hand written notes and anything else that might have been part of the original paper copy when it was microfilmed. DEMONSTRATION OF THE INTERFACE Although there was no live Internet connection available for the session, Bailey-Hainer demonstrated the search interface using Power- Point slides containing screen shots. She mentioned that the first time a user accesses the site, they are asked about the speed of their Internet connection. The system then delivers either a higher or lower resolution version of the site. Users are asked this only once but they must have cookies enabled in their Internet browser. The homepage displays the clickable map. The map is divided into regions of the state. Once the user selects a region, a list of available newspapers for the selected region appears. This search mechanism and result is analogous to using the search region tab at the top of the page. The search is fairly robust; it allows combinations of keywords using Boolean logic. In her example, Bailey-Hainer selected the eastern plains regions. Four newspaper titles displayed. This is an example of how the creation

Vision Sessions 71 of subsets reduces the problem of long lists of newspaper titles she mentioned earlier. Within the eastern plains region, she searched for newspapers from a town called Wray and selected all four papers to be searched and a date range for her search. The results returned include the headline for the article which contains the search term. While displaying the results, Bailey-Hainer reminded the audience that newspapers were formatted differently in the 1800s than they are today. Sometimes there is no headline at all for an article and sometimes articles were strung together. This sometimes makes it hard for the software to determine where an article begins or ends. But given the layout of historical newspapers, they felt that having results returned in this way is still better than forcing users to hunt through an entire page or paper for the article they seek. Search results include the date, the newspaper title, and the city and/ or county of publication, the number of pages, the page on which the article appeared, the byline, and the section of the paper in which the article appeared. Some of the sample results did not contain data in these fields. Bailey-Hainer explained that the Olive software was originally designed for modern newspaper publishers and so contains modern conventions that were not in use in the nineteenth century. By selecting the headline from the results display the user can access the full text of the article which includes a pointer and highlighting that identifies search terms in context. She also pointed out the icon on the results page used to select a view of the article in the context of the page and the newspaper in which it originally appeared. Bailey-Hainer concluded her demonstration of the search interface by describing some of the additional methods for searching. The browse all tab at the top of the page provides a display of the list of all ninetyone newspapers in the collection including the town and county of the publication. The search screen allows users to narrow their search by selecting the year and month of publication. The results screen also includes a calendar that opens up to display the dates of each edition of the newspaper selected and in which the chosen article is highlighted. There are also fairly sophisticated tools for moving around within a newspaper issue. She noted that many of the newspapers covered not only local items of interest but national and international news as well. The featured topics tab is hand-created by Colorado Virtual Library staff, allowing them to highlight and provide in-depth coverage of a new topic (often an item of Colorado history) each month.

72 MILE-HIGH VIEWS: SURVEYING THE SERIALS VISTA: NASIG 2006 ACCESS Accessibility is a continuing challenge. They continually strive to enable users to find what they are looking for. Rather than users with specific citations (their expected audience), their experience is that users often want to browse through a newspaper, often without even knowing its correct name. Historic newspapers tended to change names frequently, which makes it difficult to find a particular paper even within a single reel of microfilm. Not only the newspapers but cities and even counties changed names occasionally, exacerbating the problem. The solution is a matrix linked from the homepage, called Titles Available in The Collection that allows the user to sort by title, city, county, language, and beginning and ending dates. Bailey-Hainer used several examples to describe the ways different communities and libraries provide access to the collection. At least one has cataloged the collection and provides access through their OPAC. Many others provide a link from their Web sites as does the Colorado Virtual Library, directing traffic to the collection from a link on their home page. A local library, the Auraria Library, has cataloged the collection as a series in their ILS. This allows patrons to conduct a title search on the Colorado s Historic Newspapers Collection in their OPAC. This search returns a list of newspaper titles that are included in the collection. She demonstrated this with a screenshot of the OPAC display and the results of selecting a title from the list. This way the patron can view a bibliographic record for a particular newspaper and use the link provided to access the Colorado s Historic Newspaper Collection s home page. Another challenge they face is their desire to link the user directly to the newspaper title they are looking for rather than to the Colorado s Historic Newspaper Collection s home page. One reason they have not succeeded in this is that, in creating the database originally, Olive catered to newspaper publishers rather than librarians. Publishers have completely different searching needs than do libraries, librarians, and library patrons. Another example of how libraries are providing access to the collection is Colorado State University Libraries who include a link on their databases by subject page. The Douglas County (Public) Libraries have included a link on their local and regional resources page. The Colorado Virtual Library, a single interface providing access to library catalogs around the state, includes a link to the site as well as a featured article that changes every week.

Vision Sessions 73 CHALLENGES WITH ACCESS One of the greatest challenges to access is the frequent title changes to which historic newspapers are prone (mentioned in the previous section). In order to accommodate this they added the name of the county and/or region in parentheses to the full title. Space considerations and the limits placed on them by the database infrastructure (which only allows one title per paper) require that they have only one title in each listing. Unfortunately, this does not allow for the inclusion of alternative titles. Similarly, county boundaries changed over time raising the question: which county do you use, the current one or the historic one? Then there are the ghost towns. A number of historic Colorado towns have gone extinct, many of which published a newspaper while the town was alive but which naturally went defunct when the town went extinct. The challenge this presents is how to let people know where they are when the towns do not exist anymore. Optical character recognition (OCR) also presents challenges to providing access to the collection. Pages containing broken type face or blurry type are difficult for the computer to interpret. Although the Olive search engine does some fuzzy matching, the software guesses when it is not sure of a letter or a word, which is sometimes helpful and sometimes not. Archaic or historic language is sometimes challenging to users who are unfamiliar with it. Key word searching requires that users bear in mind the historical differences in language. For example, the database contains articles written about the Civil War, but the user is unlikely to retrieve the desired results by searching for African American. Instead they would need to search for keywords Negro or colored. The size of the database is becoming a challenge for key word searches. Some common words return hundreds of results which can reduce the overall usefulness of a search. For example, some searches are not very useful because the search terms apply to several different entities. A search for Kit Carson, which refers to a person, a town, and a county, would return results that are too numerous to be useful. COLLECTION DEVELOPMENT Collection development was originally driven by the grants that initially funded the project. Now that the grant funds have been used up, collection development is driven by practical issues. The initial concept was to digitize full runs of historic Colorado newspapers starting with

74 MILE-HIGH VIEWS: SURVEYING THE SERIALS VISTA: NASIG 2006 the earliest available published issue. What they feared were donors who would want to support one particular part or run of a paper (a specific year or period) and wanted to avoid this since it complicates the process. They wanted to digitize as many newspapers as possible because so many of them no longer exist. Now that grant funds are gone they have been forced to be more particular about what they digitize. More often than not they now wait for a donor to come to them with an offer to pay for digitization of a paper from a particular community. They have achieved the best success with newspapers from small communities where the papers are short and published two or three times a week. They find it more difficult to obtain funding for digitizing daily newspapers from larger communities because of the increased expense. Bailey-Hainer does fund-raising by selecting communities in parts of the state from which they do not have any papers digitized. She is happy to demonstrate the collection for any group that shows interest. She has also found that they have many communities that are interested in funding digitization of more recent papers if only they could clear the copyright. Copyright is a more complicated problem than they expected at first. For instance as they digitize from microfilm, they have learned that occasionally the agency that paid to microfilm the papers held copyright to the microfilm. The Colorado Historical Society had an aggressive microfilming project in the 1960s (prior to the National Newspaper Project). The Colorado s Historic Newspaper Collection project leaders decided that this would be the one place where they would try to have a really comprehensive collection. The Colorado Historical Society has graciously allowed the Colorado s Historic Newspaper Collection project to use the negative film that The Colorado Historical Society created. But the Colorado s Historic Newspaper Collection project discovered that in some cases the Colorado Historical Society did not pay for the original microfilming. A good example of this is the Greeley Tribune. Based on examining the positive service copies they thought that they had a full run of the Greeley Tribune. But when they obtained the master negative films from the archive and compared them to the service negatives, they noticed on the positives that the first four reels were missing. When they went back to negatives, they realized that the opening frame stated that the filming had been done for the Greeley Public Library (and thus held the copyrights). But the Greeley Public Library does not exist anymore; instead it is part of a larger library district. They were lucky enough to find someone at the library district who had replaced someone who had

Vision Sessions 75 been around when the newspapers were microfilmed. It turned out that the film did not belong to the Greeley Public Library either but rather belonged to the Greeley newspaper. So they are currently negotiating copyright permission with the Greeley newspaper which, of course, has changed hands several times since the film was made. For newspapers published after 1923, additional copyright questions arise. For example, who owns the content and who owns the newspaper? If the paper were owned by a family there are issues of which heir owns the copyrights. They also have to take into consideration writers, cartoonists, photographers, and newswires who did not turn copyright over to the newspaper in which their articles were published. As a further example of the difficulties that can arise over copyrights for recently published materials, Bailey-Hainer described a recent workshop that they held that included two speakers on copyright. One speaker from ProQuest shared a slide (which Bailey-Hainer showed to the audience) that contains a dauntingly complex chart of the process that ProQuest goes through to clear copyright. The other speaker, from the Guggenheim museum, described an even more complicated process. Processing is also complicated. Master negatives are stored in the State Archives which requires a written request be made asking that film be pulled for the project s use (for which there is a fee). The Historical Society then picks up the film from the Archive and delivers it to the state microfilm duplicating center (they keep a master negative for permanent archival purposes). The duplicate copy is shipped to Israel where Olive s headquarters are located and where the digitization takes place. Shipping has become much more complicated since September 11, 2001. In Israel, Olive scans the film in at 300 dpi, does distillation in order to get the best possible image from each page and creates profiles for each newspaper. Bailey-Hainer noted that if they decide to use a higher resolution later on, the higher resolution material can be used in the same database and will be displayed using the same interface they are using now. The results are shipped back to Colorado on discs in XML format with copies of the.tifs instead of trying to ftp the files. The hardest part of this process is moving the physical microfilm around; the technology part is much less difficult. USER S REACTION The public really loves this project. During the first year of operation, and without much advertising, they had 1.3 million views of articles,

76 MILE-HIGH VIEWS: SURVEYING THE SERIALS VISTA: NASIG 2006 ads, and newspaper pages. This increased in the second year and they are already (as of May 2006) at 1.1 million views for the third year. They have about 3,500 unique visitors monthly. Bailey-Hainer describes their site as a sticky site because users tend to stay for 25-30 minutes when they come to the site. They are also utilizing a user survey that was co-created by Utah, Virginia and Colorado through an IMLS grant, which the states obtained together for newspaper digitization. From the survey they have learned that 52% of users are over sixty years of age and that another 45% are between forty and sixty years of age. From this they conclude that their visitors are people interested in family history, genealogists, and educators, most of whom are from Colorado, but some are from outside the state. FUNDING MODEL The Colorado s Historic Newspapers Collection project started with a $120,000 grant from LSTA which paid for the Olive license and a server with one terabyte of storage. The IMLS grant for $250,000 then paid for the first 96,000 pages of newspapers to be digitized and some focus groups (made up of educators, librarians, and researchers). The Collaborative Digitization Project conducted the focus groups. As a result of the focus groups and some training that the Collaborative Digitization Project did, teachers created some K-12 lesson plans that were mounted on the Web site and the flyers she shared with the audience. They have reached the point now where they are receiving more in donations than they had originally in grants, in fact, the project is now driven by contributions. In order to assist donors, they have calculated the cost of getting a really good image and OCR for a newspaper. They charge a one time $400 fee for that and $1.25 per page. There is a little bit of overhead built into those prices to cover indirect costs like adding another terabyte of storage. FUTURE PLANS They still have 2.25 million pages of papers published before 1923 available to digitize. They expect to add more newspapers monthly depending on the availability of funds.

Vision Sessions 77 Other future plans include adding more featured groups of articles and a history or biography of each newspaper in the collection. In fact, they are working with an author who has created such biographies which they are planning to add to the interface. In addition to this, they would like to create a time line of events in Colorado s history in which a user would see major events and be able to click on the dates they occurred to look at newspaper articles about it. Another project is to look at the feasibility of adding keyword searching (e.g., subject headings or alternative titles) because they anticipate that searching problems are going to multiply as the database continues to grow. An interface exists that would allow them to accomplish this but it is not very time efficient. Finally, in terms of improving interoperability, Olive is working on a version of software that is OAI harvestable and they would like to upgrade to that version. They would also like to make it Z39.50 searchable as well, so that they can integrate it with library catalog searching. CONCLUSION In closing, Bailey-Hainer said that this has been one of the best projects on which she has ever worked. The public began using it the moment it was made available, and the state library was flooded with phone calls. They continue to receive great anecdotes from users by e- mail about how people have used the database to find information about their ancestors. She really encourages other libraries to take on a similar project if for no other reason than to improve relationships with their communities. QUESTIONS FROM THE AUDIENCE: There was time at the end of the presentation for questions from the audience. The first question was regarding the fact that the National Newspaper Preservation Project includes historic newspapers being fully cataloged in OCLC. The enquirer asked if the Colorado s Historic Newspapers Collection has found and used any of those records. The answer was yes, they are aware of the existence of the OCLC records but they have found that the some of those records are not necessarily accurate so they are not always useful to the project. One frustration they have faced along these lines is that the Colorado Historical Society

78 MILE-HIGH VIEWS: SURVEYING THE SERIALS VISTA: NASIG 2006 Archive s typed inventory of newspapers does not match the Historical Society s hand written inventory and trying to reconcile the records was considered too time consuming. The second audience member noted that the Collection has OCRs of the text of the articles but not textual display of them. She wondered whether there were plans to add displays of the text of an article. Bailey- Hainer responded that she would write that down as a suggestion. Then she explained that the OCR is often less than perfect because of blurry text. This has held them back because even though they have access to the OCR text, they are prevented from changing it because of the way the software maps the articles on a page. Thus they have not done it yet, but it is a really good point and they should look into it. The final question concerned the duplicates of microfilm from which they do their scanning. In the questioner s experience duplicates are sometimes not as clear as the original. She asked if they had found that to be a problem, which they had not. Olive recommends that they use the master negative and since they are using the master she thinks they are getting a clear copy. Unfortunately, state policy does not allow them to ship the master archival copy to Israel for digitization. Because this was the case, they did not really pursue it. There is only one case when they digitized from the positive and that was because the negative was no longer in existence. She thinks they are getting the best possible quality that they can. As the session ended, Bailey-Hainer invited participants to contact her one-on-one either there in the session venue or by e-mail with any additional questions they might have. Then she thanked the audience for attending and the session concluded. CONTRIBUTORS NOTES Brenda Bailey-Hainer is Director of Networking and Resource Sharing at the Colorado State Library Association. Sarah Sutton is Serials/Electronic Resources Librarian at Texas A&M University-Corpus Christi. doi:10.1300/j123v52n01_07