Digitizing simplified

Transcription

1 Digitizing simplified Large-scale digitizing for archive research Marc Holtman Stadsarchief Amsterdam English version

2 Copyright Stadsarchief Amsterdam (City Archives Amsterdam) Author Marc Holtman Senior Public Services City Archives Amsterdam Contact:

3

4

5 Introduction The Archiefbank: a system for consulting and downloading archive material using the web All archive inventories of the Stadsarchief Amsterdam (Amsterdam City Archives) have been available for consulting on the website since This search system, thanks in part to the availability of the data to search engines like Google, was being used intensively immediately after its launch. Yet from the point of view of the user this system was limited: it is true that the inventories describe the content of the archives, but in order to refer to the items themselves it was still necessary to visit the reading room. This is too great a step for the average user on the web. The current web user expects to be able to consult the items in a system like this digitally, and if this is not the case that they can at least be ordered. After all, you no longer have to go out shopping for books and music (you can, but you don t have to), so why should you have to for archive items? For this reason in 2006 the SAA began to develop the Archiefbank, a system for consulting archive material and downloading it from the web. In fact the Archiefbank is an extension of a number of existing search systems on the website. The first version was launched in July 2007 with the addition to the search system of scans of the Family Registration Cards and 3,000 digitized inventory numbers. Digitizing at the customer s request The SAA manages 32 kilometres of archives. If every sheet has to be digitized this is an incredibly large amount of paper. So where do we start? Public favourites seem to be the obvious choice, but from research into statistical data from all the requests for originals over the last five years it turns out that there is no such thing. The number of requests in this period was for only a fraction of the total of inventory numbers that are potentially requestable. There is obviously not much point digitizing inventory numbers where at present in any case there is no demand. The starting point in the Archiefbank is therefore to get the user to prioritize the digitizing. In practice this means that all public inventory numbers can be requested for digitizing via the search system. In principle all requests are honoured, unless when assessing the items it turns out that they cannot be digitized for material reasons, or they are subject to copyright or have disclosure restrictions. This why we speak of a request for digitizing in the search system and not an order.

6 How long will it take to digitize the entire Amsterdam City Archives? Making 10,000 scans each week in a total of 32 kilometres of archives and assuming 7,000 scans for each metre, it will take 431 years to digitize all of the Amsterdam City Archives. And this does not allow for vacation periods, moving the archives and growth because of the acquisition of new items. 500,000 scans of archive material from the SAA repositories were made in the pilot project. This is 0.2% of the total. However, as the number of requests is only around 5,000 to 10,000 scans, it can be concluded that there is no backlog in digitizing, and the objective that the customer is able to consult items digitally has been achieved. Conditions for consultation using the web: low cost and short delivery time Given the success of online inventories, it may be expected that consultation and digitizing on request meet a need. But the rates also have to match up to the expectation of the customer. From experience of the supply of reproductions in the reading rooms and from other search systems on the website it emerges that customers think a low price is important. This means that both the incidental and structural costs for digitizing have to remain low. Seeing that an average piece of archive research easily runs into dozens or even hundreds of documents, the purchase of large quantities of scans will still have to remain affordable too. An inventory number with 100 scans should not cost 100, because then the financial threshold usually becomes too high and the average researcher is better off coming to the reading room to get copies made there. When establishing the order of magnitude of the cost of scans, the price of an ordinary copy has to be the benchmark. This obviously has implications for the reproduction process: the average cost price of a scan that, for example, is made for the Beeldbank (Image Databank) is far greater than the price of a copy in the reading room. In other words an entirely new reproduction process will have to be developed, with large scale and quality geared to the objective as its key concepts. Only then will it be possible to digitize at low cost. In addition the delivery time for digitizing on request will have to remain short. The user will understand that producing the scans will take some time, but the research should not have to be planned weeks in advance. A delivery time longer than two weeks is really no longer acceptable. The aim of the SAA is therefore a delivery time of one to a maximum of two weeks. This can only be achieved within a streamlined, efficiently organized, large-scale reproduction process.

7 Set up and definition In 2006 the SAA published the report Digitalisering Ontrafeld, (Digitizing Unravelled). This report is an account of the research into the technical aspects of large-scale digitizing of archive contents. One of the important conclusions from this research was that storage of image files as uncompressed TIFF files for scans for archive research is not necessary. Moreover by using powerful lossy compression in files of 200 to 300 dpi (actual size) the textual information can still be very easily read and the vulnerability of the files is no greater than when they are stored uncompressed. The advantage in file size is so great that the structural costs for storage of uncompressed image files and for fast availability in good and flexible quality are higher than the costs of making new scans if this should prove necessary. Pilot project This report and the reproduction process described in it are based on results and conclusions from a pilot project in which the SAA had approximately 500,000 scans of different types of archive material made, partly based on our own selections and partly on requests from customers. In the pilot project it quite quickly became obvious that the available guidelines and best practices for digitizing often take no account of the enormous number of scans to be produced when digitizing archive material. On the basis of results from several scan trials, and in close cooperation with a digitizing partner, the SAA has therefore developed the reproduction process described in this report. This report will certainly not be the last word in large-scale digitizing. The SAA s decisions make large-scale digitizing possible within the SAA, but other options are certainly conceivable, depending on the size of the archives, the method of organization and the available facilities. In addition technological developments are happening so quickly that some options will be out of date in the not too distant future. This second report starts where Digitizing Unravelled left off and describes the organization of the reproduction process for large-scale digitizing of archive material. The report is split into two sections. Part A sets out the basic conditions required to make digitizing possible at low cost. Part B describes the specific steps in the reproduction process set up by the SAA. We have sought to describe the basic principles and steps in the process as concisely and generally as possible so that the basic principles and steps can be applied elsewhere. The emphasis in this report is on the decisions and choices made by the SAA with a view to reducing costs. The end product of the processes described in this report is a collection of scans. This report offers no guidelines for the organization of a Digital Repository. Nor, since the scanning of the documents is contracted out to an external partner, does it go into the technical set-up, hardware, settings and software for the scanning itself.

8 Aim of this report The pilot project has been particularly useful for the SAA and has resulted in a complete reproduction process, fully tested in practice. This means that the SAA is starting the next phase with absolute confidence, and dealing with the customers requests for digitizing will become a fixed part of everyday activities. One condition of the subsidy granted by the Mondrian Foundation was that we should share the knowledge we acquired with colleagues in the field. The SAA was wholeheartedly in agreement with this, and we hope that this report will contribute to the spreading of knowledge and experience of digitizing archive material on a large scale. We think that the choices made by the SAA will be widely applicable.

9 Part A: Basic Principles 1. Digitizing for archive research 2. Digitizing supports conservation and security 3. The quality standard is legibility 4. Working on the basis of existing description methods and no addition of new metadata 5. Identification of the units to be digitized base don order numbers 6. Files names of the scans are meaningless 7. Always scan the complete inventory number 8. Material preparation as simple as possible 9. The originals are not numbered 10. No fragile material or material packaged in an unconventional way 11. No copyright material 12. No storage of separate, uncompressed files 13. What is digititized is available to all users 14. Contracting out the scanning to an outside partner 15. Constant production 1. Digitizing for archive research Purpose of the scans The higher the quality standards specified, the higher the costs of producing and storing the scans. In order to keep costs low it is therefore prudent to allow the standard of quality to follow from the requirements that the end user places on the scans, and not to assume more than that. A level of quality higher than necessary for the purpose has no added value for the customer; all it does, inevitably, is push up both incidental and structural costs, and this makes no sense within the framework of the Archiefbank. Standard of quality The purpose of the scans produced in this reproduction process is archive research using the web, straight from the screen or in print. The standard of quality the scans must meet is that textual information that is legible in the originals must also be legible in the scans, that is to say when it is displayed on an ordinary screen or printed on an ordinary printer. Reproduction of details which are not part of the textual information, such as the structure of the paper, is not required. The standard required for colour reproduction is not particularly high. Obviously red has to stay red, but whether the shade exactly represents that of the original on the date of reproduction is of less importance. It gives no added value at all to the reading of the information, and anyway most screens and printers vary so much that to the user at home even the most perfect scans will never look the same as the original.

10 Document types The documents that are being digitized in this reproduction process can have the following forms: - small and large size (up to the size of an open newspaper) - bound and loose-leafed entities - card indexes - old and modern material - low and high contrast documents - text alone, text and image together - hybrid forms The same standard of quality applies to all documents, so when an inventory number contains images legibility of the visual information will still be the norm. This is obviously a somewhat difficult concept to interpret with photographs, for example, and can therefore lead to discussion, but the basic principle is that in every case a complete reproduction of the subject of the photograph ought to be possible. When details in the background could be lost as a consequence of the relatively low standard of quality for images this does not really matter. On the face of it this seems to be a somewhat strange assumption (surely you want to see everything?), but it does actually follow from the use of the original in archive research (and that is the purpose of the exercise). Firstly the album is skimmed through to get an impression of the content, and then a number of photographs may be selected for reproduction in high quality and scans made for publication or framing. The costs of these reproductions are therefore considerably higher than for an ordinary photocopy from the album. 1 Obviously skimping on the quality of scans of images (it can be better) is purely an economic decision, not one taken on principle. At present the costs of digitizing to photograph quality are on average ten times higher than scanning to the standard of legibility. If these costs were to be passed on the customer, digitizing a photograph album, for example, would become so expensive that it would be beyond the reach of many people. The experience of the SAA is that in the reading room the average user opts for an ordinary copy, possibly in colour, and only occasionally do people place made-to-measure orders for high quality reproductions. If digitizing is being done with another objective, for example scanning photographs for inclusion in the Beeldbank and providing high-quality reproductions, different requirements will have to be set and an entirely different reproduction process may have to be used. The form in which the scans are supplied will generally be different too. In an image databank, for example, you can choose a selection of a number of photographs from an album; these can be cropped as individual scans, and described separately. However when this is digitized for the Archiefbank the complete album will always be done, including the front and back of the album; the quality will be average, but the customer will be able to afford it. 1 In addition to digitizing for the Archiefbank one-offs will always be available. Reproduction of a document (or a cutout from it) to a high standard will therefore remain an option. The costs are naturally considerably higher. For the time being the reproduction will only be made for the customer concerned and will not be saved afterwards. This will only happen if and when the SAA digital repository facilities support the saving of several reproductions of the same document in different qualities.

11 From this it can also be concluded that the distinction between a request for images (Beeldbank) where the customer usually only buys a few scans but demands high quality, and a request for archive material (Archiefbank), where legibility is what counts and where sometimes dozens if not hundreds of scans are being purchased at a time, is without a doubt logical and justified. But this can mean that the original qualifies for both systems. At the SAA it works like this: if these originals are being digitized for the Beeldbank on the basis of our own selection process, the standards for images apply. If they are being digitized at the request of the user Archiefbank standards apply, the argument for this being the manageability of costs for the customer. If an inventory number digitized for the Archiefbank is to be recorded later in the Beeldbank it is digitized again. 2. Digitizing supports conservation and security The scans are made for the purposes of archive research by the user and not as a substitute for the originals. Conservation of the original remains the major concern. Digitizing archive items for the purpose of use, however, does have a real conservation function because the originals, in principle, no longer have to leave the repository. Damage or loss through use is ruled out. 3. The quality standard is legibility The objective of scans of archive material is to be able to read the textual information. The standard of quality is phrased as follows: Reproduction of all significant details (a dot on an i should be recognizable as such) The basic principle of this is that the scan should be easily legible on a screen as well as in print. A consequence of large-scale digitizing at the lowest possible cost is that hardware and software have to be adjusted in such a way that a wide range of document types can be scanned without constantly having to change the set up. This means, for example, that when an inventory number contains modern printed matter and low-contrast handwritten text on carbon copies, all the documents will be scanned on one setting. This produces an average quality, which in practice proves to be more than adequate for the purpose i.e. reading. The quality of the scans is determined visually, and not, for example, by scanning a checking card with each scan. The requirements are not so high that this needs to be checked, and this may also not be a component of the scan from the point of view of use. The SAA s experience is that the reading of the text in the scans by the customer is a reliable and efficient checking method. If there is any doubt the scans are visually compared with the originals and if necessary scanned again.

12 A case study: how legibility is perceived In the pilot project a series of inventory numbers were scanned at the request of a historical society and then printed in colour. A couple of days after the delivery of the prints they brought the package back: they couldn t read them. After comparison with the originals it did indeed turn out that both the prints and the definition of the scans on the type of screen that was being used by the customer at home had much poorer contrast and were therefore far less easy to read than the originals. In a technical respect the scans were entirely in order: details, colours and shade reproduction were excellent. The legibility was then improved by increasing the contrast (by means of adjustments in the curve) between the text and the paper, and prints were made again. The gentlemen did not mind coming back to Amsterdam a second time to pick them up, they both still had one free train ticket left, and in their opinion this was cheaper than the costs of postage. When they looked at the new prints in the archive they were completely satisfied with the result. This case is an example taken from several reactions from customers to prints and scans supplied in the pilot period. The most important conclusion from this is that what is perceived as legible is a personal matter. In general it can be stated that a somewhat harsher image provides greater legibility. As a result of these conclusions the Archiefbank application has been given a specially developed document viewer with which the user can himself set the level of contrast with the aid of a slide control.

13 Digitizing Simplified In practice: Contrast slide control in the Archiefbank Normal Contrast level 50% Contrast level 100% 9

14 4. Working on the basis of existing description methods and no addition of new metadata In all cases the unit for digitizing is the existing description unit, that is to say an inventory number. The full inventory number is always scanned too. After scanning a link is made between inventory numbers and scans using the management system. This is a relationship of 1: more. There is thus no direct relationship between scans and the originals. In other words, you cannot see from the originals which scans belong to them, and you cannot tell from the scans which archive / inventory number they relate to. In all cases the relationships are dealt with in the management system. The file names are meaningless (see 6, page 13) and the originals are not numbered (see 9, page 16). Furthermore it is a rule that no new metadata, such as page numbering or indications of recto or verso, are allotted. Even if an inventory number still has a physical hierarchy, for example with staples, paper clips or a paper cover, the scans concerned are not grouped. In theory this can happen, by adding extra metadata at the level of file names, but in practice this is 1. labour intensive, and 2. sometimes incredibly difficult as the reality is not so simple. The question is what the added value of an addition like this is at metadata level. Because it is usually possible to see on the scans themselves even in thumbnail format what any sub-units are.

15 Apart from the file name, the only thing that is recorded is the sequence of the scans. This is a separate metadata element, and by definition is not evident from the file names. The basic principle is that the sequence of the scans is initially the same as the sequence of the originals. But afterwards if desired the scans can be put in any chosen sequence. What is more, the fact that the sequence is recorded in a separate metadata element and not in the file name also makes it possible to correct an incorrect sequence. Units from uninventoried archives (still) have no descriptions and so cannot be digitized with this reproduction process. If for some reason something from the uninventoried archives does have to be digitized, this is regarded in principle as made-to-measure and goes down a different route. 5. Identification of the units to be digitized based on order numbers A unit to be digitized must be able to be identified at each step of the handling process. The existing and physically present combination of an access number / inventory number should in theory be able to be used for this, but because of inconsistencies in the two numbers this does not provide a truly consistent (and hence clear) label. As a consequence of differently applied forms of labelling and the use of different types of labels the identification of the originals based on these two existing numbers can be difficult in practice. In any event there should be a simple way for the digitizer to be clear what he is dealing with, and no doubt should exist about the units to be digitized. The SAA therefore decided to give units to be digitized a unique, meaningless order number. This number is used throughout the entire process and forms the basis for: - communication with the digitizer - scanning - attaching file names - computerized registration of the file names in the management systems - billing Order numbers are issued automatically through the internal SAA management systems. In principle this can also be done elsewhere, but because the entire pathway up to registration of the file names in the management systems is grafted on to these order numbers it is more logical and more secure as far possible to place this at the start of the handling process. All units to be digitized are given an order ticket showing the order number (see an example of an order ticket in appendix 2).

16 6. File names of the scans are meaningless The basic principle at the SAA is that file names may not contain any meaning that signifies content. For example, file names do not include an inventory number, archive number or other types of information that could be subject to change in practice. A change in this information would in fact mean that the file names of all images concerned would have to be changed. This also goes for image files stored offline on DVD. Practically speaking this is not achievable. Prior to scanning in contrast to what is often the standard when digitizing images the originals are not given the file name item by item, for instance by way of a barcode. Nonetheless when scanning it has to be obvious which file names should be attached, and it has to be possible to register this automatically to the relevant inventory number in the management systems. The most obvious option is to register file names before digitizing, but this has a practical drawback because it is never certain in advance how many scans there are in an inventory number (the number of sheets is not necessarily the same as the number of scans). The SAA has opted for a solution where the order number is part of the file name. A file name is always made up of twelve characters, the first six being reserved for the order number. The next six characters contain a serial number starting with Prior to scanning, an order ticket is added to each of the inventory numbers, on which, among other things, the first file name is given (in barcode form), for example: A (order number A02043, scan 1) After scanning the first sheet the number is increased. A random file name, for example, looks like this: A (order number 5240, scan 342) Strictly speaking a range of possible file names can be attached to each order. This is more than adequate, and in practice even excessive. An inventory number never contains that number of scans. It would therefore be possible to use fewer characters. Nevertheless this large range is useful when, for example, a large series of card systems is being scanned in one order. Obviously in the majority of cases it does mean that a huge number of potential file names will never be used. If order A00386 only contains one scan this means that the file names in the range A up to and including A will never be used. This does not matter, because the number of possible file names that can be allocated remains large enough. If order numbers have been filled, the letter A in the file name can, for example, be replaced by a B (starting at B ), and so on. The significance of the order number in the file name disappears at the moment that the file names are registered in the management system. Therefore it is a matter of meaningless file names, so at a later point in time the images can be attached to other inventory numbers without any problems, or spread over a number of inventory numbers. The sequence can also be changed without having to amend the file names.

17 In practice it has been proved that this system works extremely well. The chance of errors is minimal, and the processing of the scans after receiving them can be done without complex link tables. 7. Always scan the complete inventory number By definition the entire inventory number is digitized, never just a selection of pages. There a few simple reasons for this: - The costs of scanning are not so much a factor of the quantity, but rather of the manual processing involved in it. Therefore it is most advantageous to scan an inventory number in its entirety at one time. - If only a part is being digitized, it has to be indicated in the originals or in the metadata (which can only be done if the originals have been numbered continuously, and this is usually not the case) which part should, and which part should not be digitized. This is a time-consuming task in the preparation, and makes the scanning process particularly complex, as it must be clear without any doubt what should and what should not be scanned. - When scans are shown in the web application behind an inventory number the customer expects everything to be there. It is entirely illogical if this is only a part of the total. - The entire preparatory process has to be gone through once again when the nonscanned pages have to be digitized later. The practice: to scan or not to scan empty pages? An exception to the rule that the complete inventory number is digitized is the scan instruction that empty pages are not digitized. However from experience this instruction is only meaningful when it is immediately obvious that only some of the pages are written on. In the situation where only some of the pages are empty it becomes quite difficult, because the scan operator has to be on the alert for the appearance of a page with text on it. In these sorts of cases it is better and sometimes cheaper to simply scan the empty pages. The customer can decide for himself later whether or not to purchase the empty pages.

18 8. Material preparation as simple as possible The basic principle is that an inventory number is digitized as it is found in the repository. Only a rough check of material and content takes place there. Likewise the sequence is not checked in detail. The instruction to the digitizer is to keep the sequence of the originals when scanning. Should it turn out later that the sequence is wrong, then if need be the scans can still be put into the correct order. To remove or not to remove staples: a recurrent issue in the pilot project One of the first scan instructions was that staples, paper clips and the like may not be removed. The advantages of this are the speed in the process of preparation linked to the rapid delivery to the customer, and the internal connection is immediately obvious. In the first round of tenders from the various digitizers this instruction caused a great deal of concern. Because how do you do this? Pages always have to be turned over, and this is more difficult with a staple than when everything is in a loose-leaf pile. One candidate had found the creative solution of removing them and later putting in new staples through the original holes. In discussions internally about the progress of the pilot project and the scan instructions the removal or non-removal of staples was a constantly recurring topic of conversation. By now scanning has been done for a year in accordance with this instruction, and the digitizer is able to comply with this requirement. But after evaluation with the digitizer it has emerged that not removing staples causes complications: - pages always have to be turned over and folded back, which does not help the speed of scanning - it makes cropping scans difficult - it damages the glass plate It is therefore worthwhile once again weighing up the costs of removing the staples against the extra cost for scanning. The removal of staples obviously has implications in terms of content because the physical integrity is undone. On the other hand the removal of staples is better from the point of view of conservation, and is also standard in the present guidelines for making an inventory. For the time being the instruction remains in force, but we have not heard the last of this issue

19 Archiefbank employee checking the originals 9. The originals are not numbered Numbering the originals prior to digitizing has the advantage that the completeness of the scans compared to the originals and the completeness of the originals (at the time of numbering) can be guaranteed. But this is only true if the numbering tallies exactly. An error in the numbering is far worse than not numbering at all. If the next number is not in order, because, for example a number has been accidentally missed out ( 9, 10, 12, 13 ), this will always create problems. It may be possible to deduce from the content whether or not the number concerned was omitted, but obviously this cannot be the method used. And then again, if the completeness can be inferred from the content, is numbering really necessary? Numbering of originals therefore chiefly has added value in loose-leaf systems, where it cannot be concluded in any way whether the file is complete or not on the basis of the content. This, for example, is the case with chronologically or alphabetically classified card systems. If card number 10 relates to Baker, card number 12 is about another Baker, but card number 11 is missing, then the immediate conclusion is that card 11 has been removed from the file for some reason. However, the number could well have been inadvertently missed out during the manual numbering of the cards. Numbering must therefore be done accurately and thoroughly checked, but that makes a numbering operation particularly expensive, and in the case of large quantities essentially unworkable.

20 Case study: a trial of numbering the originals At the end of 2005 the SAA carried out a trial by numbering 20,000 market cards. This was a loose-leaf, alphabetically classified card system. The condition of the material was reasonable: size approximately A4, reasonably thick card, but sometimes frayed or torn at the top through use, so that when workers leafed through the stack two cards could easily stay stuck to one another. Three temporary employees were occupied with this numbering. Quite quickly the conclusion was reached that all three made frequent mistakes: - Jumps: from 1234 suddenly to Numbers allocated twice (card 34 followed by 34) - Forgetting to number cards (particularly because of frayed top edges) Furthermore the discovery of a mistake was one thing, but correction of the mistake would have meant that all cards numbered after the mistake would have to be numbered again (in other words: erasing an number that had already been allocated and then applying a new number). This was obviously not practical to do, and also risked damaging the originals. Another practical aspect is that the numbering should not take too long, and so three people numbered at the same time. Because this was one consecutive whole, it was impossible to give each of them series of numbers that would ultimately form a consecutive series again without destroying the original alphabetical order. The SAA therefore concluded that originals should not be numbered prior to scanning. It is hard, time-consuming work prone to mistakes, and the added value by comparison to not numbering is small.

21 The added value of numbering compared to not numbering is slight. The original file can always be referred to if there is any doubt about the completeness of the scans. Furthermore digitizing leads to a high level of certainty that the original file is complete and remains so: the originals no longer have to be used. The question therefore is why you should still want to number them. Obviously there are all kinds of what if arguments (what if someone removes a card? what if a card disappears during scanning?), but then the argument for numbering verges on paranoia: what if a card disappears during numbering? And is the checker of the numbering reliable? Digitizing, even though it is technical, is essentially the work of people, like all the other activities in the archive. There will always have to be a certain amount of trust, otherwise the work of production will deteriorate into checking the checker of the checker, and that is no good in any setting. And then again: even were the original file to be destroyed after digitizing, the added value of numbering is still only limited. In this case numbering is usually done in order to be certain that the file of scans is complete compared to the original file. However alternative solutions to guarantee this are also possible, and sometimes better, for example by scanning twice. Apart from the above example, the SAA has scanned a card system of 1,000,000 cards, where the destruction of the originals really was the point of departure. Faultless numbering of 1,000,000 items is an impossible task. In this project the completeness was therefore guaranteed by scanning all the cards twice (see page 52 for a fuller account of this project). Obviously the completeness of the digital file then has to be guaranteed, but this invites quite simple technical solutions in a digital repository environment, and is not achieved by looking at pencil numbers on the scans.

22 10. No fragile material or material packaged in an unconventional way Material like charters that is very fragile and therefore requires special handling when packaging, transporting and scanning falls outside the reproduction process described in this report. Items like these can therefore not be digitized on request. The scans will of course be offered through the Archiefbank if they are going to be digitized in a separate project (for example the oldest charters from the Iron Chapel in the Oudekerk in Amsterdam were digitized in a project like this and then entered into the Archiefbank). There are also items that are packaged or bound in unconventional ways which rule them out for large-scale digitizing. Examples of requests which could not be digitized It is often impossible to see straightaway from the inventory description that it is this sort of material. When requests for digitizing are received, the inventory number is therefore assessed in regard to a number of points prior to scanning. If it turns out that the inventory number does not belong to the standard category then the request is cancelled and the inventory number is given the status not available in digital form. However, not available, does not by definition mean that there will be no digitizing. In principle, made-to-measure will always be supplied, but then rates are charged on an hourly basis and the digitizing takes place within another reproduction process.

23 11. No copyright material The basic premise is that inventory numbers that contain items covered by copyright are in principle not supplied through the Archiefbank. This basic premise always requires interpretation and a risk analysis. One example is the situation where there are some modern photographs in an inventory number that in itself is free of title. If the photographs are by Ed van der Elsken, from the ANP or by other photographers or press agencies, then it is clear who owns the rights and digitizing will not proceed. However it is a lot more difficult when it involves an inserted newspaper photograph or material that may have copyright on it but where it is not known who the copyright holder is, or if there is still a copyright holder alive. In those situations the possibility of digitizing is looked at on a case by case basis. If the basic principle is employed too strictly a comparatively large amount of (modern) inventory numbers would not qualify for digitizing, and that would be a pity, not least for the copyright holders themselves. Experience with the Beeldbank also shows that making material available on the web is highly likely to flush out any copyright holders there may be. Experience also shows that when a copyright holder comes forward agreements can usually be made quite quickly. Mention of the name of the copyright holder is often all that is needed Furthermore the scans in the Archiefbank are not distributed free on the internet: in order to look at the scans an account has to be opened and the scans must be paid for. The quality of the scans is based on legibility, so a scan of a photo is not usable for printing, for example. In difficult cases splitting an inventory number into a copyright number and a free of copyright number can be a solution, but this is labour-intensive.

24 12. No storage of separate, uncompressed images The conclusions from an investigation into the technical aspects of large-scale digitizing are fully described in the report Digitizing Unravelled. For the time being the SAA is opting for storage in JPEG format, with a compression factor of 10 (Photoshop). This results in an image with: - first-rate reproduction in detail and colour, and hence - an extremely legible image on screen and in print, and - it provides enough flexibility for manipulation for the purposes of functionality in the Archiefbank document viewer Permanency of the files is guaranteed by redundancy in storage. For the moment the files are saved internally and on the web server. When the images are uploaded into the web application automatic derivatives are made for the purpose of: - zooming in on the various layers - allowing the individual user to determine the degree of contrast These derivatives are only on the web server. The image that the user is able to download when the highest quality is selected (and no use is made of the contrast regulator) is the same as the master image supplied by the digitizer. In due course it can be expected that these standards will be adapted with the introduction of new storage and compression formats. 13. What is digitized is available to all users All digitized inventory numbers are available to all users through the web application. This also applies to inventory numbers that are digitized at the request of the user. So digitizing on request is not done exclusively for the person who requests it. Communication about a request can therefore be kept to a minimum. Making the request is enough, there is no quotation made that would only slow down the process. If it turns out that the person who made the request does not want to buy it after digitizing this is not a problem because the scans are immediately available for other users anyway (the person who made the request does not even have to be the first buyer).

25 14. Contracting out the scanning to an outside partner Contracting out the scanning It is true that the SAA has professional facilities in-house for analogue and digital photography, but these are not designed for large-scale digitizing of archive material. The complexity of the type of material to be digitized calls for specialized scan set-ups, hardware, software, knowledge and technical infrastructure. Investing in this only makes sense where there is very high production in a commercial set-up and it can be organized on a large scale. Contracting out of this process was therefore a logical choice for the SAA. Large number of scanning companies There are a great many scanning companies with a wide variation in charges, scales of fees and approaches. It was established that many scanning companies do have experience in bulk processing but not of the degree of complexity and diversity involved in large-scale scanning of the sort of archive material managed by archive services. Example of a diary. See appendix 4 for an overview of the result after scanning Intensive collaboration produces the lowest rate In order to get the scanning of large amounts of material done quickly the processes of the archive service and digitizer will have to dovetail, and the operations will have to be integrated into the process at a logical moment. For instance, checking of completeness (scans compared to originals) is an essential step in the reproduction process. The most logical thing is to get the digitizer to do this immediately after scanning and not to wait until the originals come back to the archive service, as only then can remedial action be taken quickly and efficiently on discovery of missing scans. However this calls for a high degree of trust, and means that contracting out the scanning is more than just awarding a contract to a supplier.

26 15. Constant production A large-scale reproduction process can only be organized effectively when constant production is assumed. It is then clear to the archive institution how many hours work a week is involved in preparation and finishing, and the digitizer can plan the best use of scanner tables and personnel. The basic principle of the SAA is to work with fixed weekly batches of 10,000 scans. If too few scans are being made from customer requests this is complemented with our own selections. When there are too many scans, some of the requests will be sent with the next batch (the delivery time to the customer is of course longer). Case study: production of 10,000 scans per week In the experience of the SAA processing a batch of approximately 10,000 scans per week is quite feasible, both for the process in house and for the digitizer. This is an average of 200 inventory numbers per week, or around 1.5 metres of archive. The preparatory work lies primarily in the extraction of the inventory numbers from the repository. Certainly when it is being done on the basis of requests by the customer, or when our own selections are made on the basis of themes and subjects, it can mean that the inventory numbers are spread over all kinds of archives. This makes it difficult to evaluate the time needed for preparation. The experience of the SAA is that the preparation time for an output of 10,000 scans in a weekly transport cycle takes up one working day, and the return after digitizing around half a day. Example of a weakly selection of scans ready for transport

27 Case study: how many scans are there on average in an inventory number? There is a graph in appendix 3 in which: - the horizontal axis shows the total number of scans in an inventory number, grouped in steps of 25 scans - the vertical axis shows the number of inventory numbers (or orders) that fall within the step in question It becomes immediately clear from the graph that a relatively large number of inventory numbers contain 0 25 scans.

28 Part B: The Reproduction Process Charting and writing out the reproduction process The reproduction process can only be done efficiently if it is obvious: - what has to be done - in which framework - by whom - and when Writing out the process is important because the product scans is a sizeable and abstract entity that will run fully computerized through the process from scanning storage transport the registration of file names. There is nothing to be read from the files themselves, and there are simply too many to look at. What the digitizer provides is in fact nothing other than a medium of transport with a large quantity of files in it. The scans only get a substantive meaning at the moment the user opens them from an inventory number. And then the idea is that the correct scan is displayed straight away. If there are deviations from the conditions set in the process (e.g. an order is produced in a way other than through the procedure), then it goes irrevocably wrong. As a rule it has been shown that analyzing and rectifying a mistake like this afterwards is labour-intensive, and it is therefore better to avoid it in the first place. It is an illusion to think that all possible situations that can arise will be described in advance. This is not a bad thing, as long as there is strict observance of the rule that there is no digitizing when a situation crops up that is not described in the reproduction process. Obviously adaptation or expansion should always remain possible.

29 Case study: what happens if there are departures from the conditions within the reproduction process? A condition in the reproduction process is that only inventoried archive material can be digitized. But of course it did happen in practice: a volume of archive material that had not been inventoried was sent for digitizing. An order number and order ticket were properly made out so the volume could go through the process normally. Surely you can do it just this once? The only trouble was that after scanning it became clear that this volume was part of a delivery of 20,000 scans at the same time. And then the question was: which one was it? Of course this can be worked out from the order number, but this is still inconvenient and time-consuming. Furthermore images are stored with file names that cannot be registered in the management systems, because there is simply no inventory number to link them to. It is true that the images are on the server, but they will never be consulted. This does not matter that much, but during periodic checks on completeness these scans will always surface as problems, which is extremely annoying. In a case like this the best solution is just to destroy the scans (something that is never entirely successful when the images are stored offline too, for example on DVD). In the development of the working processes the SAA, where possible, has therefore endeavoured to build in automated checks in the reproduction process. For instance, the above example has led to a change in the process so that order numbers are only issued by the management systems; firstly checked automatically to see if the unit to be digitized meets the condition that it has an access and inventory number. If it does not no order is issued. In this set-up an item that has not been inventoried cannot be introduced into the reproduction process.

30 Reproduction process: overview Legenda Requests for digitizing Analogue Selectionfile Selections SAA Data management systems Archive number Inventory number External Check and generating order number Not ok Canceled Archive nr Inventory nr Reason End Archive bank Report Textfile Ok Orderfile Archive nr Inventory nr Repository nr Order nr Generating order tickets Check originals Not ok Cancel orders Ok Preparation originals Logistics Transport Scanning Transport Originals back to repository Scanning report Order nr Quantity of scans Upload images DVDs to repository Remove order tickets and registration order nrs Registration filenames End Originals back Order nr Export EAD Registration originals back End Import and processing Mail to customer End 26

31 Three main steps The reproduction process is divided into three main steps: 1. Preparation 2. Scanning 3. Return of originals, uploading images and registration The preparatory step is a cyclical process. This means that this step of the reproduction process can be run through several times before we move on to the second step - scanning. Transport on a weekly basis was agreed with the digitizer. During scanning, customers requests are given priority over additions based on selections by the SAA. The content of the return batch is not necessarily the same as that of the batch that was sent. This means that the maximum time from receipt of a request for digitizing to delivery of the scans to the SAA is two weeks. The third step has to be carried out as soon as possible after this so that the originals are back in their place in the repository and in particular so that they can quickly be made available to the customer. The most time-consuming part is uploading the images, when derivatives are made in addition to the automated uploading itself.

32 Requests for digitizing Digitizing at the customer s request The inventory numbers to be digitized each week are primarily determined by customers requests. These requests are collected by the digitizing employee using the management environment of the Archiefbank. Output: 10,000 scans per week!0,000 scans are made every week. If there are more than 10,000 requests for scans, some requests are carried over to the following week. If there are fewer than 10,000 scans we make up the difference with our own selections. SAA selections These selections are already determined well in advance so that they can be added to a week s batch quickly. The most important selection criteria that we use are: - interesting for the public - simple to scan - sizeable units - as far as possible physically stored together in the repository Selection file All requests are collected in a selection file. This is a simple text file with two columns: access number and inventory number.

33 SAA situation: separated systems for availability and management At system level the SAA employs a strict separation between management and availability to the public. This has a significant advantage in that the standards that have to be met by the two environments can be met with maximum effectiveness. For example, the management environment needs a high degree of security and dependability, whereas for availability the most important requirements are high performance and user-friendly presentation. Moreover development of the two environments can take place independently of one another. The management systems used by the SAA are: - Scope Archive: integral access - ABS-Archeion: repository management and visitor registration for the reading room The public systems on the SAA website are fed by means of regular exports from the management systems (for the Archiefbank: by means of EAD). The public system is Toutatis. In practice, our experience with this approach has been good. However this set-up does have a number of implications for the reproduction process that have to be taken into account. For example the requests for digitizing are situated in the public domain, but the loan registration in the repository is run on ABS-Archeion so exports / imports are unavoidable in the current reproduction process. In addition neither the management systems nor the public systems offer sufficient functionalities to be able to support the logistics for the Archiefbank. For the time being, therefore, we are working with an order processing system developed in house. In real terms, working with exports and imports is not tremendously difficult as long as we work according to strict rules and conditions, and steps are automated where possible. It can be concluded that more far-reaching automation is still certainly possible, and also desirable. For example, at present the selection file has to be stored in a folder, an SQL window has to be opened and an SQL script prompt has to be input first, and then the automated procedure starts. It would obviously be better if the employee did not have to touch the SQL window at all, and the procedure started automatically after completion of the selection file. It goes without saying that there are solutions that make this possible, but implementing them will be a second phase in automating the reproduction process.

34 Request check, loan registration and distribution of orders Digitizing Simplified

35 Importing the selection file Before the originals can be made ready for digitizing there is verification that: - the inventory numbers in the request file actually exist - it has not already been digitized - the inventory number is available in the repository As looking up and changing numbers one at a time is certainly labour-intensive in large numbers, it is done automatically and in series. The SAA s repository management system does not, however, contain standard functionality which can implement serial changes, so an import script has been developed internally that can be started by the digitizing employee, after which all steps are continued automatically. A request is accepted or declined on the basis of the results of the check steps. Request accepted When a request is accepted: - the loan registration in the management systems is changed to Archiefbank digitizing - the location is found in the repository - an order number is allocated to the inventory number this is retained throughout the entire reproduction process and forms the key for registration of the file name after scanning The script then generates a report (TXT file) which contains the: - Access number - Inventory number - Repository location - Order number Request denied The reproduction process ends if a request is denied on the basis of one or more of the checks. The script generates a report for the inventory numbers concerned which contains the: - Access number - Inventory number - Reason for refusal In each case the next step will be worked out on the basis of the report. The most common reason is that the inventory number is being used by a visitor in the reading room at the time of the request and it is therefore not available in the repository. Be that as it may, after any remedial action the reproduction process for the inventory numbers concerned will be started once again.

36 Making order tickets Order system Order tickets A unit to be digitized has to be able to be identified at every step of the handling process. This is done on the basis of order tickets. The ticket states: - the order number - access number - inventory number - the first file name to be used - special requests when digitizing the inventory number concerned

37 System for making order tickets The SAA uses a simple program developed in house for making order tickets, into which the order file can be imported. After importing, the orders are linked to the transport data, and special details can be included if necessary. Order tickets are printed from the order system, and a total overview can be generated for the digitizer. Order system screenshot

38 Assessing the originals The need for assessing the originals It cannot be assumed in advance that the registration in the management systems is complete. Before a request for digitizing can be finally accepted a simple assessment of material and content is made. The inventory number is not leafed through and looked at page by page, but it is screened briefly. The speed of the assessment depends to a significant extent on the experience of the employee. The aspects to be checked are: - material condition - copyright - publicity - privacy

39 Condition of the material For material reasons an inventory number may not qualify for the standard way of digitizing. In principle there are three situations that can cause this: - the items in the inventory number are in such a bad condition that digitizing could cause irreparable damage - the items are in good condition, but packaged in an unconventional way (e.g. receipts that have been wound on reels) - the items are in good condition, but packaged in a way that can lead to damage during transport or digitizing (such as charters that are not yet mounted) The rule is that if an inventory number falls into one of these categories the order is rejected. That does not mean to say that it will never be digitized. In some cases it could be done, for example as made-to-measure or in a separate project. Case study: request for charters that have not yet been mounted One of the customer requests in the pilot project was for a charter. In itself it was in good condition, but it had not yet been mounted according to the standard method and was therefore too fragile to be given out. In consultation with the restoration staff we looked into what was needed to mount a charter like this. Firstly the (rolled up) charter has to be flattened. It is then fixed on to cardboard and the seals are packaged separately in cotton wool. The number of actions is thus limited and not a great deal of technical knowledge is needed, but the total processing time amounts to at least one week (because of the flattening, which has to be done with weights and water). This process was gone through in its entirety for the customer concerned, but the conclusion was that it is better to cancel a request like this at the start. Actually to accept it means that the procedure has to be included in the working processes, but the departure from the standard procedure is too great. Scanning of a document like this also calls for special attention during scanning, which means that by definition it is better not to send it with standard orders. The conclusion is that for the time being charters cannot be digitized on request.

40 Copyright and publicity Copyright is only applicable to modern archive material. The basic principle is that there can be no digitizing if: - there is no doubt that an inventory number in its entirety is protected by copyright, for example in the case of a printed book - items in an inventory number were made by identified individuals If there is any doubt a risk analysis is carried out on a case by case basis. Publication is generally well documented. In case of doubt the formal transfer papers are consulted. Case study: sexual escapades documented in a diary in detail; how do you find out? Several hundred first-person documents were digitized in the pilot project. These of course are ideal media for giving vent to dissatisfaction and recording goings-on that the writer had in all likelihood meant to keep between the covers of the diary (or perhaps, of course, quite the opposite). Where modern examples are concerned, there is certainly a very real risk that people who are still alive are mentioned. However there is no time for a detailed examination to look for this sort of thing; it would mean that all diaries would have to be read first. But there is one person who does get to see all of the pages, and that is the scan operator. Now this person will not read everything either, but while scanning one of the diaries in the pilot project the scan operator commented that it might be better not just to put the book in question on the net without further ado. The diary contained records and reports of all the author s sexual escapades, including the names of those involved. It is likely that at least some of these people are still alive. A publication restriction was then placed on the diary in question. Aside from the fact that this is a nice anecdote, it clearly demonstrates that it is impossible to check in detail if an inventory number contains sensitive material. It therefore has to be assumed that this aspect was taken into account when the item was initially inventoried, and a document like this can simply be consulted in the reading room anyway. It should also be pointed out here that the scans are not just put on the net; the user has to be registered and has to pay for the scans..

41 Cancelling orders Cancelling an order A number of operations are carried out when cancelling an order: The order numbers in question are registered in a separate file (using a barcode scanner) The loan registration is automatically cancelled on the basis of this file The order tickets are removed The originals go back to the repository When it is a customer request for digitizing: the customer is informed that the request cannot be carried out and the reason why not Registration in the management system that the inventory number is not available digitally (if relevant!) Export and import of the archive in question: the inventory number is no longer requestable for digitizing through the Archiefbank (here again: if relevant).

42 In addition to the import script with which order numbers are requested, a similar script has been developed that does exactly the opposite: a summary of the orders to be cancelled is imported and the status is then automatically changed. The delivery status in the management systems is changed manually. In principle this can also be automated (and linked to cancelled loan registration), but for the time being this is not being done because there are a number of possible reasons for a cancellation, not necessarily linked to a customer request. Additions on the basis of selections by the SAA, for example, may also be cancelled for reasons of content. Communication with the customer is handled by the Archiefbank back office. An is sent to the customer with a brief explanation of the reason for cancelling. Preparatory treatment of the originals Adding an order ticket All inventory numbers are given order tickets (see step 4). The ticket is added to the cover, box or binding as the first sheet. Packaging In principle the inventory numbers are transported to the digitizer in the box in which they are packed in the repository. The inventory number to be digitized is only put into a separate box when a box contains several inventory numbers.

43 Message to the customer Communication with the customer about the processing of the request kept to a minimum Experience of supplying different types of reproductions shows that communication with the customer about orders can take up a relatively large amount of time. The basic principle of processing requests for digitizing from the Archiefbank is therefore to keep communication about the request and delivery of the scans to a minimum. By doing so: 1. a fast delivery time is guaranteed 2. the cost price for the customer is kept as low as possible In terms of the request process this actually means that: - no quotation is given: the customer makes a request, the archive goes to be scanned - the customer can buy the entire inventory number after scanning, but can also restrict himself to a few scans or not purchase anything at all if he chooses - the customer therefore does not commit himself to anything, but at the same time neither does the archive - where possible the communication is automated This means that it does not matter if a customer does not want the material after all once it has been digitized, either because he thinks it is too expensive or for some other reason, and decides not to purchase it, The experience of the SAA is that orders in other systems seldom or never lead to cancellation and furthermore, a once an inventory number has been digitized, it is available for other users.

44 Logistics Transport frequency The point of departure in large-scale digitizing is that the quantity of scans to be produced remains as constant as possible. The SAA keeps to a weekly transport schedule in which 10,000 scans are made per week. Case study: working with a buffer so that there are always items with the digitizer The scanning process runs most efficiently when the same quantity of scans is produced each week and when the scanning process can run continuously. We therefore use a buffer. This means that there are always more items with the digitizer than can be processed in that week. Making scanned orders ready for transport and preparation of new orders for scanning can then take place without the scanning coming to a standstill. The rule is that customer orders for scanning are given priority over additions based on the SAA s selection. Customer orders are specifically indicated when the orders are packed into the transport cart. The consequence of this way of working is that the content of the batch that comes back is not necessarily the same as that of the batch that was sent. This is not a problem in terms of the reproduction process.

45 Transport carts The items for transport are packed in carts or boxes specially designed for this purpose. Transport cart with a weekly production of scans Case study: always have a transport cart on hand It has been agreed with the digitizer that the SAA always has a transport cart on hand. In this way packing can be timely, efficient and safe and the time needed for handover can be kept as short as possible: empty cart out, full cart in. Communication about orders in a batch The sooner the digitizer knows what there is in a batch, the better. A provisional overview is therefore sent well before a batch is transported. A final overview is sent at the moment of handover. This overview must correspond exactly with what is really present as it describes the actual content of a batch and is also a checking tool.

46 Transport Transport contracted out to the digitizer Transport of the originals is also contracted out to the digitizer. The conditions that are placed on this are: - that the transport is insured - the presence of climate control - that there is someone with the items at all times - that there are enough methods of securing so that the transport carts cannot shift Transport of the originals and the scans transport medium A portable hard disk of (at least) 250 GB is sent with each batch. When the batch returns the content of the hard disk is the same as the content of the cart. Transport docket At each handover both parties sign a transport docket. The transport docket describes what the order is, what is being collected and / or returned.

47 Scanning Digitizing Simplified

48 Preliminary observation The SAA chooses to contract out the scanning of archive material to an external partner. Specific technical solutions that are needed to make large-scale scanning possible for a low cost price are therefore not described in this report. In 2006 the SAA described the generally applicable technical aspects in the making of a scan, such as options in the choice of a storage format and possibilities of compression, in the report Digitizing Unravelled. Factors determining costs when scanning The production costs of making a scan (by this we mean the incidental costs for making the scan, not costs for storage and the like) are determined not so much by the scanning itself (i.e. taking the shot) as by the manual operations it involves: 1. preparation of the originals by the digitizer 2. handling of originals during the scanning 3. cropping the scans 4. checking and possible remedial actions These operations will always occur and are consequently independent of the digitizer. The way in which the digitizer organizes the work is down to him. 1. Logistical preparation of the originals by the digitizer After the digitizer receives the originals, the various orders will be prepared for further processing. The better the shop floor is geared to the type of documents to be digitized, the less work is needed. For example, if a digitizer uses different systems for different sized originals (for example, up to A3 and larger than A3) then at some stage the items will have to be sorted according to size. This also applies when, for example, loose-leafed and bound units are scanned in separate systems.

49 Case study: the relationship between orders and packing units Each inventory number to be digitized is given its own order number and order ticket. This system works very well, but it has two major limitations: 1. The order ticket is placed inside the cover, binding or box, so the order number cannot be read off from the inventory number on the packing unit. 2. The packing unit for transport is usually a box, but a box can contain a number (sometimes dozens) of orders. However, the orders concerned cannot be identified from the boxes themselves. This makes the logistical preparation by the digitizer difficult, as what comes in is in simple terms a collection of boxes and bindings. The boxes, covers and bindings have to be opened first to be able to see what the content actually is. So far we have not found a satisfactory solution for this problem and the digitizer will have to find his own solution in the organization of his own processes. One option for simplifying the connection is to put the order numbers concerned on the boxes. But this still provides no information about the sequence of the items in the box, and what is more it will not work when an order is not a box but a loose binder (where you cannot just stick a number on it). Another option is to put all the orders in separate boxes, but that is particularly labour-intensive and inefficient in the use of the space. The experience of the SAA in this area also shows that: - In quotes from digitizers it is often assumed that an order is the same as a box. If clear agreements are not made beforehand this will cause problems later. - The digitizer usually uses his own system, for example, to be able to distinguish different types of originals from each other. This can lead to disruption in the communication. There is only one piece of key data for the archive organization which is used throughout the entire process, and that is the order number attached by the archive. This has to be very clearly agreed.

50 2. Handling the originals when scanning An inventory number rarely consists of a neat, easy to leaf through stack of documents. Insertions and additions, such as folded newspapers, envelopes with contents, fold-out sheets, and also staples, paper clips and the like are the rule rather than the exception. This creates an inevitable delay in scanning. The better the shop floor and the scanning tables are geared to this degree of complexity, the more efficiently the work can be done, and consequently the lower the costs. Case study: adjusting scan instructions to increase scan speed At the start of the pilot project one of the scan instructions was to record insertions in a binder, such as loose sheets of notes, separately. The advantage of this is that these scans can be moved later if it turns out that the insertion in question is in the wrong place. In terms of scanning, however, the result of this instruction is that: - the binder is scanned up to a place where there is an insertion - it then has to be removed - the insertions are then scanned separately - the binder is finally put back again and scanning can continue This delays scanning to such an extent that it was decided to scan insertions in context. This means that scanning can continue without interruption. If necessary, the insertions are placed on a grey background to distinguish them from the underlying pages.

51 3. Cropping the scans Cropping (cutting out) on the face of it appears to be a process that is easy to automate. However in practice this is not so simple: Originals with staples Originals that always have to be folded over (and where the back is not written on), because of the presence of staples cannot be automatically cropped for the simple reason that a part of the scanned object has to be cut out (unless the entire scanned object is being saved, but this produces a scan with a large area without information, which is not very practical to use). Binders with support from the back The above also applies to binders that have to be mounted with support from the back. In this case the difference between the support and the object to be scanned is not recognized by the software. The scan is therefore cropped at the top and bottom to the edges of the support. Originals differing little from their backgrounds Automated cropping is also difficult in originals that differ relatively little in colour and tint from the background on which they are mounted (for example, tissue paper or thin carbon copies, where the background can even show through to some extent). Originals not entirely straight It also occurs that the corners are cut off when scans that are not entirely straight are cropped automatically. Conclusion: cropping (for the moment) has to be done manually All in all this makes manual cropping of scans inevitable for the time being. As this is a pretty labour-intensive job, there is scope here to speed up this aspect of the operation. Experiment: automatically cropping scans within a defined area In the pilot project a series of scanned inventory numbers was cropped within a defined area. The size of the area was determined by the largest original in the inventory number in question. This works well as long as the format of the originals remains reasonably constant, but in practice this is often not the case. This way of cropping therefore produces quite a lot of scans with exceptionally large grey edges. In use this is extremely awkward because the user has to zoom in from quite a distance first before anything can be read. In a case like this making a print usually only makes sense if the grey edge is cut off first. This way of working is therefore not used.

52 4. Checking and possible remedial actions There is always a possibility that mistakes are made during scanning: pages are skipped or are scanned twice. Information can be lost because the edge does not quite fit inside the scanning area or the quality of the scan is not up to standard. Checking completeness and quality after scanning is therefore essential. The SAA opts for making a 1:1 check of original and scan. The earlier a mistake is detected, the easier it is to take remedial action, so it does not make sense to carry out the checks after the scans are delivered to the archives. By then, the cause of the mistake is difficult to discover and the originals are back in the archive repository. The entire process would have to be gone through again. The 1:1 check is therefore carried out by the digitizer himself, as part of the scanning process. If necessary remedial actions can then be taken quickly and efficiently. Case study: integrating specific checks in the reproduction process instead of a 5% sample afterwards In the second half of 2006, in collaboration with the Dienst Persoonsgegevens (Amsterdam Personal Information Department), the SAA carried out a project in which 1,000,000 archive cards (loose-leaf system, with duplicate cards of so-called Personal Index Cards) were scanned, and the basic data on the cards were entered in the system. The point of departure for this project was that the originals would be destroyed. In this case it was actually a form of preservation imaging. In collaboration with the digitizer, the SAA set up a check protocol for this project, where the basic principle was to integrate the checks in the process as far as possible and automate them where possible. This meant that the classic 5% sample afterwards could be abandoned. The checks were many times faster, more exhaustive and more specific (and as a result higher in quality). An important but difficult to gauge quality requirement was completeness: the number of scans must be exactly equal to the number of originals. But how do you guarantee that? The solution chosen was double scanning. The majority of the cards could be scanned with the aid of a continuous feed scanner. In each case in small, logical amounts this was done in duplicate: first a counting session (bi-tonal storage of scans) and then the scanning session (storage according to specifications). The number of scans from both sessions had to be the same before the next series was started. If they were not, both series were scanned again in their entirety. Both series were delivered to the SAA after scanning so that another check could be made afterwards. The check of the quality of the scans (here again the point of departure was reproduction of textual information and not, for example, the structure of the paper, as these kinds of things have no information value at all for this type of document) initially took place because the data from the scans was entered. When the information was not legible this was made known and if necessary a new scan was made. The input was also done in duplicate.

53 The SAA then checked legibility in a semi-automated way by making use of the properties of the JPEG compression algorithm: this works well on even surfaces and less well on diffuse surfaces. The scans are stored in grey tones (the originals had no colour information). By sorting the scans into file size before the check we achieved the following: - really light cards were placed at the front - really dark, rather dirty cards were placed at the back These two types of scans are the most critical as regards legibility, and are therefore the most relevant to check. Putting them at the front and back like this meant that the checking could be specifically targeted. This was done visually, working on the assumption that if the text on the scans was legible, the quality was high enough and there was no need to compare them with the originals. In conclusion it can be stated that the quality requirement of legibility in this project was of such importance that the scans in general look somewhat harsher than the originals. No significant information was lost, and legibility is as good as it can be. However, were grey tone charts to be added when checking, then the scans would probably be rejected if they were assessed based on the gradations of grey. The conclusion is that in this project the quality of a scan could not be gauged in this way. In any event continuous feed scanning (the only way to keep a project of this magnitude affordable) offers no possibility of scanning a grey tone chart together with every card. However, a barely legible card (always the same card) was scanned as a check card prior to each session. Scan instructions In order to prevent misunderstandings, a summary with short, clear, generally applicable scan instructions has been drawn up for the digitizer. If additional instructions apply to an order these are stated on the order ticket.

54 The scan Technical aspects of the scan and the requirements that can be attached to it are described at length in the report Digitizing Unravelled. The actual requirements at present set by the SAA are as follows: Colour quality Bit depth Curve ( contrast ) RGB 8 bits per channel such that the scans are legible on screen and in print File format and compression This applies to all scans made for the Archiefbank: Storage format Compression JPEG quality 10 (Photoshop) This produces files which: - remain relatively small (average 2 MB per scan) - are highly legible, even when greatly magnified - have enough flexibility to satisfy present and future uses

55 Storage and transport mediums of image files Storage For the time being the SAA is opting for: - storage in the internal network - storage on the web server - an emergency back-up on DVD The configuration of the storage on the internal network and web server meets the high requirement for security, redundancy and back-up. However this report does not go into the actual design of these environments. The emergency back-up on DVD is supplied directly by the digitizer. Images transport medium for uploading to the internal network and web server For the moment the image files for uploading to the SAA s internal network and web server are supplied by the digitizer on 250 GB portable hard disks. Case study: experience with portable hard disks has not been positive The experiences of the SAA of working with portable hard disks are none too positive. The disks themselves are fragile and, for reasons that are not clear, there were occasions when files were not copied. This made it necessary to check carefully (admittedly for a large part automatically) whether the number of images on disk was equal to the images at the location from which it was copied. There is obviously room for improvement in this area. As an alternative to Portable USB hard disks as a transport medium it is worth considering tape, for example. However the disadvantage of this is that there is no actual standard for formatting, and so the writer and the recipient have to make clear agreements. An alternative to transport on separate, physical carriers is direct uploading using the net. However transport of large quantities requires adequate bandwidth. As an example: the uploading of 10,000 scans totalling 20 GB at an upload speed of 1024 Mbps takes approximately two days. In the future much can be expected from direct transport using a (glass fibre) network. Transport using the net has the additional advantage that the delivery time for the customer can be considerably reduced because the scans can be online soon after they have been scanned and checked.

56 Returning originals, replacement in the repository and restoring loan registration Returning originals When the originals are returned they are put back in the repository. The order tickets are removed before replacement. Scanning barcodes on order tickets Orders returned are registered in a file by means of scanning the barcodes on the order tickets. Changing the loan status in the repository management system The last step is to cancel the loan registration in the repository management system. This step takes place automatically on the basis of the scanned order tickets. Case study: periodic checking of the completeness of the originals that are returned Scanning the order tickets is not a check as to whether all of the originals have actually been returned. Since by definition the batch that comes back is not necessarily the same as the batch that went to the digitizer, comparing the content of the batches is not a way of checking this. The SAA therefore opts for the periodic generation of a report from the management systems containing the inventory numbers that have had the loan status Archiefbank digitizing for longer than two weeks. These have to be investigated in more detail.

57 Uploading images Uploading images For the time being portable hard disks are used as the transport medium for image files. On receipt of a disk the scans are uploaded to: - the web server, to make them available to the customer - the internal network, for the purpose of security (image storage at several locations) Completeness check After uploading, there is a check to see whether the number of image files on the hard disk is equal to the number of images on the web server / internal network. If it is not, all the file names are compared with one another using a simple script. The conclusion from practical experience is that if there is a discrepancy the best thing to do is to start a complete new upload.