IBM Linear Tape File System Applications and considerations for broadcast adopters Software Generation Limited March 2013 Contact: Raj Patel, Product Manager Copyright IBM Corporation, 2013. All Rights Reserved. All trademarks or registered trademarks mentioned herein are the property of their respective holders
2 Table of contents Abstract...1 What is LTFS?...1 How does it work?...1 FlashNet and LTFS together create a transportable data platform...1 Making data tape technology more accessible...1 Using LTFS within a larger archive system...2 Content migration...2 Start small and build an archive from the ground up...2 Cost-effectiveness...3 Providing greater density for transportable media...3 Content delivery into the archive...3 Random access compared to sequential tape with small files...4 Offline storage and export...4 Archive as a service...5 LTFS and other media standards...5 Summary...6 Resources...7 Trademarks and special notices...8
Abstract This paper sets out to highlight some of the uses, advantages and pitfalls of the adoption of the IBM Linear Tape File System (LTFS) technology that IBM Linear Tape-Open 5 (LTO-5) technology and other emerging formats provide. This includes considerations for the applications of tape storage as either a transportation mechanism for media content or a backup solution in the traditional archive sense. What is LTFS? Linear Tape Open (LTO ) technology has been used for storing data on tape since 2000 when IBM, HP and Seagate (now Quantum) formed the Ultrium Consortium. Since then, the storage capacity of Ultrium LTO tape has grown from a mere 100 gigabytes (GB) to 1.5 terabytes (TB) per tape that is now provided by the fifth-generation LTO-5 cartridges (these capacities are for uncompressed data). As well as increasing the capacity with each generation, the IBM development team in Japan has also introduced new technologies along the way, the latest of these being IBM Linear Tape File System (LTFS ) technology, which is useful with both LTO-5 and 3592 tape formats. Contrary to some beliefs, LTFS is not a file wrapper, but a standardized method of writing data to tape and providing a way to describe its content. This means that no matter which kind of system reads that tape, it can identify the files stored on it. How does it work? The key to this self-describing nature is the addition of a metadata section on the tape, referred to as the index partition. This is at the beginning of the tape and the corresponding data is written further along the tape in a separate, larger area called the data partition. The index partition is created when a new tape is first formatted with LTFS. This means that the decision to use LTFS has to be made at the start of a tape s life cycle; after it has been formatted for LTFS, it cannot be changed unless it is reformatted back to the standard format or LTFS format. This also means that LTFS is currently compatible with LTO-5 and TS1140/3592 drives. Previous generations of LTO drives cannot read or write an LTFS-formatted tape. FlashNet and LTFS together create a transportable data platform SGL FlashNet has the flexibility to manage content in many different ways. It allows users to segregate types of media content within a much larger archive. For example, content can be sent to different groups based on whether it is news or sport related. With the advent of LTFS, FlashNet has the ability to send content to LTFS-enabled groups so that content can be exported from the archive. The same applies to the delivery of LTFS-formatted tapes into the archive. If content has been written to an LTFS tape by another system, FlashNet can read this tape and make the content on it available as part of a much larger archive. Without the FlashNet implementation of LTFS, this content could only be accessed through a standalone drive. Additionally because of the media aware nature of the FlashNet database, the integrity of newly delivered content can assured as soon as it arrives. Making data tape technology more accessible For more than 10 years, usage of data tape has, by-and-large, been limited to larger, IT-biased installations and has almost exclusively been limited to robotic tape libraries. However, with LTFS this is no longer the case. Previously it has only been possible for users to access tape technology through proprietary backup solutions that have exclusive control of the tape drive. Now it is possible to attach an LTO-5 drive using serial attached SCSI (SAS) to a standalone desktop PC or Apple Macintosh computer, download the appropriate operating system driver (Microsoft Windows, Mac OS or Linux ), and mount 1
an LTFS LTO-5 cartridge as if it were a USB or other external drive. The tape appears as a drive letter within the operating system. Suddenly this makes conventional data tape far more accessible to many more users than previously, and tape library systems such as the IBM TS3200 model are ideally positioned to meet the needs for these kinds of situations. Using LTFS within a larger archive system One key advantage of an archive storage management system such as SGL FlashNet is that it can present many disparate storage platforms to the outside world as if they were a single entity. This storage can take the form of a disk-only archive, tape-only archive, or, as is often the case, a combination of both disk and tape storage as is provided by vendors such as IBM, which is specifically designed to work in mixed media environments. Unlike many archive content management systems (CMSs), SGL FlashNet has the ability to either fully or partially integrate LTFS into a larger archive that consists of one or more scalable tape library solutions. This is achieved because of the unique way in which FlashNet communicates with the tape drives in a library. Most software vendors rely on the drive manufacturers operating system drivers to write data to tape. FlashNet does not have this dependency for any type of tape drive, instead using the SGL drivers to ensure reliable communication with tape devices. To support LTFS, SGL has fully embraced the core LTFS standard, at the time of this publication to the lowest SCSI command level. This means that FlashNet users can mix LTFS and non-ltfs media within the same archive; the archive is divided into groups internally to differentiate between media. Content migration One significant advantage of this archive grouping within SGL FlashNet is that users with an existing, non- LTFS-compatible archive can migrate to a fully compliant LTFS system by copying the content within the archive. After setting up an LTFS group within FlashNet, the migration process begins, based on userdefined rules instigated by StorageManager, a highly configurable and powerful rules engine module for SGL FlashNet. StorageManager can create duplicate copies, move content to deeper storage layers within the archive or migrate content outside of the archive (for example to free up tape slots in a tape library such as the IBM TS3500 model). Start small and build an archive from the ground up By adopting a standalone LTO-5 drive model such as the IBM TS2350 tape storage drive, users can start to build an archive based on their immediate needs without the larger capital expenditure of a full-blown archive management system. As an example, a small video production company that crafts its own content from location to finished program. Acquisition on location is multi-format (XDCAM, P2, SD card and so no), and therefore the first advantage that LTFS offers is the ability to free up these expensive acquisition storage formats. By purchasing an IBM TS2350 LTO-5 drive; connecting it to a PC workstation (through an SAS cable and PC card), they can start to copy content to this high-density storage format. Although the cost of a SAS-attached LTO-5 drive is in the region of $3000, recycling a $500 P2 memory card that can only store 32GB of data quickly represents a cost-effective solution. As a guide, it would take roughly 48 x 32 GB P2 cards (1536 GB) to fill a single LTO-5 tape, which can cost as low as $60 USD per cartridge. 1 1 Numbers based on information found December 2011 at www.tapeandmedia.com/ibm_lto_5_tape_ultrium_5_tapes.asp 2
After time, any creative organization will start to build a significant portfolio of content that it does not only own, but also wants to make available to others. This is where SGL FlashNet comes in. When an organization has built a collection of sufficient LTO-5 tapes, these can be put into a robotic tape library. SGL FlashNet can read the index information from the index partitions on each tape, build a database of content from the metadata held therein, and facilitate the search and restore of material to the outside world, on demand. As this growth continues and more complex workflows are required, an overall controlling media asset management system can be added to track content. Cost-effectiveness The important consideration is that no complex software or workflows are required. For example, footage is shot on location, the tapes or memory are brought back to a workstation, and are copied to LTO-5 tape. Due to the fact that the LTO-5 cartridge is mounted as a drive letter (just like a USB drive), the footage is simply dragged and dropped using Windows Explorer Xplore (or similar for Linux), Finder (Mac OS) or similar ways. There are of course some CMSs that claim to work on this scale and provide a database for tracking this kind of content through its manual migration (dragged and dropped). However, these are limited to device and operating system restrictions, which means the user might not be getting the full benefit of the LTO drive performance. Providing greater density for transportable media The lack of availability of Sony SR tape, has caused customers to look for alternative Media choices. One very viable alternative is LTO-5 technology, not only because of the cost (almost one third the price of SR tape), but also the density of data. The largest capacity for XDCAM SR tape is 124 minutes (4:2:2); a single LTO-5 cartridge, on the other hand, can store up to 31 hours of XDCAM HD content (at 1080p60 with 4 audio tracks). Factor in the cost of a single SR tape at more than $300 and a single LTO-5 cartridge becomes a very viable alternative. 2 Although the density of content seems very attractive, many people might argue that the lack of standardization for technology like LTFS within the broadcast media and entertainment industry makes it unattractive. However, for a growing number, the mere fact that LTFS comes from the broader and more widely adopted IT sector means that adoption could be much greater than initially thought. The ease with which users can download drivers for LTFS for a standalone drive from IBM, free of charge. The link to access this software is in the resources section of this paper.. Combine this with the self-describing nature of each LTO-5 cartridge and it becomes apparent that LTFS has the ability to deliver large quantities of material to many users effectively and with few cost implications. Content delivery into the archive From a change-management perspective, the concept of delivering content on tape is also appealing, as it fits with existing workflows. However, issues surrounding the distribution of files held on LTO tape to the archive and other systems still need to be addressed. Certainly this can be achieved manually, on a tapeby-tape basis using a standalone drive, but for larger systems this would be unworkable. The answer is offered by a CMS that is able to both read and write LTFS-compatible data, such as SGL FlashNet. The 2 Based on average pricing found on commerce sites in December 2011. 3
ability of these systems to read content written to tape by other systems means that these media can essentially be imported directly into the archive. (It is important to note that, because some systems write additional data to tape, (for example, proxies, scene change details in different folder structures and so on), FlashNet does not amend any additional data to that same tape, thus preserving the original structure). After a new tape is introduced to the archive tape library, FlashNet reads the index information on the tape and discerns its contents. Each individual piece of content is allocated a Globally Unique Identifier (GUID), which is used by FlashNet to track each asset within the archive. This means that different versions of the same content can be placed into the archive, a fundamental difference between SGL FlashNet and other simple HSM-type solutions. At this early stage, none of the overarching traditional Production Asset Management (PAM) or Media Asset Management (MAM) systems are aware of the arrival of this material. Some systems offer web services-type interfaces, through which FlashNet can push back notification of the new GUIDs, but if the controlling system does not offer such services, a new interface is required through which arrival of new content can be published. Enter the SGL Notification Service, a multi-level notification mechanism for content and archive status notification, that, in addition to the arrival of new content, offers multiple levels of notification about activities within the archive (for example the health of the archive servers and other components). As soon as FlashNet has assigned GUIDs to the content, this information is pushed back up the content chain to the relevant systems. This push notification is critical, as it eliminates the requirements for continuous polling protocols that only lead to database de-synchronization and other issues. In addition, the SGL Notification Service allows messages to be persisted in the event of crosssystem disconnects, and there are also complimentary tools, attained through the FlashNet application programming interfaces (APIs), from which established systems can request updates on the existence of content, based on certain filters and so on. One question that arises in this scenario is What if the receiving system (a PAM system, for example) was expecting the content and has a placeholder for it in its own database? SGL has developed the ability for the master PAM (or MAM) system to re-assign a GUID to any content for exactly this reason. After these systems have become aware of the new content, and more importantly, know where this material is, creative users can access the footage and begin their work. From this point on, SGL FlashNet continues its traditional role of providing content from the archive to other systems. Random access compared to sequential tape with small files Even though LTFS presents a tape cartridge as if it were a disk drive type device, it obviously does not make that tape a random-access device. Content still has to be read and written sequentially, and thus LTFS suffers in the same manner as all data tapes and drives when confronted with a large number of small files; write speeds can decrease significantly due to the constant stop/start processes that the drive undergoes when opening, writing then closing large quantities of files. To address this, the archive management software, such as SGL FlashNet, can use a small disk cache, in which it bundles small files before writing the larger, single bundled file to tape in a far more efficient process. The FlashNet database tracks the location of the original files within the bundle, and is able to retrieve individual files from tape without restoring the entire collection. In this manner, tape can be used seamlessly within a random access type environment. Offline storage and export LTFS enables an array of workflows with similarities to conventional video tape-based workflows. One 4
such workflow involves placing a master tape into the archive for duplication purposes only. Like video tape, once the content has been copied the master may be taken out of the archive tape library and held in secure storage not just to protect the content but also to prevent the archive library from becoming clogged with potentially unnecessary duplicates. After the tape has been exported, the SGL Notification Service can notify any interested systems that the tape, and its contents, are now in a location outside the library (for example, on a shelf, in an offsite disaster recovery location and so on). By taking this approach, a user can be prompted to first retrieve the correct tape from the external location before restoring the required content from the archive. This kind of notification also applies to non-ltfs workflows, where tapes can be removed from smaller-scale libraries to free up space. Archive as a service More and more organizations are investing in the necessary hardware infrastructure required to build a properly scalable archive. In many cases this is to support long-term expansion based around their immediate requirements. However, as the success of a central archive repository has been embraced throughout larger organizations, the opportunity has arisen to increase internal growth by providing this service to other departments that were not previously provisioned for. For example, a transmission archive might find itself being shared by news, sports, graphics and film production departments over time. Obviously this is only possible if the archive system has the capability to expand both storage and bandwidth required for this kind of growth. However, the growth does not always come from within the same organization. In many cases archives that were built to create a localized, private cloud are now being opened up to outside organizations as an archive service (or subscriber public cloud). In this case the business model is obvious: over-specify the size of your archive hardware and recover the capital expenditure by offering parts of the archive storage as a service to other organizations. In this manner, content makers can also become service providers. LTFS has a part to play in this business model also. When offering archiving as a service to external customers, the ownership, length of contract and life span of the content need to be taken into account. Before the advent of LTFS, at the end of a 12-month contract an archive service provider would have to transfer all of the archived content to video tape or TAR-wrapped data storage in order to hand it back to their client (the owner of the material). However, with LTFS and SGL FlashNet, archivists simply ensure that customers content is stored in a separate archive group (a virtualized container in the cloud). At the end of the contract the service provider simply exports the LTFS-compliant tapes as a complete group and passes the tapes back to the client. If the archive service contract is a rolling one, or renewed, there may not be a requirement to hold the last 12 months of content in the archive, in which case the tapes can still be extracted from the physical library in order to free up valuable resources. LTFS and other media standards The fundamental nature in which LTFS describes the content held on tape fits very well with emerging standards within the broadcast industry. One such example is AS02, which like LTFS is self-describing. As this MXF standard becomes more widely adopted there will be a requirement for the archive to deal with the discrete components described within the AS02 wrapper. For example, a finished feature film wrapped in an AS02 container could hold a single video file and two language files (for example, English and Spanish). This self-describing and complete file is then archived. Six months later a third language track (French) for the same movie is delivered on its own. The master MAM system knows that this language track belongs to the AS02-wrapped file and therefore instructs the archive to add the new track to the AS02 inventory. You ca do this virtually at first, simply by associating this new track with the previously archived content. However, when tapes are defragmented (an automatic process in SGL FlashNet that frees up unused space on tapes after content has been deleted), any fragmented AS02 files could also be repackaged for ease of access. This kind of task is second nature to systems like SGL FlashNet that has 5
defragmentation and life cycle management rules that run in the background on a daily basis. One alternative format that is sometimes considered is the Archive Exchange Format (AXF), a file rewrapper that has seen some discussion recently. Some might adopt it, but the benefits appear to be limited, given that AXF is a totally broadcast centric format, and an AXF-compliant system is still required to read tapes exported from a library. This is not the case for LTFS because its roots lie in the broader IT space, where greater interoperability is essential to ensure that systems work seamlessly. At this point, there seems little point in using both LTFS and AXF within the same archive, as AXF will still need to be adopted by other vendors (as will LTFS). This duplication of development can cause confusion within the industry. AXF does not describe the files on the LTO tape, and LTFS as a standard does not describe the media content written as files on tape. In the scenarios discussed above, however, the AS02 standard could describe the media asset and LTFS describe the assets on tape. Summary As a whole, the adoption of LTFS is likely to be significant across the broadcast industry, not least on the basis that LTO technology provides a cost-effective packing density. However, as this document argues, adopting a tape format is not sufficient; only when complementary technologies, such as the SGL FlashNet XML API and the SGL Notification Service are tied to the appropriate technologies like IBM LTO or TS3592 storage and software like LTFS are combined in a solution, can the full value of the platform be realized. For further information on LTFS, SGL FlashNet or any of SGL s content archive and storage management solutions for broadcast, contact your local IBM reseller. 6
Resources These websites provide useful references to supplement the information contained in this paper: IBM Systems on PartnerWorld ibm.com/partnerworld/systems Virtual Loaner Program ibm.com/systems/vlp IBM Linear Tape File System Customer Information Center http://publib.boulder.ibm.com/infocenter/ltfs/cust/index.jsp IBM LTFS software downloads http://publib.boulder.ibm.com/infocenter/ltfs/cust/index.jsp?topic=%2fcom.ibm.storage.holl ywood.doc%2fltfs_fix_central_upgrade.html IBM Redbooks ibm.com/redbooks IBM Publications Center www.elink.ibmlink.ibm.com/public/applications/publications/cgibin/pbi.cgi?cty=us 7
Trademarks and special notices Copyright IBM Corporation 2013. All rights Reserved. References in this document to IBM products or services do not imply that IBM intends to make them available in every country. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Linear Tape File System is a trademark of the IBM Corp. Linear Tape-Open, LTO, the LTO Logo, Ultrium and the Ultrium Logo are registered trademarks of HP, IBM and Quantum in the US and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-ibm products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-ibm list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-ibm products. Questions on the capability of non-ibm products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction. Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the 8
storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. Photographs shown are of engineering prototypes. Changes may be incorporated in production models. Any references in this information to non-ibm websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk. 9