The CineGRID collaboration

Transcription

1 The CineGRID collaboration University of Amsterdam Jeroen Roodhart 12/02/2009

2 .Plan What is CineGrid Who are involved Use cases Grand vision Storage in CineGrid context Current setup in Amsterdam Experience Lessons Future Summary

3 What is CineGrid? CineGrid is a non-profit international membership organization... CineGrid s mission is to build an interdisciplinary community focused on the research, development and demonstration of networked collaborative tools, enabling the production, use and exchange of very high-quality digital media over high-speed photonic networks. (From the site:

4 Who are involved Lots of (big) names, you find them in the members section of the site. More important:

5 Use cases (CdL) Keio/Calit2 Collaboration: Trans-Pacific 4K teleconference Used 1Gbps Dedicated Keio University Sony President Anzai NTT SGI UCSD Chancellor Fox

6 Use cases SARA (CdL)

7 Use cases Holland Festival 2007 Era la Notte (CdL)

8 Use cases Scientific visualisation Film editing processes High def. Collaboration environments Medical applications Entertainment venues Dome theatres 4K Cinema

9 Grand Vision (Cees de Laat's find the beautiful lady on the beach ) RDF describing Infrastructure Application: find video containing x, then trans-code to it view on Tiled Display RDF/CG RDF/CG RDF/VIZ RDF/ST content content PG&CdL RDF/NDL RDF/NDL RDF/CPU

10 Grand Vision (Ctd.) CineGrid will be using irods Intention to place NDL and semantic information within irods Storage/content delivery nodes/transcoding

11 Storage in CineGrid context (CdL) 3840*2160

12 Storage in CineGrid context (ctd.) (CdL) Format x Y Rate /s 720p HD p HD k SHD k Color Frame bits/pix pix Frame MByte Flow Stream MByt/s Gbit/s

13 Storage in CineGrid context Buying n x 1T disks doesn't work Traditional approach using FCAL, NFS, CIFS might not be fast enough We may now run into issues with Complexity Scalability Speed considerations Integrity

14 Why think about storage (ctd.) CineGRID Large data sets Traditional interconnecting of sites Relaying movies to display sites Display movie (from local cache) Interconnect services Streaming server transcodes 4k movie to lower res Display movie stream Both models have different storage requirements

15 Current setup in Amsterdam (CdL)

16 Current setup in Amsterdam (ctd.)

17 Current setup in Amsterdam (ctd.) Choices: Existing 10Ge interconnect (we're into networking) Sun x4500 Thumpers 48 x 1Tb data disks/2 x Opteron Running OpenSolaris Nevada ZFS filesystem Looking into upgrade to x4540, more on this later

18 Current setup in Amsterdam (ctd.) About 18T in use, 13T left RAIDZ1 for now (speed/space considerations) Both thumpers are synced, using ZFS snapshot streaming 10Ge connection to streaming node node41 and (suitcees/node41) to Optiputer net Syncing of thumpers may stop if other use dictates

19 Experience Ease of administration #!/usr/bin/bash for i in ; do if [ $i = "1" ]; then zpool_command="zpool create -f mypool raidz2 " else zpool_command="zpool add mypool raidz2 " fi $zpool_command c0t${i}d0 c1t${i}d0 c4t${i}d0 c5t${i}d0 c6t${i}d0 c7t${i}d0 done zpool add mypool spare c0t0d0 c1t0d0 c6t0d0 c7t0d0 Wait two minutes --> approx. 32T filesystem

20 Experience (ctd.) Syncing filesystems # Make a point in time snapshot of the pool >zfs snapshot mypool@ _1 # Stream the snapshot using RBUDP to the other host >zfs send mypool@ _1 \ sendstream m 8000 # On the other host, receive the stream: >recvstream \ zfs receive mypool/basketcees@ _1 This may take longer...

21 Experience (ctd.)

22 Experience (ctd.) Got a Thor to play with :) Can we move the Thumper disks to the Thor?

23 Experience (ctd.) Yes you can! And you'll even keep your ZFS volumes if you export them nicely ;) Probably you don't want to move the OS disks though... So let's look at some tests...

24 Experience (ctd.) Thor

25 Experience (ctd.) Thor

26 Experience (ctd.) Thor Not much faster! But there's some strange variation here... So what if we would split per controller...

27 Experience (ctd.) Thor

28 Experience (ctd.) Thor

29 Lessons Raw storage system speed may be enough for uncompressed 4K, but that doesn't scale to concurrent streaming Using Thor-s it can help to think about controller/disk assignment With the ixgb NICs we max out at 6Gbps on a 10Ge link! (comparable using irods) This needs to improve to use Thor-speed! Considering the CineGRID application We don't yet have a standard solution from storage to Film display Conventional streaming tools use file system paradigm

30 Future Filesystems Linux: Ext4, BTRFS looks promising Solaris: ZFS remains very strong in benchmarks and usability Clustering Lustre GlusterFS pnfs Networking RDMA/iWARP interconnect (nice if we went 100Ge)

31 Summary Storage backend speed of individual modern systems may be sufficient for 1 stream: _But_ we will likely want more We need to consider the entire component stack of the CineGRID application We probably need an approach where streaming nodes can access data using cluster technology Fast interconnect (RDMA/QDR Infinib.) More than one storage server New technologies may lead to more elegant designs (e.g. SSD/ZFS/Lustre)

32 Backup slides

33 Experience (ctd.)

34 Sun x4500 Thumper architectuur

35 Effective filesystem size (Using 500MB disks) ZFS config Z2_14x3 Z1_6x7 Z2_10x4 Z1_2x3x7 Z2_6x7 mirror_3way mirror_3wayx2 Size (TB) x x 3.1

36 Mirrorset variances mirror_3wayx2/c0-c2/iozone_encore.science.uva.nl_snv_103_mirrored_3wayx2_0_variance_w.txt mirror_3wayx2/c3-c5/iozone_encore.science.uva.nl_snv_103_mirrored_3wayx2_1_variance_w.txt mirror_3way/iozone_encore.science.uva.nl_snv_103_mirrored_3way_variance_w.txt mirror_3wayx2/c0-c2/iozone_encore.science.uva.nl_snv_103_mirrored_3wayx2_0_variance_r.txt mirror_3wayx2/c3-c5/iozone_encore.science.uva.nl_snv_103_mirrored_3wayx2_1_variance_r.txt mirror_3way/iozone_encore.science.uva.nl_snv_103_mirrored_3way_variance_r.txt