Big Data, Big Networks, Big Displays the Next WAVE of Cyberinfrastructure

Transcription

1 Big Data, Big Networks, Big Displays the Next WAVE of Cyberinfrastructure Tom DeFanti, Greg Dawe, Falko Kuester, Joe Keefe, Larry Smarr, and Phil Papadopoulos Qualcomm Institute, California Institute for Telecommunications and Information Technology, UCSD

2 Great Data, Great Networks, Great Displays the Next WAVE of Cyberinfrastructure What about Great Storage? There Will be a Test!

3 My Career

4 Rapid Evolution of 10Gb Port Prices Makes Campus-Scale 10Gbps CI Affordable $80K/port Chiaro (60 Max) Port Pricing is Falling Density is Rising Dramatically Cost of 10Gb and 40Gb NICs = graphics cards! $ 5K Force 10 (40 max) $ 500 Arista 48 ports $ 400/port 576 ports (2013) Source: Philip Papadopoulos, SDSC/Calit2

5 Feeding the Network with Inexpensive High Performance Storage Sizes of working data sets continue to increase Real or near-real time information delivery is one of the defining characteristics of big data analytics. Latency is therefore avoided whenever and wherever possible. Data in memory is good ( How fast you can retrieve it? How much can you hold locally? Data Size Retrieval 1 Gbps Retrieval Time@ 40Gbps 1 gigabyte 10 seconds ¼ second 10 gigabytes 100 seconds 2.5 seconds 100 gigabytes 16 minutes 25 sec 1 terabyte 3 hours 4 minutes For 10Mbps, multiply 1Gbps Column by 100 Source: Philip Papadopoulos, SDSC/Calit2

6 Creating a Great Data Freeway System Connecting Instruments, Computers, & Storage PRISM CC-NIE Award Phil Papadopoulos, PI Larry Smarr co-pi CHERuB CC-NIE Award Mike Norman, PI

7 Pink New CENIC 100G SAN-SEA Green CENIC 100G LA-SEA Blue ESnet/I2 100G 100G to Europe and Brazil

8 Great Data to Great Display Challenges Optimizing transfer via 40G, 100G and rdma GPUs synchronizing screens for megapixel movies fed at 30/60 fps (a terabit/sec or more problem!) Measuring performance Understanding resulting bottlenecks as they shift Extending this capability nationwide and worldwide FIONA is our interim impedance matcher and test system (see next slides)

9 FIONA Flash I/O Network Appliance Combination of Desktop and Server building blocks $6K & up Flash up to 16TB RAID Drives up to 48TB 10GbE/40GbE Adapters Tested speed ~30Gbs Developed by UCSD s Phil Papadopoulos Tom DeFanti 9 X 256GB 510MB/sec 8 X 3TB 125MB/sec 2 x 40GbE 2 TB Cache 24TB Disk Joe Keefe FIONA 3+GB/s Data Appliance, 32GB

10 FIONA Building Blocks 1X - RAID Controller LSI HBA SAS ~ $300 each Desktop Form Factor With Single Socket Server $3K Total Cost - $6K Add $750 for good graphics Much more for 16 1TB SSDs 1 X - Dual 10GbE Myricom, Inc or Dual Mellanox 40GbE ~$800 each 8 X 3TB SATA Drive Hitachi/Seagate $ each 8 x 240GB SSD $ each 500+MB/sec

11 Fast Uptake at UCSD Using these for many of the Walls/Portables/WAVE/WAVElets at UCSD. Reliability and Performance has been quite good Propagating quickly in labs Scratching the surface on trying to understand basic performance in a variety of contexts

12 KB/sec Baseline 8 Flash-Disk File System Performance (Local only, What Applications Observe) 64 GB File Maximum Expected Record Length (KB) 0 Write Read Random Read Random Write

13 Performance Measurements so Far Streaming read/write performance Within 10% of expected peak Serial-attached SCSI Host Bus Adapters (SAS HBA) running at ~80% of max throughput Random write performance Good only when basic record size >= stripe size of RAID Random read performance Much poorer than expected. Need to find the cause, but likely culprit is overly aggressive read-ahead Result would be to read data and then toss it out 1KB random record lengths, application observes 5-10MB/sec, disks are 500+MB/sec aggregate A FIONA has been installed at UvA in Amsterdam with 40Gb/s connectivity and will be used for measuring big data transfers and image streaming over very long distances

14 64-Megapixel UCSD 9 X 256GB 510MB/sec X 3TB 125MB/sec 40 Other Local Equip. 2 x 40GbE 2 TB Cache 24TB Disk FIONA 3+GB/s Data Appliance 40 Prism Core Switch - Arista 7504E 80 External Networks 60 Gordon (300TF) NSF Data Oasis 7.5PB, 100GB/s NSF & UCSD nnn Gbit/s 128 x 10GbE 128 x 10GbE 48x10GbE, 36x40GbE, 2 x100gbe Campus Labs Arista 7508 Oasis Switches (576 x 10GbE) Cloud Storage 3PB UCSD 96 x 10GbE

15 Creating a Great Data Freeway System Connecting Instruments, Computers, & Storage Phil Papadopoulos, PI Larry Smarr co-pi WAVE 80Gbps 25x40Gbs in the WAVElab

16 UCSD s RCI Program UCSD 18 x 1TB 510MB/sec 50 WAVE FIONA Grande HDMIs 40x25 Other Local Equip. 40GbE 16 TB flash In local server Nodes have 2TB flash 24 FIONA Grandes 1 Local FIONA Server 3+GB/s each 80 Prism Core Switch - Arista 7504E 80 External Networks 60 Gordon (300TF) NSF Data Oasis 7.5PB, 100GB/s NSF & UCSD nn =Gbit/s 128 x 10GbE 128 x 10GbE 48x10GbE, 36x40GbE, 2x100GbE Campus Labs Arista 7508 Oasis Switches (576 x 10GbE) Cloud Storage 3PB UCSD 96 x 10GbE

17

18 Great 110 Foldable 4K Stereo on Wheels

19 Shipping Great Displays for Collaboration Worldwide

20 Great Displays can be as Wide as You Want They can be in 2D or 3D, with or without tracking, gesture recognition, and audio

21 Great Collaboration: UCSD is Linked to CICESE at 10G Coupling OptIPortals via SAGE at Many Sites

22 Great Displays Stack

23 32 Screens Stacked in QI s VRoom

24 Dynamically Showing Billions of XYZ Lidar Points Vid Petrovic, CISA3, UCSD QI

25 The CAVEcam: 3D Stereo Panoramic Photography Adding Photorealism to VR With the CAVEcam, in addition to photographically capturing an image that says: See what I have seen, we can now capture a complete 3D photo environment, recreate it in a VR setting, and claim: Be where I have been. Source: Dick Ainsworth

26 CAVEcam April 29, 2011 Photo: Greg Wickham

27 Image Acquisition: Collect 72 images for Each Eye

28 Image Acquisition & Processing: 0ne for Each Eye

29 Displays Correctly in the StarCAVE

30

31 The Great New WAVE 70 Megapixels WAVE VR display 7 high by 5 wide HD panels built for the SME Building, UCSD 125,000 Cores (50 nvidia 780s), 200TF (single precision), 1 Terabit CAD Design by Greg Dawe, as Always

32 Last Week in Greece

33

34 Camlot First Concept Rendering by Greg Dawe

35

36 Rotation on Center of Convergence Wood Pole ~5ft Lamp ~10ft AOV 50

37 Rotation 6 From Center of Convergence Wood Pole ~5ft Lamp ~10ft AOV 50 P JPG

38 Camlot: Raspberry PI test system 8 cameras Produces a 3-frame stereo panorama ~ 90 wide by 57 high. Still images and videos

39 Camlot Prototype

40 One Ring of Cameras

41 Two Rings of Cameras

42 100 Cameras in 5 Rings

43 72 Cameras in 4 Rings

44 Exploded Close-Up with Ribbon Cables

45 Exploded View with Ribbon Cables

46 Goals of final Camlot system Goals of final system 4 PI steradian stereo capture 30 fps video and a higher resolution still capture mode Approximately 20/20 acuity or higher Reproducible by other scientific laboratories Applicable to diverse areas of scientific and cultural research Monitoring of natural environments Underwater studies Biological population surveys studies Archeological dig surveys Cultural heritage High quality capture of as-built environments

47 Full 360 by 180 Video Panoramas Conclusions There are excellent solutions for still stereo, like the CAVEcam There are solutions for 360 by <180 monoscopic video, like Google Street View, GoPro cameras There are no good solutions yet for 360 by 180 stereo video It is a active and challenging research area We need help with stitching, streaming, etc.

48 Camlot Team QI Tom DeFanti, Truong Nguyen, Greg Dawe, Andrew Prudhomme, Jason Juang, Jurgen Schulze, Dan Sandin EVL Maxine Brown, Luc Renambot, Jonas Talandis, Jason Leigh, Dan Sandin, Lance Long

49 Our planning, research, and education efforts are made possible by current funding and support from: US National Science Foundation (NSF) awards CNS and CNS Calit2 Director s Office Calit2 UCSD Division (Qualcomm Institute) RCI Program, UCSD University of Illinois at Chicago, Argonne National Laboratory, and Northwestern University for StarLight networking and management Pacific Wave and CENIC NTT Network Innovations Lab Pacific Interface, Inc. Many corporate donors

50 Nothing New Here Today, I Swear

51 My 4 Questions for Dinner Time 1. What is the difference between Big Data and Great Data? 2. In a 5/5/14 NYT article on response to "What far-off technology will be commonplace in a decade?" Peter Thiel answered "Fully immersive virtual reality. Why is it going to take a decade? 3. Will 3D in movies (conventional or animated) or video games be commercially successful? 4. What makes collaborative VR experiences easy/good and what makes them difficult/bad?

52 4 Questions for Now 1. How can visualization be better integrated with data and computation? 2. What are some important considerations for doing visualization at the exascale? 3. How should one go about developing GPU codes which make use of the GPU(s) for both computation and visualization? 4. What are the major bottlenecks/challenges in the visualization field?