Managing and Monitoring SUSE Enterprise Storage

Transcription

1 Managing and Monitoring SUSE Enterprise Storage Tim Serong Eric Jackson Senior Clustering Engineer Senior Software Developer Distributed Storage

2 SUSE Enterprise Storage...in 30 seconds or less 2 Massively scalable No bottlenecks or single points of failure Object storage, block storage Based on Ceph

3 SUSE Enterprise Storage...in 30 seconds or less 3 Data stored redundantly Lots of disks (OSDs) in lots of storage nodes A few monitor (MON) nodes All on commodity hardware

4 Deployment

5 DIY Deployment 5 Boot a bunch of nodes Install SUSE Linux Enterprise Server 12 Add SUSE Enterprise Storage Run ceph-deploy

6 DIY Deployment # ceph-deploy new node1 node2 node3 # ceph-deploy mon create-initial # ceph-deploy osd prepare node1:sdb # ceph-deploy osd prepare node1:sdc # ceph-deploy calamari --master node0 \ connect node1 node2 node3 6

7 Or, Take a Crowbar to it 7 Same technology as SUSE OpenStack Cloud Install one admin node PXE boot everything else Click to deploy

8 Or, Take a Crowbar to it 8

9 Or, Take a Crowbar to it 9

10 Or, Take a Crowbar to it 10

11 Or, Take a Crowbar to it 11

12 Or, Take a Crowbar to it 12

13 Or, Take a Crowbar to it 13

14 Or, Take a Crowbar to it 14

15 Or, Take a Crowbar to it 15

16 Or, Take a Crowbar to it 16

17 Or, Take a Crowbar to it 17

18 Or, Take a Crowbar to it 18

19 Or, Take a Crowbar to it 19

20 iscsi

21 Configuring iscsi today RBD mapped devices Targetcli - interactive - command line 21 Iblock backstore

22 Configuring iscsi challenges 22 Simple configuration requires a dozen steps Sometimes command order matters Locally saved configuration Synchronizing redundancy across gateways Experimentation can be cumbersome

23 Lrbd 23 Uses rbd and targetcli Configuration stored in Ceph Synchronization is automatic Experimentation is quick Configuration format is JSON Command line options, man pages

24 Lrbd information Github Wiki tutorial 30 configuration samples - /usr/share/doc/packages/lrbd/samples 24

25 Monitoring

26 Is it Working? # ceph status cluster 565bbaaf-11e a-6b468f0b7b7e health HEALTH_OK monmap e1: 1 mons at {node1= :6789/0} election epoch 1, quorum 0 node1 osdmap e12: 2 osds: 2 up, 2 in pgmap v118: 64 pgs, 1 pools, 1024 kb data, 3 objects kb used, MB / MB avail 64 active+clean 26

27 Is it Working? # ceph status cluster 565bbaaf-11e a-6b468f0b7b7e health HEALTH_WARN 33 pgs degraded; 35 pgs stuck... monmap e1: 3 mons at {ceph2=...,ceph3=...,ceph4=...}, election epoch 22, quorum 0,1,2 ceph2,ceph3,... osdmap e411: 52 osds: 52 up, 52 in pgmap v1014: 4288 pgs, 4 pools, 0 bytes data, MB used, GB / GB avail 33 active+degraded 2 active+remapped 4253 active+clean 27

28 Is it Working? # ceph status cluster c9d3ae97-2f4c-4d91-a3f7-ff42bce754df health HEALTH_WARN 2174 pgs backfill; 367 pgs backfilling; 3271 pgs degraded; 23 pgs down; 57 pgs peering; 35 pgs recovering; 188 pgs recovery_wait; 227 pgs stale; 26 pgs stuck inactive; 3065 pgs stuck unclean; recovery / objects degraded (17.988%); 1/148 in osds are down monmap e3: 3 mons at {a001= :6789/0,a002= :6789/0,a003= :6789/0}, election... osdmap e51357: 168 osds: 147 up, 148 in pgmap v : pgs, 5 pools, 1183 GB data, 5609 objects 3330 GB used, GB / GB avail / objects degraded (17.988%) 9 inactive active+clean 10 degraded+remapped 9 active+degraded+remapped 361 active+degraded+remapped+backfilling 39 stale+active+degraded+remapped+wait_backfill 30 peering 154 active+recovery_wait 171 stale+active+clean 22 active+recovery_wait+degraded+remapped 14 active+remapped+wait_backfill 23 down+peering 3 stale+active+degraded+remapped+backfilling 3 stale+active+recovery_wait 2 active+remapped 19 degraded 673 active+degraded 3 active+remapped+backfilling 4 remapped+peering 6 active+recovery_wait+remapped 3 active+recovery_wait+degraded 11 stale+active+degraded 2121 active+degraded+remapped+wait_backfill 35 active+recovering 28

29 Is it Working? 29 What do we care about, right now? Overall cluster health MON quorum OSD status PG status Disk used/free Is anything dead?

30 Is it Working Well? 30 What do we care about, long term? What's CPU bound? What's disk bound? What's network bound?

31 Enter Calamari and Romana 31 Ceph GUI, included with SUSE Enterprise Storage Calamari is the backend (REST API) Romana is the frontend (GUI) Provides monitoring and some management

32 Cluster Status 32

33 Cluster Status 33

34 Cluster Performance 34

35 Cluster Performance 35

36 Cluster Performance 36

37 Management

38 When Everything is Fine 38 OSD management Pool / Placement Group management iscsi targets Users, authentication Adding new nodes, disks, etc.

39 Cluster Settings 39

40 OSD Management 40

41 Pool / Placement Group Management 41

42 Pool / Placement Group Management 42

43 CLI Tools for iscsi targets Users, authentication Adding new nodes, disks, etc.

44 When Everything is Not Fine 44 Dead disks Dead nodes Half the building is on fire...

45 Calamari will tell you about it... 45

46 ...and help you find the problem 46

47 Again, Other Tools for Redeploying Adding new nodes Replacing OSDs

48 Beneath the Surface

49 Calamari Consists Of Romana (the frontend) Calamari (the backend REST API) Salt (communication and minor configuration) Graphite (metrics, graphs)

50 Usually... # zypper in romana # calamari-ctl initialize [INFO] Loading configuration [INFO] Starting/enabling salt Username (leave blank to use 'root'):... # ceph-deploy calamari --master node0 \ connect node1 node2 node3 50

51 In Practice, Right Now Calamari Node: Storage Nodes: 51 Romana, Calamari, Salt Master, Graphite/Carbon Salt Minion, Diamond

52 Variations More Salt with your Calamari? Longer metric retention /etc/carbon/storage-schemas.conf 52 BYO Graphite Less Salt with your Calamari?

53 Alternatives

54 Alternatives 54 Intel Virtual Storage Manager InkScope openattic

55 Virtual Storage Manager Intel Openstack Horizon Interface Django, Python Released 2.0 beta 1 on August 21 Packages incomplete at OBS

56 InkScope Orange Labs AngularJS and Python REST api Rados gateway user management Packages at OBS - Images from 56

57 57

58 58

59 59

60 openattic IT-Novum GmbH AngularJS and Python REST api Released on July 21 Complete Linux storage management system Demo

61 61

62 62

63 63

64 Questions?

65 For more information about SUSE Enterprise Storage: Thank you. 65

66 66

67 67 Corporate Headquarters (Worldwide) Join us on: Maxfeldstrasse Nuremberg Germany

68 Unpublished Work of SUSE LLC. All Rights Reserved. This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE LLC. Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.