Managing and Monitoring SUSE Enterprise Storage Tim Serong Eric Jackson Senior Clustering Engineer tserong@suse.com Senior Software Developer Distributed Storage ejackson@suse.com
SUSE Enterprise Storage...in 30 seconds or less 2 Massively scalable No bottlenecks or single points of failure Object storage, block storage Based on Ceph
SUSE Enterprise Storage...in 30 seconds or less 3 Data stored redundantly Lots of disks (OSDs) in lots of storage nodes A few monitor (MON) nodes All on commodity hardware
Deployment
DIY Deployment 5 Boot a bunch of nodes Install SUSE Linux Enterprise Server 12 Add SUSE Enterprise Storage Run ceph-deploy
DIY Deployment # ceph-deploy new node1 node2 node3 # ceph-deploy mon create-initial # ceph-deploy osd prepare node1:sdb # ceph-deploy osd prepare node1:sdc # ceph-deploy calamari --master node0 \ connect node1 node2 node3 6
Or, Take a Crowbar to it 7 Same technology as SUSE OpenStack Cloud Install one admin node PXE boot everything else Click to deploy
Or, Take a Crowbar to it 8
Or, Take a Crowbar to it 9
Or, Take a Crowbar to it 10
Or, Take a Crowbar to it 11
Or, Take a Crowbar to it 12
Or, Take a Crowbar to it 13
Or, Take a Crowbar to it 14
Or, Take a Crowbar to it 15
Or, Take a Crowbar to it 16
Or, Take a Crowbar to it 17
Or, Take a Crowbar to it 18
Or, Take a Crowbar to it 19
iscsi
Configuring iscsi today RBD mapped devices Targetcli - interactive - command line 21 Iblock backstore
Configuring iscsi challenges 22 Simple configuration requires a dozen steps Sometimes command order matters Locally saved configuration Synchronizing redundancy across gateways Experimentation can be cumbersome
Lrbd 23 Uses rbd and targetcli Configuration stored in Ceph Synchronization is automatic Experimentation is quick Configuration format is JSON Command line options, man pages
Lrbd information Github https://github.com/suse/lrbd Wiki tutorial https://github.com/suse/lrbd.wiki 30 configuration samples - /usr/share/doc/packages/lrbd/samples 24
Monitoring
Is it Working? # ceph status cluster 565bbaaf-11e9-4105-934a-6b468f0b7b7e health HEALTH_OK monmap e1: 1 mons at {node1=192.168.124.81:6789/0} election epoch 1, quorum 0 node1 osdmap e12: 2 osds: 2 up, 2 in pgmap v118: 64 pgs, 1 pools, 1024 kb data, 3 objects 74192 kb used, 38817 MB / 38889 MB avail 64 active+clean 26
Is it Working? # ceph status cluster 565bbaaf-11e9-4105-934a-6b468f0b7b7e health HEALTH_WARN 33 pgs degraded; 35 pgs stuck... monmap e1: 3 mons at {ceph2=...,ceph3=...,ceph4=...}, election epoch 22, quorum 0,1,2 ceph2,ceph3,... osdmap e411: 52 osds: 52 up, 52 in pgmap v1014: 4288 pgs, 4 pools, 0 bytes data,... 2466 MB used, 12659 GB / 12662 GB avail 33 active+degraded 2 active+remapped 4253 active+clean 27
Is it Working? # ceph status cluster c9d3ae97-2f4c-4d91-a3f7-ff42bce754df health HEALTH_WARN 2174 pgs backfill; 367 pgs backfilling; 3271 pgs degraded; 23 pgs down; 57 pgs peering; 35 pgs recovering; 188 pgs recovery_wait; 227 pgs stale; 26 pgs stuck inactive; 3065 pgs stuck unclean; recovery 2519083/14004502 objects degraded (17.988%); 1/148 in osds are down monmap e3: 3 mons at {a001=172.16.25.1:6789/0,a002=172.168.25.2:6789/0,a003=172.16.25.3:6789/0}, election... osdmap e51357: 168 osds: 147 up, 148 in pgmap v7243525: 20480 pgs, 5 pools, 1183 GB data, 5609 objects 3330 GB used, 88846 GB / 92177 GB avail 2519083/14004502 objects degraded (17.988%) 9 inactive 16755 active+clean 10 degraded+remapped 9 active+degraded+remapped 361 active+degraded+remapped+backfilling 39 stale+active+degraded+remapped+wait_backfill 30 peering 154 active+recovery_wait 171 stale+active+clean 22 active+recovery_wait+degraded+remapped 14 active+remapped+wait_backfill 23 down+peering 3 stale+active+degraded+remapped+backfilling 3 stale+active+recovery_wait 2 active+remapped 19 degraded 673 active+degraded 3 active+remapped+backfilling 4 remapped+peering 6 active+recovery_wait+remapped 3 active+recovery_wait+degraded 11 stale+active+degraded 2121 active+degraded+remapped+wait_backfill 35 active+recovering 28
Is it Working? 29 What do we care about, right now? Overall cluster health MON quorum OSD status PG status Disk used/free Is anything dead?
Is it Working Well? 30 What do we care about, long term? What's CPU bound? What's disk bound? What's network bound?
Enter Calamari and Romana 31 Ceph GUI, included with SUSE Enterprise Storage Calamari is the backend (REST API) Romana is the frontend (GUI) Provides monitoring and some management
Cluster Status 32
Cluster Status 33
Cluster Performance 34
Cluster Performance 35
Cluster Performance 36
Management
When Everything is Fine 38 OSD management Pool / Placement Group management iscsi targets Users, authentication Adding new nodes, disks, etc.
Cluster Settings 39
OSD Management 40
Pool / Placement Group Management 41
Pool / Placement Group Management 42
CLI Tools for... 43 iscsi targets Users, authentication Adding new nodes, disks, etc.
When Everything is Not Fine 44 Dead disks Dead nodes Half the building is on fire...
Calamari will tell you about it... 45
...and help you find the problem 46
Again, Other Tools for... 47 Redeploying Adding new nodes Replacing OSDs
Beneath the Surface
Calamari Consists Of... 49 Romana (the frontend) Calamari (the backend REST API) Salt (communication and minor configuration) Graphite (metrics, graphs)
Usually... # zypper in romana # calamari-ctl initialize [INFO] Loading configuration [INFO] Starting/enabling salt...... Username (leave blank to use 'root'):... # ceph-deploy calamari --master node0 \ connect node1 node2 node3 50
In Practice, Right Now Calamari Node: Storage Nodes: 51 Romana, Calamari, Salt Master, Graphite/Carbon Salt Minion, Diamond
Variations More Salt with your Calamari? Longer metric retention /etc/carbon/storage-schemas.conf 52 BYO Graphite Less Salt with your Calamari?
Alternatives
Alternatives 54 Intel Virtual Storage Manager InkScope openattic
Virtual Storage Manager Intel Openstack Horizon Interface Django, Python Released 2.0 beta 1 on August 21 Packages incomplete at OBS - 55 https://build.opensuse.org/project/show/home:swiftgist:vsm
InkScope Orange Labs AngularJS and Python REST api Rados gateway user management Packages at OBS - https://build.opensuse.org/package/show/home:swiftgist/inkscope Images from https://github.com/inkscope/inkscope/tree/master/screenshots 56
57
58
59
openattic IT-Novum GmbH AngularJS and Python REST api Released 2.0.1 on July 21 Complete Linux storage management system Demo - http://demo.openattic.org/openattic/ 60
61
62
63
Questions?
For more information about SUSE Enterprise Storage: http://suse.com/storage Thank you. 65
66
67 Corporate Headquarters +49 911 740 53 0 (Worldwide) Join us on: Maxfeldstrasse 5 90409 Nuremberg Germany www.suse.com www.opensuse.org
Unpublished Work of SUSE LLC. All Rights Reserved. This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE LLC. Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.