CentOS7.6 enable Ceph nautilus's dashboard failure handling

Keywords: Linux Ceph Python ansible pip

[TOC]

1. Environmental description

Software Edition
operating system CentOS7.6
ceph nautilus, v14.2.4

The nautilus version of CEPH deploy is installed the same as the luminous version. Please refer to my previous article:
https://blog.51cto.com/ygqygq2/2161917

2. Failed to enable ceph dashboard

There is an error when the mgr module turns on the dashboard. Although the -- force surface is enabled, the dashboard service is not actually started because the dashboard user command prompt setting is not supported.

3. Troubleshooting process

mgr node view mgr startup log

Detailed log:

2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  e: '/usr/bin/ceph-mgr'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  0: '/usr/bin/ceph-mgr'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  1: '-f'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  2: '--cluster'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  3: 'ceph'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  4: '--id'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  5: 'utyun-node2'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  6: '--setuser'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  7: 'ceph'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  8: '--setgroup'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  9: 'ceph'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn respawning with exe /usr/bin/ceph-mgr
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  exe_path /proc/self/exe
2019-11-08 12:44:23.737 7fb0b7715b80  0 ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable), process ceph-mgr, pid 158648
2019-11-08 12:44:23.738 7fb0b7715b80  0 pidfile_write: ignore empty --pid-file
2019-11-08 12:44:23.776 7fb0b7715b80  1 mgr[py] Loading python module 'ansible'
2019-11-08 12:44:23.834 7fb0b7715b80 -1 mgr[py] Module not found: 'ansible'
2019-11-08 12:44:23.834 7fb0b7715b80 -1 mgr[py] Traceback (most recent call last):
  File "/usr/share/ceph/mgr/ansible/__init__.py", line 5, in <module>
    from .module import Module
  File "/usr/share/ceph/mgr/ansible/module.py", line 10, in <module>
    import requests
  File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 58, in <module>
    from . import utils
  File "/usr/lib/python2.7/site-packages/requests/utils.py", line 32, in <module>
    from .exceptions import InvalidURL
  File "/usr/lib/python2.7/site-packages/requests/exceptions.py", line 10, in <module>
    from .packages.urllib3.exceptions import HTTPError as BaseHTTPError
  File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py", line 95, in load_module
    raise ImportError("No module named '%s'" % (name,))
ImportError: No module named 'requests.packages.urllib3'

2019-11-08 12:44:23.835 7fb0b7715b80 -1 mgr[py] Class not found in module 'ansible'
2019-11-08 12:44:23.835 7fb0b7715b80 -1 mgr[py] Error loading module 'ansible': (2) No such file or directory
2019-11-08 12:44:23.835 7fb0b7715b80  1 mgr[py] Loading python module 'balancer'
2019-11-08 12:44:23.850 7fb0b7715b80  1 mgr[py] Loading python module 'crash'
2019-11-08 12:44:23.864 7fb0b7715b80  1 mgr[py] Loading python module 'dashboard'
2019-11-08 12:44:24.133 7fb0b7715b80 -1 mgr[py] Module not found: 'dashboard'
2019-11-08 12:44:24.133 7fb0b7715b80 -1 mgr[py] Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/__init__.py", line 38, in <module>
    from .module import Module, StandbyModule
  File "/usr/share/ceph/mgr/dashboard/module.py", line 75, in <module>
    from .grafana import push_local_dashboards
  File "/usr/share/ceph/mgr/dashboard/grafana.py", line 7, in <module>
    import requests
  File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 58, in <module>
    from . import utils
  File "/usr/lib/python2.7/site-packages/requests/utils.py", line 32, in <module>
    from .exceptions import InvalidURL
  File "/usr/lib/python2.7/site-packages/requests/exceptions.py", line 10, in <module>
    from .packages.urllib3.exceptions import HTTPError as BaseHTTPError
  File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py", line 95, in load_module
    raise ImportError("No module named '%s'" % (name,))
ImportError: No module named 'requests.packages.urllib3'

2019-11-08 12:44:24.133 7fb0b7715b80 -1 mgr[py] Class not found in module 'dashboard'
2019-11-08 12:44:24.133 7fb0b7715b80 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory
2019-11-08 12:44:24.134 7fb0b7715b80  1 mgr[py] Loading python module 'deepsea'
2019-11-08 12:44:24.197 7fb0b7715b80 -1 mgr[py] Module not found: 'deepsea'
2019-11-08 12:44:24.197 7fb0b7715b80 -1 mgr[py] Traceback (most recent call last):
  File "/usr/share/ceph/mgr/deepsea/__init__.py", line 1, in <module>
    from .module import DeepSeaOrchestrator
  File "/usr/share/ceph/mgr/deepsea/module.py", line 13, in <module>
    import requests
  File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 58, in <module>
    from . import utils
  File "/usr/lib/python2.7/site-packages/requests/utils.py", line 32, in <module>
    from .exceptions import InvalidURL
  File "/usr/lib/python2.7/site-packages/requests/exceptions.py", line 10, in <module>
    from .packages.urllib3.exceptions import HTTPError as BaseHTTPError
  File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py", line 95, in load_module
    raise ImportError("No module named '%s'" % (name,))
ImportError: No module named 'requests.packages.urllib3'

2019-11-08 12:44:24.198 7fb0b7715b80 -1 mgr[py] Class not found in module 'deepsea'
2019-11-08 12:44:24.198 7fb0b7715b80 -1 mgr[py] Error loading module 'deepsea': (2) No such file or directory
2019-11-08 12:44:24.198 7fb0b7715b80  1 mgr[py] Loading python module 'devicehealth'
2019-11-08 12:44:24.211 7fb0b7715b80  1 mgr[py] Loading python module 'influx'
2019-11-08 12:44:24.226 7fb0b7715b80  1 mgr[py] Loading python module 'insights'
2019-11-08 12:44:24.240 7fb0b7715b80  1 mgr[py] Loading python module 'iostat'
2019-11-08 12:44:24.254 7fb0b7715b80  1 mgr[py] Loading python module 'localpool'
2019-11-08 12:44:24.268 7fb0b7715b80  1 mgr[py] Loading python module 'orchestrator_cli'
2019-11-08 12:44:24.304 7fb0b7715b80  1 mgr[py] Loading python module 'pg_autoscaler'
2019-11-08 12:44:24.350 7fb0b7715b80  1 mgr[py] Loading python module 'progress'
2019-11-08 12:44:24.383 7fb0b7715b80  1 mgr[py] Loading python module 'prometheus'
2019-11-08 12:44:24.446 7fb0b7715b80  1 mgr[py] Loading python module 'rbd_support'
2019-11-08 12:44:24.523 7fb0b7715b80  1 mgr[py] Loading python module 'restful'
2019-11-08 12:44:24.674 7fb0b7715b80  1 mgr[py] Loading python module 'selftest'
2019-11-08 12:44:24.689 7fb0b7715b80  1 mgr[py] Loading python module 'status'
2019-11-08 12:44:24.714 7fb0b7715b80  1 mgr[py] Loading python module 'telegraf'
2019-11-08 12:44:24.737 7fb0b7715b80  1 mgr[py] Loading python module 'telemetry'
2019-11-08 12:44:24.784 7fb0b7715b80 -1 mgr[py] Module not found: 'telemetry'
2019-11-08 12:44:24.784 7fb0b7715b80 -1 mgr[py] Traceback (most recent call last):
  File "/usr/share/ceph/mgr/telemetry/__init__.py", line 1, in <module>
    from .module import Module
  File "/usr/share/ceph/mgr/telemetry/module.py", line 10, in <module>
    import requests
  File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 58, in <module>
    from . import utils
  File "/usr/lib/python2.7/site-packages/requests/utils.py", line 32, in <module>
    from .exceptions import InvalidURL
  File "/usr/lib/python2.7/site-packages/requests/exceptions.py", line 10, in <module>
    from .packages.urllib3.exceptions import HTTPError as BaseHTTPError
  File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py", line 95, in load_module
    raise ImportError("No module named '%s'" % (name,))
ImportError: No module named 'requests.packages.urllib3'

2019-11-08 12:44:24.784 7fb0b7715b80 -1 mgr[py] Class not found in module 'telemetry'
2019-11-08 12:44:24.784 7fb0b7715b80 -1 mgr[py] Error loading module 'telemetry': (2) No such file or directory
2019-11-08 12:44:24.784 7fb0b7715b80  1 mgr[py] Loading python module 'test_orchestrator'
2019-11-08 12:44:24.826 7fb0b7715b80  1 mgr[py] Loading python module 'volumes'
2019-11-08 12:44:24.940 7fb0b7715b80  1 mgr[py] Loading python module 'zabbix'
2019-11-08 12:44:24.957 7fb0b7715b80 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: ansible, dashboard, deepsea, telemetry
2019-11-08 12:44:24.961 7fb0a2b49700  0 ms_deliver_dispatch: unhandled message 0x5601d8ebaa00 mon_map magic: 0 v1 from mon.0 v2:172.16.138.26:3300/0

Try to install the module manually by pip. No matter what plug-ins are installed, an error is reported. Upgrade Python to solve the problem of pip installation: yum -y update python.

However, the installation of ANSI ble module in pip fails, and the dashboard still prompts importerror: no module named 'requests. Packages. Urlib3'


From the log point of view, it seems that ceph mgr enables the module. In principle, python uses urlib3 to download and install it from the network. Therefore, to change the direction, first solve the problem of urlib3 module.

pip uninstall urlib3 and install it in yum mode instead.

Check the log when the dashboard module is enabled in real time again,
tail -f /var/log/ceph/ceph-mgr.utyun-node2.log

Finally, as you wish, the dashboard was enabled successfully without any error.

[root@utyun-node1 external-ceph]# ceph mgr module ls |more
{
    "enabled_modules": [
        "dashboard",
        "iostat",
        "pg_autoscaler",
        "restful"
    ],

4. summary

The following is a summary of the steps for ceph nautilus to open the dashboard:

mgr node operation:

yum -y install ceph-mgr-dashboard
pip uninstall urllib3
yum install python-urllib3 -y

ceph management node operation:

ceph mgr module enable dashboard
ceph config set mgr mgr/dashboard/server_addr 0.0.0.0
ceph config set mgr mgr/dashboard/server_port 7000
ceph config set mgr mgr/dashboard/ssl false
ceph dashboard ac-user-create admin password administrator

reference material:
[1] https://docs.ceph.com/docs/nautilus/mgr/dashboard/

Posted by dougp23 on Fri, 15 Nov 2019 09:49:29 -0800