1. 故障现象
登录时一直提示用户名密码错误,但确认密码和用户名没有问题.之前1个月其实已经通过控制台重置过证书日期,并看到证书日期是2年后的了.于是怀疑是不是证书过期了.
当时急着排除故障就没有截图了.引用官网的2张图
2. 排除过程
2.1 确认故障原因
2.2.1 登录vcenter命令行
2.2.2 查看证书日期
果然,证书日期是今天凌晨过期的
root@pana-vc [ ~ ]# for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list);do echo STORE $i;sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text|egrep "Alias|Not After"; done
STORE MACHINE_SSL_CERT
Alias : __MACHINE_CERT
Not After : Sep 16 05:18:40 2024 GMT
STORE TRUSTED_ROOTS
Alias : 79f3d5a2a7ee50b15c6698c7c796b104ff2dc855
Not After : Oct 4 08:49:45 2028 GMT
Alias : f7083e9abad114d69d3bb1eb0657bb0f904c4a59
Not After : Oct 15 04:35:10 2030 GMT
STORE TRUSTED_ROOT_CRLS
Alias : 8900883067e989a792fdd6aad2182d8ab965e7bc
Alias : be6990d6d91b1ec33f0b1a1d16acc1e4bcb71a14
STORE machine
Alias : machine
Not After : Oct 15 04:35:10 2030 GMT
STORE vsphere-webclient
Alias : vsphere-webclient
Not After : Oct 15 04:35:10 2030 GMT
STORE vpxd
Alias : vpxd
Not After : Sep 16 05:19:07 2024 GMT
STORE vpxd-extension
Alias : vpxd-extension
Not After : Oct 15 04:35:10 2030 GMT
STORE SMS
Alias : sms_self_signed
Not After : Oct 10 08:54:35 2028 GMT
STORE BACKUP_STORE
Alias : bkp___MACHINE_CERT
Not After : Oct 9 20:49:45 2020 GMT
Alias : bkp_machine
Not After : Oct 9 08:40:35 2020 GMT
Alias : bkp_vsphere-webclient
Not After : Oct 9 08:40:35 2020 GMT
Alias : bkp_vpxd
Not After : Oct 9 08:40:35 2020 GMT
Alias : bkp_vpxd-extension
Not After : Oct 9 08:40:36 2020 GMT
Alias : bkp__MACHINE_CERT
Not After : Oct 20 04:25:11 2022 GMT
STORE BACKUP_STORE_H5C
Alias : bkpmachine
Not After : Oct 20 04:26:33 2022 GMT
Alias : bkpvsphere-webclient
Not After : Oct 20 04:26:33 2022 GMT
Alias : bkpvpxd-extension
Not After : Oct 20 04:26:34 2022 GMT
Alias : bkpvpxd
Not After : Oct 20 04:26:34 2022 GMT
2.2 故障排除
2.2.1 工具准备
我们需要准备2个软件,当时一直在正文里找这2个文件的下载链接.浪费了很多时间,其实在右测有下载地址.
实在找不到的朋友可以访问我的百度网盘
链接：https://pan.baidu.com/s/1PCOP_PQ6HXe-5hdRWG4LlA?pwd=MQiu
提取码：MQiu
2.2.1.1 checksts.py
https://kb.vmware.com/s/article/79248
2.2.1.2 fixsts.sh
https://kb.vmware.com/s/article/76719
2.2.1.3 工具上传
用winscp一直报错,官方好像也有KB说怎么解决.我看到winscp有报错就直接放弃使用其他服务进行上传
# scp checksts.py fixsts.sh 192.168.101.200:/root
2.2.2 检查证书日期
# cd /root
# python checksts.py
# for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list);do echo STORE $i;sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text|egrep "Alias|Not After"; done
至此确诊了证书过期
2.2.3 证书续订
# chmod +x fixsts.sh
# ./fixsts.sh
注意!!!
这里需要输入一次Vcenter控制台登录密码,我第一次搞错了输入了vcenter的root密码,导致服务启动失败.
root@pana-vc [ ~ ]# ./fixsts.sh
NOTE: This works on external and embedded PSCs
This script will do the following
1: Regenerate STS certificate
What is needed?
1: Offline snapshots of VCs/PSCs
2: SSO Admin Password
IMPORTANT: This script should only be run on a single PSC per SSO domain
==================================
Resetting STS certificate for pana-vc.pana.dc started on Fri Oct 21 01:08:07 UTC 2022
Detected DN: cn=192.168.101.200,ou=Domain Controllers,dc=vsphere,dc=local
Detected PNID: 192.168.101.200
Detected PSC: 192.168.101.200
Detected SSO domain name: vsphere.local
Detected Machine ID: c3d30267-b3c9-4108-8580-54a6890b4133
Detected IP Address: 192.168.101.200
Domain CN: dc=vsphere,dc=local
==================================
==================================
Detected Root's certificate expiration date: 2030 Oct 15
Detected today's date: 2022 Oct 21
==================================
Exporting and generating STS certificate
Status : Success
Using config file : /tmp/vmware-fixsts/certool.cfg
Status : Success
Enter password for [email protected]:
Highest tenant credentials index : 1
Exporting tenant 1 to /tmp/vmware-fixsts
Deleting tenant 1
Highest trusted cert chains index: 1
Exporting trustedcertchain 1 to /tmp/vmware-fixsts
Deleting trustedcertchain 1
Applying newly generated STS certificate to SSO domain
adding new entry "cn=TenantCredential-1,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"
adding new entry "cn=TrustedCertChain-1,cn=TrustedCertificateChains,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"
Replacement finished - Please restart services on all vCenters and PSCs in your SSO domain
==================================
IMPORTANT: In case you're using HLM (Hybrid Linked Mode) without a gateway, you would need to re-sync the certs from Cloud to On-Prem after following this procedure
==================================
==================================
2.2.4 重启服务
2.2.4.1 停止所有服务
root@pana-vc [ ~ ]# service-control --stop --all
Perform stop operation. vmon_profile=ALL, svc_names=None, include_coreossvcs=True, include_leafossvcs=True
2022-10-21T00:51:44.164Z Service vmware-vmon does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2022-10-21T00:53:46.043Z Done running commandsbin/service', u'vmware-vmon', 'stop']
2022-10-21T00:53:46.043Z Successfully stopped service vmware-vmon
Successfully stopped vmon services. Profile ALL.
2022-10-21T00:53:46.054Z Service vmware-psc-client does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2022-10-21T00:53:46.054Z Running command: ['/sbin/service', u'vmware-psc-client', 'status']
2022-10-21T00:53:46.700Z Done running command
Successfully stopped service vmware-psc-client
2022-10-21T00:53:48.889Z Service vmdnsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2022-10-21T00:53:48.889Z Running command: ['/sbin/service', u'vmdnsd', 'status']
2022-10-21T00:53:49.349Z Done running command
Successfully stopped service vmdnsd
2022-10-21T00:53:49.506Z Service vmware-stsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2022-10-21T00:53:49.507Z Running command: ['/sbin/service', u'vmware-stsd', 'status']
2022-10-21T00:53:49.959Z Done running command
Successfully stopped service vmware-stsd
2022-10-21T00:53:54.357Z Service vmware-sts-idmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2022-10-21T00:53:54.357Z Running command: ['/sbin/service', u'vmware-sts-idmd', 'status']
2022-10-21T00:53:54.636Z Done running command
Successfully stopped service vmware-sts-idmd
2022-10-21T00:53:55.722Z Service vmcad does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2022-10-21T00:53:55.722Z Running command: ['/sbin/service', u'vmcad', 'status']
2022-10-21T00:53:56.223Z Done running command
Successfully stopped service vmcad
2022-10-21T00:53:56.333Z Service vmdird does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2022-10-21T00:53:56.333Z Running command: ['/sbin/service', u'vmdird', 'status']
2022-10-21T00:53:56.696Z Done running command
Successfully stopped service vmdird
2022-10-21T00:53:56.808Z Service vmafdd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2022-10-21T00:53:56.808Z Running command: ['/sbin/service', u'vmafdd', 'status']
2022-10-21T00:53:57.406Z Done running command
Successfully stopped service vmafdd
2022-10-21T00:53:57.603Z Service lwsmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2022-10-21T00:53:57.603Z Running command: ['/sbin/service', u'lwsmd', 'status']
2022-10-21T00:53:58.235Z Done running command
Successfully stopped service lwsmd
2.2.4.2 启动所有服务
root@pana-vc [ ~ ]# service-control --start --all
Perform start operation. vmon_profile=ALL, svc_names=None, include_coreossvcs=True, include_leafossvcs=True
2022-10-21T01:08:47.990Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'lwsmd']
2022-10-21T01:08:47.995Z Done running command
2022-10-21T01:08:47.999Z Service lwsmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2022-10-21T01:08:47.999Z Running command: ['/sbin/service', u'lwsmd', 'status']
2022-10-21T01:08:48.041Z Done running command
Successfully started service lwsmd
2022-10-21T01:08:48.050Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmafdd']
2022-10-21T01:08:48.062Z Done running command
2022-10-21T01:08:48.067Z Service vmafdd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2022-10-21T01:08:48.067Z Running command: ['/sbin/service', u'vmafdd', 'status']
2022-10-21T01:08:48.104Z Done running command
Successfully started service vmafdd
2022-10-21T01:08:48.108Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdird']
2022-10-21T01:08:48.119Z Done running command
2022-10-21T01:08:48.123Z Service vmdird does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2022-10-21T01:08:48.123Z Running command: ['/sbin/service', u'vmdird', 'status']
2022-10-21T01:08:48.162Z Done running command
Successfully started service vmdird
2022-10-21T01:08:48.168Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmcad']
2022-10-21T01:08:48.180Z Done running command
2022-10-21T01:08:48.188Z Service vmcad does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2022-10-21T01:08:48.188Z Running command: ['/sbin/service', u'vmcad', 'status']
2022-10-21T01:08:48.229Z Done running command
Successfully started service vmcad
2022-10-21T01:08:48.236Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-sts-idmd']
2022-10-21T01:08:48.247Z Done running command
2022-10-21T01:08:48.251Z Service vmware-sts-idmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2022-10-21T01:08:48.251Z Running command: ['/sbin/service', u'vmware-sts-idmd', 'status']
2022-10-21T01:08:48.287Z Done running command
Successfully started service vmware-sts-idmd
2022-10-21T01:08:48.291Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-stsd']
2022-10-21T01:08:48.304Z Done running command
2022-10-21T01:08:48.308Z Service vmware-stsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2022-10-21T01:08:48.308Z Running command: ['/sbin/service', u'vmware-stsd', 'status']
2022-10-21T01:08:48.344Z Done running command
Successfully started service vmware-stsd
2022-10-21T01:08:48.348Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdnsd']
2022-10-21T01:08:48.354Z Done running command
2022-10-21T01:08:48.357Z Service vmdnsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2022-10-21T01:08:48.357Z Running command: ['/sbin/service', u'vmdnsd', 'status']
2022-10-21T01:08:48.390Z Done running command
Successfully started service vmdnsd
2022-10-21T01:08:48.398Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-psc-client']
2022-10-21T01:08:48.409Z Done running command
2022-10-21T01:08:48.412Z Service vmware-psc-client does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2022-10-21T01:08:48.412Z Running command: ['/sbin/service', u'vmware-psc-client', 'status']
2022-10-21T01:08:48.453Z Done running command
Successfully started service vmware-psc-client
Successfully started vmon services. Profile ALL.
之前由于密码错误启动时有这样的报错,看到这个报错,一般都是前面fixsts的时候密码搞错了,再次确认下是控制台的密码,不是ssh的root密码
root@pana-vc [ ~ ]# service-control --start --all
略
Service-control failed. Error Failed to start vmon services.vmon-cli RC=1, stderr=Failed to start vapi-endpoint, vpxd-svcs, cm services. Error: Operation timed out
3. 故障解决
此时EXPIRED的证书已经没有了.可以看到
root@pana-vc [ ~ ]# python checksts.py
2 VALID CERTS
================
LEAF CERTS:
[] Certificate 33:87:6B:66:90:ED:24:90:23:66:54:B3:65:EF:8F:68:C5:6F:5C:64 will expire in 730 days (2.0 years).
ROOT CERTS:
[] Certificate F7:08:3E:9A:BA:D1:14:D6:9D:3B:B1:EB:06:57:BB:0F:90:4C:4A:59 will expire in 2916 days (7.0 years).
0 EXPIRED CERTS
================
LEAF CERTS:
None
ROOT CERTS:
None
此时Vcenter已经可以正常登陆.到Vcenter配置中可以看到STS证书日期已经变成2年后.
至此Vcenter证书更新全部完成.