Kubernetes (K8s)安装部署过程(四)之创建高可用etcd集群

其他 2021-01-29 17:59:30 阅读次数: 0

kuberntes 系统使用 etcd 存储所有数据，本文档介绍部署一个三节点高可用 etcd 集群的步骤，这三个节点复用 kubernetes master 机器

1、TLS 认证文件

需要为 etcd 集群创建加密通信的 TLS 证书，etcd集群认证用，除了本机有，分发到其他node节点

[root@k8s_Master ssl]# scp ca.pem kubernetes-key.pem kubernetes.pem [email protected]:/etc/kubernetes/ssl/
[email protected]'s password: 
ca.pem                                                                                                                                                                                                                                      100% 1359   992.7KB/s   00:00    
kubernetes-key.pem                                                                                                                                                                                                                          100% 1679     1.8MB/s   00:00    
kubernetes.pem                                                                                                                                                                                                                              100% 1619     1.8MB/s   00:00    
[root@k8s_Master ssl]# scp ca.pem kubernetes-key.pem kubernetes.pem [email protected]:/etc/kubernetes/ssl/
[email protected]'s password: 
ca.pem                                                                                                                                                                                                                                      100% 1359   860.8KB/s   00:00    
kubernetes-key.pem                                                                                                                                                                                                                          100% 1679     2.1MB/s   00:00    
kubernetes.pem

2、下载包文件

[root@k8s_Master package]# wget https://github.com/etcd-io/etcd/releases/download/v3.4.10/etcd-v3.4.10-linux-arm64.tar.gz
[root@k8s_Master package]# cd etcd-v3.4.10-linux-amd64
[root@k8s_Master etcd-v3.4.10-linux-amd64]# mv etcd* /usr/local/bin/

3、创建 etcd 的 systemd unit 文件

在/usr/lib/systemd/system/目录下创建文件etcd.service，内容如下。注意替换IP地址为你自己的etcd集群的主机IP。

# mkdir /var/lib/etcd
# chmod -R 700 /var/lib/etcd
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/local/bin/etcd \
  --name ${ETCD_NAME} \
  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
  --listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \
  --listen-client-urls ${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
  --advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \
  --initial-cluster-token ${ETCD_INITIAL_CLUSTER_TOKEN} \
  --initial-cluster infra1=https://192.168.0.221:2380,infra2=https://192.168.0.222:2380,infra3=https://192.168.0.223:2380 \
  --initial-cluster-state new \
  --data-dir=${ETCD_DATA_DIR}
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

配置注意事项：所有节点都必须配置此文件，并且注意下面4个注意事项。

  • 指定 etcd 的工作目录为 /var/lib/etcd，数据目录为 /var/lib/etcd，需在启动服务前创建这个目录，否则启动服务的时候会报错“Failed at step CHDIR spawning /usr/bin/etcd: No such file or directory”；
  • 为了保证通信安全，需要指定 etcd 的公私钥(cert-file和key-file)、Peers 通信的公私钥和 CA 证书(peer-cert-file、peer-key-file、peer-trusted-ca-file)、客户端的CA证书（trusted-ca-file）；
  • 创建 kubernetes.pem 证书时使用的 kubernetes-csr.json 文件的 hosts 字段包含所有 etcd 节点的IP，否则证书校验会出错；
  • --initial-cluster-state 值为 new 时，--name 的参数值必须位于 --initial-cluster 列表中；

4、创建etcd环境变量文件

文件位置：/etc/etcd/etcd.conf，yum安装完之后该文件会存在，删除重建即可。

[root@k8s_Master ssl]# vim /etc/etcd/etcd.conf

# [member]
ETCD_NAME=infra1
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.0.221:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.221:2379"

#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.221:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.221:2379"

这是192.168.0.221节点的配置，其他两个etcd节点只要将上面的IP地址改成相应节点的IP地址即可。ETCD_NAME换成对应节点的infra1/2/3。

5、启动 etcd 服务

1) 系统命令启动

[root@k8s_Master ~]# systemctl enable etcd
[root@k8s_Master ~]# systemctl start etcd
[root@k8s_Master ~]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2020-08-24 16:32:20 CST; 17s ago
     Docs: https://github.com/coreos
 Main PID: 3408 (etcd)
    Tasks: 13 (limit: 17529)
   Memory: 27.1M
   CGroup: /system.slice/etcd.service
           └─3408 /usr/local/bin/etcd --name etcd1 --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --trusted-ca-f>

Aug 24 16:32:20 k8s_Node1 etcd[3408]: established a TCP streaming connection with peer 5d3da0a181835f54 (stream Message writer)
Aug 24 16:32:20 k8s_Node1 etcd[3408]: ready to serve client requests
Aug 24 16:32:20 k8s_Node1 etcd[3408]: ready to serve client requests
Aug 24 16:32:20 k8s_Node1 etcd[3408]: published {Name:etcd1 ClientURLs:[https://192.168.0.222:2379]} to cluster ed83350b82359b62
Aug 24 16:32:20 k8s_Node1 etcd[3408]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
Aug 24 16:32:20 k8s_Node1 etcd[3408]: serving client requests on 192.168.0.222:2379
Aug 24 16:32:20 k8s_Node1 systemd[1]: Started Etcd Server.
Aug 24 16:32:20 k8s_Node1 etcd[3408]: established a TCP streaming connection with peer 1b23aa32587ec13e (stream MsgApp v2 writer)
Aug 24 16:32:20 k8s_Node1 etcd[3408]: established a TCP streaming connection with peer 1b23aa32587ec13e (stream Message writer)
Aug 24 16:32:27 k8s_Node1 etcd[3408]: /health OK (status code 200)
/usr/local/bin/etcd     --name etcd0     --cert-file=/etc/kubernetes/ssl/kubernetes.pem     --key-file=/etc/kubernetes/ssl/kubernetes-key.pem     --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem     --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem     --trusted-ca-file=/etc/kubernetes/ssl/ca.pem     --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem     --initial-advertise-peer-urls https://192.168.0.221:2380     --listen-peer-urls https://192.168.0.221:2380     --listen-client-urls https://192.168.0.221:2379,http://127.0.0.1:2379     --advertise-client-urls https://192.168.0.221:2379     --initial-cluster-token etcd-cluster     --initial-cluster etcd0=https://192.168.0.221:2380,etcd1=https://192.168.0.222:2380,etcd2=https://192.168.0.223:2380     --initial-cluster-state new     --data-dir=/var/lib/etcd

6、执行最后一个对应的日志如下

2020-08-14 03:41:15.728340 I | etcdmain: etcd Version: 3.4.10
2020-08-14 03:41:15.728385 I | etcdmain: Git SHA: 18dfb9cca
2020-08-14 03:41:15.728389 I | etcdmain: Go Version: go1.12.17
2020-08-14 03:41:15.728392 I | etcdmain: Go OS/Arch: linux/amd64
2020-08-14 03:41:15.728395 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
2020-08-14 03:41:15.728429 N | etcdmain: the server is already initialized as member before, starting as etcd member...
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2020-08-14 03:41:15.728456 I | embed: peerTLS: cert = /etc/kubernetes/ssl/kubernetes.pem, key = /etc/kubernetes/ssl/kubernetes-key.pem, trusted-ca = /etc/kubernetes/ssl/ca.pem, client-cert-auth = false, crl-file = 
2020-08-14 03:41:15.729004 W | embed: The scheme of client url http://127.0.0.1:2379 is HTTP while peer key/cert files are presented. Ignored key/cert files.
2020-08-14 03:41:15.729081 I | embed: name = etcd2
2020-08-14 03:41:15.729088 I | embed: data dir = /var/lib/etcd
2020-08-14 03:41:15.729091 I | embed: member dir = /var/lib/etcd/member
2020-08-14 03:41:15.729094 I | embed: heartbeat = 100ms
2020-08-14 03:41:15.729097 I | embed: election = 1000ms
2020-08-14 03:41:15.729099 I | embed: snapshot count = 100000
2020-08-14 03:41:15.729104 I | embed: advertise client URLs = https://192.168.0.223:2379
2020-08-14 03:41:15.736925 I | etcdserver: starting member 1b23aa32587ec13e in cluster ed83350b82359b62
raft2020/08/14 03:41:15 INFO: 1b23aa32587ec13e switched to configuration voters=()
raft2020/08/14 03:41:15 INFO: 1b23aa32587ec13e became follower at term 0
raft2020/08/14 03:41:15 INFO: newRaft 1b23aa32587ec13e [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2020/08/14 03:41:15 INFO: 1b23aa32587ec13e became follower at term 1
raft2020/08/14 03:41:15 INFO: 1b23aa32587ec13e switched to configuration voters=(1955593796418715966)
raft2020/08/14 03:41:15 INFO: 1b23aa32587ec13e switched to configuration voters=(1955593796418715966 6718702834629697364)
raft2020/08/14 03:41:15 INFO: 1b23aa32587ec13e switched to configuration voters=(1955593796418715966 6718702834629697364 15280391182966381234)
2020-08-14 03:41:15.737712 W | auth: simple token is not cryptographically signed
2020-08-14 03:41:15.740245 I | rafthttp: starting peer 5d3da0a181835f54...
2020-08-14 03:41:15.740297 I | rafthttp: started HTTP pipelining with peer 5d3da0a181835f54
2020-08-14 03:41:15.748504 I | rafthttp: started streaming with peer 5d3da0a181835f54 (writer)
2020-08-14 03:41:15.748835 I | rafthttp: started streaming with peer 5d3da0a181835f54 (writer)
2020-08-14 03:41:15.749324 I | rafthttp: started peer 5d3da0a181835f54
2020-08-14 03:41:15.749356 I | rafthttp: added peer 5d3da0a181835f54
2020-08-14 03:41:15.749369 I | rafthttp: starting peer d40edad269b8dab2...
2020-08-14 03:41:15.749381 I | rafthttp: started HTTP pipelining with peer d40edad269b8dab2
2020-08-14 03:41:15.750767 I | rafthttp: started streaming with peer 5d3da0a181835f54 (stream MsgApp v2 reader)
2020-08-14 03:41:15.750813 I | rafthttp: started streaming with peer d40edad269b8dab2 (writer)
2020-08-14 03:41:15.750974 I | rafthttp: started streaming with peer 5d3da0a181835f54 (stream Message reader)
2020-08-14 03:41:15.751182 I | rafthttp: started streaming with peer d40edad269b8dab2 (writer)
2020-08-14 03:41:15.752148 I | rafthttp: started peer d40edad269b8dab2
2020-08-14 03:41:15.752180 I | rafthttp: added peer d40edad269b8dab2
2020-08-14 03:41:15.752197 I | rafthttp: started streaming with peer d40edad269b8dab2 (stream MsgApp v2 reader)
2020-08-14 03:41:15.752217 I | rafthttp: started streaming with peer d40edad269b8dab2 (stream Message reader)
2020-08-14 03:41:15.752315 I | etcdserver: starting server... [version: 3.4.10, cluster version: to_be_decided]
raft2020/08/14 03:41:15 INFO: 1b23aa32587ec13e switched to configuration voters=(1955593796418715966 6718702834629697364 15280391182966381234)
2020-08-14 03:41:15.753175 I | etcdserver/membership: added member 1b23aa32587ec13e [https://192.168.0.223:2380] to cluster ed83350b82359b62
raft2020/08/14 03:41:15 INFO: 1b23aa32587ec13e switched to configuration voters=(1955593796418715966 6718702834629697364 15280391182966381234)
2020-08-14 03:41:15.753291 I | etcdserver/membership: added member 5d3da0a181835f54 [https://192.168.0.221:2380] to cluster ed83350b82359b62
raft2020/08/14 03:41:15 INFO: 1b23aa32587ec13e switched to configuration voters=(1955593796418715966 6718702834629697364 15280391182966381234)
2020-08-14 03:41:15.753418 I | etcdserver/membership: added member d40edad269b8dab2 [https://192.168.0.222:2380] to cluster ed83350b82359b62
2020-08-14 03:41:15.754986 I | embed: ClientTLS: cert = /etc/kubernetes/ssl/kubernetes.pem, key = /etc/kubernetes/ssl/kubernetes-key.pem, trusted-ca = /etc/kubernetes/ssl/ca.pem, client-cert-auth = false, crl-file = 
2020-08-14 03:41:15.755097 I | embed: listening for peers on 192.168.0.223:2380
2020-08-14 03:41:15.757189 I | rafthttp: peer 5d3da0a181835f54 became active
2020-08-14 03:41:15.757216 I | rafthttp: established a TCP streaming connection with peer 5d3da0a181835f54 (stream Message reader)
2020-08-14 03:41:15.760649 I | rafthttp: established a TCP streaming connection with peer 5d3da0a181835f54 (stream MsgApp v2 reader)
2020-08-14 03:41:15.762092 I | rafthttp: peer d40edad269b8dab2 became active
2020-08-14 03:41:15.762124 I | rafthttp: established a TCP streaming connection with peer d40edad269b8dab2 (stream Message reader)
2020-08-14 03:41:15.762147 I | rafthttp: established a TCP streaming connection with peer d40edad269b8dab2 (stream MsgApp v2 reader)
raft2020/08/14 03:41:15 INFO: 1b23aa32587ec13e [term: 1] received a MsgHeartbeat message with higher term from 5d3da0a181835f54 [term: 173]
raft2020/08/14 03:41:15 INFO: 1b23aa32587ec13e became follower at term 173
raft2020/08/14 03:41:15 INFO: raft.node: 1b23aa32587ec13e elected leader 5d3da0a181835f54 at term 173
2020-08-14 03:41:15.817316 I | etcdserver: 1b23aa32587ec13e initialized peer connection; fast-forwarding 8 ticks (election ticks 10) with 2 active peer(s)

[root@k8s_Node2 ~]# 2020-08-14 03:41:15.998998 N | etcdserver/membership: set the initial cluster version to 3.0
2020-08-14 03:41:15.999051 I | etcdserver/api: enabled capabilities for version 3.0
2020-08-14 03:41:16.000181 I | etcdserver: published {Name:etcd2 ClientURLs:[https://192.168.0.223:2379]} to cluster ed83350b82359b62
2020-08-14 03:41:16.001285 I | rafthttp: established a TCP streaming connection with peer d40edad269b8dab2 (stream MsgApp v2 writer)
2020-08-14 03:41:16.002148 I | embed: ready to serve client requests
2020-08-14 03:41:16.002786 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
2020-08-14 03:41:16.002841 I | embed: ready to serve client requests
2020-08-14 03:41:16.003850 I | embed: serving client requests on 192.168.0.223:2379
2020-08-14 03:41:16.004068 I | rafthttp: established a TCP streaming connection with peer 5d3da0a181835f54 (stream Message writer)
2020-08-14 03:41:16.005011 I | rafthttp: established a TCP streaming connection with peer 5d3da0a181835f54 (stream MsgApp v2 writer)
2020-08-14 03:41:16.064656 I | rafthttp: established a TCP streaming connection with peer d40edad269b8dab2 (stream Message writer)
2020-08-14 03:41:17.711425 N | etcdserver/membership: updated the cluster version from 3.0 to 3.4
2020-08-14 03:41:17.711501 I | etcdserver/api: enabled capabilities for version 3.4

检查etcd服务集群是否正常

3.3的版本如下

[root@k8s_Master ~]# /usr/local/bin/etcdctl --ca-file=/etc/kubernetes/ssl/ca.pem  --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem --endpoints=https://192.168.0.221:2379,https://192.168.0.222:2379,https://192.168.0.223:2379 cluster-health 
member 1b23aa32587ec13e is healthy: got healthy result from https://192.168.0.223:2379
member 5d3da0a181835f54 is healthy: got healthy result from https://192.168.0.221:2379
member d40edad269b8dab2 is healthy: got healthy result from https://192.168.0.222:2379

3.4的版本 

[root@k8s_Master package]# etcdctl \
> --cacert=/etc/kubernetes/ssl/ca.pem  \
> --cert=/etc/kubernetes/ssl/kubernetes.pem \
> --key=/etc/kubernetes/ssl/kubernetes-key.pem \
> --endpoints=https://192.168.0.221:2379,https://192.168.0.222:2379,https://192.168.0.223:2379 \
> endpoint health


https://192.168.0.221:2379 is healthy: successfully committed proposal: took = 7.787866ms
https://192.168.0.222:2379 is healthy: successfully committed proposal: took = 9.082548ms
https://192.168.0.223:2379 is healthy: successfully committed proposal: took = 11.033587ms

检测结果如下

[root@k8s_Master package]# etcdctl --cacert=/etc/kubernetes/ssl/ca.pem  --cert=/etc/kubernetes/ssl/kubernetes.pem --key=/etc/kubernetes/ssl/kubernetes-key.pem --endpoints=https://192.168.0.221:2379 endpoint health
https://192.168.0.221:2379 is healthy: successfully committed proposal: took = 6.312375ms
[root@k8s_Master package]# etcdctl --cacert=/etc/kubernetes/ssl/ca.pem  --cert=/etc/kubernetes/ssl/kubernetes.pem --key=/etc/kubernetes/ssl/kubernetes-key.pem --endpoints=https://192.168.0.222:2379 endpoint health
https://192.168.0.222:2379 is healthy: successfully committed proposal: took = 7.003448ms
[root@k8s_Master package]# etcdctl --cacert=/etc/kubernetes/ssl/ca.pem  --cert=/etc/kubernetes/ssl/kubernetes.pem --key=/etc/kubernetes/ssl/kubernetes-key.pem --endpoints=https://192.168.0.223:2379 endpoint health
https://192.168.0.223:2379 is healthy: successfully committed proposal: took = 7.049361ms

最后查询集群状态

etcd3.4

[root@k8s_Master ssl]# etcdctl --write-out=table --cacert=/etc/kubernetes/ssl/ca.pem  --cert=/etc/kubernetes/ssl/kubernetes.pem --key=/etc/kubernetes/ssl/kubernetes-key.pem --endpoints=https://192.168.0.221:2379,https://192.168.0.222:2379,https://192.168.0.223:2379 endpoint status
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.0.221:2379 | 5d3da0a181835f54 |  3.4.10 |   20 kB |     false |      false |       192 |         14 |                 14 |        |
| https://192.168.0.222:2379 | d40edad269b8dab2 |  3.4.10 |   16 kB |      true |      false |       192 |         14 |                 14 |        |
| https://192.168.0.223:2379 | 1b23aa32587ec13e |  3.4.10 |   16 kB |     false |      false |       192 |         14 |                 14 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

列出集群成员

etcd3.3版本

[root@k8s_Master ~]# etcdctl --ca-file=/etc/kubernetes/ssl/ca.pem  --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem --endpoints=https://192.168.0.221:2379,https://192.168.0.222:2379,https://192.168.0.223:2379 member list
1b23aa32587ec13e: name=etcd2 peerURLs=https://192.168.0.223:2380 clientURLs=https://192.168.0.223:2379 isLeader=false
5d3da0a181835f54: name=etcd0 peerURLs=https://192.168.0.221:2380 clientURLs=https://192.168.0.221:2379 isLeader=true
d40edad269b8dab2: name=etcd1 peerURLs=https://192.168.0.222:2380 clientURLs=https://192.168.0.222:2379 isLeader=false

etcd3.4 

[root@k8s_Master ssl]# etcdctl --write-out=table --cacert=/etc/kubernetes/ssl/ca.pem  --cert=/etc/kubernetes/ssl/kubernetes.pem --key=/etc/kubernetes/ssl/kubernetes-key.pem --endpoints=https://192.168.0.221:2379,https://192.168.0.222:2379,https://192.168.0.223:2379 member list -w table           
+------------------+---------+-------+----------------------------+----------------------------+------------+
|        ID        | STATUS  | NAME  |         PEER ADDRS         |        CLIENT ADDRS        | IS LEARNER |
+------------------+---------+-------+----------------------------+----------------------------+------------+
| 1b23aa32587ec13e | started | etcd2 | https://192.168.0.223:2380 | https://192.168.0.223:2379 |      false |
| 5d3da0a181835f54 | started | etcd0 | https://192.168.0.221:2380 | https://192.168.0.221:2379 |      false |
| d40edad269b8dab2 | started | etcd1 | https://192.168.0.222:2380 | https://192.168.0.222:2379 |      false |
+------------------+---------+-------+----------------------------+----------------------------+------------+

猜你喜欢

转载自blog.csdn.net/baidu_38432732/article/details/107986539
今日推荐
国产云输入法——仅华为无云端数据上传安全问题
开源日报 | 工业开源项目OGG 1.0；姐姐，你要和我一起配置火狐吗；苹果AI遥遥落后？Fedora 40
开放签电子签章：停止新增，优化体验，前进更进（五一假期前工作）
开源日报 | 中学生开源前端动画引擎；全球首个Llama3 8B中文版开源模型；联想电脑恐出局；Linus讽刺AI炒作
“百模大战”必有一战 | 2024中国“百模大战”竞争格局分析
最强开源大模型 Llama 3 上架 Gitee AI
周排行
自媒体文章如何提高原创度以及如何检测原创度
开启qq邮箱的smtp服务
Qt程序单次启动（QSingleApplication类）
国外的外包网站
更新IDEA主题——放飞代码风格
cocos2dx 实现搓牌效果（翻牌效果），包括铺平动画
dict和json之间的互相转换
angular的一些思考
. Fibonacci数列是这样定义的： F[0] = 0 F[1] = 1 for each i ≥ 2: F[i] = F[i-1] + F[i-2] 因此，Fibonacci数列就形如：0, 1
洛谷P1064 金明的预算方案
每日归档
更多
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)
2024-04-22(39)
2024-04-21(0)
2024-04-20(6)
2024-04-19(5)
2024-04-18(0)
2024-04-17(5)
2024-04-16(70)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022