首页 > 其他分享> > OpenStack Newton HA高可用搭建

OpenStack Newton HA高可用搭建

2021-08-03 15:34:49 作者：互联网

OpenStack Newton HA高可用搭建

更新源：

yum clean all

yum makecache

修改网卡名称：

vi /etc/default/grub

net.ifnames=0

grub2-mkconfig -o /boot/grub2/grub.cfg

修改网卡文件名称

安装常用工具：

yum install vim net-tools wget ntpdate ntp bash-completion -y

vim /etc/hosts配置

10.1.1.141 controller1

10.1.1.142 controller2

10.1.1.143 controller3

10.1.1.144 compute1

10.1.1.146 cinder1

网络配置

external:10.254.15.128/27

admin mgt:10.1.1.0/24

tunnel:10.2.2.0/24

时间同步

ntpdate 10.6.0.2

vim /etc/ntp.conf

添加：

server 10.6.0.2 iburst

systemctl enable ntpd

systemctl restart ntpd

systemctl status ntpd

ntpq -p

免密码登陆：

ssh-keygen -t rsa

ssh-copy-id -i /root/.ssh/id_rsa.pub -p 22 root@10.254.15.141

ssh-copy-id -i /root/.ssh/id_rsa.pub -p 22 root@10.254.15.142

ssh-copy-id -i /root/.ssh/id_rsa.pub -p 22 root@10.254.15.143

一、搭建mariadb galera cluster

1. mariadb galera cluster集群介绍

mariadb galera cluster是mysql高可用性和可扩展性的解决方案

官网：http://galeracluster.com/products/

mariadb galera cluster是一套在mysql innodb存储引擎上面实现multi-master及数据实时同步的系统结构，业务层面无需做读写分离工作，数据库读写压力都能按照既定的规则分发到各节点上去。在数据方面完全兼容mariadb和mysql。

特性：

(1).同步复制synchronous replication

(2).active-active multi-master拓扑逻辑

(3).可对集群中任一节点进行数据读写

(4).自动成员控制，故障节点自动从集群中移除

(5).自动节点加入

(6).真正并行的复制，基于行级

(7).直接客户端连接，原生的mysql接口

(8).每个节点都包含完整的数据副本

(9).多台数据库中数据同步由wsrep接口实现

缺点：

(1).目前的复制仅仅支持innodb存储引擎，任何写入其他引擎的表，包括mysql.*表将不会复制，但ddl语句会被复制的，因此创建用户将会被复制，但是insert into mysql.user..将不会被复制的。

(2).delete操作不支持没有主键的表，没有主键的表在不同的节点顺序将不同，如果执行select...limit...将出现不同的结果集。

(3).在多主环境下lock/unlock tables不支持，以及锁函数get_lock(),release_lock()...

(4).查询日志不能保存表中。如果开启查询日志，只能保存到文件中。

(5).允许最大的事务大小由wsrep_max_ws_rows和wsrep_max_ws_size定义。任何大型操作将被拒绝。如大型的load data操作。

(6).由于集群是乐观的并发控制，事务commit可能在该阶段中止。如果有两个事务向在集群中不同的节点向同一行写入并提交，失败的节点将中止。对于集群别的中止，集群返回死锁错误代码(error:1213 sqlstate:40001(er_lock_deadlock))。

(7).xa事务不支持，由于在提交上可能回滚。

(8).整个集群的写入吞吐量是由最弱的节点限制，如果有一个节点变得缓慢，那么整个集群将是缓慢的。为了稳定的高性能要求，所有的节点应使用统一硬件。

(9).集群节点最少3个。

(10).如果ddl语句有问题将破坏集群。

2.安装mariadb galera

在三个node上分别执行

yum install -y MariaDB-server MariaDB-client galera xinetd rsync ntp ntpdate bash-completion percona-xtrabackup socat gcc gcc-c++ vim

systemctl start mariadb.service

mysql_secure_installation （执行设置mysql密码）

3.在三个节点上创建sst用户，目的是给xtrabakcup-v2数据同步使用

mysql -uroot -p

grant all privileges on *.* to 'sst'@'localhost' identified by 'gdxc1902';

flush privileges;

4.配置mariadb cluster集群

第一个节点上，添加内容如下：

vim /etc/my.cnf.d/client.cnf

[client]

port = 3306

socket = /var/lib/mysql/mysql.sock

vim /etc/my.cnf.d/server.cnf

添加内容

[isamchk]

key_buffer_size = 16M

#key_buffer_size这个参数是用来设置索引块（index blocks）缓存的大小，它被所有线程共享，严格说是它决定了数据库索引处理的速度，尤其是索引读的速度。

[mysqld]

datadir=/var/lib/mysql

#指定数据data目录绝对路径

innodb-data-home-dir = /var/lib/mysql

#指定innodb数据存放的家目录

basedir = /usr

#指定mariadb的安装路径，填写全路径可以解决相对路径所造成的问题

binlog_format=ROW

#该参数可以有三种设置值：row、statement和mixed。row代表二进制日志中记录数据表每一行经过写操作后被修改的最终值。各个参与同步的salve节点，也会参照这个最终值，将自己数据表上的数据进行修改；statement形式是在日志中记录数据操作过程，而非最终执行结果。各个参与同步的salve节点会解析这个过程，并形成最终记录；mixed设置值，是以上两种记录方式的混合体，mysql服务会自动选择当前运行状态下最适合的日志记录方式。

character-set-server = utf8

#设置数据库的字符集

collation-server = utf8_general_ci

#collation是描述数据在数据库中是按照什么规则来描述字符的，以及字符是如何被排序和比较，这里我们设置的字符格式是utf8_general_ci

max_allowed_packet = 256M

#设置mysql接受数据包最大值，比如你执行的sql语句过大，可能会执行失败，这个参数就是让能执行的sql语句大小调高

max_connections = 10000

#设置mysql集群最大的connection连接数，这个在openstack环境里非常重要

ignore-db-dirs = lost+found

#设置忽略把lost+found当做数据目录

init-connect = SET NAMES utf8

#设置初始化字符集编码（仅对非超级用户有效）

innodb_autoinc_lock_mode = 2

#这种模式下任何类型的inserts都不采用auto-inc锁，性能最好，但是在同一条语句内部产生auto-increment值间隙

innodb_buffer_pool_size = 2000M

#设置缓冲池字节大小，innodb缓存表和索引数据的内存区域；这个值设置的越大，在不止一次的访问相同的数据表数据时，消耗的磁盘i/o就越少。在一个专用的数据库服务器，则可能将其设置为高达80%的机器物理内存大小。不过在实际的测试中，发现无限的增大这个值，带来的性能提升也并不显著，对cpu的压力反而增大，设置合理的值才是最优。参考：https://my.oschina.net/realfighter/blog/368225

innodb_doublewrite = 0

#设置0是禁用doublewrite，一般在不关心数据一致性（比如使用了raid0）或文件系统可以保证不会出现部分写失效，你可以通过将innodb_doublewrite参数设置为0还禁用doublewrite。

innodb_file_format = Barracuda #设置文件格式为barracuda，barracude是innodb-plugin后引入的文件格式，同时barracude也支持antelope文件格式，barracude在数据压缩上优于antelope，配合下面的innodb_file_per_table=1使用

innodb_file_per_table = 1

#开启独立的表空间，使每个innodb的表，有自己的独立空间。如删除文件后可以回收那部分空间。

innodb_flush_log_at_trx_commit = 2

#默认值1：每一次事务提交或事务外的指令都需要把日志写入（flush）硬盘，这是很费时的。特别是使用电池供电缓存（battery backed up cache）时。设成2：意思是不写入硬盘而是写入系统缓存，日志仍然会每秒flush到硬盘，所以你一般不会丢失超过1-2秒的更新。设成0写入会更快一点，但安全方面比较差，mysql挂了可能会丢失事务的数据。设置为2只会在整个操作系统挂了时才可能丢失数据，所以在openstack环境里设置为2安全点。

innodb_flush_method = O_DIRECT

#设置打开、刷写模式，设置O_DIRECT的意思是innodb使用O_DIRECT打开数据文件，使用fsync()更新日志和数据文件；文件的打开方式为O_DIRECT会最小化缓冲对io的影响，该文件的io是直接在用户空间的buffer上操作的，并且io操作是同步的，因此不管是read()系统调用还是write()系统调用，数据都保证是从磁盘上读取的，所以单纯从写入的角度讲，O_DIRECT模式性能最差，但是在openstack环境下，设置这个模式可以减少操作系统级别vfs的缓存使用内存过多和innodb本身的buffer的缓存冲突，同时也算是给操作系统减少点压力。

innodb_io_capacity = 500

#这个参数数据控制innodb checkpoint时的io能力，一般可以按一块sas 15000转的磁盘200个计算，6块盘的sas做的raid10这个值可以配到600即可。如果是普通的sata一块盘只能按100算。这里要注意，对于普通机械硬盘，由于其随机io的iops最多也就是300，所以innodb_io_capacity开的过大，反而会造成磁盘io不均匀；如果是ssd场景，由于io能力大大增强，所以innodb_io_capacity可以调高，可以配置到2000以上，但是目前的openstack环境为了节约成本都是普通机械硬盘，所以这里一般根据自己的环境情况调试值

innodb_locks_unsafe_for_binlog = 1

#开启事物锁机制，强制mysql使用多版本数据一致性读。

innodb_log_file_size = 2000M

#如果对innodb数据表有大量的写入操作，那么选择合适的innodb_log_file_size值对提升mysql性能很重要。然而设置太大了，就会增加恢复的时间，因此在mysql崩溃或者突然断电等情况会令mysql服务器花很长时间来恢复。在笔者维护的openstack环境里，计算节点是12台，虚拟机在500左右。

innodb_read_io_threads = 8

#设置数据库从磁盘读文件的线程数，用于并发；根据服务器cpu核心数以及读写频率设置

innodb_write_io_threads = 8

#设置数据库写磁盘文件的线程数，用于并发；根据服务器cpu核心数以及读写频率设置

key_buffer_size = 64

#这个参数是用来设置索引块（index blocks）缓存的大小，它被所有线程共享，严格说是它决定了数据库索引处理的速度，尤其是索引读的速度。那我们怎么才能知道key_buffer_size的设置是否合理呢，一般可以检查状态值key_read_requests和key_reads，比例key_reads/key_read_requests应该尽可能的低，比如1:100,1:1000，1:10000。其值可以用show status like 'key_read%';命令查得。

myisam-recover-options = BACKUP

#设置自动修复myisam表的方式，backup模式会自动修复；这种模式如果在恢复过程中，数据文件被更改了，会将tbl_name.myd文件备份为tbl_name-datetime.bak。

myisam_sort_buffer_size = 64M

#设置myisam表发生变化时重新排序所需的缓冲值，一般64M足够

open_files_limit = 102400

#设置最大文件打开数，需要参照os的ulimit值和max_aonnections的大小

performance_schema = on

#打开收集数据库性能参数的数据库

query_cache_limit = 1M

#设置单个查询能够使用的缓冲区大小

query_cache_size = 0

#关闭query_cache

query_cache_type = 0

#关闭query_cache_type

skip-external-locking

#设置跳过“外部锁定”，当“外部锁定”起作用时，每个进程若要访问数据表，则必须等待之前的进程完成操作并解绑锁定；由于服务器访问数据表时经常需要等待解锁，因此会让mysql性能下降。所以这里设置跳过。

skip-name-resolve

#禁用DNS主机名查找

socket = /var/lib/mysql/mysql.sock

#设置mysql.sock的绝对路径

table_open_cache = 10000

#描述符缓存大小，可减少文件打开/关闭次数

thread_cache_size = 8

#可以用show status like 'open%tables';查看当前open_tables的值是多少，然后适当调整

thread_stack = 256K

#用来存放每个线程的标识信息，如线程id，线程运行时环境等，可以通过设置thread_stack来决定给每个线程分配多大内存

tmpdir = /tmp

#设置mysql临时文件存放的目录

user = mysql

#设置mysql数据的系统账户名

wait_timeout = 1800

#设置单个connection空闲连接的超时时间，mysql默认是8小时，这里我们设置为1800秒，也就是说，一个connection如果空闲超过30分钟，那么就会被释放

[galera]

wsrep_on=ON

wsrep_provider=/usr/lib64/galera/libgalera_smm.so

wsrep_cluster_address="gcomm://10.1.1.141,10.1.1.142,10.1.1.143"

#gcomm是特殊的地址，仅仅是galera cluster初始化启动时候使用

wsrep_cluster_name = openstack

#mysql cluster集群名称

wsrep_node_name=controller1

wsrep_node_address=10.1.1.141

#此节点IP地址

wsrep_sst_method=xtrabackup-v2

#有xtrabakcup模式和rsync模式，新版本的支持xtrabackup-v2模式。rsync在数据同步（sst和ist）的时候，速度最快，但是会锁住提供数据的节点，然后无法提供访问；xtrabackup只会短暂的锁住节点，基本不影响访问。SST:state snapshot transfer,节点初始化的方式，做数据的全量同步；IST：incremental state transfer,当一个节点加入，他当前的guid与现集群相同，且确实的数据能够在donor的writeset的cache中找到，则可以进行ist，否则只能全部初始化数据走sst模式。经过相关调研，目前xtravackup-v2模式是最好sst方式。

wsrep_sst_auth=sst:gdxc1902

#设置sst同步数据所需的mysql用户和密码，因为上面我们用的是xtrabackup-v2模式，那么xtrabackup-v2模式就会用到sst这个mysql用户去做节点之间的验证，gdxc是密码

wsrep_slave_threads=4

#指定线程数量，建议每个core启动4个复制线程，这个参数很大程度受到i/o能力的影响（本人维护的openstack集群服务器配置是：cpu 48core disk 1.8T 10000转的，这个参数设置的是12）

default_storage_engine=InnoDB

#设置数据库默认引擎为innodb

bind-address=10.1.1.141

#mysql服务绑定的IP

[mysqld_safe]

nice = 0

#调用系统的nice命令设置进程优先级，linux系统的普通用户只能在0-19中设置，mysql用户为普通用户，设置为0应该就是让mysql进程优先级最高了。

socket = /var/lib/mysql/mysql.sock

syslog

vim /etc/my.cnf.d/mysql-clients.cnf

添加内容

[mysqldump]

max_allowed_packet = 16M

#mysql根据配置文件会限制server接受的数据包大小。有时候大的插入和更新会受max_allowed_packet参数限制，导致写入或者更新失败。

quick

#强制mysqldump从服务器查询取得记录直接输出而不是取得所有记录后将他们缓存到内存中

quote-names

#使用（）引起表和列名。默认为打开状态，使用--skip-quote-names取消该选项。

注：相关详细参数可以参考官网：http://galeracluster.com/documentation-webpages/mysqlwsrepoptions.html

第二个和第三个节点的my.cnf配置如下，注意改下相关ip和节点名称：

vim /etc/my.cnf.d/client.cnf

[client]

port = 3306

socket = /var/lib/mysql/mysql.sock

vim /etc/my.cnf.d/server.cnf

[isamchk]

key_buffer_size = 16M

[mysqld]

datadir=/var/lib/mysql

innodb-data-home-dir = /var/lib/mysql

basedir = /usr

binlog_format=ROW

character-set-server = utf8

collation-server = utf8_general_ci

max_allowed_packet = 256M

max_connections = 10000

ignore-db-dirs = lost+found

init-connect = SET NAMES utf8

innodb_autoinc_lock_mode = 2

innodb_buffer_pool_size = 2000M

innodb_doublewrite = 0

innodb_file_format = Barracuda

innodb_file_per_table = 1

innodb_flush_log_at_trx_commit = 2

innodb_flush_method = O_DIRECT

innodb_io_capacity = 500

innodb_locks_unsafe_for_binlog = 1

innodb_log_file_size = 2000M

innodb_read_io_threads = 8

innodb_write_io_threads = 8

key_buffer_size = 64

myisam-recover-options = BACKUP

myisam_sort_buffer_size = 64M

open_files_limit = 102400

performance_schema = on

query_cache_limit = 1M

query_cache_size = 0

query_cache_type = 0

skip-external-locking

skip-name-resolve

socket = /var/lib/mysql/mysql.sock

table_open_cache = 10000

thread_cache_size = 8

thread_stack = 256K

tmpdir = /tmp

user = mysql

wait_timeout = 1800

[galera]

wsrep_on=ON

wsrep_provider=/usr/lib64/galera/libgalera_smm.so

wsrep_cluster_address="gcomm://10.1.1.141,10.1.1.142,10.1.1.143"

wsrep_cluster_name = openstack

wsrep_node_name=controller2

wsrep_node_address=10.1.1.142

wsrep_sst_method=xtrabackup-v2

wsrep_sst_auth=sst:gdxc1902

wsrep_slave_threads=4

default_storage_engine=InnoDB

bind-address=10.1.1.142

[mysqld_safe]

nice = 0

socket = /var/lib/mysql/mysql.sock

syslog

vim /etc/my.cnf.d/mysql-clients.cnf

[mysqldump]

max_allowed_packet = 16M

quick

quote-names

4.设置mysql最大连接数

修改完server.cnf，然后修改下mysql.service文件，让数据库最大支持连接数调整到10000（这样做是很有用的，笔者在维护openstack环境中，在vm数量比较大负载较高的时候，经常出现数据库连接数不够导致访问界面出现各种内容刷不出的情况）

vim /usr/lib/systemd/system/mariadb.service

在[Service]添加两行如下参数：

LimitNOFILE = 10000

LimitNPROC = 10000

都修改完毕后，执行

systemctl daemon-reload

等启动了mariadb服务就能通过show variables like 'max_connections';可查看当前连接数值

5.关于mysql服务的启动顺序

三个节点my.cnf都配置完成后，全部执行

systemctl stop mariadb.service

然后在第一个节点用下面的命令初始化启动mariadb集群服务

/usr/sbin/mysqld --wsrep-new-cluster --user=root &

其他两个节点分别启动mariadb

systemctl start mariadb.service

systemctl status mariadb.service

上面无法连接可尝试下面的命令：

/usr/sbin/mysqld --wsrep-cluster-address="gcomm://10.1.1.141:4567"

最后，其他两个节点启动成功了，再回到第一个节点执行：

pkill -9 mysql

systemctl start mariadb.service

systemctl status mariadb.service

注意：如果遇到启动不了服务的情况，看下具体的错误，如果是[ERROR]Can't init tc log错误可以通过以下方法解决：

cd /var/lib/mysql

chown mysql:mysql *

然后再重启服务即可，这是因为/var/lib/mysql/tc.log用户组和用户名不是mysql，更改下权限就可以了

6.查看mariadb数据库集群状态

mysql -uroot -p

show status like 'wsrep_cluster_size%';

show variables like 'wsrep_sst_meth%';

登陆这两节点的mysql里，发现mysql句群数变成了3，说明这个集群有3个节点了，集群已经搭建成功！

7.测试

下面来测试一把，在ctr3中创建一张表，并插入记录，看ctr1和ctr2中能否查询得到。

具体状况截图：

ctr3：创建test数据库

GREATE DATABASE test;

ctr2：使用ctr3创建的test数据库，并且创建一个名为example的表

USE TEST;

GREATE TABLE example (node_id INT PRIMARY KEY,node_name VARCHAR(30));

ctr1：插入一条测试数据

INSERT INTO test.example VALUES (1,'TEST1');

SELECT * FROM test.example;

ctr1 ctr2 ctr3分别查询这条数据，发现都能查询到

SELECT * FROM test.example;

二、安装rabbitmq cluster集群

1.每个节点都安装erlang

yum install -y erlang

2.每个节点都安装rabbitmq

yum install -y rabbitmq-server

3.每个节点都启动rabbitmq及设置开机启动

systemctl enable rabbitmq-server.service

systemctl restart rabbitmq-server.service

systemctl status rabbitmq-server.service

systemctl list-unit-files |grep rabbitmq-server.service

4.创建openstack，注意替换为自己合适的密码

rabbitmqctl add_user openstack gdxc1902

5.将openstack用户赋予权限

rabbitmqctl set_permissions openstack ".*" ".*" ".*"

rabbitmqctl set_user_tags openstack administrator

rabbitmqctl list_users

6.看下监听端口，rabbitmq用的是5672端口

netstat -ntlp |grep 5672

7.查看rabbitmq插件

/usr/lib/rabbitmq/bin/rabbitmq-plugins list

8.每个节点都打开rabbitmq相关插件

/usr/lib/rabbitmq/bin/rabbitmq-plugins enable rabbitmq_management mochiweb webmachine rabbitmq_web_dispatch amqp_client rabbitmq_management_agent

打开相关插件后，重启下rabbitmq服务

systemctl restart rabbitmq-server

浏览器输入：http://10.254.15.141:15672 默认用户名密码：guest/guest

通过这个界面，我们能直观的看到rabbitmq的运行和负载情况

当然我们可以不用guest，我们换一个另外用户，比如mqadmin

rabbitmqctl add_user mqadmin mqadmin

rabbitmqctl set_user_tags mqadmin administrator

rabbitmqctl set_permissions -p / mqadmin ".*" ".*" ".*"

我们还可以通过命令把密码换了，比如把guest用户的密码变成passw0rd

rabbitmqctl change_password guest passw0rd

9.查看rabbitma状态

rabbitmqctl status

10.集群配置

cat /var/lib/rabbitmq/.erlang.cookie 到每个节点上查看cookie

在controller1上操作：

scp /var/lib/rabbitmq/.erlang.cookie controller2:/var/lib/rabbitmq/.erlang.cookie

scp /var/lib/rabbitmq/.erlang.cookie controller3:/var/lib/rabbitmq/.erlang.cookie

在controller2上操作：

systemctl restart rabbitmq-server

rabbitmqctl stop_app

rabbitmqctl join_cluster --ram rabbit@controller1

rabbitmqctl start_app

在controller3上操作：

systemctl restart rabbitmq-server

rabbitmqctl stop_app

rabbitmqctl join_cluster --ram rabbit@controller1

rabbitmqctl start_app

查看集群状态：

rabbitmqctl cluster_status

11.集群管理

如果遇到rabbitmq脑裂情况，按以下步骤操作，重新设置集群：

登陆没加入集群的节点：

rabbitmqctl stop_app

rabbitmqctl reset

rabbitmqctl start_app

最后再重新执行添加集群操作即可！

如果某个节点下面这个路径/var/lib/rabbitmq 有多余的文件，请全部删除掉！

正常下面只有mnesia这一个文件夹

如果有一堆这样的文件就状态不对：

12.rabbitmq优化

rabbitmq一般优化的地方比较少，本人总结了下往上的一些资料以及mirantis官方的一篇博客，对rabbitmq的优化总结了下面几点，大家可以采纳：

a.尽可能的吧rabbitmq部署在单独的服务器中

因为使用专用节点，rabbitmq服务能尽全部的享受cpu资源，这样性能更高

b.让rabbitmq跑在hipe模式下

rabbitmq是用erlang语言编写的，而开启hipe能让erlang预编译运行，这样性能可以提升30%以上（关于测试结果，可以参考：https://github.com/binarin/rabbit-simple-benchmark/blob/master/report.md）。

但是开启hipe模式会让rabbitmq第一次启动很慢，大概需要2分钟；另外就是如果启用了hipe，rabbitmq的调试可能变得很难，因为hipe可以破坏错误回溯，使它们不可读。

enable hipe方法：

vim /etc/rabbitmq/rabbitmq.config

去掉{hipe_compile, true}前面注释（包括后面的逗号）即可，然后重启rabbitmq服务（重启过程中，你会发现启动过程很慢）。

scp -p /etc/rabbitmq/rabbitmq.config controller2:/etc/rabbitmq/rabbitmq.config

scp -p /etc/rabbitmq/rabbitmq.config controller3:/etc/rabbitmq/rabbitmq.config

c.不要对rpc队列使用队列镜像

研究表明，在3节点集群上启用队列镜像会使消息吞吐量下降两倍。另一方面，rpc消息生命周期会变短，如果消息变短丢失，它只导致当前正在进行的操作失败，因此没有镜像的整体rpc队列似乎是一个很好的权衡，不过也不是所有的消息队列都不启用镜像，ceilometer队列可以启用队列镜像，因为ceilometer的消息必须保留；但是如果你的环境装了ceilometer组件，最好给ceilometer单独一个rabbitmq集群，因为在通常情况下，ceilometer不会产生大量的消息队列，但是，如果ceilometer卡住有问题，那么关于ceilometer的消息队列就会很多溢出，这会造成rabbitmq集群的崩溃，这样必然导致其他openstack服务中断。

d.减少发送的指标数量或者频率

在openstack环境下运行rabbitmq的另一个最佳实践是减少发送的指标数量和或其频率。减少了相关指标数量和或频率，也自然减少了消息在rabbitmq服务中堆积的机会，这样rabbitmq就可以把更多的资源用来处理重要的openstack服务队列，以间接的提高rabbitmq性能。一般ceilometer和mongodb的消息队列可以尽量的挪开。

e.增加rabbitmq socket最大打开数

vim /etc/sysctl.conf

最下面添加：fs.file-max = 1000000

sysctl -p 执行生效

scp -p /etc/sysctl.conf controller2:/etc/sysctl.conf

scp -p /etc/sysctl.conf controller3:/etc/sysctl.conf

设置ulimit最大打开数

vim /etc/security/limits.conf

添加两行：

* soft nofile 655350

* hard nofile 655350

scp -p /etc/security/limits.conf controller2:/etc/security/limits.conf

scp -p /etc/security/limits.conf controller3:/etc/security/limits.conf

设置systemctl管理的服务文件最大打开数为1024000

vim /etc/systemd/system.conf

添加两行：

DefaultLimitNOFILE=1024000

DefaultLimitNPROC=1024000

scp -p /etc/systemd/system.conf controller2:/etc/systemd/system.conf

scp -p /etc/systemd/system.conf controller3:/etc/systemd/system.conf

改完后服务器都重启下，重启完毕查看值是否更改好，运行ulimit -Hn 查看

修改后，登陆rabbitmq web插件可以看到最大文件打开数和socket数都变大，默认值是最大文件打开数是1024和socket数是829.

参考：https://www.mirantis.com/blog/best-practices-rabbitmq-openstack/

https://www.qcloud.com/community/article/135

pcs restart导致rabbitmq用户丢失的bug：https://access.redhat.com/solutions/2374351

三、安装pacemaker

三个ctr节点需要安装以下包：

pacemaker

pcs(centos or rhel) or crmsh

corosync

fence-agent (centos or rhel) or cluster-glue

resource-agents

yum install -y lvm2 cifs-utils quota psmisc

yum install -y pcs pacemaker corosync fence-agents-all resource-agents

yum install -y crmsh

1.三个ctl节点都设置pcs服务开机启动

systemctl enable pcsd

systemctl enable corosync

systemctl start pcsd

systemctl status pcsd

2.设置hacluster用户密码，每个节点都需要设置

passwd hacluster

3.配置编写corosync.conf文件

vim /etc/corosync/corosync.conf

添加以下内容：

totem{

version:2

secauth:off

cluster_name:openstack_cluster

transport:udpu

}

nodelist{

node{

ring0_addr:controller1

nodeid:1

}

node{

ring0_addr:controller2

nodeid:2

}

node{

ring0_addr:controller3

nodeid:3

}

quorum{

provider:corosync_votequorum

}

logging{

to_logfile:yes

logfile:/var/log/cluster/corosync.log

to_syslog:yes

}

scp -p /etc/corosync/corosync.conf controller2:/etc/corosync/corosync.conf

scp -p /etc/corosync/corosync.conf controller3:/etc/corosync/corosync.conf

在3个节点上分别启动corosync服务

systemctl enable corosync

systemctl restart corosync

systemctl status corosync

4.设置集群相互验证，在controller1上操作即可

pcs cluster auth controller1 controller2 controller3 -u hacluster -p gdxc1902 --force

5.在controller1上创建并启动名为openstack_cluster的集群，其中controller1 controller2 controller3为集群成员

pcs cluster setup --force --name openstack_cluster controller1 controller2 controller3

6.设置集群自启动

pcs cluster enable --all

7.设置集群启动

pcs cluster start --all

8.查看并设置集群属性

pcs cluster status

9.检查pacemaker服务

ps aux |grep pacemaker

10.检验corosync的安装及当前corosync状态

corosync-cfgtool -s

corosync-cmapctl |grep members

pcs status corosync

11.检查配置是否正确（假若没有输出任何则配置正确）

crm_verify -L -V

如果想忽略这个错误，那么做以下操作

禁用STONITH

pcs property set stonith-enabled=false

无法仲裁时候，选择忽略

pcs property set no-quorum-policy=ignore

12.pcs其他命令

查看pcs支持的资源代理标准

pcs resource providers

13.通过crm设置VIP

crm

configure

crm(live)configure# primitive vip_public ocf:heartbeat:IPaddr2 params ip="10.254.15.140" cidr_netmask="27" nic=eth0 op monitor interval="30s"

crm(live)configure# primitive vip_management ocf:heartbeat:IPaddr2 params ip="10.1.1.140" cidr_netmask="24" nic=eth1 op monitor interval="30s"

commit

或者

pcs resource create vip_public ocf:heartbeat:IPaddr2 params ip="10.254.15.140" cidr_netmask="27" nic=eth0 op monitor interval="30s"

pcs resource create primitive vip_management ocf:heartbeat:IPaddr2 params ip="10.1.1.140" cidr_netmask="24" nic=eth1 op monitor interval="30s"

四、安装haproxy

1.安装haproxy

在三个节点上分别安装haproxy

yum install -y haproxy

systemctl enable haproxy.service

2.跟rsyslog集合配置haproxy日志，在三个节点上都操作

cd /etc/rsyslog.d/

vim haproxy.conf

添加：

$ModLoad imudp

$UDpServerRun 514

$template Haproxy,"%rawmsg% \n"

local0.=info -/var/log/haproxy.log;Haproxy

local0.notice -var/log/haproxy-status.log;Haproxy

local0.* ~

scp -p /etc/rsyslog.d/haproxy.conf controller2:/etc/rsyslog.d/haproxy.conf

scp -p /etc/rsyslog.d/haproxy.conf controller3:/etc/rsyslog.d/haproxy.conf

systemctl restart rsyslog.service

systemctl status rsyslog.service

3.在三个节点上配置haproxy.cfg

cd /etc/haproxy

mv haproxy.cfg haproxy.cfg.orig

vim haproxy.cfg

添加下面内容：

global

log 127.0.0.1 local0

log 127.0.0.1 local1 notice

maxconn 16000

chroot /usr/share/haproxy

user haproxy

group haproxy

daemon

defaults

log global

mode http

option tcplog

option dontlognull

retries 3

option redispatch

maxconn 10000

contimeout 5000

clitimeout 50000

srvtimeout 50000

frontend stats-front

bind *:8088

mode http

default_backend stats-back

backend stats-back

mode http

balance source

stats uri /haproxy/stats

stats auth admin:gdxc1902

listen RabbitMQ-Server-Cluster

bind 10.1.1.140:56720

mode tcp

balance roundrobin

option tcpka

server controller1 controller1:5672 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

server controller2 controller2:5672 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

server controller3 controller3:5672 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

listen RabbitMQ-Web

bind 10.254.15.140:15673

mode tcp

balance roundrobin

option tcpka

server controller1 controller1:15672 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

server controller2 controller2:15672 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

server controller3 controller3:15672 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

listen Galera-Cluster

bind 10.1.1.140:3306

balance leastconn

mode tcp

option tcplog

option httpchk

server controller1 controller1:3306 check port 9200 inter 20s fastinter 2s downinter 2s rise 3 fall 3

server controller2 controller2:3306 check port 9200 inter 20s fastinter 2s downinter 2s rise 3 fall 3

server controller3 controller3:3306 check port 9200 inter 20s fastinter 2s downinter 2s rise 3 fall 3

listen keystone_admin_cluster

bind 10.1.1.140:35357

balance source

option httpchk

option httplog

option httpclose

server controller1 controller1:35358 check inter 10s fastinter 2s downinter 2s rise 30 fall 3

server controller2 controller2:35358 check inter 10s fastinter 2s downinter 2s rise 30 fall 3

server controller3 controller3:35358 check inter 10s fastinter 2s downinter 2s rise 30 fall 3

listen keystone_public_internal_cluster

bind 10.1.1.140:5000

balance source

option httpchk

option httplog

option httpclose

server controller1 controller1:5002 check inter 10s fastinter 2s downinter 2s rise 30 fall 3

server controller2 controller2:5002 check inter 10s fastinter 2s downinter 2s rise 30 fall 3

server controller3 controller3:5002 check inter 10s fastinter 2s downinter 2s rise 30 fall 3

listen Memcache_Servers

bind 10.1.1.140:22122

balance roundrobin

mode tcp

option tcpka

server controller1 controller1:11211 check inter 10s fastinter 2s downinter 2s rise 30 fall 3

server controller2 controller2:11211 check inter 10s fastinter 2s downinter 2s rise 30 fall 3

server controller3 controller3:11211 check inter 10s fastinter 2s downinter 2s rise 30 fall 3

listen dashboard_cluster

bind 10.254.15.140:80

balance source

option httpchk

option httplog

option httpclose

server controller1 controller1:8080 check inter 2000 fall 3

server controller2 controller2:8080 check inter 2000 fall 3

server controller3 controller3:8080 check inter 2000 fall 3

listen glance_api_cluster

bind 10.1.1.140:9292

balance source

option httpchk

option httplog

option httpclose

server controller1 controller1:9393 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

server controller2 controller2:9393 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

server controller3 controller3:9393 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

listen glance_registry_cluster

bind 10.1.1.140:9090

balance roundrobin

mode tcp

option tcpka

server controller1 controller1:9191 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

server controller2 controller2:9191 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

server controller3 controller3:9191 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

listen nova_compute_api_cluster

bind 10.1.1.140:8774

balance source

option httpchk

option httplog

option httpclose

server controller1 controller1:9774 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

server controller2 controller2:9774 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

server controller3 controller3:9774 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

listen nova-metadata-api_cluster

bind 10.1.1.140:8775

balance source

option httpchk

option httplog

option httpclose

server controller1 controller1:9775 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

server controller2 controller2:9775 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

server controller3 controller3:9775 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

listen nova_vncproxy_cluster

bind 10.1.1.140:6080

balance source

option tcpka

option tcplog

server controller1 controller1:6080 check inter 2000 rise 2 fall 5

server controller2 controller2:6080 check inter 2000 rise 2 fall 5

server controller3 controller3:6080 check inter 2000 rise 2 fall 5

listen neutron_api_cluster

bind 10.1.1.140:9696

balance source

option httpchk

option httplog

option httpclose

server controller1 controller1:9797 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

server controller2 controller2:9797 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

server controller3 controller3:9797 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

listen cinder_api_cluster

bind 10.1.1.140:8776

balance source

option httpchk

option httplog

option httpclose

server controller1 controller1:8778 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

server controller2 controller2:8778 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

server controller3 controller3:8778 check inter 10s fastinter 2s downinter 3s rise 3 fall 3

scp -p /etc/haproxy/haproxy.cfg controller2:/etc/haproxy/haproxy.cfg

scp -p /etc/haproxy/haproxy.cfg controller3:/etc/haproxy/haproxy.cfg

systemctl restart haproxy.service

systemctl status haproxy.service

参数解释，具体的看分享的《haproxy终极参考手册》：

inter<delay>:设定健康状态检查的时间间隔，单位为毫秒，默认为2000；也可以使用fastinter和downinter来根据服务器状态优化此时间延迟。

rise<count>:设定健康状态检查中，某离线的server从离线状态转换至正常状态需要成功检查次数。

fall<count>:默认server从正常转换为不可用状态需要检查的次数。

3.配置haproxy能监控galera数据库集群

在controller1上进入mysql，创建clustercheck

grant process on *.* to 'clustercheckuser'@'localhost' identified by 'gdxc1902';

flush privileges;

三个节点分别创建clustercheck文本，里面是clustercheckuser用户密码

vim /etc/sysconfig/clustercheck

添加：

MYSQL_USERNAME=clustercheckuser

MYSQL_PASSWORD=gdxc1902

MYSQL_HOST=localhost

MYSQL_PORT=3306

scp -p /etc/sysconfig/clustercheck controller2:/etc/sysconfig/clustercheck

scp -p /etc/sysconfig/clustercheck controller3:/etc/sysconfig/clustercheck

确认下是否存在/usr/bin/clustercheck,如果没有从网上下载一个，然后放到/usr/bin目录下面，记得chmod +x /usr/bin/clustercheck 赋予权限

这个脚本的作用就是让haproxy能监控galera cluster状态

scp -p /usr/bin/clustercheck controller2:/usr/bin/clustercheck

scp -p /usr/bin/clustercheck controller3:/usr/bin/clustercheck

在controller1上检查haproxy服务状态

clustercheck （存在/usr/bin/clustercheck可以直接运行clustercheck命令）

结合xinetd监控galera服务（三个节点安装xinetd）

yum install -y xinetd

vim /etc/xinetd.d/mysqlchk

添加以下内容：

# default: on

# description: mysqlchk

service mysqlchk

{

# this is a config for xinetd, place it in /etc/xinetd.d/

disable = no

flags = REUSE

socket_type = stream

port = 9200

wait = no

user = nobody

server = /usr/bin/clustercheck

log_on_failure = USERID

only_from = 0.0.0.0/0

# recommended to put the IPs that need

# to connect exclusively (security purposes)

per_source = UNLIMITED

}

scp -p /etc/xinetd.d/mysqlchk controller2:/etc/xinetd.d/mysqlchk

scp -p /etc/xinetd.d/mysqlchk controller3:/etc/xinetd.d/mysqlchk

vim /etc/services

最后一行添加：mysqlchk 9200/tcp # mysqlchk

scp -p /etc/services controller2:/etc/services

scp -p /etc/services controller3:/etc/services

重启xinetd服务

systemctl restart xinetd.service

systemctl status xinetd.service

5.三个节点修改内核参数

echo 'net.ipv4.ip_nonlocal_bind = 1'>>/etc/sysctl.conf

echo 'net.ipv4.ip_forward=1'>>/etc/sysctl.conf

sysctl -p

第一个参数的意思是设置haproxy能够绑定到不属于本地网卡的地址上。

第二个参数的意思是内核是否转发数据包，默认是禁止的，这里我们设置打开。

注意！如果不设置这两个参数，你的第二个和第三个ctr节点haproxy服务将启动不了

6.三个节点启动haproxy服务

systemctl restart haproxy.service

systemctl status haproxy.service

7.访问haproxy前端web平台

http://10.254.15.140:8088/haproxy/stats admin/gdxc1902

五、安装配置keystone

1.在controller1上创建keystone数据库

CREATE DATABASE keystone;

2.在congtroller1上创建数据库用户及赋予权限

GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' IDENTIFIED BY 'gdxc1902';

GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'gdxc1902';

注意替换为自己的数据库密码

3.在三个节点上分别安装keystone和memcached

yum install -y openstack-keystone httpd mod_wsgi python-openstackclient memcached python-memcached openstack-utils

4.优化配置memcached

vim /etc/sysconfig/memcached

PORT="11211"

#定义端口

USER="memcached"

#定义运行memcache的用户

MAXCONN="8192"

#定义最大连接数

CACHESIZE="1024"

#定义最大内存使用值

OPTIONS="-l 127.0.0.1,::1,10.1.1.141 -t 4 -I 10m"

# -|设置服务绑定IP，-t设置线程数，-I调整分配slab页大小

注意！OPTIONS中10.1.1.141改成各个节点对应的IP

scp -p /etc/sysconfig/memcached controller2:/etc/sysconfig/memcached

scp -p /etc/sysconfig/memcached controller3:/etc/sysconfig/memcached

5.在三个节点上分别启动memcache服务并设置开机自启动

systemctl enable memcached.service

systemctl restart memcached.service

systemctl status memcached.service

6.配置/etc/keystone/keystone.conf文件

cp /etc/keystone/keystone.conf /etc/keystone/keystone.conf.bak

>/etc/keystone/keystone.conf

openstack-config --set /etc/keystone/keystone.conf DEFAULT debug false

openstack-config --set /etc/keystone/keystone.conf DEFAULT verbose true

openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_endpoint http://10.1.1.140:35357

openstack-config --set /etc/keystone/keystone.conf DEFAULT public_endpoint http://10.1.1.140:5000

openstack-config --set /etc/keystone/keystone.conf eventlet_server public_bind_host 10.1.1.141

openstack-config --set /etc/keystone/keystone.conf eventlet_server admin_bind_host 10.1.1.141

openstack-config --set /etc/keystone/keystone.conf cache backend oslo_cache.memcache_pool

openstack-config --set /etc/keystone/keystone.conf cache enabled true

openstack-config --set /etc/keystone/keystone.conf cache memcache_servers controller1:11211,controller2:11211,controller3:11211

openstack-config --set /etc/keystone/keystone.conf cache memcache_dead_retry 60

openstack-config --set /etc/keystone/keystone.conf cache memcache_socket_timeout 1

openstack-config --set /etc/keystone/keystone.conf cache memcache_pool_maxsize 1000

openstack-config --set /etc/keystone/keystone.conf cache memcache_pool_unused_timeout 60

openstack-config --set /etc/keystone/keystone.conf catalog template_file /etc/keystone/default_catalog.templates

openstack-config --set /etc/keystone/keystone.conf catalog driver sql

openstack-config --set /etc/keystone/keystone.conf database connection mysql://keystone:gdxc1902@10.1.1.140/keystone

openstack-config --set /etc/keystone/keystone.conf database idle_timeout 3600

openstack-config --set /etc/keystone/keystone.conf database max_pool_size 30

openstack-config --set /etc/keystone/keystone.conf database max_retries -1

openstack-config --set /etc/keystone/keystone.conf database retry_interval 2

openstack-config --set /etc/keystone/keystone.conf database max_overflow 60

openstack-config --set /etc/keystone/keystone.conf identity driver sql

openstack-config --set /etc/keystone/keystone.conf identity caching false

openstack-config --set /etc/keystone/keystone.conf fernet_tokens key_repository /etc/keystone/fernet-keys/

openstack-config --set /etc/keystone/keystone.conf fernet_tokens max_active_keys 3

openstack-config --set /etc/keystone/keystone.conf memcache servers controller1:11211,controller2:11211,controller3:11211

openstack-config --set /etc/keystone/keystone.conf memcache dead_retry 60

openstack-config --set /etc/keystone/keystone.conf memcache socket_timeout 1

openstack-config --set /etc/keystone/keystone.conf memcache pool_maxsize 1000

openstack-config --set /etc/keystone/keystone.conf memcache pool_unused_timeout 60

openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_hosts controller1:5672,controller2:5672,controller3:5672

openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_userid openstack

openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_password gdxc1902

openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_use_ssl false

openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_ha_queues true

openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_retry_interval 1

openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_retry_backoff 2

openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_max_retries 0

openstack-config --set /etc/keystone/keystone.conf token expiration 3600

openstack-config --set /etc/keystone/keystone.conf token caching False

openstack-config --set /etc/keystone/keystone.conf token provider fernet

scp -p /etc/keystone/keystone.conf controller2:/etc/keystone/keystone.conf

scp -p /etc/keystone/keystone.conf controller3:/etc/keystone/keystone.conf

7.配置httpd.conf文件

vim /etc/httpd/conf/httpd.conf

servername controller1 （如果是controller2那就写congtroller2）

listen 8080 （80->8080 haproxy里用了80，不修改启动不了）

sed -i "s/#ServerName www.example.com:80/ServerName controller1/" /etc/httpd/conf/httpd.conf

sed -i "s/Listen 80/Listen 8080/" /etc/httpd/conf/httpd.conf

8.配置keystone与httpd结合

vim /etc/httpd/conf.d/wsgi-keystone.conf

Listen 5002

Listen 35358

WSGIDaemonProcess keystone-public processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}

WSGIProcessGroup keystone-public

WSGIScriptAlias / /usr/bin/keystone-wsgi-public

WSGIApplicationGroup %{GLOBAL}

WSGIPassAuthorization On

ErrorLogFormat "%{cu}t %M"

ErrorLog /var/log/httpd/keystone-error.log

CustomLog /var/log/httpd/keystone_access.log combined

Require all granted

</Directory>

</VirtualHost>

WSGIDaemonProcess keystone-admin processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}

WSGIProcessGroup keystone-admin

WSGIScriptAlias / /usr/bin/keystone-wsgi-admin

WSGIApplicationGroup %{GLOBAL}

WSGIPassAuthorization On

ErrorLogFormat "%{cu}t %M"

ErrorLog /var/log/httpd/keystone-error.log

CustomLog /var/log/httpd/keystone_access.log combined

Require all granted

</Directory>

</VirtualHost>

把这个文件拷贝到另外两个节点上：

scp -p /etc/httpd/conf.d/wsgi-keystone.conf controller2:/etc/httpd/conf.d/wsgi-keystone.conf

scp -p /etc/httpd/conf.d/wsgi-keystone.conf controller3:/etc/httpd/conf.d/wsgi-keystone.conf

9.在controller1上设置数据库同步

su -s /bin/sh -c "keystone-manage db_sync" keystone

10.三个节点都初始化fernet

keystone-manage fernet_setup --keystone-user keystone --keystone-group keystone

keystone-manage credential_setup --keystone-user keystone --keystone-group keystone

11.同步三个节点fernet信息，在congtroller1操作

scp -p /etc/keystone/fernet-keys/* controller2:/etc/keystone/fernet-keys/

scp -p /etc/keystone/fernet-keys/* controller3:/etc/keystone/fernet-keys/

scp -p /etc/keystone/credential-keys/* controller2:/etc/keystone/credential-keys/

scp -p /etc/keystone/credential-keys/* controller3:/etc/keystone/credential-keys/

12.三个节点都要启动httpd，并设置httpd开机启动

systemctl enable httpd.service

systemctl restart httpd.service

systemctl status httpd.service

systemctl list-unit-files |grep httpd.service

13.在congtroller1上创建admin用户角色

keystone-manage bootstrap \

--bootstrap-password gdxc1902 \

--bootstrap-role-name admin \

--bootstrap-service-name keystone \

--bootstrap-admin-url http://10.1.1.140:35357/v3/ \

--bootstrap-internal-url http://10.1.1.140:35357/v3/ \

--bootstrap-public-url http://10.1.1.140:5000/v3/ \

--bootstrap-region-id RegionOne

这样，就可以在openstack命令行里使用admin账号登陆了。

验证。测试是否配置合理：

openstack project list --os-username admin --os-project-name admin --os-user-domain-id default --os-project-domain-id default --os-identity-api-version 3 --os-auth-url http://10.1.1.140:5000 --os-password gdxc1902

14.在controller1上创建admin用户环境变量，创建/root/admin-openrc文件并写入如下内容：

vim /root/admin-openrc

添加以下内容：

export OS_USER_DOMAIN_ID=default

export OS_PROJECT_DOMAIN_ID=default

export OS_USERNAME=admin

export OS_PROJECT_NAME=admin

export OS_PASSWORD=gdxc1902

export OS_IDENTITY_API_VERSION=3

export OS_AUTH_URL=http://10.1.1.140:35357/v3

15.在controller1上创建service项目

source /root/admin-openrc

openstack project create --domain default --description "Service Project" service

16.在controller1上创建demo项目

openstack project create --domain default --description "Demo Project" demo

17.在controller1上创建demo用户

openstack user create --domain default demo --password gdxc1902

注意：gdxc1902为demo用户密码

18.在controller1上创建user角色将demo用户赋予user角色

openstack role create user

openstack role add --project demo --user demo user

19.在controller1上验证keystone

unset OS_TOKEN OS_URL

openstack --os-auth-url http://10.1.1.140:35357/v3 --os-project-domain-name default --os-user-domain-name default --os-project-name admin --os-username admin token issue --os-password gdxc1902

openstack --os-auth-url http://10.1.1.140:35357/v3 --os-project-domain-name default --os-user-domain-name default --os-project-name demo --os-username demo token issue --os-password gdxc1902

20.在controller1上创建demo用户环境变量，创建/root/demo-openrc文件并写入下列内容：

vim /root/demo-openrc

添加：

export OS_USER_DOMAIN_ID=default

export OS_PROJECT_DOMAIN_ID=default

export OS_USERNAME=demo

export OS_PROJECT_NAME=demo

export OS_PASSWORD=gdxc1902

export OS_IDENTITY_API_VERSION=3

export OS_AUTH_URL=http://10.1.1.140:35357/v3

scp -p /root/admin-openrc controller2:/root/admin-openrc

scp -p /root/admin-openrc controller3:/root/admin-openrc

scp -p /root/demo-openrc controller2:/root/demo-openrc

scp -p /root/demo-openrc controller3:/root/demo-openrc

六、安装配置glance

1.在controller1上创建glance数据库

CREATE DATABASE glance;

2.在controller1上创建数据库用户并赋予权限

GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'localhost' IDENTIFIED BY 'gdxc1902';

GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'%' IDENTIFIED BY 'gdxc1902';

3.在controller1上创建glance用户并赋予admin权限

source /root/admin-openrc

openstack user create --domain default glance --password gdxc1902

openstack role add --project service --user glance admin

4.在controller1上创建image服务

openstack service create --name glance --description "OpenStack Image service" image

5.在controller1上创建glance的endpoint

openstack endpoint create --region RegionOne image public http://10.1.1.140:9292

openstack endpoint create --region RegionOne image internal http://10.1.1.140:9292

openstack endpoint create --region RegionOne image admin http://10.1.1.140:9292

6.在三个节点上安装glance相关rpm包

yum install -y openstack-glance

7.在三个节点上修改glance配置文件/etc/glance/glance-api.conf

注意红色的密码设置成你自己的

cp /etc/glance/glance-api.conf /etc/glance/glance-api.conf.bak

>/etc/glance/glance-api.conf

openstack-config --set /etc/glance/glance-api.conf DEFAULT debug False

openstack-config --set /etc/glance/glance-api.conf DEFAULT verbose True

openstack-config --set /etc/glance/glance-api.conf DEFAULT bind_host controller1

openstack-config --set /etc/glance/glance-api.conf DEFAULT bind_port 9393

openstack-config --set /etc/glance/glance-api.conf DEFAULT registry_host controller1

openstack-config --set /etc/glance/glance-api.conf DEFAULT registry_port 9191

openstack-config --set /etc/glance/glance-api.conf DEFAULT auth_region RegionOne

openstack-config --set /etc/glance/glance-api.conf DEFAULT registry_client_protocol http

openstack-config --set /etc/glance/glance-api.conf DEFAULT show_image_direct_url False

openstack-config --set /etc/glance/glance-api.conf DEFAULT workers 4

openstack-config --set /etc/glance/glance-api.conf DEFAULT backlog 4096

openstack-config --set /etc/glance/glance-api.conf DEFAULT image_cache_dir /var/lib/glance/image-cache

openstack-config --set /etc/glance/glance-api.conf DEFAULT rpc_backend rabbit

openstack-config --set /etc/glance/glance-api.conf DEFAULT scrub_time 43200

openstack-config --set /etc/glance/glance-api.conf DEFAULT delayed_delete False

openstack-config --set /etc/glance/glance-api.conf DEFAULT enable_v1_api False

openstack-config --set /etc/glance/glance-api.conf DEFAULT enable_v2_api True

openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_hosts controller1:5672,controller2:5672,controller3:5672

openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_userid openstack

openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_password gdxc1902

openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_use_ssl False

openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_ha_queues True

openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_retry_interval 1

openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_retry_backoff 2

openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_max_retries 0

openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit amqp_durable_queues False

openstack-config --set /etc/glance/glance-api.conf oslo_concurrency lock_path /var/lock/glance

openstack-config --set /etc/glance/glance-api.conf database connection mysql+pymysql://glance:gdxc1902@10.1.1.140/glance

openstack-config --set /etc/glance/glance-api.conf database idle_timeout 3600

openstack-config --set /etc/glance/glance-api.conf database max_pool_size 30

openstack-config --set /etc/glance/glance-api.conf database max_retries -1

openstack-config --set /etc/glance/glance-api.conf database retry_interval 2

openstack-config --set /etc/glance/glance-api.conf database max_overflow 60

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_uri http://10.1.1.140:5000

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_url http://10.1.1.140:35357

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken memcached_servers controller1:11211,controller2:11211,controller3:11211

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_type password

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken project_domain_name default

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken user_domain_name default

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken username glance

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken password gdxc1902

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken project_name service

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken token_cache_time -1

openstack-config --set /etc/glance/glance-api.conf paste_deploy flavor keystone

openstack-config --set /etc/glance/glance-api.conf glance_store stores file,http

openstack-config --set /etc/glance/glance-api.conf glance_store default_store file

openstack-config --set /etc/glance/glance-api.conf glance_store filesystem_store_datadir /var/lib/glance/images/

scp -p /etc/glance/glance-api.conf controller2:/etc/glance/glance-api.conf

scp -p /etc/glance/glance-api.conf controller3:/etc/glance/glance-api.conf

8.在三个节点上修改glance配置文件/etc/glance/glance-registry.conf

cp /etc/glance/glance-registry.conf /etc/glance/glance-registry.conf.bak

>/etc/glance/glance-registry.conf

openstack-config --set /etc/glance/glance-registry.conf DEFAULT debug False

openstack-config --set /etc/glance/glance-registry.conf DEFAULT verbose True

openstack-config --set /etc/glance/glance-registry.conf DEFAULT bind_host controller1

openstack-config --set /etc/glance/glance-registry.conf DEFAULT bind_port 9191

openstack-config --set /etc/glance/glance-registry.conf DEFAULT workers 4

openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_hosts controller1:5672,controller2:5672,controller3:5672

openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_userid openstack

openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_password gdxc1902

openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_use_ssl False

openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_ha_queues True

openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_retry_interval 1

openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_retry_backoff 2

openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_max_retries 0

openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit amqp_durable_queues False

openstack-config --set /etc/glance/glance-registry.conf database connection mysql+pymysql://glance:gdxc1902@10.1.1.140/glance

openstack-config --set /etc/glance/glance-registry.conf database idle_timeout 3600

openstack-config --set /etc/glance/glance-registry.conf database max_pool_size 30

openstack-config --set /etc/glance/glance-registry.conf database max_retries -1

openstack-config --set /etc/glance/glance-registry.conf database retry_interval 2

openstack-config --set /etc/glance/glance-registry.conf database max_overflow 60

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_uri http://10.1.1.140:5000

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_url http://10.1.1.140:35357

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken memcached_servers controller1:11211,controller2:11211,controller3:11211

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_type password

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken project_domain_name default

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken user_domain_name default

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken project_name service

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken username glance

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken password gdxc1902

openstack-config --set /etc/glance/glance-registry.conf paste_deploy flavor keystone

openstack-config --set /etc/glance/glance-registry.conf glance_store filesystem_store_datadir /var/lib/glance/images/

openstack-config --set /etc/glance/glance-registry.conf glance_store os_region_name RegionOne

scp -p /etc/glance/glance-registry.conf controller2:/etc/glance/glance-registry.conf

scp -p /etc/glance/glance-registry.conf controller3:/etc/glance/glance-registry.conf

9.在controller1上同步glance数据库

su -s /bin/sh -c "glance-manage db_sync" glance

10.在三个节点上启动glance及设置开机启动

systemctl enable openstack-glance-api.service openstack-glance-registry.service

systemctl restart openstack-glance-api.service openstack-glance-registry.service

systemctl status openstack-glance-api.service openstack-glance-registry.service

11.在三个节点上将glance版本号写入环境变量openrc文件中

echo " " >> /root/admin-openrc && \

echo " " >> /root/demo-openrc && \

echo "export OS_IMAGE_API_VERSION=2"|tee -a /root/admin-openrc /root/demo-openrc

12.搭建glance后端存储

因为是ha环境，3个控制节点必须要有一个共享的后端存储，不然request发起请求的时候不确定会去调用哪个控制节点的glance服务，如果没有共享存储池存镜像，那么会遇到创建vm时候image找不到的问题。

这里我们采用nfs的方式把glance的后端存储建立起来，当然在实际的生产环境当中，一般会用ceph、glusterfs等方式，这里我们以nfs为例子来讲述后端存储的搭建。

首先准备一台物理机或者虚拟机，要求空间要大，网络最好是在万兆。

这里我们用10.1.1.125 这台虚拟机

首先在这台机器上安装glance组件：

yum install -y openstack-glance python-glance python-glanceclient python-openstackclient openstack-nova-compute

其次安装nfs服务

yum install -y nfs-utils rpcbind

创建glance images的存储路径并赋予glance用户相应的权限：

mkdir -p /var/lib/glance/images

chown -R glance:glance /var/lib/glance/images

配置nfs把/var/lib/glance目录共享出去

vim /etc/exports

/var/lib/glance *(rw,sync,no_root_squash)

启动相关服务，并把nfs设置开机启动

systemctl enable rpcbind

systemctl enable nfs-server.service

systemctl restart rpcbind

systemctl restart nfs-server.service

systemctl status nfs-server.service

让nfs共享目录生效

showmount -e

接着在3个controller节点上做如下操作：

mount -t nfs 10.1.1.125:/var/lib/glance/images /var/lib/glance/images

echo "/usr/bin/mount -t nfs 10.1.1.125:/var/lib/glance/ /var/lib/glance/" >> /etc/rc.d/rc.local

chmod +x /etc/rc.d/rc.local

df -h

13.在controller1上下载测试镜像文件

wget http://10.254.15.138/images/cirros-0.3.4-x86_64-disk.img

14.在controller1上传镜像到glance

source /root/admin-openrc

glance image-create --name "cirros-0.3.4-x86_64" --file cirros-0.3.4-x86_64-disk.img --disk-format qcow2 --container-format bare --visibility public --progress

如果你做好了一个centos6.7系统镜像，也可以用这命令操作，例如：

glance image-create --name "CentOS6.7-x86_64" --file CentOS6.7.qcow2 --disk-format qcow2 --container-format bare --visibility public --progress

查看镜像列表：

glance image-list

七、安装配置nova

1.在controller1上创建nova数据库

CREATE DATABASE nova;

CREATE DATABASE nova_api;

2.在controller1上创建数据库用户并赋予权限

GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'localhost' IDENTIFIED BY 'gdxc1902';

GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'%' IDENTIFIED BY 'gdxc1902';

GRANT ALL PRIVILEGES ON nova_api.* TO 'nova'@'localhost' IDENTIFIED BY 'gdxc1902';

GRANT ALL PRIVILEGES ON nova_api.* TO 'nova'@'%' IDENTIFIED BY 'gdxc1902';

3.在controller1上创建nova用户及赋予admin权限

source /root/admin-openrc

openstack user create --domain default nova --password gdxc1902

openstack role add --project service --user nova admin

4.在controller1上创建computer服务

openstack service create --name nova --description "OpenStack Compute" compute

5.在controller1上创建nova的endpoint

openstack endpoint create --region RegionOne compute public http://10.1.1.140:8774/v2.1/%$tenant_id$s

openstack endpoint create --region RegionOne compute internal http://10.1.1.140:8774/v2.1/%$tenant_id$s

openstack endpoint create --region RegionOne compute admin http://10.1.1.140:8774/v2.1/%$tenant_id$s

6.在三台控制节点上安装nova组件

yum install -y openstack-nova-api openstack-nova-conductor openstack-nova-cert openstack-nova-console openstack-nova-novncproxy openstack-nova-scheduler

7.在三台控制节点上配置nova的配置文件/etc/nova/nova.conf

cp /etc/nova/nova.conf /etc/nova/nova.conf.bak

>/etc/nova/nova.conf

openstack-config --set /etc/nova/nova.conf DEFAULT debug False

openstack-config --set /etc/nova/nova.conf DEFAULT verbose True

openstack-config --set /etc/nova/nova.conf DEFAULT enabled_apis osapi_compute,metadata

openstack-config --set /etc/nova/nova.conf DEFAULT osapi_compute_listen_port 9774

openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen_port 9775

openstack-config --set /etc/nova/nova.conf DEFAULT my_ip 10.1.1.141

openstack-config --set /etc/nova/nova.conf DEFAULT use_neutron True

openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver

openstack-config --set /etc/nova/nova.conf DEFAULT rpc_backend rabbit

openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone

openstack-config --set /etc/nova/nova.conf DEFAULT scheduler_use_baremetal_filters False

openstack-config --set /etc/nova/nova.conf DEFAULT scheduler_default_filters RetryFilter,AvailabilityZoneFilter,RamFilter,CoreFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter

openstack-config --set /etc/nova/nova.conf DEFAULT scheduler_weight_classes nova.scheduler.weights.all_weighers

openstack-config --set /etc/nova/nova.conf DEFAULT scheduler_host_subset_size 30

openstack-config --set /etc/nova/nova.conf DEFAULT scheduler_driver nova.scheduler.filter_scheduler.FilterScheduler

openstack-config --set /etc/nova/nova.conf DEFAULT scheduler_max_attempts 3

openstack-config --set /etc/nova/nova.conf DEFAULT scheduler_available_filters nova.scheduler.filters.all_filters

openstack-config --set /etc/nova/nova.conf DEFAULT ram_allocation_ratio 3.0

openstack-config --set /etc/nova/nova.conf DEFAULT disk_allocation_ratio 1.0

openstack-config --set /etc/nova/nova.conf DEFAULT cpu_allocation_ratio 16.0

openstack-config --set /etc/nova/nova.conf DEFAULT service_down_time 180

openstack-config --set /etc/nova/nova.conf DEFAULT osapi_compute_workers 4

openstack-config --set /etc/nova/nova.conf DEFAULT metadata_workers 4

openstack-config --set /etc/nova/nova.conf DEFAULT rootwrap_config /etc/nova/rootwrap.conf

openstack-config --set /etc/nova/nova.conf DEFAULT notify_on_state_change vm_and_task_state

openstack-config --set /etc/nova/nova.conf DEFAULT allow_resize_to_same_host True

openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_host 10.1.1.141

openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_port 6080

openstack-config --set /etc/nova/nova.conf database connection mysql+pymysql://nova:gdxc1902@10.1.1.140/nova

openstack-config --set /etc/nova/nova.conf database idle_timeout 3600

openstack-config --set /etc/nova/nova.conf database max_pool_size 30

openstack-config --set /etc/nova/nova.conf database retry_interval 2

openstack-config --set /etc/nova/nova.conf database max_retries -1

openstack-config --set /etc/nova/nova.conf api_database connection mysql+pymysql://nova:gdxc1902@10.1.1.140/nova_api

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts controller1:5672,controller2:5672,controller3:5672

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_userid openstack

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_password gdxc1902

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_use_ssl False

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues True

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_retry_interval 1

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_retry_backoff 2

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_max_retries 0

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit amqp_durable_queues False

openstack-config --set /etc/nova/nova.conf oslo_concurrency lock_path /var/lib/nova/tmp

openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_uri http://10.1.1.140:5000

openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_url http://10.1.1.140:35357

openstack-config --set /etc/nova/nova.conf keystone_authtoken memcached_servers controller1:11211,controller2:11211,controller3:11211

openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_type password

openstack-config --set /etc/nova/nova.conf keystone_authtoken project_domain_name default

openstack-config --set /etc/nova/nova.conf keystone_authtoken user_domain_name default

openstack-config --set /etc/nova/nova.conf keystone_authtoken project_name service

openstack-config --set /etc/nova/nova.conf keystone_authtoken username nova

openstack-config --set /etc/nova/nova.conf keystone_authtoken password gdxc1902

openstack-config --set /etc/nova/nova.conf glance api_servers http://10.1.1.140:9292

openstack-config --set /etc/nova/nova.conf conductor use_local False

openstack-config --set /etc/nova/nova.conf conductor workers 4

openstack-config --set /etc/nova/nova.conf vnc enabled True

openstack-config --set /etc/nova/nova.conf vnc vncserver_listen 0.0.0.0

openstack-config --set /etc/nova/nova.conf vnc vncserver_proxyclient_address 10.1.1.141

openstack-config --set /etc/nova/nova.conf vnc novncproxy_base_url http://10.1.1.140:6080/vnc_auto.html

注意：其他节点上记得替换IP，还有密码。

scp -p /etc/nova/nova.conf controller2:/etc/nova/nova.conf

scp -p /etc/nova/nova.conf controller3:/etc/nova/nova.conf

8.在controller1上同步nova数据

su -s /bin/sh -c "nova-manage api_db sync" nova

su -s /bin/sh -c "nova-manage db sync" nova

9.在controller1上设置开机启动

systemctl enable openstack-nova-api.service openstack-nova-cert.service openstack-nova-consoleauth.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service

controller2、3上设置开机启动（注意比起ctr1节点少了openstack-nova-consoleauth.service）：

systemctl enable openstack-nova-api.service openstack-nova-cert.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service

controller1上启动nova服务：

systemctl restart openstack-nova-api.service openstack-nova-cert.service openstack-nova-consoleauth.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service

systemctl status openstack-nova-api.service openstack-nova-cert.service openstack-nova-consoleauth.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service

controller2、3上启动nova服务：

systemctl restart openstack-nova-api.service openstack-nova-cert.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service

systemctl status openstack-nova-api.service openstack-nova-cert.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service

systemctl list-unit-files |grep openstack-nova-*

10.随便一个节点上验证nova服务

unset OS_TOKEN OS_URL

echo "export OS_REGION_NAME=RegionOne" >> admin-openrc

source /root/admin-openrc

nova service-list

openstack endpoint list

八、安装配置neutron

1.在controller1上创建neutron数据库

CREATE DATABASE neutron;

2.在controller1上创建数据库用户并赋予权限

GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY 'gdxc1902';

GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'%' IDENTIFIED BY 'gdxc1902';

3.在controller1上创建neutron用户及赋予admin权限

source /root/admin-openrc

openstack user create --domain default neutron --password gdxc1902

openstack role add --project service --user neutron admin

4.在controller1上创建network服务

openstack service create --name neutron --description "OpenStack Networking" network

5.在controller1上创建nova的endpoint

openstack endpoint create --region RegionOne network public http://10.1.1.140:9696

openstack endpoint create --region RegionOne network internal http://10.1.1.140:9696

openstack endpoint create --region RegionOne network admin http://10.1.1.140:9696

6.在三台控制节点上安装neutron相关软件

yum install -y openstack-neutron openstack-neutron-ml2 openstack-neutron-linuxbridge ebtables

7.在三台控制节点上配置neutron的配置文件/etc/neutron/neutron.conf

cp /etc/neutron/neutron.conf /etc/neutron/neutron.conf.bak

>/etc/neutron/neutron.conf

openstack-config --set /etc/neutron/neutron.conf DEFAULT debug False

openstack-config --set /etc/neutron/neutron.conf DEFAULT verbose true

openstack-config --set /etc/neutron/neutron.conf DEFAULT bind_host controller1

openstack-config --set /etc/neutron/neutron.conf DEFAULT bind_port 9797

openstack-config --set /etc/neutron/neutron.conf DEFAULT core_plugin neutron.plugins.ml2.plugin.Ml2Plugin

openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins neutron.services.l3_router.l3_router_plugin.L3RouterPlugin,neutron.services.metering.metering_plugin.MeteringPlugin

openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone

openstack-config --set /etc/neutron/neutron.conf DEFAULT allow_overlapping_ips True

openstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_status_changes True

openstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_data_changes True

openstack-config --set /etc/neutron/neutron.conf DEFAULT advertise_mtu True

openstack-config --set /etc/neutron/neutron.conf DEFAULT rpc_response_timeout 180

openstack-config --set /etc/neutron/neutron.conf DEFAULT mac_generation_retries 32

openstack-config --set /etc/neutron/neutron.conf DEFAULT dhcp_lease_duration 600

openstack-config --set /etc/neutron/neutron.conf DEFAULT global_physnet_mtu 1500

openstack-config --set /etc/neutron/neutron.conf DEFAULT control_exchange neutron

openstack-config --set /etc/neutron/neutron.conf DEFAULT api_workers 4

openstack-config --set /etc/neutron/neutron.conf DEFAULT rpc_workers 4

openstack-config --set /etc/neutron/neutron.conf DEFAULT agent_down_time 75

openstack-config --set /etc/neutron/neutron.conf DEFAULT dhcp_agents_per_network 2

openstack-config --set /etc/neutron/neutron.conf DEFAULT router_distributed False

openstack-config --set /etc/neutron/neutron.conf DEFAULT router_scheduler_driver neutron.scheduler.l3_agent_scheduler.ChanceScheduler

openstack-config --set /etc/neutron/neutron.conf DEFAULT allow_automatic_l3agent_failover True

openstack-config --set /etc/neutron/neutron.conf DEFAULT l3_ha True

openstack-config --set /etc/neutron/neutron.conf DEFAULT max_l3_agents_per_router 0

openstack-config --set /etc/neutron/neutron.conf DEFAULT min_l3_agents_per_router 2

openstack-config --set /etc/neutron/neutron.conf database connection mysql+pymysql://neutron:gdxc1902@10.1.1.140/neutron

openstack-config --set /etc/neutron/neutron.conf database idle_timeout 3600

openstack-config --set /etc/neutron/neutron.conf database max_pool_size 30

openstack-config --set /etc/neutron/neutron.conf database max_retries -1

openstack-config --set /etc/neutron/neutron.conf database retry_interval 2

openstack-config --set /etc/neutron/neutron.conf database max_overflow 60

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts controller1:5672,controller2:5672,controller3:5672

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_userid openstack

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_password gdxc1902

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues True

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_retry_interval 1

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_retry_backoff 2

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_max_retries 0

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit amqp_durable_queues False

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_uri http://10.1.1.140:5000

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_url http://10.1.1.140:35357

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken memcached_servers controller1:11211,controller2:11211,controller3:11211

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_type password

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken project_domain_name default

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken user_domain_name default

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken project_name service

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken username neutron

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken password gdxc1902

openstack-config --set /etc/neutron/neutron.conf nova auth_url http://10.1.1.140:35357

openstack-config --set /etc/neutron/neutron.conf nova auth_type password

openstack-config --set /etc/neutron/neutron.conf nova project_domain_name default

openstack-config --set /etc/neutron/neutron.conf nova user_domain_name default

openstack-config --set /etc/neutron/neutron.conf nova region_name RegionOne

openstack-config --set /etc/neutron/neutron.conf nova project_name service

openstack-config --set /etc/neutron/neutron.conf nova username nova

openstack-config --set /etc/neutron/neutron.conf nova password gdxc1902

openstack-config --set /etc/neutron/neutron.conf oslo_concurrency lock_path /var/lib/neutron/tmp

openstack-config --set /etc/neutron/neutron.conf agent report_interval 30

openstack-config --set /etc/neutron/neutron.conf agent root_helper sudo\ neutron-rootwrap\ /etc/neutron/rootwrap.conf

scp -p /etc/neutron/neutron.conf controller2:/etc/neutron/neutron.conf

scp -p /etc/neutron/neutron.conf controller3:/etc/neutron/neutron.conf

8.在三个控制节点上配置/etc/neutron/plugins/ml2/ml2_conf.ini

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 type_drivers flat,vlan,vxlan

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 mechanism_drivers linuxbridge,l2population

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 extension_drivers port_security

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 tenant_network_types vxlan

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 path_mtu 1500

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_flat flat_networks provider

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vxlan vni_ranges 1:1000

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup enable_ipset True

scp -p /etc/neutron/plugins/ml2/ml2_conf.ini controller2:/etc/neutron/plugins/ml2/ml2_conf.ini

scp -p /etc/neutron/plugins/ml2/ml2_conf.ini controller3:/etc/neutron/plugins/ml2/ml2_conf.ini

9.在三个控制节点上配置/etc/neutron/plugins/ml2/linuxbridge_agent.ini

openstack-config --set /etc/neutron/plugins/ml2/linuxbridge_agent.ini DEFAULT debug false

openstack-config --set /etc/neutron/plugins/ml2/linuxbridge_agent.ini linux_bridge physical_interface_mappings provider:eth0

openstack-config --set /etc/neutron/plugins/ml2/linuxbridge_agent.ini vxlan enable_vxlan True

openstack-config --set /etc/neutron/plugins/ml2/linuxbridge_agent.ini vxlan local_ip 10.2.2.141

openstack-config --set /etc/neutron/plugins/ml2/linuxbridge_agent.ini vxlan l2_population True

openstack-config --set /etc/neutron/plugins/ml2/linuxbridge_agent.ini agent prevent_arp_spoofing True

openstack-config --set /etc/neutron/plugins/ml2/linuxbridge_agent.ini securitygroup enable_security_group True

openstack-config --set /etc/neutron/plugins/ml2/linuxbridge_agent.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.IptablesFirewallDriver

scp -p /etc/neutron/plugins/ml2/linuxbridge_agent.ini controller2:/etc/neutron/plugins/ml2/linuxbridge_agent.ini

scp -p /etc/neutron/plugins/ml2/linuxbridge_agent.ini controller3:/etc/neutron/plugins/ml2/linuxbridge_agent.ini

注意eth0是public网卡，一般这里写的网卡名都是能访问外网的，如果不是外网网卡，那么vm就会与外界隔离。

10.在三个控制节点上配置/etc/neutron/l3_agent.ini

openstack-config --set /etc/neutron/l3_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.BridgeInterfaceDriver

openstack-config --set /etc/neutron/l3_agent.ini DEFAULT external_network_bridge

openstack-config --set /etc/neutron/l3_agent.ini DEFAULT debug false

scp -p /etc/neutron/l3_agent.ini controller2:/etc/neutron/l3_agent.ini

scp -p /etc/neutron/l3_agent.ini controller3:/etc/neutron/l3_agent.ini

11.在三个控制节点上配置/etc/neutron/dhcp_agent.ini

openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.BridgeInterfaceDriver

openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dhcp_driver neutron.agent.linux.dhcp.Dnsmasq

openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT enable_isolated_metadata True

openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT verbose True

openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT debug false

scp -p /etc/neutron/dhcp_agent.ini controller2:/etc/neutron/dhcp_agent.ini

scp -p /etc/neutron/dhcp_agent.ini controller3:/etc/neutron/dhcp_agent.ini

12.在三个控制节点上重新配置/etc/nova/nova.conf，配置这步的目的是让compute节点能使用neutron网络

openstack-config --set /etc/nova/nova.conf neutron url http://10.1.1.140:9696

openstack-config --set /etc/nova/nova.conf neutron auth_url http://10.1.1.140:35357

openstack-config --set /etc/nova/nova.conf neutron auth_plugin password

openstack-config --set /etc/nova/nova.conf neutron project_domain_id default

openstack-config --set /etc/nova/nova.conf neutron user_domain_id default

openstack-config --set /etc/nova/nova.conf neutron region_name RegionOne

openstack-config --set /etc/nova/nova.conf neutron project_name service

openstack-config --set /etc/nova/nova.conf neutron username neutron

openstack-config --set /etc/nova/nova.conf neutron password gdxc1902

openstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True

openstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret gdxc1902

openstack-config --set /etc/nova/nova.conf cinder os_region_name RegionOne

13.在三个控制节点上将dhcp-option-force=26,1450写入/etc/neutron/dnsmasq-neutron.conf

echo "dhcp-option-force=26,1450" > /etc/neutron/dnsmasq-neutron.conf

14.在三个控制节点上配置/etc/neutron/metadata_agent.ini

openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_ip 10.1.1.140

openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_proxy_shared_secret gdxc1902

openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_workers 4

openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT verbose True

openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT debug false

openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_protocol http

scp -p /etc/neutron/metadata_agent.ini controller2:/etc/neutron/metadata_agent.ini

scp -p /etc/neutron/metadata_agent.ini controller3:/etc/neutron/metadata_agent.ini

15.在三个控制节点上创建软链接

ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini

16.在controller1上同步数据库

su -s /bin/sh -c "neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head" neutron

17.在三个控制节点上重启nova服务，因为刚才改了nova.conf

systemctl restart openstack-nova-api.service

systemctl status openstack-nova-api.service

18.在三个控制节点上重启neutron服务并设置开机启动

systemctl enable neutron-server.service neutron-linuxbridge-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service

systemctl restart neutron-server.service neutron-linuxbridge-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service

systemctl status neutron-server.service neutron-linuxbridge-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service

19.在三个控制节点上启动neutron-l3-agent.service并设置开机启动

systemctl enable neutron-l3-agent.service

systemctl restart neutron-l3-agent.service

systemctl status neutron-l3-agent.service

20.随便一节点上执行验证

source /root/admin-openrc

neutron agent-list

21.创建vxlan模式网络，让虚拟机能外出

a.首先执行环境变量

source /root/admin-openrc

b.创建flat模式的public网络，注意这个public是外出网络，必须是flat模式

neutron --debug net-create --shared provider --router:external True --provider:network_type flat --provider:physical_network provider

执行完这步，在界面里进行操作，把public网络设置为共享和外部网络，创建后，结果为：

c.创建public网络子网，名为public-sub，网段就是10.254.15.160/27，并且IP范围是162-190（这个一般是给vm用的floating IP了），DNS设置为218.30.26.68，网关为10.254.15.161

neutron subnet-create provider 10.254.15.160/27 --name provider-sub --allocation-pool start=10.254.15.162,end=10.254.15.190 --dns-nameserver 218.30.26.68 --gateway 10.254.15.161

d.创建名为private的私有网络，网络模式为vxlan

neutron net-create private-test --provider:network_type vxlan --router:external False --shared

e.创建名为private-subnet的私有网络子网，网段为192.168.1.0，这个网段就是虚拟机获取的自由的IP地址

neutron subnet-create private-test --name private-subnet --gateway 192.168.1.1 192.168.1.0/24

例如你们公司的私有云环境是用于不同的业务，比如行政、销售、技术等，那么你可以创建3个不同名称的私有网络

neutron net-create private-office --provider:network_type vxlan --router:external False --shared

neutron subnet-create private-office --name office-subnet --gateway 192.168.2.1 192.168.2.0/24

neutron net-create private-sale --provider:network_type vxlan --router:external False --shared

neutron subnet-create private-sale --name sale-subnet --gateway 192.168.3.1 192.168.3.0/24

neutron net-create private-tachnology --provider:network_type vxlan --router:external False --shared

neutron subnet-create private-tachnology --name tachnology-subnet --gateway 192.168.4.1 192.168.4.0/24

22.检查网络服务

neutron agent-list

九、安装dashboard

1.安装dashboard相关软件包

yum install openstack-dashboard -y

2.修改配置文件/etc/openstack-dashboard/local_settings

wget http://10.254.15.147/local_settings

修改配置文件

cp local_settings /etc/openstack-dashboard/

scp -p /etc/openstack-dashboard/local_settings controller2:/etc/openstack-dashboard/local_settings

scp -p /etc/openstack-dashboard/local_settings controller3:/etc/openstack-dashboard/local_settings

3.启动dashboard服务并设置开机启动

systemctl enable httpd.service memcached.service

systemctl restart httpd.service memcached.service

systemctl status httpd.service memcached.service

十、安装配置cinder

1.在controller1上创建数据库用户并赋予权限

CREATE DATABASE cinder;

GRANT ALL PRIVILEGES ON cinder.* TO 'cinder'@'localhost' IDENTIFIED BY 'gdxc1902';

GRANT ALL PRIVILEGES ON cinder.* TO 'cinder'@'%' IDENTIFIED BY 'gdxc1902';

2.在controller1上创建neutron用户及赋予admin权限

source /root/admin-openrc

openstack user create --domain default cinder --password gdxc1902

openstack role add --project service --user cinder admin

3.在controller1上创建network服务

openstack service create --name cinder --description "OpenStack Block Storage" volume

openstack service create --name cinderv2 --description "OpenStack Block Storage" volumev2

4.在controller1上创建nova的endpoint

openstack endpoint create --region RegionOne volume public http://10.1.1.140:8776/v1/%$tenant_id$s

openstack endpoint create --region RegionOne volume internal http://10.1.1.140:8776/v1/%$tenant_id$s

openstack endpoint create --region RegionOne volume admin http://10.1.1.140:8776/v1/%$tenant_id$s

openstack endpoint create --region RegionOne volumev2 public http://10.1.1.140:8776/v2/%$tenant_id$s

openstack endpoint create --region RegionOne volumev2 internal http://10.1.1.140:8776/v2/%$tenant_id$s

openstack endpoint create --region RegionOne volumev2 admin http://10.1.1.140:8776/v2/%$tenant_id$s

5.在三台控制节点上安装neutron相关软件

yum install -y openstack-cinder

6.在三台控制节点上配置cinder配置文件/etc/cinder/cinder.conf

cp /etc/cinder/cinder.conf /etc/cinder/cinder.conf.bak

>/etc/cinder/cinder.conf

openstack-config --set /etc/cinder/cinder.conf DEFAULT debug False

openstack-config --set /etc/cinder/cinder.conf DEFAULT verbose True

openstack-config --set /etc/cinder/cinder.conf DEFAULT my_ip 10.1.1.141

openstack-config --set /etc/cinder/cinder.conf DEFAULT osapi_volume_listen_port 8778

openstack-config --set /etc/cinder/cinder.conf DEFAULT auth_strategy keystone

openstack-config --set /etc/cinder/cinder.conf DEFAULT enable_v1_api True

openstack-config --set /etc/cinder/cinder.conf DEFAULT enable_v2_api True

openstack-config --set /etc/cinder/cinder.conf DEFAULT enable_v3_api True

openstack-config --set /etc/cinder/cinder.conf DEFAULT glance_api_servers http://10.1.1.140:9292

openstack-config --set /etc/cinder/cinder.conf DEFAULT glance_api_version 2

openstack-config --set /etc/cinder/cinder.conf DEFAULT storage_availability_zone nova

openstack-config --set /etc/cinder/cinder.conf DEFAULT default_availability_zone nova

openstack-config --set /etc/cinder/cinder.conf DEFAULT allow_availability_zone_fallback True

openstack-config --set /etc/cinder/cinder.conf DEFAULT service_down_time 180

openstack-config --set /etc/cinder/cinder.conf DEFAULT report_interval 10

openstack-config --set /etc/cinder/cinder.conf DEFAULT osapi_volume_workers 4

openstack-config --set /etc/cinder/cinder.conf DEFAULT enable_force_upload True

openstack-config --set /etc/cinder/cinder.conf DEFAULT rootwrap_config /etc/cinder/rootwrap.conf

openstack-config --set /etc/cinder/cinder.conf database connection mysql+pymysql://cinder:gdxc1902@10.1.1.140/cinder

openstack-config --set /etc/cinder/cinder.conf database idle_timeout 3600

openstack-config --set /etc/cinder/cinder.conf database max_pool_size 30

openstack-config --set /etc/cinder/cinder.conf database max_retries -1

openstack-config --set /etc/cinder/cinder.conf database retry_interval 2

openstack-config --set /etc/cinder/cinder.conf database max_overflow 60

openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_hosts controller1:5672,controller2:5672,controller3:5672

openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_userid openstack

openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_password gdxc1902

openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_ha_queues True

openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_use_ssl False

openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_retry_interval 1

openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_retry_backoff 2

openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_max_retries 0

openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit amqp_durable_queues False

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_uri http://10.1.1.140:5000

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_url http://10.1.1.140:35357

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken memcached_servers controller1:11211,controller2:11211,controller3:11211

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_type password

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken project_domain_name default

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken user_domain_name default

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken project_name service

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken username cinder

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken password gdxc1902

openstack-config --set /etc/cinder/cinder.conf oslo_concurrency lock_path /var/lib/cinder/tmp

scp -p /etc/cinder/cinder.conf controller2:/etc/cinder/cinder.conf

scp -p /etc/cinder/cinder.conf controller3:/etc/cinder/cinder.conf

7.在controller1上同步数据库

su -s /bin/sh -c "cinder-manage db sync" cinder

8.在三台控制节点上启动cinde服务，并设置开机启动

systemctl enable openstack-cinder-api.service openstack-cinder-scheduler.service

systemctl restart openstack-cinder-api.service openstack-cinder-scheduler.service

systemctl status openstack-cinder-api.service openstack-cinder-scheduler.service

9.安装cinder节点，cinder节点这里我们需要额外的添加一个硬盘(/dev/sdb)用作cinder的存储服务（注意！这一步是在cinder节点操作的）

yum install lvm2 -y

10.启动服务并设置开机启动（注意！这一步是在cinder节点操作的）

systemctl enable lvm2-lvmetad.service

systemctl start lvm2-lvmetad.service

systemctl status lvm2-lvmetad.service

11.创建lvm，这里的/dev/sdb就是额外添加的硬盘注意！（注意！这一步是在cinder节点操作的）

fdisk -l

pvcreate /dev/sdb

vgcreate cinder-volumes /dev/sdb

12.编辑存储节点lvm.conf文件

vim /etc/lvm/lvm.conf

在devices下面添加filter = ["a/sda/","a/sdb/","r/.*/"],129行

然后重启lvm2服务

systemctl restart lvm2-lvmetad.service

systemctl status lvm2-lvmetad.service

13.安装openstack-cinder、targetcli（注意！这一步是在cinder节点操作的）

yum install openstack-cinder openstack-utils python-keystone scsi-target-utils targetcli ntpdate -y

14.配置cinder配置文件（注意！这一步是在cinder节点操作的）

cp /etc/cinder/cinder.conf /etc/cinder/cinder.conf.bak

>/etc/cinder/cinder.conf

openstack-config --set /etc/cinder/cinder.conf DEFAULT debug False

openstack-config --set /etc/cinder/cinder.conf DEFAULT verbose True

openstack-config --set /etc/cinder/cinder.conf DEFAULT rpc_backend rabbit

openstack-config --set /etc/cinder/cinder.conf DEFAULT auth_strategy keystone

openstack-config --set /etc/cinder/cinder.conf DEFAULT my_ip 10.1.1.146

openstack-config --set /etc/cinder/cinder.conf DEFAULT enabled_backends lvm