corosync+ pacemaker实现pg流复制自动切换(一)

前端之家收集整理的这篇文章主要介绍了corosync+ pacemaker实现pg流复制自动切换(一)前端之家小编觉得挺不错的,现在分享给大家,也给大家做个参考。

corosync + pacemaker + postgres_streaming_replication

说明:

该文档用于说明以corosync+pacemaker的方式实现Postgresql流复制自动切换。注意内容包括有关corosync/pacemaker知识总结以及整个环境的搭建过程和问题处理。

一、介绍

Corosync

Corosync是由OpenAIS项目分离独立出来的项目,分离后能实现HA信息传输功能的就成为了Corosync,因此Corosync 60%代码来源于OpenAIS

Corosync与分离出来的Heartbeat类似,都属于集群信息层,负责传送集群信息以及节点间的心跳信息,单纯HA软件都不存在管理资源的功能,而是需要依赖上层的CRM来管理资源。目前最著名的资源管理器为PacemakerCorosync+Pacemaker也成为高可用集群方案中的最佳组合。

Pacemaker

Pacemaker,即Cluster Resource ManagerCRM),管理整个HA,客户端通过pacemaker管理监控整个集群。

常用的集群管理工具:

1)基于命令行

crm shell/pcs

2)基于图形化

pygui/hawk/lcmc/pcs

Pacemaker内部组件、模块关系图:

095431_RmOs_1011289.png

二、环境

2.1 OS

  1. #cat/etc/issue
  2. CentOSrelease6.4(Final)
  3. Kernel\ronan\m
  4.  
  5. #uname-a
  6. Linuxnode12.6.32-358.el6.x86_64#1SMPFriFeb2200:31:26UTC2013x86_64x86_64x86_64GNU/Linux


2.2 IP

node1:

eth0 192.168.100.201/24 GW 192.168.100.1 ---真实地址

eth1 10.10.10.1/24 ---心跳地址

eth2 192.168.1.1/24 ---流复制地址

node2:

eth0 192.168.100.202/24 GW 192.168.100.1 ---真实地址

eth1 10.10.10.2/24 ---心跳地址

eth2 192.168.1.2/24 ---流复制地址

虚拟地址:

eth0:0 192.168.100.213/24 ---vip-master

eth0:0 192.168.100.214/24 ---vip-slave

eth2:0 192.168.1.3/24 ---vip-rep

2.3软件版本

  1. #rpm-qa|grepcorosync
  2. corosync-1.4.5-2.3.x86_64
  3. corosynclib-1.4.5-2.3.x86_64
  4.  
  5. #rpm-qa|greppacemaker
  6. pacemaker-libs-1.1.10-14.el6_5.2.x86_64
  7. pacemaker-cli-1.1.10-14.el6_5.2.x86_64
  8. pacemaker-1.1.10-14.el6_5.2.x86_64
  9. pacemaker-cluster-libs-1.1.10-14.el6_5.2.x86_64
  10.  
  11. #rpm-qa|grepcrmsh
  12. crmsh-1.2.6-6.1.x86_64



Postgresql Version9.1.4

三、安装

3.1设置YUM

  1. #cat/etc/yum.repos.d/ha-clustering.repo
  2. [haclustering]
  3. name=HAClustering
  4. baseurl=http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/
  5. enabled=1
  6. gpgcheck=0


3.2安装pacemaker/corosync/crmsh

  1. #yuminstallpacemakercorosynccrmsh

安装后会在/usr/lib/ocf/resource.d生成相应的ocf资源脚本,如下:

  1. #cd/usr/lib/ocf/resource.d/
  2. [root@node1resource.d]#ls
  3. heartbeatpacemakerredhat

通过命令查看资源脚本:

  1. [root@node1resource.d]#crmralistocf
  2. ASEHAagent.shAoEtargetAudibleAlarmCTDBClusterMonDelayDummy
  3. EvmsSCCEvmsdFilesystemHealthcpuHealthSMARTICPIPaddr
  4. IPaddr2IPsrcaddrIPv6addrLVMLinuxSCSIMailToManageRAID
  5. ManageVEPure-FTPdRaid1RouteSAPDatabaseSAPInstanceSendArp
  6. ServeRAIDSphinxSearchDaemonSquidStatefulSysInfoSystemHealthVIPArip
  7. VirtualDomainWASWAS6WinPopupXenXinetdanything
  8. apacheapache.shasteriskclusterfs.shconntrackdcontrolddb2
  9. dhcpddrbddrbd.sheDir88ethmonitorexportfsfio
  10. fs.shiSCSILogicalUnitiSCSITargetidsip.shiscsijboss
  11. ldirectordlvm.shlvm_by_lv.shlvm_by_vg.shlxcMysqLMysqL-proxy
  12. MysqL.shnamednamed.shnetfs.shnfsclient.shnfsexport.shnfsserver
  13. nfsserver.shNginxocf-shellfuncsopenldap.shoracleoracledb.shorainstance.sh
  14. oralistener.shoralsnrpgsqlpingpingdportblockpostfix
  15. postgres-8.shpoundproftpdremotersyncdrsyslogsamba.sh
  16. script.shscsi2reservationservice.shsfexslapdsmb.shsvclib_nfslock
  17. symlinksyslog-ngtomcattomcat-5.shtomcat-6.shvarnishvm.sh
  18. vmwarezabbixserver

启动corosync

  1. [root@node1~]#servicecorosyncstart
  2. StartingCorosyncClusterEngine(corosync):[OK]
  3. [root@node2~]#servicecorosyncstart
  4. StartingCorosyncClusterEngine(corosync):[OK]
  5. [root@node2~]#crmstatus
  6. Lastupdated:SatJan1807:00:342014
  7. Lastchange:SatJan1806:58:112014viacrmdonnode1
  8. Stack:classicopenais(withplugin)
  9. CurrentDC:node1-partitionwithquorum
  10. Version:1.1.10-14.el6_5.2-368c726
  11. 2Nodesconfigured,2expectedvotes
  12. 0Resourcesconfigured
  13.  
  14.  
  15. Online:[node1node2]

若出现以下错误可先禁止stonith,该错误是因为stonith未配置导致,错误如下:

crm_verify[4921]: 2014/01/10_07:34:34 ERROR: unpack_resources: Resource start-up disabled since no STONITH resources have been defined

crm_verify[4921]: 2014/01/10_07:34:34 ERROR: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option

crm_verify[4921]: 2014/01/10_07:34:34 ERROR: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity

禁止stonith(只在一个节点上执行即可):

  1. [root@node1~]#crmconfigurepropertystonith-enabled=false

3.3安装Postgresql

安装目录为/opt/pgsql

{安装过程略}

postgres用户配置环境变量:

  1. [postgres@node1~]$cat.bash_profile
  2. #.bash_profile
  3.  
  4. #Getthealiasesandfunctions
  5. if[-f~/.bashrc];then
  6. .~/.bashrc
  7. fi
  8.  
  9. #Userspecificenvironmentandstartupprograms
  10.  
  11.  
  12. exportPATH=/opt/pgsql/bin:$PATH:$HOME/bin
  13. exportPGDATA=/opt/pgsql/data
  14. exportPGUSER=postgres
  15. exportPGPORT=5432
  16. exportLD_LIBRARY_PATH=/opt/pgsql/lib:$LD_LIBRARY_PATH


四、配置

4.1 hosts设置

  1. #vim/etc/hosts
  2. 192.168.100.201node1
  3. 192.168.100.202node2

4.2配置corosync

  1. [root@node1~]#cd/etc/corosync/
  2. [root@node1corosync]#ls
  3. corosync.conf.examplecorosync.conf.example.udpuservice.duidgid.d
  4. [root@node1corosync]#cpcorosync.conf.examplecorosync.conf
  5. [root@node1corosync]#vimcorosync.conf
  6. compatibility:whitetank//兼容旧版本
  7.  
  8. totem{//节点间通信协议定义
  9. version:2
  10. secauth:on//是否开启安全认证
  11. threads:0
  12. interface{//心跳配置
  13. ringnumber:0
  14. bindnetaddr:10.10.10.0//绑定网络
  15. mcastaddr:226.94.1.1//向外发送多播的地址
  16. mcastport:5405//多播端口
  17. ttl:1
  18. }
  19. }
  20.  
  21. logging{//日志设置
  22. fileline:off
  23. to_stderr:no//是否发送错误信息到标准输出
  24. to_logfile:yes//是否记录到日志文件
  25. to_syslog:yes//是否记录到系统日志
  26. logfile:/var/log/cluster/corosync.log//日志文件,注意/var/log/cluster目录必须存在
  27. debug:off
  28. timestamp:on//日志中是否标记时间
  29. logger_subsys{
  30. subsys:AMF
  31. debug:off
  32. }
  33. }
  34.  
  35. amf{
  36. mode:disabled
  37. }
  38.  
  39. service{
  40. ver:0
  41. name:pacemaker//启用pacemaker
  42. }
  43.  
  44. aisexec{
  45. user:root
  46. group:root
  47. }

4.3生成密钥

{默认利用random生成,但如果中断的系统随机数不够用就需要较长的时间,此时可以通过urandom来替代random}

  1. [root@node1corosync]#mv/dev/random/dev/random.bak
  2. [root@node1corosync]#ln-s/dev/urandom/dev/random
  3. [root@node1corosync]#corosync-keygen
  4. CorosyncClusterEngineAuthenticationkeygenerator.
  5. Gathering1024bitsforkeyfrom/dev/random.
  6. Presskeysonyourkeyboardtogenerateentropy.
  7. Writingcorosynckeyto/etc/corosync/authkey.

4.4 SSH互信配置

node1 -> node2 :

  1. [root@node1~]#cd.ssh/
  2. [root@node1.ssh]#ssh-keygen-trsa
  3. Generatingpublic/privatersakeypair.
  4. Enterfileinwhichtosavethekey(/root/.ssh/id_rsa):
  5. Enterpassphrase(emptyfornopassphrase):
  6. Entersamepassphraseagain:
  7. Youridentificationhasbeensavedin/root/.ssh/id_rsa.
  8. Yourpublickeyhasbeensavedin/root/.ssh/id_rsa.pub.
  9. Thekeyfingerprintis:
  10. 2c:ed:1e:a6:a7:cd:e3:b2:7c:de:aa:ff:63:28:9a:19root@node1
  11. Thekey'srandomartimageis:
  12. +--[RSA2048]----+
  13. ||
  14. ||
  15. ||
  16. |o|
  17. |.S|
  18. |o|
  19. |E+.|
  20. |=o*=oo|
  21. |+.*%O=o.|
  22. +-----------------+
  23. [root@node1.ssh]#ssh-copy-id-iid_rsa.pubnode2
  24. Theauthenticityofhost'node2(192.168.100.202)'can'tbeestablished.
  25. RSAkeyfingerprintisbe:76:cd:29:af:59:76:11:6a:c7:7d:72:27:df:d1:02.
  26. Areyousureyouwanttocontinueconnecting(yes/no)?yes
  27. Warning:Permanentlyadded'node2,192.168.100.202'(RSA)tothelistofknownhosts.
  28. root@node2'spassword:
  29. Nowtryloggingintothemachine,with"ssh'node2'",andcheckin:
  30.  
  31. .ssh/authorized_keys
  32.  
  33. tomakesurewehaven'taddedextrakeysthatyouweren'texpecting.
  34. [root@node1.ssh]#sshnode2date
  35. SatJan1806:36:21CST2014

node2 -> node1 :

  1. [root@node2~]#cd.ssh/
  2. [root@node2.ssh]#ssh-keygen-trsa
  3. [root@node2.ssh]#ssh-copy-id-iid_rsa.pubnode1
  4. [root@node2.ssh]#sshnode1date
  5. SatJan1806:37:31CST2014

4.5同步配置

  1. [root@node1corosync]#scpauthkeycorosync.confnode2:/etc/corosync/
  2. authkey100%1280.1KB/s00:00
  3. corosync.conf100%28082.7KB/s00:00

4.6下载替换脚本

虽然安装了上述软件后会生成pgsql资源脚本,但是其版本过旧,且自带pgsql不能实现自动切换功能,所以在安装了pacemaker/corosync之后需要从网上下载进行替换,如下:

https://github.com/ClusterLabs/resource-agents/tree/master/heartbeat

下载pgsqlocf-shellfuncs.in

替换:

  1. #cppgsql/usr/lib/ocf/resource.d/heartbeat/
  2. #cpocf-shellfuncs.in/usr/lib/ocf/lib/heartbeat/ocf-shellfuncs

{注意要将ocf-shellfuncs.in名称改为ocf-shellfuncs,否则pgsql可能会找不到要用的函数。新下载的函数定义文件添加了一些新功能函数,如ocf_local_nodename}

pgsql资源脚本特性:

主节点失效切换

master宕掉时,RA检测到该问题并将master标记stop,随后将slave提升为新的master

异步与同步切换

如果slave宕掉或者LAN中存在问题,那么当设置为同步复制时包含写操作的事务将会被终止,也就意味着服务将停止。因此,为防止服务停止RA将会动态地将同步转换为异步复制。

初始启动时自动识别新旧数据

当两个或多个节点上的Pacemaker同时初始启动时,RA通过每个节点上最近的replay location进行比较,找出最新数据节点。这个拥有最新数据的节点将被认为是master。当然,若在一个节点上启动pacemaker或者该节点上的pacemaker是第一个被启动的,那么它也将成为masterRA依据停止前的数据状态进行裁定。

负载均衡

由于slave节点可以处理只读事务,因此对于读操作可以通过虚拟另一个虚拟IP来实现读操作的负载均衡。

4.7启动corosync

启动:

  1. [root@node1~]#servicecorosyncstart
  2. [root@node2~]#servicecorosyncstart

检测状态:

  1. [root@node1~]#crmstatus
  2. Lastupdated:TueJan2123:55:132014
  3. Lastchange:TueJan2123:37:362014viacrm_attributeonnode1
  4. Stack:classicopenais(withplugin)
  5. CurrentDC:node1-partitionwithquorum
  6. Version:1.1.10-14.el6_5.2-368c726
  7. 2Nodesconfigured,'宋体';font-size:13px;white-space:normal;">{corosync启动成功}

  8. 4.8配置流复制

  9. node1/node2上配置postgresql.conf/pg_hba.conf

  10. postgresql.conf :

  11. listen_addresses = '*'

  12. port = 5432

  13. wal_level = hot_standby

  14. archive_mode = on

  15. archive_command = 'test ! -f /opt/archivelog/%f && cp %p /opt/archivelog/%f'

  16. max_wal_senders = 4

  17. wal_keep_segments = 50

  18. hot_standby = on

  19. pg_hba.conf :

  20. host replication postgres 192.168.1.0/24 trust

  21. node2上执行基础同步:

  22. [postgres@node2data]$pg_basebackup-h192.168.1.1-Upostgres-D/opt/pgsql/data-P
  23. 若需测试流复制是否能够成功,可在此处手工配置(corosync启动数据库自动生成,若已经存在将会被覆盖)recovery.conf进行测试:

  24. standby_mode = 'on'

  25. primary_conninfo = 'host=192.168.1.1 port=5432 user=postgres application_name=node2 keepalives_idle=60 keepalives_interval=5 keepalives_count=5'

  26. restore_command = 'cp /opt/archivelog/%f %p'

  27. recovery_target_timeline = 'latest'

  28. [postgres@node2data]$pg_ctlstart
  29. [postgres@node1pgsql]$psql
  30. postgres=#selectclient_addr,sync_statefrompg_stat_replication;
  31. client_addr|sync_state
  32. -------------+------------
  33. 192.168.1.2|sync
  34. (1row)
  35. 停止数据库

  36. [postgres@node2~]$pg_ctlstop-mf
  37. [postgres@node1~]$pg_ctlstop-mf
  38. 4.9配置pacemaker

  39. {关于pacemaker的配置可通过多种方式,如crmshhb_guipcs等,该实验使用crmsh配置}

  40. 编写crm配置脚本:

  41. [root@node1~]#catpgsql.crm
  42. property\//设置全局属性
  43. no-quorum-policy="ignore"\//关闭法定投票人数策略,多节点时启用
  44. stonith-enabled="false"\//禁用stonith设备检测
  45. crmd-transition-delay="0s"
  46. rsc_defaults\//资源默认属性配置
  47. resource-stickiness="INFINITY"\//资源留在所处位置的自愿程度,INFINITY为无限自愿
  48. migration-threshold="1"//设置资源发生多少次故障时节点将失去管理该资源的资格
  49. msmsPostgresqlpgsql\//
  50. Meta\
  51. master-max="1"\
  52. master-node-max="1"\
  53. clone-max="2"\
  54. clone-node-max="1"\
  55. notify="true"
  56. cloneclnPingCheckpingCheck//克隆资源
  57. groupmaster-group\//定义资源组
  58. vip-master\
  59. vip-rep
  60. primitivevip-masterocf:heartbeat:IPaddr2\//定义vip-master资源
  61. params\
  62. ip="192.168.100.213"\
  63. nic="eth0"\
  64. cidr_netmask="24"\
  65. opstarttimeout="60s"interval="0s"on-fail="stop"\
  66. opmonitortimeout="60s"interval="10s"on-fail="restart"\
  67. opstoptimeout="60s"interval="0s"on-fail="block"
  68. primitivevip-repocf:heartbeat:IPaddr2\//定义vip-rep资源
  69. params\
  70. ip="192.168.1.3"\
  71. nic="eth2"\
  72. cidr_netmask="24"\
  73. Meta\
  74. migration-threshold="0"\
  75. opstarttimeout="60s"interval="0s"on-fail="restart"\
  76. opmonitortimeout="60s"interval="10s"on-fail="restart"\
  77. opstoptimeout="60s"interval="0s"on-fail="block"
  78. primitivevip-slaveocf:heartbeat:IPaddr2\//定义vip-slave资源
  79. params\
  80. ip="192.168.100.214"\
  81. nic="eth0"\
  82. cidr_netmask="24"\
  83. Meta\
  84. resource-stickiness="1"\
  85. opstarttimeout="60s"interval="0s"on-fail="restart"\
  86. opmonitortimeout="60s"interval="10s"on-fail="restart"\
  87. opstoptimeout="60s"interval="0s"on-fail="block"
  88. primitivepgsqlocf:heartbeat:pgsql\//定义pgsql资源
  89. params\//设置相关参数
  90. pgctl="/opt/pgsql/bin/pg_ctl"\
  91. psql="/opt/pgsql/bin/psql"\
  92. pgdata="/opt/pgsql/data/"\
  93. start_opt="-p5432"\
  94. rep_mode="sync"\
  95. node_list="node1node2"\
  96. restore_command="cp/opt/archivelog/%f%p"\
  97. primary_conninfo_opt="keepalives_idle=60keepalives_interval=5keepalives_count=5"\
  98. master_ip="192.168.1.3"\
  99. stop_escalate="0"\
  100. opstarttimeout="60s"interval="0s"on-fail="restart"\
  101. opmonitortimeout="60s"interval="7s"on-fail="restart"\
  102. opmonitortimeout="60s"interval="2s"on-fail="restart"role="Master"\
  103. oppromotetimeout="60s"interval="0s"on-fail="restart"\
  104. opdemotetimeout="60s"interval="0s"on-fail="stop"\
  105. opstoptimeout="60s"interval="0s"on-fail="block"\
  106. opnotifytimeout="60s"interval="0s"
  107. primitivepingCheckocf:pacemaker:ping\//定义pingCheck资源
  108. params\
  109. name="default_ping_set"\
  110. host_list="192.168.100.1"\
  111. multiplier="100"\
  112. opstarttimeout="60s"interval="0s"on-fail="restart"\
  113. opmonitortimeout="60s"interval="10s"on-fail="restart"\
  114. opstoptimeout="60s"interval="0s"on-fail="ignore"
  115. locationrsc_location-1vip-slave\//定义资源vip-slave选择位置
  116. rule200:pgsql-statuseq"HS:sync"\
  117. rule100:pgsql-statuseq"PRI"\
  118. rule-inf:not_definedpgsql-status\
  119. rule-inf:pgsql-statusne"HS:sync"andpgsql-statusne"PRI"
  120. locationrsc_location-2msPostgresql\//定义资源msPostgresql选择位置
  121. rule-inf:not_defineddefault_ping_setordefault_ping_setlt100
  122. colocationrsc_colocation-1inf:msPostgresqlclnPingCheck//定义在相同节点上运行的资源
  123. colocationrsc_colocation-2inf:master-groupmsPostgresql:Master
  124. orderrsc_order-10:clnPingCheckmsPostgresql//定义对资源的操作顺序
  125. orderrsc_order-20:msPostgresql:promotemaster-group:startsymmetrical=false
  126. orderrsc_order-30:msPostgresql:demotemaster-group:stopsymmetrical=false
  127. 注:该脚本针对网上的配置方式做了一点修改,因为网上是针对pacemaker-1.0.*进行配置的,而本实验使用的是pacemaker-1.1.10

  128. 导入配置脚本:

  129. [root@node1~]#crmconfigureloadupdatepgsql.crm
  130. WARNING:pgsql:specifiedtimeout60sforstopissmallerthantheadvised120
  131. WARNING:pgsql:specifiedtimeout60sforstartissmallerthantheadvised120
  132. WARNING:pgsql:specifiedtimeout60sfornotifyissmallerthantheadvised90
  133. WARNING:pgsql:specifiedtimeout60sfordemoteissmallerthantheadvised120
  134. WARNING:pgsql:specifiedtimeout60sforpromoteissmallerthantheadvised120
  135. 一段时间后查看ha状态:

  136. sql[pgsql]
  137. Masters:[node1]
  138. Slaves:[node2]
  139. CloneSet:clnPingCheck[pingCheck]
  140. Started:[node1node2]
  141. [root@node1~]#crm_mon-Afr-1
  142. Lastupdated:TueJan2123:37:202014
  143. Lastchange:TueJan2123:37:362014viacrm_attributeonnode1
  144. Stack:classicopenais(withplugin)
  145. CurrentDC:node1-partitionwithquorum
  146. Version:1.1.10-14.el6_5.2-368c726
  147. 2Nodesconfigured,2expectedvotes
  148. 7Resourcesconfigured
  149. Online:[node1node2]
  150. Fulllistofresources:
  151. vip-slave(ocf::heartbeat:IPaddr2):Startednode2
  152. ResourceGroup:master-group
  153. vip-master(ocf::heartbeat:IPaddr2):Startednode1
  154. vip-rep(ocf::heartbeat:IPaddr2):Startednode1
  155. Master/SlaveSet:msPostgresql[pgsql]
  156. Masters:[node1]
  157. Slaves:[node2]
  158. CloneSet:clnPingCheck[pingCheck]
  159. Started:[node1node2]
  160. NodeAttributes:
  161. *Nodenode1:
  162. +default_ping_set:100
  163. +master-pgsql:1000
  164. +pgsql-data-status:LATEST
  165. +pgsql-master-baseline:0000000006000078
  166. +pgsql-status:PRI
  167. *Nodenode2:
  168. +default_ping_set:100
  169. +master-pgsql:100
  170. +pgsql-data-status:STREAMING|SYNC
  171. +pgsql-status:HS:sync
  172. Migrationsummary:
  173. *Nodenode2:
  174. *Nodenode1:
  175. 注:刚启动时两节点均为slave,一段时间后node1自动切换为master

猜你在找的Postgre SQL相关文章