简介

本文以工业界测试模型TPC-C为测试模型,介绍PostgreSQL数据库从系统层面的优化到数据库层面的优化方法。 TPmC从 256195.32 提升到 606466.31 是如何做到的。

测试环境介绍

  • 16核开HT共32线程
  • 256G 1600MHz 内存
  • 万兆网卡
  • 3 块 6.4TB AliFlash PCI-E SSD
  • 逻辑卷条带
  • XFS
  • 数据块对齐

XFS文件系统优化

主要分3块:

  1. 逻辑卷优化部分
  2. XFS mkfs 优化部分
  3. XFS mount 优化部分

以上几个部分都可以通过man手册查看,了解原理和应用场景后着手优化。

  1. man lvcreate
  2. man xfs
  3. man mkfs.xfs
  4. man mount

逻辑卷优化部分

对于不在lvm列表的设备,可能要先修改lvm.conf,添加设备号才行。否则不能创建PV。

  1. # cat /proc/devices
  2. 252 shannon
  3. [root@localhost ~]# vi /etc/lvm/lvm.conf
  4. # types = [ "fd", 16 ]
  5. types = [ "shannon", 252 ]
  1. 创建PV前,将块设备对齐(对齐的目的是避免双写,因为SSD有最小写入单元,如果没有对齐,可能出现SSD写多个块),前面1MB最好不要分配,从2048 sector开始分配。(使用pvcreate的–dataalignment参数也可以达到同样的目的。)

    1. fdisk -c -u /dev/dfa
    2. start 2048
    3. end + (2048*n) - 1

    或者使用parted创建分区。

    LVM的layout,创建PV时,也需要对齐DATA的数据。 从4MB处开始分配DATA EXTENSION: # pvcreate --dataalignment 4M /dev/sdc

    1st PE 即数据开始位置。

    1. [root@digoal ~]# pvs -o+pe_start
    2. PV VG Fmt Attr PSize PFree 1st PE
    3. /dev/sda2 centos lvm2 a-- 19.51g 40.00m 1.00m
    4. /dev/sdc lvm2 --- 20.00g 20.00g 4.00m

    列出所有可以查看的flag pvs -o+

  2. 创建lv主要指定2个参数,条带数量,和pv数量一致即可,如果PV本身是一个RAID设备,根据RAID的块设备个数来定条带数。 例如RAID5 5块盘,去除1个校验数据,取4作为条带数。RAID10 10块盘,取5作为条带数。RAID0 10块盘,取10作为条带数。

    1. -i, --stripes Stripes
    2. Gives the number of stripes. This is equal to the number of physical volumes to scatter the logical volume.

    条带大小,和数据库块大小一致,例如postgresql默认为 8KB。

    1. -I, --stripesize StripeSize
    2. Gives the number of kilobytes for the granularity of the stripes.
    3. StripeSize must be 2^n (n = 2 to 9) for metadata in LVM1 format. For metadata in LVM2 format, the stripe size may be a larger power of 2 but must not exceed the physical extent size.

    创建快照时,指定的参数chunksize, 最好和数据库的块大小一致, 例如postgresql默认为 8KB。

    1. -c, --chunksize ChunkSize
    2. Power of 2 chunk size for the snapshot logical volume between 4k and 512k.

    例如: 预留2GB给xfs的LOG DEV

    1. #lvcreate -i 3 -I 8 -n lv02 -L 2G vgdata01
    2. Logical volume "lv02" created
    3. #lvcreate -i 3 -I 8 -n lv01 -l 100%FREE vgdata01
    4. Logical volume "lv01" created
    5. #lvs
    6. LV VG Attr LSize Origin Snap% Move Log Copy% Convert
    7. lv01 vgdata01 -wi-a- 17.29t
    8. lv02 vgdata01 -wi-a- 2g

XFS mkfs 优化部分

首先要搞清楚XFS的layout

xfs包含3个section,data、log、realtime files。默认情况下 log存在data里面,没有realtime。所有的section都是由最小单位block组成,初始化xfs是-b指定block size。

  1. data 包含 metadata(inode, 目录, 间接块), user file data, non-realtime files data被拆分成多个allocation group,mkfs.xfs时可以指定group的个数,以及单个group的SIZE。 group越多,可以并行进行的文件和块的allocation就越多。你可以认为单个组的操作是串行的,多个组是并行的。但是组越多,消耗的CPU会越多,需要权衡。对于并发写很高的场景,可以多一些组(例如一台主机跑了很多小的数据库,每个数据库都很繁忙的场景下)

  2. log 存储metadata的log,修改metadata前,必须先记录log,然后才能修改data section中的metadata。也用于crash后的恢复。

  3. realtime 被划分为很多个小的extents, 要将文件写入到 realtime section中,必须使用xfsctl改一下文件描述符的bit位,并且一定要在数据写入前完成。在realtime中的文件大小是realtime extents的倍数关系。

mkfs.xfs优化

  1. data section allocation group count数量和AGSIZE相乘等于块设备大小。 AG count数量多少和用户需求的并行度相关。 同时AG SIZE的取值范围是16M到1TB,PostgreSQL 建议1GB左右。 -b size=8192 与数据库块大小一致 (但不是所有的xfs版本都支持大于4K的block size,所以如果你发现mount失败并且告知只支持4K以下的BLOCK,那么请重新格式化) -d agcount=9000,sunit=16,swidth=48 假设有9000个并发写操作,使用9000个allocation groups (单位512 bytes) 与lvm或RAID块设备的条带大小对齐 与lvm或RAID块设备条带跨度大小对齐,以上对应3*8 例如 -i 3 -I 8。

  2. log section 最好放在SSD上,速度越快越好。最好不要使用cgroup限制LOG块设备的iops操作。

  3. realtime section 不需要的话,不需要创建。

agsize绝对不能是条带宽度的倍数。(假设条带数为3,条带大小为8K,则宽度为24K) 如果根据指定agcount算出的agsize是swidth的倍数,会弹出警告: 例如下面的例子, agsize=156234 blks 是 swidth=6 blks 的倍数 26039。 给出的建议是减掉一个stripe unit即8K,即156234 blks - sunit 2 blks = 156232 blks。 156232 blks换算成字节数= 156232*4096 = 639926272 bytes 或 156232*4 = 624928K

  1. #mkfs.xfs -f -b size=4096 -l logdev=/dev/mapper/vgdata01-lv01,size=2136997888,sunit=16 -d agcount=30000,sunit=16,swidth=48 /dev/mapper/vgdata01-lv02
  2. Warning: AG size is a multiple of stripe width. This can cause performance
  3. problems by aligning all AGs on the same disk. To avoid this, run mkfs with
  4. an AG size that is one stripe unit smaller, for example 156232.
  5. meta-data=/dev/mapper/vgdata01-lv02 isize=256 agcount=30000, agsize=156234 blks
  6. = sectsz=4096 attr=2, projid32bit=1
  7. = crc=0 finobt=0
  8. data = bsize=4096 blocks=4686971904, imaxpct=5
  9. = sunit=2 swidth=6 blks
  10. naming =version 2 bsize=4096 ascii-ci=0 ftype=0
  11. log =/dev/mapper/vgdata01-lv01 bsize=4096 blocks=521728, version=2
  12. = sectsz=512 sunit=2 blks, lazy-count=1
  13. realtime =none extsz=4096 blocks=0, rtextents=0

对于上面这个mkfs.xfs操作,改成以下

  1. #mkfs.xfs -f -b size=4096 -l logdev=/dev/mapper/vgdata01-lv01,size=2136997888,sunit=16 -d agsize=639926272,sunit=16,swidth=48 /dev/mapper/vgdata01-lv02

  1. #mkfs.xfs -f -b size=4096 -l logdev=/dev/mapper/vgdata01-lv01,size=2136997888,sunit=16 -d agsize=624928k,sunit=16,swidth=48 /dev/mapper/vgdata01-lv02

输出如下

  1. meta-data=/dev/mapper/vgdata01-lv02 isize=256 agcount=30001, agsize=156232 blks (约600MB)
  2. = sectsz=4096 attr=2, projid32bit=1
  3. = crc=0 finobt=0
  4. data = bsize=4096 blocks=4686971904, imaxpct=5
  5. = sunit=2 swidth=6 blks
  6. naming =version 2 bsize=4096 ascii-ci=0 ftype=0
  7. log =/dev/mapper/vgdata01-lv01 bsize=4096 blocks=521728, version=2
  8. = sectsz=512 sunit=2 blks, lazy-count=1
  9. realtime =none extsz=4096 blocks=0, rtextents=0

XFS mount 优化部分

nobarrier largeio 针对数据仓库,流媒体这种大量连续读的应用 nolargeio 针对OLTP logbsize=262144 指定 log buffer logdev= 指定log section对应的块设备,用最快的SSD。 noatime,nodiratime swalloc 条带对齐 allocsize=16M delayed allocation writeout的buffer io大小 inode64 Indicates that XFS is allowed to create inodes at any location in the filesystem

mount.xfs 例子

  1. #mount -t xfs -o allocsize=16M,inode64,nobarrier,nolargeio,logbsize=262144,noatime,nodiratime,swalloc,logdev=/dev/mapper/vgdata01-lv02 /dev/mapper/vgdata01-lv01 /data01

xfsctl 优化部分

排错

  1. #mount -o noatime,swalloc /dev/mapper/vgdata01-lv01 /data01
  2. mount: Function not implemented

原因是用了不支持的块大小

  1. [ 5736.642924] XFS (dm-0): File system with blocksize 8192 bytes. Only pagesize (4096) or less will currently work.
  2. [ 5736.695146] XFS (dm-0): SB validate failed with error -38.

问题排除

  1. # mkfs.xfs -f -b size=4096 -l logdev=/dev/mapper/vgdata01-lv02,size=2136997888,sunit=16 -d agcount=9000,sunit=16,swidth=48 /dev/mapper/vgdata01-lv01
  2. meta-data=/dev/mapper/vgdata01-lv01 isize=256 agcount=9000, agsize=515626 blks
  3. = sectsz=512 attr=2
  4. data = bsize=4096 blocks=4640621568, imaxpct=5
  5. = sunit=2 swidth=6 blks
  6. naming =version 2 bsize=4096 ascii-ci=0
  7. log =/dev/mapper/vgdata01-lv02 bsize=4096 blocks=521728, version=2
  8. = sectsz=512 sunit=2 blks, lazy-count=1
  9. realtime =none extsz=4096 blocks=0, rtextents=0

mount时指定logdev

  1. #mount -t xfs -o allocsize=16M,inode64,nobarrier,nolargeio,logbsize=262144,noatime,nodiratime,swalloc,logdev=/dev/mapper/vgdata01-lv02 /dev/mapper/vgdata01-lv01 /data01

安装benchmarksql

http://sourceforge.net/projects/benchmarksql/

下载安装 JDK7

  1. http://www.oracle.com/technetwork/cn/java/javase/downloads/jdk7-downloads-1880260.html
  2. wget http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.rpm
  3. rpm -ivh jdk-7u79-linux-x64.rpm

检查包安装位置(使用rpm安装时也可以直接指定位置)

  1. rpm -ql jdk
  2. ...
  3. /usr/java/jdk1.7.0_79/bin/java
  4. ...

配置JAVA环境变量

  1. $ export JAVA_HOME=/usr/java/jdk1.7.0_79
  2. $ export PATH=$JAVA_HOME/bin:$PATH
  3. $ export CLASSPATH=.:$CLASSPATH

下载最新java版本对应的postgresql jdbc jar

  1. wget https://jdbc.postgresql.org/download/postgresql-9.4.1207.jre7.jar
  2. mv postgresql-9.4.1207.jre7.jar benchmarksql-4.1.0/lib/

配置benchmarksql,使用新的postgresql java驱动

  1. $ vi runBenchmark.sh
  2. java -cp .:../lib/postgresql-9.4.1207.jre7.jar:../lib/log4j-1.2.17.jar:../lib/apache-log4j-extras-1.1.jar:../dist/BenchmarkSQL-4.1.jar -Dprop=$1 jTPCC
  3. $ vi runLoader.sh
  4. java -cp .:../lib/postgresql-9.4.1207.jre7.jar:../dist/BenchmarkSQL-4.1.jar -Dprop=$1 LoadData $2 $3 $4 $5
  5. $ vi runSQL.sh
  6. myCP="../lib/postgresql-9.4.1207.jre7.jar"
  7. myCP="$myCP:../dist/BenchmarkSQL-4.1.jar"
  8. myOPTS="-Dprop=$1"
  9. myOPTS="$myOPTS -DcommandFile=$2"
  10. java -cp .:$myCP $myOPTS ExecJDBC

修改log4j,减少日志打印量。priority改成info,只输出最终结果,不输出产生订单的日志。

  1. $ vi log4j.xml
  2. <?xml version="1.0" encoding="UTF-8" ?>
  3. <!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
  4. <log4j:configuration xmlns:log4j='http://jakarta.apache.org/log4j/'>
  5. <appender name="console" class="org.apache.log4j.ConsoleAppender">
  6. <param name="Threshold" value="info"/>
  7. <layout class="org.apache.log4j.PatternLayout">
  8. <param name="ConversionPattern" value="%d %5p - %m%n"/>
  9. </layout>
  10. </appender>
  11. <appender name="R" class="org.apache.log4j.rolling.RollingFileAppender">
  12. <param name="Append" value="True" />
  13. <rollingPolicy class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
  14. <param name="FileNamePattern" value="log/archive/benchmarksql.%d{yyyyMMddHHmm}.log"/>
  15. <param name="ActiveFileName" value="log/benchmarksql.log"/>
  16. </rollingPolicy>
  17. <triggeringPolicy class="org.apache.log4j.rolling.SizeBasedTriggeringPolicy">
  18. <param name="MaxFileSize" value="1"/>
  19. </triggeringPolicy>
  20. <layout class="org.apache.log4j.PatternLayout">
  21. <param name="ConversionPattern" value="%5p\t[%d{yyyy-MM-dd HH:mm:ss.SSS}]\t%t \t%m%n"/>
  22. </layout>
  23. <filter class="org.apache.log4j.filter.StringMatchFilter">
  24. <param name="StringToMatch" value ="\n" />
  25. <param name="AcceptOnMatch" value="false" />
  26. </filter>
  27. </appender>
  28. <appender name="E" class="org.apache.log4j.rolling.RollingFileAppender">
  29. <param name="Append" value="True" />
  30. <param name="Threshold" value="warn"/>
  31. <rollingPolicy class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
  32. <param name="FileNamePattern" value="log/BenchmarkSQLError.%d.log"/>
  33. <param name="ActiveFileName" value="log/BenchmarkSQLError.log"/>
  34. </rollingPolicy>
  35. <layout class="org.apache.log4j.PatternLayout">
  36. <param name="ConversionPattern" value="%5p\t[%d{yyyy-MM-dd HH:mm:ss.SSS}]\t%t \t%m%n"/>
  37. </layout>
  38. </appender>
  39. <root>
  40. <priority value="info"/>
  41. <appender-ref ref="R"/>
  42. <appender-ref ref="E"/>
  43. </root>
  44. </log4j:configuration>

系统配置优化

  1. 内核配置
  2. /etc/grub.conf
  3. numa=off
  4. elevator=deadline
  5. 编译器版本
  6. gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC)
  7. /etc/sysctl.conf
  8. vm.swappiness = 0
  9. kernel.shmmax=135497418752
  10. net.core.rmem_max = 4194304
  11. net.core.wmem_max = 4194304
  12. net.core.rmem_default = 262144
  13. net.core.wmem_default = 262144
  14. net.ipv4.ip_local_port_range = 9000 65535
  15. kernel.sem = 50100 64128000 50100 1280
  16. vm.dirty_background_bytes = 102400000
  17. vm.dirty_ratio = 80
  18. vm.nr_hugepages = 102352
  19. /etc/security/limits.conf
  20. * soft nofile 655360
  21. * hard nofile 655360
  22. * soft nproc 655360
  23. * hard nproc 655360
  24. * soft stack unlimited
  25. * hard stack unlimited
  26. * soft memlock 250000000
  27. * hard memlock 250000000
  28. 块设备预读
  29. blockdev --setra 16384 /dev/dfa
  30. blockdev --setra 16384 /dev/dfb
  31. blockdev --setra 16384 /dev/dfc
  32. blockdev --setra 16384 /dev/dm-0

安装PostgreSQL

PostgreSQL编译项

  1. ./configure --prefix=/u02/digoal/soft_bak/pgsql9.5 --with-blocksize=8 --with-pgport=1921 --with-perl --with-python --with-tcl --with-openssl --with-pam --with-ldap --with-libxml --with-libxslt --enable-thread-safety
  2. gmake world -j32
  3. gmake install-world -j32

配置postgres环境变量

  1. $ vi env_pg.sh
  2. export PS1="$USER@`/bin/hostname -s`-> "
  3. export PGPORT=1921
  4. export PGDATA=/data01/pgdata/pg_root
  5. export LANG=en_US.utf8
  6. export PGHOME=/u02/digoal/soft_bak/pgsql9.5
  7. export LD_LIBRARY_PATH=$PGHOME/lib:/lib64:/usr/lib64:/usr/local/lib64:/lib:/usr/lib:/usr/local/lib:$LD_LIBRARY_PATH
  8. export DATE=`date +"%Y%m%d%H%M"`
  9. export PATH=$PGHOME/bin:$PATH:.
  10. export MANPATH=$PGHOME/share/man:$MANPATH
  11. export PGHOST=$PGDATA
  12. export PGDATABASE=postgres
  13. export PGUSER=postgres
  14. alias rm='rm -i'
  15. alias ll='ls -lh'
  16. unalias vi

配置postgresql.conf

  1. $ vi $PGDATA/postgresql.conf
  2. port = 1921 # (change requires restart)
  3. max_connections = 300 # (change requires restart)
  4. unix_socket_directories = '.' # comma-separated list of directories
  5. shared_buffers = 32GB # min 128kB
  6. huge_pages = try # on, off, or try
  7. maintenance_work_mem = 2GB # min 1MB
  8. dynamic_shared_memory_type = posix # the default is the first option
  9. bgwriter_delay = 10ms # 10-10000ms between rounds
  10. wal_level = minimal # minimal, archive, hot_standby, or logical
  11. synchronous_commit = off # synchronization level;
  12. full_page_writes = off # recover from partial page writes, 有备份和归档就可以关闭它, crash后从备份恢复, 放partial write
  13. wal_buffers = 16MB # min 32kB, -1 sets based on shared_buffers
  14. wal_writer_delay = 10ms # 1-10000 milliseconds
  15. max_wal_size = 32GB
  16. effective_cache_size = 240GB
  17. log_destination = 'csvlog' # Valid values are combinations of
  18. logging_collector = on # Enable capturing of stderr and csvlog
  19. log_truncate_on_rotation = on # If on, an existing log file with the

编辑benchmarksql连接配置和压测配置

1000 个仓库,约5亿数据量。

  1. $ vi props.pg
  2. driver=org.postgresql.Driver
  3. conn=jdbc:postgresql://localhost:1921/postgres
  4. user=postgres
  5. password=123
  6. warehouses=1000
  7. terminals=96
  8. //To run specified transactions per terminal- runMins must equal zero
  9. runTxnsPerTerminal=0
  10. //To run for specified minutes- runTxnsPerTerminal must equal zero
  11. runMins=1
  12. //Number of total transactions per minute
  13. limitTxnsPerMin=0
  14. //The following five values must add up to 100
  15. //The default percentages of 45, 43, 4, 4 & 4 match the TPC-C spec
  16. newOrderWeight=40
  17. paymentWeight=36
  18. orderStatusWeight=8
  19. deliveryWeight=8
  20. stockLevelWeight=8

生成测试数据

配置postgres用户默认搜索路径

  1. $ psql
  2. psql (9.5.0)
  3. Type "help" for help.
  4. postgres=# alter role postgres set search_path='benchmarksql','public';

创建用于存放生成CSV的目录

  1. $ mkdir /u02/digoal/soft_bak/benchcsv

修改benchmarksql sqlTableCopies,指定目录

  1. $ vi sqlTableCopies
  2. copy benchmarksql.warehouse
  3. (w_id, w_ytd, w_tax, w_name, w_street_1, w_street_2, w_city, w_state, w_zip)
  4. from '/u02/digoal/soft_bak/benchcsv/warehouse.csv' WITH CSV;
  5. copy benchmarksql.item
  6. (i_id, i_name, i_price, i_data, i_im_id)
  7. from '/u02/digoal/soft_bak/benchcsv/item.csv' WITH CSV;
  8. copy benchmarksql.stock
  9. (s_i_id, s_w_id, s_quantity, s_ytd, s_order_cnt, s_remote_cnt, s_data,
  10. s_dist_01, s_dist_02, s_dist_03, s_dist_04, s_dist_05,
  11. s_dist_06, s_dist_07, s_dist_08, s_dist_09, s_dist_10)
  12. from '/u02/digoal/soft_bak/benchcsv/stock.csv' WITH CSV;
  13. copy benchmarksql.district
  14. (d_id, d_w_id, d_ytd, d_tax, d_next_o_id, d_name, d_street_1,
  15. d_street_2, d_city, d_state, d_zip)
  16. from '/u02/digoal/soft_bak/benchcsv/district.csv' WITH CSV;
  17. copy benchmarksql.customer
  18. (c_id, c_d_id, c_w_id, c_discount, c_credit, c_last, c_first, c_credit_lim,
  19. c_balance, c_ytd_payment, c_payment_cnt, c_delivery_cnt, c_street_1,
  20. c_street_2, c_city, c_state, c_zip, c_phone, c_since, c_middle, c_data)
  21. from '/u02/digoal/soft_bak/benchcsv/customer.csv' WITH CSV;
  22. copy benchmarksql.history
  23. (hist_id, h_c_id, h_c_d_id, h_c_w_id, h_d_id, h_w_id, h_date, h_amount, h_data)
  24. from '/u02/digoal/soft_bak/benchcsv/cust-hist.csv' WITH CSV;
  25. copy benchmarksql.oorder
  26. (o_id, o_w_id, o_d_id, o_c_id, o_carrier_id, o_ol_cnt, o_all_local, o_entry_d)
  27. from '/u02/digoal/soft_bak/benchcsv/order.csv' WITH CSV;
  28. copy benchmarksql.order_line
  29. (ol_w_id, ol_d_id, ol_o_id, ol_number, ol_i_id, ol_delivery_d,
  30. ol_amount, ol_supply_w_id, ol_quantity, ol_dist_info)
  31. from '/u02/digoal/soft_bak/benchcsv/order-line.csv' WITH CSV;
  32. copy benchmarksql.new_order
  33. (no_w_id, no_d_id, no_o_id)
  34. from '/u02/digoal/soft_bak/benchcsv/new-order.csv' WITH CSV;

建立表结构

  1. $ cd benchmarksql-4.1.0/run
  2. $ ./runSQL.sh props.pg sqlTableCreates

生成CSV

  1. $ ./runLoader.sh props.pg numWarehouses 1000 fileLocation /u02/digoal/soft_bak/benchcsv/

1000个仓库的数据量69 GB:

  1. total 69G
  2. -rw-r--r-- 1 digoal users 2.0G Jan 9 15:53 cust-hist.csv
  3. -rw-r--r-- 1 digoal users 16G Jan 9 15:53 customer.csv
  4. -rw-r--r-- 1 digoal users 898K Jan 9 15:12 district.csv
  5. -rw-r--r-- 1 digoal users 7.0M Jan 9 14:22 item.csv
  6. -rw-r--r-- 1 digoal users 95M Jan 9 16:14 new-order.csv
  7. -rw-r--r-- 1 digoal users 1.3G Jan 9 16:14 order.csv
  8. -rw-r--r-- 1 digoal users 22G Jan 9 16:14 order-line.csv
  9. -rw-r--r-- 1 digoal users 28G Jan 9 15:12 stock.csv
  10. -rw-r--r-- 1 digoal users 84K Jan 9 14:22 warehouse.csv

导入数据库

  1. $ ./runSQL.sh props.pg sqlTableCopies

创建约束和索引

  1. $ ./runSQL.sh props.pg sqlIndexCreates

备份

  1. $ pg_dump -f /u02/digoal/soft_bak/benchmarksql.dmp -F c -n benchmarksql postgres

阶段1 TPC-C 压测

  1. nohup ./runBenchmark.sh props.pg >/dev/null 2>./errrun.log &

测试结果

  1. INFO [2016-01-09 22:03:39.963] Thread-7 Term-00, Measured tpmC (NewOrders) = 102494.46
  2. INFO [2016-01-09 22:03:39.963] Thread-7 Term-00, Measured tpmTOTAL = 256195.32
  3. INFO [2016-01-09 22:03:39.964] Thread-7 Term-00, Session Start = 2016-01-09 21:53:39
  4. INFO [2016-01-09 22:03:39.964] Thread-7 Term-00, Session End = 2016-01-09 22:03:39
  5. INFO [2016-01-09 22:03:39.964] Thread-7 Term-00, Transaction Count = 2563088

主机信息,截取压测第9分钟的top, iostat数据。

  1. TOP
  2. top - 22:02:09 up 3 days, 12:55, 3 users, load average: 19.23, 15.97, 8.37
  3. Tasks: 619 total, 10 running, 609 sleeping, 0 stopped, 0 zombie
  4. Cpu(s): 35.0%us, 9.4%sy, 0.0%ni, 52.6%id, 0.1%wa, 0.0%hi, 2.9%si, 0.0%st
  5. Mem: 264643396k total, 241719372k used, 22924024k free, 36672k buffers
  6. Swap: 18825200k total, 0k used, 18825200k free, 196557376k cached
  7. iostat -x
  8. avg-cpu: %user %nice %system %iowait %steal %idle
  9. 35.07 0.00 12.30 0.12 0.00 52.51
  10. Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
  11. dfa 0.00 0.00 57.40 743.40 918.40 11849.00 15.94 0.02 0.03 0.03 2.08
  12. dfb 0.00 0.00 57.20 740.40 915.20 11829.00 15.98 0.02 0.03 0.03 2.04
  13. dfc 0.00 0.00 58.40 730.80 934.40 11675.80 15.98 0.03 0.03 0.03 2.52
  14. dm-0 0.00 0.00 173.00 2213.20 2768.00 35331.40 15.97 0.08 0.03 0.03 7.02

PostgreSQL可以使用oprofile或perf top跟踪统计,可以参考这里,找到需要优化的代码就靠它了。

  1. CPU: Intel Ivy Bridge microarchitecture, speed 2600 MHz (estimated)
  2. Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000
  3. vma samples % app name symbol name
  4. 007a7780 751274 5.1565 /soft/digoal/soft_bak/pgsql9.5/bin/postgres hash_search_with_hash_value
  5. 004a92f0 574315 3.9419 /soft/digoal/soft_bak/pgsql9.5/bin/postgres _bt_compare
  6. 006a4bd0 514473 3.5312 /soft/digoal/soft_bak/pgsql9.5/bin/postgres LWLockAcquire
  7. 0078a090 510962 3.5071 /soft/digoal/soft_bak/pgsql9.5/bin/postgres SearchCatCache
  8. 007bc3a0 484601 3.3262 /soft/digoal/soft_bak/pgsql9.5/bin/postgres AllocSetAlloc
  9. 006969c0 442341 3.0361 /soft/digoal/soft_bak/pgsql9.5/bin/postgres GetSnapshotData
  10. 00498930 352134 2.4170 /soft/digoal/soft_bak/pgsql9.5/bin/postgres heap_hot_search_buffer
  11. 005b8f70 279718 1.9199 /soft/digoal/soft_bak/pgsql9.5/bin/postgres ExecInitExpr
  12. 006895d0 249377 1.7117 /soft/digoal/soft_bak/pgsql9.5/bin/postgres PinBuffer
  13. 006a4220 168770 1.1584 /soft/digoal/soft_bak/pgsql9.5/bin/postgres LWLockRelease
  14. 007ac620 161861 1.1110 /soft/digoal/soft_bak/pgsql9.5/bin/postgres pg_encoding_mbcliplen
  15. 007a2180 161090 1.1057 /soft/digoal/soft_bak/pgsql9.5/bin/postgres FunctionCall2Coll
  16. 004aaa80 153079 1.0507 /soft/digoal/soft_bak/pgsql9.5/bin/postgres _bt_checkkeys
  17. 007a3950 147078 1.0095 /soft/digoal/soft_bak/pgsql9.5/bin/postgres fmgr_info_cxt_security
  18. 0049bce0 136680 0.9381 /soft/digoal/soft_bak/pgsql9.5/bin/postgres heap_page_prune_opt
  19. 0048c8f0 130807 0.8978 /soft/digoal/soft_bak/pgsql9.5/bin/postgres hash_any
  20. 006b2e50 130564 0.8962 /soft/digoal/soft_bak/pgsql9.5/bin/postgres PostgresMain
  21. 0046c790 121776 0.8358 /soft/digoal/soft_bak/pgsql9.5/bin/postgres slot_deform_tuple
  22. ......

阶段1 PostgreSQL 9.5.0 TPmC : 256195.32

阶段1 性能瓶颈分析

从系统统计信息可以分析得到, 系统还有大量空闲CPU,IO资源,所以极限性能应该不止于此。

阶段2 TPC-C 优化

benchmarksql放到另一台主机,主机间万兆网同一交换机下互联。

为了突破测试程序的极限,开4个schema,每个schema负责1000个仓库,数据量总共20亿左右,入库后的数据总量约400GB。每个测试程序对付一个schema。终端数保持一致,每个测试程序开24个终端,一共96个终端。

让benchmarksql支持多个schema

benchmarksql 默认编译好的,还有配置都是用的benchmarksql 这个schema,如果我们想对一个数据库用多个schema来压性能,就需要开多个benchmarksql终端来压。这里就涉及到benchmarksql需要支持多个schema,每个benchmarksql连一个schema。 目录结构:

  1. drwxr-xr-x 2 digoal users 4096 Jan 10 13:24 build
  2. -rwxr-xr-x 1 digoal users 1112 Jan 10 13:24 build.xml
  3. drwxr-xr-x 2 digoal users 4096 Jan 10 13:24 dist
  4. -rw-r--r-- 1 digoal users 128 Jan 10 13:24 env_java.sh
  5. -rwxr-xr-x 1 digoal users 1927 Jan 10 13:24 HOW-TO-RUN.txt
  6. drwxr-xr-x 2 digoal users 4096 Jan 10 13:24 lib
  7. -rwxr-xr-x 1 digoal users 2825 Jan 10 13:24 README.txt
  8. drwxr-xr-x 3 digoal users 4096 Jan 10 13:24 run
  9. drwxr-xr-x 6 digoal users 4096 Jan 10 13:24 src

需要修改的地方:

  1. src/LoadData/LoadData.java
  2. src/client/jTPCCTerminal.java
  3. run/props.ora
  4. run/props.pg
  5. run/sqlIndexCreates
  6. run/sqlIndexDrops
  7. run/sqlTableCopies
  8. run/sqlTableCreates
  9. run/sqlTableDrops
  10. run/sqlTableTruncates

把所有的benchmarksql替换成新的schema name,例如 test01

  1. sed -i "s/benchmarksql/test01/g" src/LoadData/LoadData.java
  2. sed -i "s/benchmarksql/test01/g" src/client/jTPCCTerminal.java
  3. sed -i "s/benchmarksql/test01/g" run/props.ora
  4. sed -i "s/benchmarksql/test01/g" run/props.pg
  5. sed -i "s/benchmarksql/test01/g" run/sqlIndexCreates
  6. sed -i "s/BENCHMARKSQL/TEST01/g" run/sqlIndexCreates
  7. sed -i "s/benchmarksql/test01/g" run/sqlIndexDrops
  8. sed -i "s/benchmarksql/test01/g" run/sqlTableCopies
  9. sed -i "s/benchmarksql/test01/g" run/sqlTableCreates
  10. sed -i "s/benchmarksql/test01/g" run/sqlTableDrops
  11. sed -i "s/benchmarksql/test01/g" run/sqlTableTruncates

然后使用ant重新打包工程,如果没有安装ant,可以用yum install -y ant安装它。

使用ant重新打包benchmarksql.jar

  1. $ant -buildfile ./build.xml
  2. Buildfile: ./build.xml
  3. Trying to override old definition of task javac
  4. init:
  5. compile:
  6. [javac] Compiling 16 source files to /soft/digoal/soft_bak/benchmarksql-4.1.0_oracle01/build
  7. [javac] Note: /soft/digoal/soft_bak/benchmarksql-4.1.0_oracle01/src/client/jTPCCTerminal.java uses unchecked or unsafe operations.
  8. [javac] Note: Recompile with -Xlint:unchecked for details.
  9. dist:
  10. [jar] Building jar: /soft/digoal/soft_bak/benchmarksql-4.1.0_oracle01/dist/BenchmarkSQL-4.1.jar
  11. BUILD SUCCESSFUL
  12. Total time: 2 seconds

现在benchmarksql使用的是test01这个schema。使用同样的方法,生成支持test02,test03,test04 schema的benchmarksql版本。

创建4个数据库,分别为test01、test02、test03、test04,将阶段1 pg_dump导出的数据导入到这4个数据库, 并将schema重命名为对应的test01、test02、test03、test04。

测试数据量

  1. postgres=# \l+
  2. List of databases
  3. Name | Owner | Encoding | Collate | Ctype | Access privileges | Size | Tablespace | Description
  4. -----------+----------+----------+---------+-------+-----------------------+---------+------------+--------------------------------------------
  5. test01 | test01 | UTF8 | C | C | | 100 GB | pg_default |
  6. test02 | test02 | UTF8 | C | C | | 100 GB | pg_default |
  7. test03 | test03 | UTF8 | C | C | | 100 GB | pg_default |
  8. test04 | test04 | UTF8 | C | C | | 100 GB | pg_default |

benchmarksql软件目录

  1. $ ll
  2. drwxr-xr-x 7 digoal users 4.0K Jan 10 14:41 benchmarksql-4.1.0_pg01
  3. drwxr-xr-x 7 digoal users 4.0K Jan 10 14:41 benchmarksql-4.1.0_pg02
  4. drwxr-xr-x 7 digoal users 4.0K Jan 10 14:41 benchmarksql-4.1.0_pg03
  5. drwxr-xr-x 7 digoal users 4.0K Jan 10 14:41 benchmarksql-4.1.0_pg04

配置每个benchmarksql的props.pg,修改对应的连接。 例如:

  1. $cat run/props.pg
  2. driver=org.postgresql.Driver
  3. conn=jdbc:postgresql://xxx.xxx.xxx.xxx:1921/test01?preparedStatementCacheSizeMiB=10
  4. user=test01
  5. password=123
  6. warehouses=1000
  7. terminals=20
  8. //To run specified transactions per terminal- runMins must equal zero
  9. runTxnsPerTerminal=0
  10. //To run for specified minutes- runTxnsPerTerminal must equal zero
  11. runMins=10
  12. //Number of total transactions per minute
  13. limitTxnsPerMin=0
  14. //The following five values must add up to 100
  15. //The default percentages of 45, 43, 4, 4 & 4 match the TPC-C spec
  16. newOrderWeight=40
  17. paymentWeight=36
  18. orderStatusWeight=8
  19. deliveryWeight=8
  20. stockLevelWeight=8

配置数据库pg_hba.conf,允许测试机连接。

  1. vi $PGDATA/pg_hba.conf
  2. host all all 0.0.0.0/0 md5
  3. pg_ctl reload

阶段2 TPC-C 压测

  1. cd benchmarksql-4.1.0_pg01/run
  2. nohup ./runBenchmark.sh props.pg >/dev/null 2>./errrun.log &
  3. cd ../../benchmarksql-4.1.0_pg02/run
  4. nohup ./runBenchmark.sh props.pg >/dev/null 2>./errrun.log &
  5. cd ../../benchmarksql-4.1.0_pg03/run
  6. nohup ./runBenchmark.sh props.pg >/dev/null 2>./errrun.log &
  7. cd ../../benchmarksql-4.1.0_pg04/run
  8. nohup ./runBenchmark.sh props.pg >/dev/null 2>./errrun.log &
  9. cd ../..

阶段2 PostgreSQL 9.5.0 TPmC : 453058.64

  1. $ cat benchmarksql-4.1.0_pg01/run/log/benchmarksql.log
  2. INFO [2016-01-10 17:54:04.925] Thread-22 Term-00, Measured tpmC (NewOrders) = 45416.28
  3. INFO [2016-01-10 17:54:04.925] Thread-22 Term-00, Measured tpmTOTAL = 113487.61
  4. INFO [2016-01-10 17:54:04.925] Thread-22 Term-00, Session Start = 2016-01-10 17:44:04
  5. INFO [2016-01-10 17:54:04.925] Thread-22 Term-00, Session End = 2016-01-10 17:54:04
  6. INFO [2016-01-10 17:54:04.925] Thread-22 Term-00, Transaction Count = 1134913
  7. $ cat benchmarksql-4.1.0_pg02/run/log/benchmarksql.log
  8. INFO [2016-01-10 17:54:04.943] Thread-12 Term-00, Measured tpmC (NewOrders) = 45292.48
  9. INFO [2016-01-10 17:54:04.943] Thread-12 Term-00, Measured tpmTOTAL = 113269.54
  10. INFO [2016-01-10 17:54:04.943] Thread-12 Term-00, Session Start = 2016-01-10 17:44:04
  11. INFO [2016-01-10 17:54:04.944] Thread-12 Term-00, Session End = 2016-01-10 17:54:04
  12. INFO [2016-01-10 17:54:04.944] Thread-12 Term-00, Transaction Count = 1132770
  13. $ cat benchmarksql-4.1.0_pg03/run/log/benchmarksql.log
  14. INFO [2016-01-10 17:54:04.955] Thread-12 Term-00, Measured tpmC (NewOrders) = 45336.15
  15. INFO [2016-01-10 17:54:04.955] Thread-12 Term-00, Measured tpmTOTAL = 113247.19
  16. INFO [2016-01-10 17:54:04.956] Thread-12 Term-00, Session Start = 2016-01-10 17:44:04
  17. INFO [2016-01-10 17:54:04.956] Thread-12 Term-00, Session End = 2016-01-10 17:54:04
  18. INFO [2016-01-10 17:54:04.956] Thread-12 Term-00, Transaction Count = 1132537
  19. $ cat benchmarksql-4.1.0_pg04/run/log/benchmarksql.log
  20. INFO [2016-01-10 17:54:04.986] Thread-23 Term-00, Measured tpmC (NewOrders) = 45231.67
  21. INFO [2016-01-10 17:54:04.987] Thread-23 Term-00, Measured tpmTOTAL = 113054.3
  22. INFO [2016-01-10 17:54:04.987] Thread-23 Term-00, Session Start = 2016-01-10 17:44:04
  23. INFO [2016-01-10 17:54:04.987] Thread-23 Term-00, Session End = 2016-01-10 17:54:04
  24. INFO [2016-01-10 17:54:04.987] Thread-23 Term-00, Transaction Count = 1130640
  25. TPM
  26. 113487.61 + 113269.54 + 113247.19 + 113054.3 = 453058.64

第9分钟操作系统统计信息

  1. TOP
  2. top - 17:38:27 up 4 days, 8:32, 4 users, load average: 78.54, 68.64, 37.22
  3. Tasks: 658 total, 34 running, 624 sleeping, 0 stopped, 0 zombie
  4. Cpu(s): 70.2%us, 15.7%sy, 0.0%ni, 5.5%id, 1.5%wa, 0.0%hi, 7.1%si, 0.0%st
  5. Mem: 264643396k total, 229866068k used, 34777328k free, 59652k buffers
  6. Swap: 18825200k total, 0k used, 18825200k free, 183529592k cached
  7. iostat -x
  8. avg-cpu: %user %nice %system %iowait %steal %idle
  9. 71.39 0.00 22.47 1.26 0.00 4.88
  10. Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
  11. dfa 0.00 0.00 3659.33 7008.67 58538.67 112050.67 15.99 5.85 0.55 0.06 68.17
  12. dfb 0.00 0.00 3714.67 6888.67 59418.67 110173.33 15.99 5.98 0.56 0.06 67.87
  13. dfc 0.00 0.00 3709.00 6974.33 59328.00 111504.00 15.99 5.63 0.52 0.07 71.60
  14. dm-0 0.00 0.00 11083.00 20870.33 177285.33 333706.67 15.99 17.60 0.55 0.03 92.10

测试过程oprofile报告

  1. #/home/digoal/oprof/bin/opreport -l -f -w -x -t 0.5
  2. Using /soft/digoal/soft_bak/oprof_test/oprofile_data/samples/ for samples directory.
  3. WARNING! Some of the events were throttled. Throttling occurs when
  4. the initial sample rate is too high, causing an excessive number of
  5. interrupts. Decrease the sampling frequency. Check the directory
  6. /soft/digoal/soft_bak/oprof_test/oprofile_data/samples/current/stats/throttled
  7. for the throttled event names.
  8. CPU: Intel Ivy Bridge microarchitecture, speed 2600 MHz (estimated)
  9. Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000
  10. vma samples % app name symbol name
  11. 007a7780 2632700 5.2511 /soft/digoal/soft_bak/pgsql9.5/bin/postgres hash_search_with_hash_value
  12. 004a92f0 1895924 3.7816 /soft/digoal/soft_bak/pgsql9.5/bin/postgres _bt_compare
  13. 006969c0 1844371 3.6787 /soft/digoal/soft_bak/pgsql9.5/bin/postgres GetSnapshotData
  14. 0078a090 1775031 3.5404 /soft/digoal/soft_bak/pgsql9.5/bin/postgres SearchCatCache
  15. 006a4bd0 1725350 3.4413 /soft/digoal/soft_bak/pgsql9.5/bin/postgres LWLockAcquire
  16. 007bc3a0 1565190 3.1219 /soft/digoal/soft_bak/pgsql9.5/bin/postgres AllocSetAlloc
  17. 00498930 1406694 2.8058 /soft/digoal/soft_bak/pgsql9.5/bin/postgres heap_hot_search_buffer
  18. 005b8f70 965646 1.9261 /soft/digoal/soft_bak/pgsql9.5/bin/postgres ExecInitExpr
  19. 006895d0 767078 1.5300 /soft/digoal/soft_bak/pgsql9.5/bin/postgres PinBuffer
  20. 004aaa80 617741 1.2321 /soft/digoal/soft_bak/pgsql9.5/bin/postgres _bt_checkkeys
  21. 007a2180 588043 1.1729 /soft/digoal/soft_bak/pgsql9.5/bin/postgres FunctionCall2Coll
  22. 006a4220 575864 1.1486 /soft/digoal/soft_bak/pgsql9.5/bin/postgres LWLockRelease
  23. 007ac620 485162 0.9677 /soft/digoal/soft_bak/pgsql9.5/bin/postgres pg_encoding_mbcliplen
  24. 007a3950 471102 0.9396 /soft/digoal/soft_bak/pgsql9.5/bin/postgres fmgr_info_cxt_security
  25. 0046c790 441548 0.8807 /soft/digoal/soft_bak/pgsql9.5/bin/postgres slot_deform_tuple
  26. 0048c8f0 425867 0.8494 /soft/digoal/soft_bak/pgsql9.5/bin/postgres hash_any
  27. 006b2e50 404548 0.8069 /soft/digoal/soft_bak/pgsql9.5/bin/postgres PostgresMain
  28. 007bd0f0 396510 0.7909 /soft/digoal/soft_bak/pgsql9.5/bin/postgres palloc
  29. 0049bce0 394201 0.7863 /soft/digoal/soft_bak/pgsql9.5/bin/postgres heap_page_prune_opt
  30. 007bce00 353243 0.7046 /soft/digoal/soft_bak/pgsql9.5/bin/postgres pfree
  31. 0049b300 335896 0.6700 /soft/digoal/soft_bak/pgsql9.5/bin/postgres heap_page_prune
  32. 0046c580 313145 0.6246 /soft/digoal/soft_bak/pgsql9.5/bin/postgres heap_getsysattr
  33. 006b14a0 311776 0.6219 /soft/digoal/soft_bak/pgsql9.5/bin/postgres exec_bind_message
  34. 007cb070 292106 0.5826 /soft/digoal/soft_bak/pgsql9.5/bin/postgres HeapTupleSatisfiesMVCC
  35. 007bd210 275282 0.5491 /soft/digoal/soft_bak/pgsql9.5/bin/postgres MemoryContextAllocZeroAligned
  36. 005b8530 273199 0.5449 /soft/digoal/soft_bak/pgsql9.5/bin/postgres ExecProject
  37. 00494ba0 266495 0.5315 /soft/digoal/soft_bak/pgsql9.5/bin/postgres heap_update
  38. 007bca10 265556 0.5297 /soft/digoal/soft_bak/pgsql9.5/bin/postgres AllocSetFree

阶段2 性能瓶颈分析

  1. 单次IO请求响应较高,在0.06毫秒;
  2. 系统调用占用的CPU百分比较高;
  3. 数据库获取快照占用CPU较高,需要PostgreSQL代码层优化,本文暂时不涉及。

阶段3 TPC-C 优化

  1. 开启PostgreSQL 预读, 预读数(n-1), 仅仅对hash 聚合有效, 有兴趣的同学可以看看PostgreSQL代码。n是条带数, 所以本例case effective_io_concurrency = 2 这个使用xfs的largeio参数效果是类似的,还有块设备的预读功能。 开启预读可能存在IO浪费的情况,例如全BUFFER命中的情况下。预读对于OLAP非常有效。

  2. 开启大页支持, 开到168G;

    1. /etc/sysctl.conf
    2. vm.nr_hugepages = 102352
    3. sysctl -p
    4. /etc/security/limits.conf
    5. * soft memlock 250000000
    6. * hard memlock 250000000
    7. # 配置 memlock 大于 nr_hugepages 大于 shared_buffers
  3. 使用数据块分组提交, commit_delay = 10, commit_siblings = 16 平滑检查点到0.8个周期,减少fsync dirty page IO影响。

    1. http://blog.163.com/digoal@126/blog/static/1638770402016011115141697/
    2. shared_buffers = 164GB # min 128kB
    3. huge_pages = on # on, off, or try
    4. maintenance_work_mem = 2GB # min 1MB
    5. wal_buffers = 16MB # min 32kB, -1 sets based on shared_buffers
    6. wal_writer_delay = 10ms # 1-10000 milliseconds
    7. commit_delay = 10 # range 0-100000, in microseconds
    8. commit_siblings = 16 # range 1-1000
    9. checkpoint_timeout = 35min # range 30s-1h
    10. max_wal_size = 320GB
    11. checkpoint_completion_target = 0.8 # checkpoint target duration, 0.0 - 1.0
    12. effective_cache_size = 240GB
    13. log_destination = 'csvlog' # Valid values are combinations of
    14. logging_collector = on # Enable capturing of stderr and csvlog
    15. log_truncate_on_rotation = on # If on, an existing log file with the

阶段3 TPC-C 压测

  1. $tail -n 5 benchmarksql-4.1.0_pg01/run/log/benchmarksql.log
  2. INFO [2016-01-11 13:33:55.917] Thread-14 Term-00, Measured tpmC (NewOrders) = 48151.07
  3. INFO [2016-01-11 13:33:55.917] Thread-14 Term-00, Measured tpmTOTAL = 120215.48
  4. INFO [2016-01-11 13:33:55.917] Thread-14 Term-00, Session Start = 2016-01-11 13:23:55
  5. INFO [2016-01-11 13:33:55.917] Thread-14 Term-00, Session End = 2016-01-11 13:33:55
  6. INFO [2016-01-11 13:33:55.917] Thread-14 Term-00, Transaction Count = 1202222
  7. $tail -n 5 benchmarksql-4.1.0_pg02/run/log/benchmarksql.log
  8. INFO [2016-01-11 13:33:55.971] Thread-16 Term-00, Measured tpmC (NewOrders) = 48505.54
  9. INFO [2016-01-11 13:33:55.971] Thread-16 Term-00, Measured tpmTOTAL = 121182.26
  10. INFO [2016-01-11 13:33:55.971] Thread-16 Term-00, Session Start = 2016-01-11 13:23:55
  11. INFO [2016-01-11 13:33:55.972] Thread-16 Term-00, Session End = 2016-01-11 13:33:55
  12. INFO [2016-01-11 13:33:55.972] Thread-16 Term-00, Transaction Count = 1211858
  13. $tail -n 5 benchmarksql-4.1.0_pg03/run/log/benchmarksql.log
  14. INFO [2016-01-11 13:33:55.985] Thread-4 Term-00, Measured tpmC (NewOrders) = 48119.61
  15. INFO [2016-01-11 13:33:55.985] Thread-4 Term-00, Measured tpmTOTAL = 120523.98
  16. INFO [2016-01-11 13:33:55.985] Thread-4 Term-00, Session Start = 2016-01-11 13:23:55
  17. INFO [2016-01-11 13:33:55.985] Thread-4 Term-00, Session End = 2016-01-11 13:33:55
  18. INFO [2016-01-11 13:33:55.985] Thread-4 Term-00, Transaction Count = 1205271
  19. $tail -n 5 benchmarksql-4.1.0_pg04/run/log/benchmarksql.log
  20. INFO [2016-01-11 13:33:55.958] Thread-21 Term-00, Measured tpmC (NewOrders) = 48087.55
  21. INFO [2016-01-11 13:33:55.958] Thread-21 Term-00, Measured tpmTOTAL = 120461.29
  22. INFO [2016-01-11 13:33:55.958] Thread-21 Term-00, Session Start = 2016-01-11 13:23:55
  23. INFO [2016-01-11 13:33:55.958] Thread-21 Term-00, Session End = 2016-01-11 13:33:55
  24. INFO [2016-01-11 13:33:55.958] Thread-21 Term-00, Transaction Count = 1204638
  25. TPM:
  26. 120215.48 + 121182.26 + 120523.98 + 120461.29 = 482383.01

阶段3 PostgreSQL 9.5.0 TPmC : 482383.01

阶段3 性能瓶颈分析

  1. 操作系统后台刷脏页的数据量太大,容易带来抖动;
  2. 优化并发数,减少事务快照CPU开销;
  3. 优化work_mem,减少文件排序;
  4. 优化分组提交阈值。

阶段4 TPC-C 优化

  • 优化分组提交的时延,最小结束点并发事务数量,work_mem等
  • 操作系统内核参数优化
  • 优化老化脏页刷新唤醒间隔
  • vm.dirty_writeback_centisecs\=10
  • 优化老化脏页阈值, 60秒以前的脏页将被刷到磁盘
  • vm.dirty_expire_centisecs\=6000
  • 优化用户进程刷脏页阈值, 脏页超过80%才需要用户进程刷脏页, 尽量避免用户进程刷脏页.
  • vm.dirty_ratio\=80
  • 优化内核进程刷脏页阈值, 尽量让系统后台进程刷脏页.
  • vm.dirty_background_bytes\=102400000
  • 优化终端数,每个benchmarksql 20个终端,一共80个终端。

PostgreSQL参数

  1. listen_addresses = '0.0.0.0' # what IP address(es) to listen on;
  2. port = 1921 # (change requires restart)
  3. max_connections = 300 # (change requires restart)
  4. unix_socket_directories = '.' # comma-separated list of directories
  5. shared_buffers = 164GB # min 128kB
  6. huge_pages = on # on, off, or try
  7. work_mem = 256MB # min 64kB
  8. maintenance_work_mem = 2GB # min 1MB
  9. autovacuum_work_mem = 2GB # min 1MB, or -1 to use maintenance_work_mem
  10. dynamic_shared_memory_type = mmap # the default is the first option
  11. vacuum_cost_delay = 10 # 0-100 milliseconds
  12. vacuum_cost_limit = 10000 # 1-10000 credits
  13. bgwriter_delay = 10ms # 10-10000ms between rounds
  14. bgwriter_lru_maxpages = 1000# 0-1000 max buffers written/round
  15. bgwriter_lru_multiplier = 10.0 # 0-10.0 multipler on buffers scanned/round
  16. effective_io_concurrency = 2 # 1-1000; 0 disables prefetching
  17. wal_level = minimal # minimal, archive, hot_standby, or logical
  18. synchronous_commit = off # synchronization level;
  19. full_page_writes = off # recover from partial page writes
  20. wal_buffers = 1GB # min 32kB, -1 sets based on shared_buffers
  21. wal_writer_delay = 10ms # 1-10000 milliseconds
  22. commit_delay = 10 # range 0-100000, in microseconds
  23. commit_siblings = 6 # range 1-1000
  24. checkpoint_timeout = 55min # range 30s-1h
  25. max_wal_size = 320GB
  26. checkpoint_completion_target = 0.99 # checkpoint target duration, 0.0 - 1.0
  27. random_page_cost = 1.0 # same scale as above
  28. effective_cache_size = 240GB
  29. log_destination = 'csvlog' # Valid values are combinations of
  30. logging_collector = on # Enable capturing of stderr and csvlog
  31. log_truncate_on_rotation = on # If on, an existing log file with the
  32. log_timezone = 'PRC'
  33. update_process_title = off
  34. track_activities = off
  35. autovacuum = on# Enable autovacuum subprocess? 'on'

重启数据库

  1. pg_ctl restart

将数据加载到shared buffer

  1. psql
  2. \c test01 test01
  3. explain analyze select * from customer; explain analyze select * from stock;
  4. \c test02 test02
  5. explain analyze select * from customer; explain analyze select * from stock;
  6. \c test03 test03
  7. explain analyze select * from customer; explain analyze select * from stock;
  8. \c test04 test04
  9. explain analyze select * from customer; explain analyze select * from stock;

阶段4 TPC-C 压测

  1. $ tail -n 5 benchmarksql-4.1.0_pg01/run/log/benchmarksql.log
  2. INFO [2016-01-12 11:55:09.461] Thread-12 Term-00, Measured tpmC (NewOrders) = 57995.55
  3. INFO [2016-01-12 11:55:09.461] Thread-12 Term-00, Measured tpmTOTAL = 144975.59
  4. INFO [2016-01-12 11:55:09.461] Thread-12 Term-00, Session Start = 2016-01-12 11:45:09
  5. INFO [2016-01-12 11:55:09.461] Thread-12 Term-00, Session End = 2016-01-12 11:55:09
  6. INFO [2016-01-12 11:55:09.462] Thread-12 Term-00, Transaction Count = 1449796
  7. $ tail -n 5 benchmarksql-4.1.0_pg02/run/log/benchmarksql.log
  8. INFO [2016-01-12 11:55:09.499] Thread-0 Term-00, Measured tpmC (NewOrders) = 58013.75
  9. INFO [2016-01-12 11:55:09.499] Thread-0 Term-00, Measured tpmTOTAL = 145006.74
  10. INFO [2016-01-12 11:55:09.499] Thread-0 Term-00, Session Start = 2016-01-12 11:45:09
  11. INFO [2016-01-12 11:55:09.500] Thread-0 Term-00, Session End = 2016-01-12 11:55:09
  12. INFO [2016-01-12 11:55:09.500] Thread-0 Term-00, Transaction Count = 1450110
  13. $ tail -n 5 benchmarksql-4.1.0_pg03/run/log/benchmarksql.log
  14. INFO [2016-01-12 11:55:09.541] Thread-14 Term-00, Measured tpmC (NewOrders) = 57322.05
  15. INFO [2016-01-12 11:55:09.541] Thread-14 Term-00, Measured tpmTOTAL = 143227.03
  16. INFO [2016-01-12 11:55:09.542] Thread-14 Term-00, Session Start = 2016-01-12 11:45:09
  17. INFO [2016-01-12 11:55:09.542] Thread-14 Term-00, Session End = 2016-01-12 11:55:09
  18. INFO [2016-01-12 11:55:09.542] Thread-14 Term-00, Transaction Count = 1432298
  19. $ tail -n 5 benchmarksql-4.1.0_pg04/run/log/benchmarksql.log
  20. INFO [2016-01-12 11:55:09.574] Thread-7 Term-00, Measured tpmC (NewOrders) = 57863.92
  21. INFO [2016-01-12 11:55:09.574] Thread-7 Term-00, Measured tpmTOTAL = 144596.45
  22. INFO [2016-01-12 11:55:09.575] Thread-7 Term-00, Session Start = 2016-01-12 11:45:09
  23. INFO [2016-01-12 11:55:09.575] Thread-7 Term-00, Session End = 2016-01-12 11:55:09
  24. INFO [2016-01-12 11:55:09.575] Thread-7 Term-00, Transaction Count = 1445978
  25. TPM
  26. 144975.59 + 145006.74 + 143227.03 + 144596.45 = 577805.81

阶段4 PostgreSQL 9.5.0 TPmC : 577805.81

阶段4 性能瓶颈分析

无明显瓶颈,从编译器,代码方面入手优化。

阶段5 TPC-C 优化

使用最新版本的gcc, 同时使用O3优化代码。

使用gcc 4.9.3版本,使用O3优化CFLAGS,重新编译

  1. $ export LD_LIBRARY_PATH=/u02/digoal/gcc4.9.3/lib:/u02/digoal/cloog/lib:/u02/digoal/gmp/lib:/u02/digoal/isl/lib:/u02/digoal/mpc/lib:/u02/digoal/mpfr/lib:$LD_LIBRARY_PATH
  2. $ export PATH=/u02/digoal/gcc4.9.3/bin:$PATH
  3. $ CFLAGS="-O3 -march=native -flto" CC=/u02/digoal/gcc4.9.3/bin/gcc ./configure --prefix=/u02/digoal/soft_bak/pgsql9.5 --with-blocksize=8 --with-pgport=1921 --with-perl --with-python --with-tcl --with-openssl --with-pam --with-ldap --with-libxml --with-libxslt --enable-thread-safety --with-wal-segsize=64
  4. $ make world -j 32
  5. $ make install-world -j 32

阶段5 TPC-C 压测

  1. digoal tail -n 5 benchmarksql-4.1.0_pg01/run/log/benchmarksql.log
  2. INFO [2016-01-13 02:00:49.699] Thread-15 Term-00, Measured tpmC (NewOrders) = 59092.33
  3. INFO [2016-01-13 02:00:49.699] Thread-15 Term-00, Measured tpmTOTAL = 147832.44
  4. INFO [2016-01-13 02:00:49.699] Thread-15 Term-00, Session Start = 2016-01-13 01:50:49
  5. INFO [2016-01-13 02:00:49.699] Thread-15 Term-00, Session End = 2016-01-13 02:00:49
  6. INFO [2016-01-13 02:00:49.699] Thread-15 Term-00, Transaction Count = 1478385
  7. digoal tail -n 5 benchmarksql-4.1.0_pg02/run/log/benchmarksql.log
  8. INFO [2016-01-13 02:00:49.704] Thread-0 Term-00, Measured tpmC (NewOrders) = 60051.49
  9. INFO [2016-01-13 02:00:49.704] Thread-0 Term-00, Measured tpmTOTAL = 150231.54
  10. INFO [2016-01-13 02:00:49.704] Thread-0 Term-00, Session Start = 2016-01-13 01:50:49
  11. INFO [2016-01-13 02:00:49.704] Thread-0 Term-00, Session End = 2016-01-13 02:00:49
  12. INFO [2016-01-13 02:00:49.704] Thread-0 Term-00, Transaction Count = 1502367
  13. digoal tail -n 5 benchmarksql-4.1.0_pg03/run/log/benchmarksql.log
  14. INFO [2016-01-13 02:00:49.693] Thread-16 Term-00, Measured tpmC (NewOrders) = 60273.99
  15. INFO [2016-01-13 02:00:49.694] Thread-16 Term-00, Measured tpmTOTAL = 150601.93
  16. INFO [2016-01-13 02:00:49.694] Thread-16 Term-00, Session Start = 2016-01-13 01:50:49
  17. INFO [2016-01-13 02:00:49.694] Thread-16 Term-00, Session End = 2016-01-13 02:00:49
  18. INFO [2016-01-13 02:00:49.694] Thread-16 Term-00, Transaction Count = 1506066
  19. digoal tail -n 5 benchmarksql-4.1.0_pg04/run/log/benchmarksql.log
  20. INFO [2016-01-13 02:00:49.715] Thread-18 Term-00, Measured tpmC (NewOrders) = 60180.69
  21. INFO [2016-01-13 02:00:49.715] Thread-18 Term-00, Measured tpmTOTAL = 150591.78
  22. INFO [2016-01-13 02:00:49.716] Thread-18 Term-00, Session Start = 2016-01-13 01:50:49
  23. INFO [2016-01-13 02:00:49.716] Thread-18 Term-00, Session End = 2016-01-13 02:00:49
  24. INFO [2016-01-13 02:00:49.716] Thread-18 Term-00, Transaction Count = 1505962
  25. TPM
  26. 599257.69

阶段5 PostgreSQL 9.5.0 TPmC : 599257.69

阶段5 性能瓶颈分析

使用CLANG编译器。

阶段6 TPC-C 优化

clang的安装参考这里。 使用clang编译

  1. CC=/u02/digoal/llvm/bin/clang CFLAGS="-O2 -fstrict-enums" ./configure --prefix=/u02/digoal/soft_bak/pgsql9.5 --with-pgport=1921 --with-perl --with-python --with-tcl --with-openssl --with-pam --with-ldap --with-libxml --with-libxslt --enable-thread-safety
  2. make world -j 32
  3. make install-world -j 32

阶段6 TPC-C 压测

  1. $ tail -n 5 benchmarksql-4.1.0_pg01/run/log/benchmarksql.log
  2. INFO [2016-01-16 07:21:58.070] Thread-12 Term-00, Measured tpmC (NewOrders) = 60519.19
  3. INFO [2016-01-16 07:21:58.070] Thread-12 Term-00, Measured tpmTOTAL = 151235.02
  4. INFO [2016-01-16 07:21:58.070] Thread-12 Term-00, Session Start = 2016-01-16 07:11:58
  5. INFO [2016-01-16 07:21:58.071] Thread-12 Term-00, Session End = 2016-01-16 07:21:58
  6. INFO [2016-01-16 07:21:58.071] Thread-12 Term-00, Transaction Count = 1512377
  7. $ tail -n 5 benchmarksql-4.1.0_pg02/run/log/benchmarksql.log
  8. INFO [2016-01-16 07:21:58.180] Thread-15 Term-00, Measured tpmC (NewOrders) = 60924.87
  9. INFO [2016-01-16 07:21:58.180] Thread-15 Term-00, Measured tpmTOTAL = 152126.73
  10. INFO [2016-01-16 07:21:58.180] Thread-15 Term-00, Session Start = 2016-01-16 07:11:58
  11. INFO [2016-01-16 07:21:58.180] Thread-15 Term-00, Session End = 2016-01-16 07:21:58
  12. INFO [2016-01-16 07:21:58.180] Thread-15 Term-00, Transaction Count = 1521312
  13. $ tail -n 5 benchmarksql-4.1.0_pg03/run/log/benchmarksql.log
  14. INFO [2016-01-16 07:21:58.198] Thread-0 Term-00, Measured tpmC (NewOrders) = 60481.19
  15. INFO [2016-01-16 07:21:58.198] Thread-0 Term-00, Measured tpmTOTAL = 151294.63
  16. INFO [2016-01-16 07:21:58.199] Thread-0 Term-00, Session Start = 2016-01-16 07:11:58
  17. INFO [2016-01-16 07:21:58.199] Thread-0 Term-00, Session End = 2016-01-16 07:21:58
  18. INFO [2016-01-16 07:21:58.199] Thread-0 Term-00, Transaction Count = 1512968
  19. $ tail -n 5 benchmarksql-4.1.0_pg04/run/log/benchmarksql.log
  20. INFO [2016-01-16 07:21:58.200] Thread-5 Term-00, Measured tpmC (NewOrders) = 60715.57
  21. INFO [2016-01-16 07:21:58.200] Thread-5 Term-00, Measured tpmTOTAL = 151809.93
  22. INFO [2016-01-16 07:21:58.200] Thread-5 Term-00, Session Start = 2016-01-16 07:11:58
  23. INFO [2016-01-16 07:21:58.200] Thread-5 Term-00, Session End = 2016-01-16 07:21:58
  24. INFO [2016-01-16 07:21:58.200] Thread-5 Term-00, Transaction Count = 1518149
  25. TPM:
  26. 606466.31

阶段6 PostgreSQL 9.5.0 TPmC : 606466.31

当前perf top

  1. samples pcnt function DSO
  2. _______ _____ _________________________________ __________________________________________
  3. 15900.00 3.2% hash_search_with_hash_value /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  4. 13970.00 2.8% _bt_compare /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  5. 13215.00 2.6% AllocSetAlloc /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  6. 10678.00 2.1% LWLockAcquire /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  7. 10298.00 2.1% memcpy /lib64/libc-2.12.so
  8. 9016.00 1.8% SearchCatCache /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  9. 8577.00 1.7% heap_hot_search_buffer/u02/digoal/soft_bak/pgsql9.5/bin/postgres
  10. 8059.00 1.6% GetSnapshotData /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  11. 6975.00 1.4% ExecInitExpr /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  12. 6517.00 1.3% fmgr_info_cxt_security/u02/digoal/soft_bak/pgsql9.5/bin/postgres
  13. 5232.00 1.0% PostgresMain /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  14. 4328.00 0.9% LWLockRelease /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  15. 4044.00 0.8% PinBuffer /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  16. 4037.00 0.8% _int_malloc /lib64/libc-2.12.so
  17. 4026.00 0.8% StrategyGetBuffer /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  18. 3777.00 0.8% slot_deform_tuple /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  19. 3755.00 0.7% FunctionCall2Coll /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  20. 3741.00 0.7% __GI_vfprintf /lib64/libc-2.12.so
  21. 3403.00 0.7% __strncpy_ssse3 /lib64/libc-2.12.so
  22. 3305.00 0.7% aliflash_reconfig_task[aliflash]
  23. 3090.00 0.6% _bt_checkkeys /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  24. 3012.00 0.6% __memset_sse2 /lib64/libc-2.12.so
  25. 2881.00 0.6% palloc /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  26. 2698.00 0.5% __strlen_sse42 /lib64/libc-2.12.so
  27. 2585.00 0.5% _int_free /lib64/libc-2.12.so
  28. 2505.00 0.5% heap_page_prune /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  29. 2495.00 0.5% hash_any /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  30. 2442.00 0.5% heap_page_prune_opt /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  31. 2437.00 0.5% __schedule[kernel.kallsyms]
  32. 2210.00 0.4% MemoryContextAllocZeroAligned /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  33. 2111.00 0.4% pfree /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  34. 2048.00 0.4% heap_update /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  35. 2012.00 0.4% update_blocked_averages [kernel.kallsyms]
  36. 1937.00 0.4% __switch_to [kernel.kallsyms]
  37. 1925.00 0.4% heap_getsysattr /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  38. 1916.00 0.4% TupleDescInitEntry /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  39. 1905.00 0.4% irq_entries_start [kernel.kallsyms]
  40. 1863.00 0.4% AllocSetFree /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  41. 1821.00 0.4% _wordcopy_bwd_aligned /lib64/libc-2.12.so
  42. 1761.00 0.4% _raw_spin_lock [kernel.kallsyms]
  43. 1758.00 0.4% check_stack_depth /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  44. 1749.00 0.3% _bt_binsrch /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  45. 1748.00 0.3% ReadBuffer_common /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  46. 1747.00 0.3% expression_tree_walker/u02/digoal/soft_bak/pgsql9.5/bin/postgres
  47. 1651.00 0.3% __GI___libc_malloc /lib64/libc-2.12.so
  48. 1608.00 0.3% __memcmp_sse4_1 /lib64/libc-2.12.so
  49. 1586.00 0.3% LockAcquireExtended /u02/digoal/soft_bak/pgsql9.5/bin/postgres
  50. ------------------------------------------------------------------------------------------------------------

阶段6 性能瓶颈分析

需要从PG代码层进行优化.

其他本文未尝试的优化手段

有兴趣的朋友可以试试:

  1. 使用interl的icc编译一下,看看性能还能不能提升;
  2. 关闭表的自动analyze, 关闭日志表的autovacuum和auto analyze;
  3. PostgreSQL jdbc有一些参数可以优化,本文还未处理。例如防止类型转换(benchmarksql的代码已处理了这块),QUERY plan CACHE size。 参考这里
  4. PostgreSQL 代码层也有优化的空间,例如使用分区表, 以及分区表的代码,快照的优化(应对高并发), LLVM。

请关注阿里云AliCloudDB for PgSQL,我们会持续从代码层面优化社区版本的PostgreSQL。

总结

内核参数优化总结,以及每项配置的原理。

  1. kernel.shmmax = 135497418752
  2. kernel.shmmni = 4096
  3. fs.file-max = 7672460
  4. fs.aio-max-nr = 1048576
  5. vm.zone_reclaim_mode=0 # 禁用 numa, 或者在vmlinux中禁止.
  6. vm.swappiness = 0 # 关闭交换分区
  7. kernel.shmmax=135497418752 # 最大共享内存段大小
  8. net.core.rmem_max = 4194304 # The maximum receive socket buffer size in bytes
  9. net.core.wmem_max = 4194304 # The maximum send socket buffer size in bytes.
  10. net.core.rmem_default = 262144 # The default setting of the socket receive buffer in bytes.
  11. net.core.wmem_default = 262144 # The default setting (in bytes) of the socket send buffer.
  12. net.ipv4.ip_local_port_range = 9000 65535 # 本地自动分配的TCP UDP端口号范围
  13. kernel.sem = 50100 64128000 50100 1280 # 信号量
  14. vm.dirty_background_bytes = 102400000 # 系统脏页到达这个值,系统后台刷脏页调度进程 pdflush(或其他) 自动将(dirty_expire_centisecs/100)秒前的脏页刷到磁盘
  15. vm.dirty_expire_centisecs = 6000 # 比这个值老的脏页,将被刷到磁盘。6000表示60秒。
  16. vm.dirty_writeback_centisecs = 50 # pdflush(或其他)后台刷脏页进程的唤醒间隔, 50表示0.5秒。
  17. vm.dirty_ratio = 80 # 如果系统进程刷脏页太慢,使得系统脏页超过内存 80 % 时,则用户进程如果有写磁盘的操作(如fsync, fdatasync等调用),则需要主动把系统脏页刷出。
  18. vm.nr_hugepages = 102352 # 大页数量,乘以/proc/meminfo Hugepagesize就是内存数量。
  19. vm.overcommit_memory = 0 # 在分配内存时,允许少量over malloc
  20. vm.overcommit_ratio = 90 # 当overcommit_memory = 2 时,用于参与计算允许指派的内存大小。

内存分配策略解释,参考这里

  1. vm.overcommit_memory=0时,允许用户轻微的overcommit
  2. vm.overcommit_memory=1时,任何情况下都允许申请内存overcommit, 比较危险,常用于一些科学计算应用。
  3. vm.overcommit_memory=2时,Committed_AS不能大于CommitLimit
  4. 申请内存的限制 计算方法
  5. The CommitLimit is calculated with the following formula:
  6. CommitLimit = ([total RAM pages] - [total huge TLB pages]) *
  7. overcommit_ratio / 100 + [total swap pages]
  8. For example, on a system with 1G of physical RAM and 7G
  9. of swap with a `vm.overcommit_ratio` of 30 it would
  10. yield a CommitLimit of 7.3G.
  11. [root@digoal postgresql-9.4.4]# free
  12. total used free shared buffers cached
  13. Mem: 1914436 713976 1200460 72588 32384 529364
  14. -/+ buffers/cache: 152228 1762208
  15. Swap: 1048572 542080 506492
  16. [root@digoal ~]# cat /proc/meminfo |grep Commit
  17. CommitLimit: 2005788 kB
  18. Committed_AS: 132384 kB
  19. 这个例子的2G就是以上公式计算得来。
  20. overcommit限制的初衷是malloc后,内存并不是立即使用掉,所以如果多个进程同时申请一批内存的话,不允许OVERCOMMIT可能导致某些进程申请内存失败,但实际上内存是还有的。
  21. 所以Linux内核给出了几种选择,
  22. 2是比较靠谱或者温柔的做法。
  23. 1的话风险有点大,虽然可以申请内存,但是实际上可能已经没有足够的内存给程序使用,最终可能会导致OOM
  24. 0是最常见的,允许少量的overcommit,但是对于需要超很多内存的情况,不允许。
  25. 还可以参考代码 :
  26. security/commoncap.c::cap_vm_enough_memory()
  27. 所以当数据库无法启动时,要么你降低一下数据库申请内存的大小(例如降低shared_buffer或者max conn),要么就是修改一下overcommit的风格。

vi /etc/security/limits.conf

  1. # add by digoal.zhou
  2. * soft nofile 131072
  3. * hard nofile 131072
  4. * soft nproc 131072
  5. * hard nproc 131072
  6. * soft core unlimited
  7. * hard core unlimited
  8. * soft memlock 500000000
  9. * hard memlock 500000000

内核启动参数优化总结 关闭numa 使用deadline调度IO

  1. kernel /vmlinuz-3.18.24 numa=off elevator=deadline intel_idle.max_cstate=0 scsi_mod.scan=sync

块设备优化总结,预读

  1. blockdev --setra 16384 /dev/dfa
  2. blockdev --setra 16384 /dev/dfb
  3. blockdev --setra 16384 /dev/dfc
  4. blockdev --setra 16384 /dev/dm-0

数据库参数优化总结

  1. max_connections = 300 # (change requires restart)
  2. unix_socket_directories = '.' # comma-separated list of directories
  3. shared_buffers = 194GB # 尽量用数据库管理内存,减少双重缓存,提高使用效率
  4. huge_pages = on # on, off, or try ,使用大页
  5. work_mem = 256MB # min 64kB , 减少外部文件排序的可能,提高效率
  6. maintenance_work_mem = 2GB # min 1MB , 加速建立索引
  7. autovacuum_work_mem = 2GB # min 1MB, or -1 to use maintenance_work_mem , 加速垃圾回收
  8. dynamic_shared_memory_type = mmap # the default is the first option
  9. vacuum_cost_delay = 0 # 0-100 milliseconds , 垃圾回收不妥协,极限压力下,减少膨胀可能性
  10. bgwriter_delay = 10ms # 10-10000ms between rounds , 刷shared buffer脏页的进程调度间隔,尽量高频调度,减少用户进程申请不到内存而需要主动刷脏页的可能(导致RT升高)。
  11. bgwriter_lru_maxpages = 1000 # 0-1000 max buffers written/round , 一次最多刷多少脏页
  12. bgwriter_lru_multiplier = 10.0 # 0-10.0 multipler on buffers scanned/round 一次扫描多少个块,上次刷出脏页数量的倍数
  13. effective_io_concurrency = 2 # 1-1000; 0 disables prefetching , 执行节点为bitmap heap scan时,预读的块数。从而
  14. wal_level = minimal # minimal, archive, hot_standby, or logical , 如果现实环境,建议开启归档。
  15. synchronous_commit = off # synchronization level; , 异步提交
  16. wal_sync_method = open_sync # the default is the first option , 因为没有standby,所以写xlog选择一个支持O_DIRECT的fsync方法。
  17. full_page_writes = off # recover from partial page writes , 生产中,如果有增量备份和归档,可以关闭,提高性能。
  18. wal_buffers = 1GB # min 32kB, -1 sets based on shared_buffers ,wal buffer大小,如果大量写wal buffer等待,则可以加大。
  19. wal_writer_delay = 10ms # 1-10000 milliseconds wal buffer调度间隔,和bg writer delay类似。
  20. commit_delay = 20 # range 0-100000, in microseconds ,分组提交的等待时间
  21. commit_siblings = 9 # range 1-1000 , 有多少个事务同时进入提交阶段时,就触发分组提交。
  22. checkpoint_timeout = 55min # range 30s-1h 时间控制的检查点间隔。
  23. max_wal_size = 320GB # 2个检查点之间最多允许产生多少个XLOG文件
  24. checkpoint_completion_target = 0.99 # checkpoint target duration, 0.0 - 1.0 ,平滑调度间隔,假设上一个检查点到现在这个检查点之间产生了100个XLOG,则这次检查点需要在产生100*checkpoint_completion_target个XLOG文件的过程中完成。PG会根据这些值来调度平滑检查点。
  25. random_page_cost = 1.0 # same scale as above , 离散扫描的成本因子,本例使用的SSD IO能力足够好
  26. effective_cache_size = 240GB # 可用的OS CACHE
  27. log_destination = 'csvlog' # Valid values are combinations of
  28. logging_collector = on # Enable capturing of stderr and csvlog
  29. log_truncate_on_rotation = on # If on, an existing log file with the
  30. update_process_title = off
  31. track_activities = off
  32. autovacuum = on # Enable autovacuum subprocess? 'on'
  33. autovacuum_max_workers = 4 # max number of autovacuum subprocesses ,允许同时有多少个垃圾回收工作进程。
  34. autovacuum_naptime = 6s # time between autovacuum runs , 自动垃圾回收探测进程的唤醒间隔
  35. autovacuum_vacuum_cost_delay = 0 # default vacuum cost delay for , 垃圾回收不妥协

其他优化总结:

  1. 尽量减少费的IO请求,所以本文从块设备,到逻辑卷,到文件系统的块大小都尽量和数据库块大小靠齐;
  2. 通过对齐,减少IO覆盖写;
  3. 通过大页减少内存管理开销;
  4. 通过多个客户端将数据库硬件资源充分利用起来;
  5. 减少客户端输出日志的开销,降低客户端性能干扰;
  6. 使用新的编译器,优化编译后的可执行程序质量。