Hey guys. I'm posting here because the Software section of the forum seems to be more "entertainment" oriented.... @Mods please move my thread if needed.
So I have this Centos database VM that needs to move to Rocky Linux due to Centos expiring in Dec 2021. I am using Proxmox 7 as hypervisor, and the database VM uses a Proxmox logical volume (named "lv1") passed directly to the VM via its "VMID.conf" file. This logical volume acts as the database server's datastore for MariaDB.
Instead of doing an in-place migration from Centos to Rocky Linux, I opted to start from a clean slate and create a new VM from scratch. This is because the existing Centos VM has many config flaws and is a bit messy. I also figured upgrading entire OS's is often prone to issues.
This is my plan of action:
Do you guys see anything bad with this?
Next question is regarding current database server. Using phpMyAdmin I see the following alerts:
Aborted clients: I guess I cant do anything about this. Could this be due to poorly coded ap[plications using MariaDB that are NOT closing their connections properly?
Aborted connects : Same as above?
Created tmp disk tables: I believe this one is a problem. The server is sometimes slow. According to some sources the value of "tmp_table_size" should be around 1% of the physical RAM. My server has 16GB which would make this variable around 160MB. Current value in my.cnf is 128M. Should I increase it? According to "free -m", there is plenty of physical RAM to use:
Handler read rnd: This number is insanely high, and I dont know what to do about it...
Handler read rnd next: Not sure what to do about this one too.... The description states "your tables are not properly indexed". How can I troubleshoot to see if this the case? The description also states "your queries are not written to take advantage of the indexes you have". I believe I have to control over that if its because the applications using MariaDB are poorly coded....
Innodb buffer pool reads: This looks worrysome. What to do about it?
Innodb row lock time max: Is this related to hard drive (storage) speed? If so, perhaps making MariaDB work more in RAM would help?
Innodb row lock waits: Same as above?
Opened tables: The variable "table_open_cache" is set to 2048 in my.cnf. Should I increase it?
Qcache lowmem prunes: The variable "query_cache_size" is set to 160M in my.cnf. Should I increase it?
Select full join: "The number of joins that do not use indexes. If this value is not 0, you should carefully check the indexes of your tables" Ouch....
Slow queries: Probably storage related?
My.cnf
mysqltuner.pl
mariaDB version
Final note: sorry for the basic questions... I'm extremely risk adverse and "unknowledgeable" with MySQL/MariaDB in general so instead of trying all kind of stuff and end up with data corruption or loss I prefer to have a conversation with knowledgeable people...
Thanks!
EDIT: Ran the latest mysqltuner.pl and replaced results
So I have this Centos database VM that needs to move to Rocky Linux due to Centos expiring in Dec 2021. I am using Proxmox 7 as hypervisor, and the database VM uses a Proxmox logical volume (named "lv1") passed directly to the VM via its "VMID.conf" file. This logical volume acts as the database server's datastore for MariaDB.
Instead of doing an in-place migration from Centos to Rocky Linux, I opted to start from a clean slate and create a new VM from scratch. This is because the existing Centos VM has many config flaws and is a bit messy. I also figured upgrading entire OS's is often prone to issues.
This is my plan of action:
- Shutdown all VM's using the DB server
- Shutdown MariaDB
- Make complete backup of the datastore (lv1)
- Backup my.cnf (and other config files for mariadb)
- Shutdown existing database server VM and make a backup in case something doesnt work
- Destroy existing database VM
- Create new VM based on Rocky Linux and manually attach "lv1" via its VMID.conf
- Boot new database VM and make sure all NFS shares and LV's are mounted
- Install Mariadb and restore my.cnf
- Start MariaDB and confirm all OK using "lv1" datastore and that MariaDB runs as expected
Do you guys see anything bad with this?
Next question is regarding current database server. Using phpMyAdmin I see the following alerts:
Variable | Value | Description |
---|---|---|
Aborted clients | 70 | The number of connections that were aborted because the client died without closing the connection properly. |
Aborted connects | 1 | The number of failed attempts to connect to the MySQL server. |
Created tmp disk tables | 85.8 k | The number of temporary tables on disk created automatically by the server while executing statements. If Created_tmp_disk_tables is big, you may want to increase the tmp_table_size value to cause temporary tables to be memory-based instead of disk-based. |
Handler read rnd | 9.4 M | The number of requests to read a row based on a fixed position. This is high if you are doing a lot of queries that require sorting of the result. You probably have a lot of queries that require MySQL to scan whole tables or you have joins that don't use keys properly. |
Handler read rnd next | 982 M | The number of requests to read the next row in the data file. This is high if you are doing a lot of table scans. Generally this suggests that your tables are not properly indexed or that your queries are not written to take advantage of the indexes you have. |
Innodb buffer pool reads | 28.7 k | The number of logical reads that InnoDB could not satisfy from buffer pool and had to do a single-page read. |
Innodb row lock time max | 707 | The maximum time to acquire a row lock, in milliseconds. |
Innodb row lock waits | 547 | The number of times a row lock had to be waited for. |
Opened tables | 1.2 k | The number of tables that have been opened. If opened tables is big, your table cache value is probably too small. |
Qcache lowmem prunes | 411.3 k | The number of queries that have been removed from the cache to free up memory for caching new queries. This information can help you tune the query cache size. The query cache uses a least recently used (LRU) strategy to decide which queries to remove from the cache. |
Select full join | 44.1 k | The number of joins that do not use indexes. If this value is not 0, you should carefully check the indexes of your tables. |
Slow queries | 2 | The number of queries that have taken more than long_query_time seconds. |
Aborted clients: I guess I cant do anything about this. Could this be due to poorly coded ap[plications using MariaDB that are NOT closing their connections properly?
Aborted connects : Same as above?
Created tmp disk tables: I believe this one is a problem. The server is sometimes slow. According to some sources the value of "tmp_table_size" should be around 1% of the physical RAM. My server has 16GB which would make this variable around 160MB. Current value in my.cnf is 128M. Should I increase it? According to "free -m", there is plenty of physical RAM to use:
Code:
total used free shared buff/cache available
Mem: 15884 7900 3346 24 4637 7629
Swap: 3071 0 3071
Handler read rnd next: Not sure what to do about this one too.... The description states "your tables are not properly indexed". How can I troubleshoot to see if this the case? The description also states "your queries are not written to take advantage of the indexes you have". I believe I have to control over that if its because the applications using MariaDB are poorly coded....
Innodb buffer pool reads: This looks worrysome. What to do about it?
Innodb row lock time max: Is this related to hard drive (storage) speed? If so, perhaps making MariaDB work more in RAM would help?
Innodb row lock waits: Same as above?
Opened tables: The variable "table_open_cache" is set to 2048 in my.cnf. Should I increase it?
Qcache lowmem prunes: The variable "query_cache_size" is set to 160M in my.cnf. Should I increase it?
Select full join: "The number of joins that do not use indexes. If this value is not 0, you should carefully check the indexes of your tables" Ouch....
Slow queries: Probably storage related?
My.cnf
Code:
[client]
port=3306
socket=/mnt/sql-databases/mysql.sock
[mysqld]
collation-server=utf8_unicode_ci
init-connect='SET NAMES utf8'
character-set-server=utf8
port=3306
socket=/mnt/sql-databases/mysql.sock
pid-file=/var/run/mariadb/mariadb.pid
log-error=/var/log/mariadb/mariadb.log
datadir=/mnt/sql-databases
user=mysql
symbolic-links=0
back_log = 50
# Modified this line
max_connections=100
max_connect_errors=10
table_open_cache=2048
max_allowed_packet=128M
binlog_cache_size=1M
max_heap_table_size=128M
read_buffer_size=2M
#Modified these 2 lines
read_rnd_buffer_size=24M
sort_buffer_size=16M
join_buffer_size=8M
thread_cache_size=8
#Modified these 2 lines
query_cache_size=160M
query_cache_limit=4M
ft_min_word_len=4
default-storage-engine=INNODB
thread_stack=192K
transaction_isolation=REPEATABLE-READ
tmp_table_size=128M
group_concat_max_len=8192
#log-bin=mysql-bin
max_binlog_size=1073741824
#expire_logs_days=1
binlog_format=mixed
log_warnings
slow_query_log
long_query_time=5
key_buffer_size=128M
bulk_insert_buffer_size=64M
myisam_sort_buffer_size=128M
myisam_max_sort_file_size=10G
myisam_repair_threads=1
myisam_recover
# Added this line
innodb_file_per_table
innodb_flush_method=O_DIRECT
innodb_additional_mem_pool_size=16M
# Changed this from 7G to 9G
innodb_buffer_pool_size=9G
innodb_data_file_path=ibdata1:10M:autoextend
innodb_thread_concurrency=0
innodb_flush_log_at_trx_commit=0
innodb_log_buffer_size=32M
# Changed this from 512M to 1G
innodb_log_file_size=2G
innodb_log_files_in_group=3
innodb_max_dirty_pages_pct=90
innodb_lock_wait_timeout=120
[mysqldump]
quick
max_allowed_packet=16M
[mysql]
no-auto-rehash
[myisamchk]
key_buffer_size=512M
sort_buffer_size=512M
read_buffer=8M
write_buffer=8M
[mysqlhotcopy]
interactive-timeout
[mysqld_safe]
open-files-limit=8192
Code:
>> MySQLTuner 1.8.5 - Major Hayden <major@mhtx.net>
>> Bug reports, feature requests, and downloads at http://mysqltuner.pl/
>> Run with '--help' for additional options and output filtering
[--] Skipped version check for MySQLTuner script
[OK] Logged in using credentials passed on the command line
[!!] Your MySQL version 5.5.68-MariaDB is EOL software! Upgrade soon!
[OK] Operating on 64-bit architecture
-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/log/mariadb/mariadb.log exists
[--] Log file: /var/log/mariadb/mariadb.log(1020B)
[OK] Log file /var/log/mariadb/mariadb.log is not empty
[OK] Log file /var/log/mariadb/mariadb.log is smaller than 32 Mb
[OK] Log file /var/log/mariadb/mariadb.log is readable.
[!!] /var/log/mariadb/mariadb.log contains 10 warning(s).
[OK] /var/log/mariadb/mariadb.log doesn't contain any error.
[--] 0 start(s) detected in /var/log/mariadb/mariadb.log
[--] 0 shutdown(s) detected in /var/log/mariadb/mariadb.log
-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +Aria +BLACKHOLE +CSV +FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA
[--] Data in MyISAM tables: 13.8M (Tables: 44)
[--] Data in InnoDB tables: 790.2M (Tables: 851)
[OK] Total fragmented tables: 0
-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: ON
[!!] Stat are updated during querying INFORMATION_SCHEMA.
-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] User 'xxxxxxx'@% does not specify hostname restrictions.
[!!] User 'xxxxxxx'@% does not specify hostname restrictions.
[!!] There is no basic password file list!
-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined
-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 13d 1h 27m 20s (42M q [37.437 qps], 33K conn, TX: 20G, RX: 5G)
[--] Reads / Writes: 71% / 29%
[--] Binary logging is disabled
[--] Physical Memory : 15.5G
[--] Max MySQL memory : 27.0G
[--] Other process memory: 0B
[--] Total buffers: 9.6G global + 178.2M per thread (100 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[!!] Maximum reached memory usage: 15.0G (96.52% of installed RAM)
[!!] Maximum possible memory usage: 27.0G (173.92% of installed RAM)
[!!] Overall possible memory usage with other process exceeded memory
[OK] Slow queries: 0% (2/42M)
[OK] Highest usage of available connections: 31% (31/100)
[OK] Aborted connections: 0.03% (9/33339)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[!!] Query cache efficiency: 16.0% (4M cached / 27M selects)
[!!] Query cache prunes per day: 31490
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 1M sorts)
[!!] Joins performed without indexes: 44138
[OK] Temporary tables created on disk: 6% (85K on disk / 1M total)
[OK] Thread cache hit rate: 99% (102 created / 33K connections)
[OK] Table cache hit rate: 85% (1K hits / 1K requests)
[!!] table_definition_cache(400) is lower than number of tables(1033)
[OK] Open file limit used: 1% (140/8K)
[OK] Table locks acquired immediately: 100% (29M immediate / 29M locks)
-------- Performance schema ------------------------------------------------------------------------
[--] Performance schema is disabled.
[--] Memory used by P_S: 0B
[--] Sys schema isn't installed.
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 4 thread(s).
[--] Using default value is good enough for your version (5.5.68-MariaDB)
-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.3% (24M used / 134M cache)
[OK] Key buffer size / total MyISAM indexes: 128.0M/2.9M
[OK] Read Key buffer hit rate: 99.5% (23K cached / 117 reads)
[!!] Write Key buffer hit rate: 50.5% (216 cached / 109 writes)
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 9.0G/790.2M
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (66.6666666666667 %): 2.0G * 3/9.0G should be equal to 25%
[!!] InnoDB buffer pool instances: 1
[--] InnoDB Buffer Pool Chunk Size not used or defined in your version
[OK] InnoDB Read buffer efficiency: 100.00% (4605401370 hits/ 4605430103 total)
[OK] InnoDB Write log efficiency: 95.41% (5119556 hits/ 5365936 total)
[OK] InnoDB log waits: 0.00% (0 waits / 246380 writes)
-------- Aria Metrics ------------------------------------------------------------------------------
[--] Aria Storage Engine is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/0B
[OK] Aria pagecache hit rate: 99.9% (9M cached / 12K reads)
-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.
-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.
-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.
-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: MIXED
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
Check warning line(s) in /var/log/mariadb/mariadb.log file
Restrict Host for 'xxxxxxx'@'%' to 'xxxxxxx'@LimitedIPRangeOrLocalhost
RENAME USER 'xxxxxxx'@'%' TO 'xxxxxxx'@LimitedIPRangeOrLocalhost;
Restrict Host for 'xxxxxxx'@'%' to 'xxxxxxx'@LimitedIPRangeOrLocalhost
RENAME USER 'xxxxxxx'@'%' TO 'xxxxxxx'@LimitedIPRangeOrLocalhost;
Reduce your overall MySQL memory footprint for system stability
Dedicate this server to your database for highest performance.
Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
Increasing the query_cache size over 128M may reduce performance
We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.
See https://dev.mysql.com/doc/internals/en/join-buffer-size.html
(specially the conclusions at the bottom of the page).
For MySQL 5.6.2 and lower, Max combined innodb_log_file_size should have a ceiling of (4096MB / log files in group) - 1MB.
Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: https://bit.ly/2TcGgtU
Variables to adjust:
*** MySQL's maximum memory usage is dangerously high ***
*** Add RAM before increasing MySQL buffer variables ***
SET innodb_stats_on_metadata = OFF
query_cache_size (=0)
query_cache_type (=0)
query_cache_limit (> 4M, or use smaller result sets)
query_cache_size (> 160M) [see warning above]
join_buffer_size (> 8.0M, or always use indexes with JOINs)
table_definition_cache(400) > 1033 or -1 (autosizing if supported)
innodb_log_file_size should be (=768M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
innodb_buffer_pool_instances(=9)
Code:
mysql Ver 15.1 Distrib 5.5.68-MariaDB, for Linux (x86_64) using readline 5.1
Thanks!
EDIT: Ran the latest mysqltuner.pl and replaced results
Last edited: