MariaDB optimization needed?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
lpallard

lpallard

Member
Aug 17, 2013
276
11
18
Hey guys. I'm posting here because the Software section of the forum seems to be more "entertainment" oriented.... @Mods please move my thread if needed.

So I have this Centos database VM that needs to move to Rocky Linux due to Centos expiring in Dec 2021. I am using Proxmox 7 as hypervisor, and the database VM uses a Proxmox logical volume (named "lv1") passed directly to the VM via its "VMID.conf" file. This logical volume acts as the database server's datastore for MariaDB.

Instead of doing an in-place migration from Centos to Rocky Linux, I opted to start from a clean slate and create a new VM from scratch. This is because the existing Centos VM has many config flaws and is a bit messy. I also figured upgrading entire OS's is often prone to issues.

This is my plan of action:
  • Shutdown all VM's using the DB server
  • Shutdown MariaDB
  • Make complete backup of the datastore (lv1)
  • Backup my.cnf (and other config files for mariadb)
  • Shutdown existing database server VM and make a backup in case something doesnt work
  • Destroy existing database VM
  • Create new VM based on Rocky Linux and manually attach "lv1" via its VMID.conf
  • Boot new database VM and make sure all NFS shares and LV's are mounted
  • Install Mariadb and restore my.cnf
  • Start MariaDB and confirm all OK using "lv1" datastore and that MariaDB runs as expected
Of course on top of backing up the entire datastore, I also have nightly and weekly DB dumps on a Freenas server.

Do you guys see anything bad with this?

Next question is regarding current database server. Using phpMyAdmin I see the following alerts:

VariableValueDescription
Aborted clients70The number of connections that were aborted because the client died without closing the connection properly.
Aborted connects1The number of failed attempts to connect to the MySQL server.
Created tmp disk tables85.8 kThe number of temporary tables on disk created automatically by the server while executing statements. If Created_tmp_disk_tables is big, you may want to increase the tmp_table_size value to cause temporary tables to be memory-based instead of disk-based.
Handler read rnd9.4 MThe number of requests to read a row based on a fixed position. This is high if you are doing a lot of queries that require sorting of the result. You probably have a lot of queries that require MySQL to scan whole tables or you have joins that don't use keys properly.
Handler read rnd next982 MThe number of requests to read the next row in the data file. This is high if you are doing a lot of table scans. Generally this suggests that your tables are not properly indexed or that your queries are not written to take advantage of the indexes you have.
Innodb buffer pool reads28.7 kThe number of logical reads that InnoDB could not satisfy from buffer pool and had to do a single-page read.
Innodb row lock time max707The maximum time to acquire a row lock, in milliseconds.
Innodb row lock waits547The number of times a row lock had to be waited for.
Opened tables1.2 kThe number of tables that have been opened. If opened tables is big, your table cache value is probably too small.
Qcache lowmem prunes411.3 kThe number of queries that have been removed from the cache to free up memory for caching new queries. This information can help you tune the query cache size. The query cache uses a least recently used (LRU) strategy to decide which queries to remove from the cache.
Select full join44.1 kThe number of joins that do not use indexes. If this value is not 0, you should carefully check the indexes of your tables.
Slow queries2The number of queries that have taken more than long_query_time seconds.

Aborted clients: I guess I cant do anything about this. Could this be due to poorly coded ap[plications using MariaDB that are NOT closing their connections properly?

Aborted connects : Same as above?

Created tmp disk tables: I believe this one is a problem. The server is sometimes slow. According to some sources the value of "tmp_table_size" should be around 1% of the physical RAM. My server has 16GB which would make this variable around 160MB. Current value in my.cnf is 128M. Should I increase it? According to "free -m", there is plenty of physical RAM to use:
Code: 
              total        used        free      shared  buff/cache   available
Mem:          15884        7900        3346          24        4637        7629
Swap:          3071           0        3071
Handler read rnd: This number is insanely high, and I dont know what to do about it...

Handler read rnd next: Not sure what to do about this one too.... The description states "your tables are not properly indexed". How can I troubleshoot to see if this the case? The description also states "your queries are not written to take advantage of the indexes you have". I believe I have to control over that if its because the applications using MariaDB are poorly coded....

Innodb buffer pool reads: This looks worrysome. What to do about it?

Innodb row lock time max: Is this related to hard drive (storage) speed? If so, perhaps making MariaDB work more in RAM would help?

Innodb row lock waits: Same as above?

Opened tables: The variable "table_open_cache" is set to 2048 in my.cnf. Should I increase it?

Qcache lowmem prunes: The variable "query_cache_size" is set to 160M in my.cnf. Should I increase it?

Select full join: "The number of joins that do not use indexes. If this value is not 0, you should carefully check the indexes of your tables" Ouch....

Slow queries: Probably storage related?

My.cnf
Code: 
[client]
port=3306
socket=/mnt/sql-databases/mysql.sock

[mysqld]
collation-server=utf8_unicode_ci
init-connect='SET NAMES utf8'
character-set-server=utf8
port=3306
socket=/mnt/sql-databases/mysql.sock
pid-file=/var/run/mariadb/mariadb.pid
log-error=/var/log/mariadb/mariadb.log
datadir=/mnt/sql-databases
user=mysql
symbolic-links=0
back_log = 50

# Modified this line
max_connections=100

max_connect_errors=10
table_open_cache=2048
max_allowed_packet=128M
binlog_cache_size=1M
max_heap_table_size=128M
read_buffer_size=2M

#Modified these 2 lines
read_rnd_buffer_size=24M
sort_buffer_size=16M

join_buffer_size=8M
thread_cache_size=8

#Modified these 2 lines
query_cache_size=160M
query_cache_limit=4M

ft_min_word_len=4
default-storage-engine=INNODB
thread_stack=192K
transaction_isolation=REPEATABLE-READ
tmp_table_size=128M
group_concat_max_len=8192
#log-bin=mysql-bin
max_binlog_size=1073741824
#expire_logs_days=1
binlog_format=mixed
log_warnings
slow_query_log
long_query_time=5
key_buffer_size=128M
bulk_insert_buffer_size=64M
myisam_sort_buffer_size=128M
myisam_max_sort_file_size=10G
myisam_repair_threads=1
myisam_recover

# Added    this line
innodb_file_per_table
innodb_flush_method=O_DIRECT

innodb_additional_mem_pool_size=16M

# Changed this from 7G to 9G
innodb_buffer_pool_size=9G

innodb_data_file_path=ibdata1:10M:autoextend
innodb_thread_concurrency=0
innodb_flush_log_at_trx_commit=0
innodb_log_buffer_size=32M

# Changed this from 512M to 1G
innodb_log_file_size=2G

innodb_log_files_in_group=3
innodb_max_dirty_pages_pct=90
innodb_lock_wait_timeout=120

[mysqldump]
quick
max_allowed_packet=16M

[mysql]
no-auto-rehash

[myisamchk]
key_buffer_size=512M
sort_buffer_size=512M
read_buffer=8M
write_buffer=8M

[mysqlhotcopy]
interactive-timeout

[mysqld_safe]
open-files-limit=8192
mysqltuner.pl
Code: 
 >>  MySQLTuner 1.8.5 - Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.pl/
 >>  Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script
[OK] Logged in using credentials passed on the command line
[!!] Your MySQL version 5.5.68-MariaDB is EOL software!  Upgrade soon!
[OK] Operating on 64-bit architecture
 
-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/log/mariadb/mariadb.log exists
[--] Log file: /var/log/mariadb/mariadb.log(1020B)
[OK] Log file /var/log/mariadb/mariadb.log is not empty
[OK] Log file /var/log/mariadb/mariadb.log is smaller than 32 Mb
[OK] Log file /var/log/mariadb/mariadb.log is readable.
[!!] /var/log/mariadb/mariadb.log contains 10 warning(s).
[OK] /var/log/mariadb/mariadb.log doesn't contain any error.
[--] 0 start(s) detected in /var/log/mariadb/mariadb.log
[--] 0 shutdown(s) detected in /var/log/mariadb/mariadb.log
 
-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +Aria +BLACKHOLE +CSV +FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA 
[--] Data in MyISAM tables: 13.8M (Tables: 44)
[--] Data in InnoDB tables: 790.2M (Tables: 851)
[OK] Total fragmented tables: 0
 
-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: ON
[!!] Stat are updated during querying INFORMATION_SCHEMA.
 
-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] User 'xxxxxxx'@% does not specify hostname restrictions.
[!!] User 'xxxxxxx'@% does not specify hostname restrictions.
[!!] There is no basic password file list!
 
-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined
 
-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 13d 1h 27m 20s (42M q [37.437 qps], 33K conn, TX: 20G, RX: 5G)
[--] Reads / Writes: 71% / 29%
[--] Binary logging is disabled
[--] Physical Memory     : 15.5G
[--] Max MySQL memory    : 27.0G
[--] Other process memory: 0B
[--] Total buffers: 9.6G global + 178.2M per thread (100 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[!!] Maximum reached memory usage: 15.0G (96.52% of installed RAM)
[!!] Maximum possible memory usage: 27.0G (173.92% of installed RAM)
[!!] Overall possible memory usage with other process exceeded memory
[OK] Slow queries: 0% (2/42M)
[OK] Highest usage of available connections: 31% (31/100)
[OK] Aborted connections: 0.03%  (9/33339)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[!!] Query cache efficiency: 16.0% (4M cached / 27M selects)
[!!] Query cache prunes per day: 31490
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 1M sorts)
[!!] Joins performed without indexes: 44138
[OK] Temporary tables created on disk: 6% (85K on disk / 1M total)
[OK] Thread cache hit rate: 99% (102 created / 33K connections)
[OK] Table cache hit rate: 85% (1K hits / 1K requests)
[!!] table_definition_cache(400) is lower than number of tables(1033) 
[OK] Open file limit used: 1% (140/8K)
[OK] Table locks acquired immediately: 100% (29M immediate / 29M locks)
 
-------- Performance schema ------------------------------------------------------------------------
[--] Performance schema is disabled.
[--] Memory used by P_S: 0B
[--] Sys schema isn't installed.
 
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 4 thread(s).
[--] Using default value is good enough for your version (5.5.68-MariaDB)
 
-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.3% (24M used / 134M cache)
[OK] Key buffer size / total MyISAM indexes: 128.0M/2.9M
[OK] Read Key buffer hit rate: 99.5% (23K cached / 117 reads)
[!!] Write Key buffer hit rate: 50.5% (216 cached / 109 writes)
 
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 9.0G/790.2M
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (66.6666666666667 %): 2.0G * 3/9.0G should be equal to 25%
[!!] InnoDB buffer pool instances: 1
[--] InnoDB Buffer Pool Chunk Size not used or defined in your version
[OK] InnoDB Read buffer efficiency: 100.00% (4605401370 hits/ 4605430103 total)
[OK] InnoDB Write log efficiency: 95.41% (5119556 hits/ 5365936 total)
[OK] InnoDB log waits: 0.00% (0 waits / 246380 writes)
 
-------- Aria Metrics ------------------------------------------------------------------------------
[--] Aria Storage Engine is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/0B
[OK] Aria pagecache hit rate: 99.9% (9M cached / 12K reads)
 
-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.
 
-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.
 
-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.
 
-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: MIXED
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server
 
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Check warning line(s) in /var/log/mariadb/mariadb.log file
    Restrict Host for 'xxxxxxx'@'%' to 'xxxxxxx'@LimitedIPRangeOrLocalhost
    RENAME USER 'xxxxxxx'@'%' TO 'xxxxxxx'@LimitedIPRangeOrLocalhost;
    Restrict Host for 'xxxxxxx'@'%' to 'xxxxxxx'@LimitedIPRangeOrLocalhost
    RENAME USER 'xxxxxxx'@'%' TO 'xxxxxxx'@LimitedIPRangeOrLocalhost;
    Reduce your overall MySQL memory footprint for system stability
    Dedicate this server to your database for highest performance.
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    Increasing the query_cache size over 128M may reduce performance
    We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.
             See https://dev.mysql.com/doc/internals/en/join-buffer-size.html
             (specially the conclusions at the bottom of the page).
    For MySQL 5.6.2 and lower, Max combined innodb_log_file_size should have a ceiling of (4096MB / log files in group) - 1MB.
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: https://bit.ly/2TcGgtU
Variables to adjust:
  *** MySQL's maximum memory usage is dangerously high ***
  *** Add RAM before increasing MySQL buffer variables ***
    SET innodb_stats_on_metadata = OFF
    query_cache_size (=0)
    query_cache_type (=0)
    query_cache_limit (> 4M, or use smaller result sets)
    query_cache_size (> 160M) [see warning above]
    join_buffer_size (> 8.0M, or always use indexes with JOINs)
    table_definition_cache(400) > 1033 or -1 (autosizing if supported)
    innodb_log_file_size should be (=768M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
    innodb_buffer_pool_instances(=9)
mariaDB version
Code: 
mysql  Ver 15.1 Distrib 5.5.68-MariaDB, for Linux (x86_64) using readline 5.1
Final note: sorry for the basic questions... I'm extremely risk adverse and "unknowledgeable" with MySQL/MariaDB in general so instead of trying all kind of stuff and end up with data corruption or loss I prefer to have a conversation with knowledgeable people...

Thanks!

EDIT: Ran the latest mysqltuner.pl and replaced results
 
Last edited:
A

amalurk

Active Member
Dec 16, 2016
311
116
43
102
You probably don't need a MySQL database admin expert, you need an expert in the SQL language and database design. It appears from what you posted that your database structure and the way it is being queried is very suboptimal. So a SQL expert could fix the indexes making new ones where appropriate, changing table structures and layout and/or if the apps hitting the DB can be modified, refactor the queires in those apps to hopefully not do things like a bunch of joins that aren't on indexes or large table scans.
 
lpallard

lpallard

Member
Aug 17, 2013
276
11
18
Okay, this DB server is home for the following programs and web base services (in order of usage frequency and what I can see in phpMyAdmin's processes):
  • Zoneminder
  • TinyTinyRSS
  • OpenKM
  • Horde groupware
  • kMyMoney
  • OwnCloud server
  • Coppermine gallery
  • XWiki
  • Drupal
I do not use my database server to store data myself, but for a DB storage back-end for the above services and programs. That's why I mentioned that I have little to no play in how these programs are storing data in, and using MariaDB..... All I can do is optimize the DB server and try to make it as efficient as possible.

I just noticed that the mysqltuner.pl script mentions that RAM usage is extremely high. Before anything else, I will increase the available RAM to the VM from 16GB to 32GB and see if things are getting better....
 
T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,640
2,057
113
- Even though they're not your apps you can still check to make sure they're all using InnoDB some of those apps are OLD (coppermine) and may have created myIsam tables. InnoDB offered fulltext search awhile ago and I can't think of any other reason to use myisam back then...

- It noted concurrency, but if not set it still should so I'd check the docs for the version you're planning to run, specify it if needed

- Your slow queries are only 2, so that's not bad... but your long query time is 5 seconds.... drop that to 2 seconds and log slow queries. Then check for indexes, you can add indexes still even if it's not your app, maybe they forgot or you added plugins that the devs sucked on querying the underlying data itself.

- If the above still doesn't help you could try:
-- Percona Server for MySQL to see if that helps with performance
-- Run your DB on an Intel Optane

For comparison on my shared hosting server with a lot of similar inefficient software and more I allocate 28GB to InnoDB BP and it usually has around 18GB in there but all my other stats vs. yours are MUCH worse.... nothing I can do about the various software, but it's on Optane and flys :D
 
Z

zrav

New Member
Sep 26, 2019
3
2
3
The analytics provided by MariaDBs performance schema has been extremely valuable to me in the past. Enable via
Code: 
[mariadb]
performance_schema = ON
Restart the DB and let your apps run for a while. Then check the performance_schema. As a starting point for finding issues the table table_io_waits_summary_by_index_usage is excellent. Sort by SUM_TIMER_WAIT and you'll see which table accesses consume the most time and whether they are accessed by index or not (INDEX_NAME = NULL).
Manually adding missing indexes to application DBs is hit and miss, some notice the schema is not what was expected and complain.
 
You must log in or register to reply here.
Top Bottom