Buffer pool & memory
The buffer pool is where MySQL actually lives. Every page read and every page written passes through it, and the single biggest determinant of whether your database feels fast is whether the pages you need are already in it. This part covers how it decides what to keep — including the clever modification that stops one careless full table scan from evicting everything.
Buffer pool structure
A large block of memory holding copies of pages from disk. Every page InnoDB touches must be here first — there is no path that reads a row directly from a file.
Three lists, not one
| List | Contains | Used for |
|---|---|---|
| LRU list | All pages currently cached | Deciding what to evict when space is needed |
| Flush list | Dirty pages, ordered by oldest modification LSN | Deciding what to write to disk next — checkpointing walks this list |
| Free list | Unused frames | Instant allocation without eviction |
A dirty page appears on both the LRU list and the flush list simultaneously. The distinction matters: eviction order is by recency of use, while flush order is by age of modification. Those are different questions, so they need different lists.
Instances and chunks
A single pool means a single set of mutexes, and on a many-core server that
becomes contention. innodb_buffer_pool_instances splits the pool
into independent sub-pools, each with its own lists and locks. Pages are
assigned to an instance by hashing space id and page number.
Rule of thumb: instances only help if the pool is at least 1GB, and each instance should be at least 1GB. Eight instances on a 512MB pool is worse than one. MySQL 8 sets it sensibly by default; leave it alone unless you have measured mutex contention.
innodb_buffer_pool_size can be changed at runtime. It
happens in innodb_buffer_pool_chunk_size units (default 128MB),
and the pool size is rounded to a multiple of
chunk_size × instances. That is why setting 9GB on a server with
8 instances can silently give you something slightly different — check what you
actually got rather than assuming.SELECT @@innodb_buffer_pool_size / 1024/1024/1024 AS gb,
@@innodb_buffer_pool_instances AS instances,
@@innodb_buffer_pool_chunk_size / 1024/1024 AS chunk_mb;
-- The single most important health metric on the server.
-- Aim for > 99% on an OLTP workload.
SELECT ROUND(
(1 - (
SUM(IF(Variable_name='Innodb_buffer_pool_reads', Variable_value, 0)) /
SUM(IF(Variable_name='Innodb_buffer_pool_read_requests', Variable_value, 0))
)) * 100, 3) AS hit_rate_pct
FROM performance_schema.global_status
WHERE Variable_name IN
('Innodb_buffer_pool_reads', 'Innodb_buffer_pool_read_requests');
-- read_requests = logical reads (asked the pool)
-- reads = physical reads (had to hit disk)
-- What is actually in there, by table?
SELECT table_name, index_name, COUNT(*) AS pages,
ROUND(COUNT(*)*16/1024,1) AS mb
FROM information_schema.innodb_buffer_page
WHERE table_name IS NOT NULL
GROUP BY table_name, index_name
ORDER BY pages DESC LIMIT 15;The midpoint LRU
A plain LRU has a fatal flaw for a database. Run one
SELECT * FROM big_table and every page of that table is inserted
at the head of the list, evicting your entire working set. One reporting query
destroys cache locality for everyone, and the pages it loaded will never be
read again.
InnoDB's fix: insert new pages in the middle, not at the head.
The list is split at the midpoint:
- Young sublist (the head, default 63%) — the hot working set.
- Old sublist (the tail, default 37%) — newly read pages on probation.
A newly read page enters at the head of the old sublist. It is promoted
to the young sublist only if it is accessed again after
innodb_old_blocks_time milliseconds have passed (default
1000).
The time window distinguishes many accesses in one burst — a scan — from accesses spread over time — genuine reuse. Only the latter earns promotion. That one-second delay is what makes a scan cheap for everyone else.
SELECT @@innodb_old_blocks_pct, @@innodb_old_blocks_time;
-- Split of the pool right now.
SELECT pool_id, database_pages, old_database_pages,
pages_made_young, pages_not_made_young
FROM information_schema.innodb_buffer_pool_stats;
-- Take a baseline, run a big scan, then re-check.
SELECT COUNT(*) FROM some_huge_table;
-- pages_not_made_young should jump a lot;
-- pages_made_young should barely move. That is the
-- protection working: the scan filled the old sublist
-- and was evicted from it without ever reaching young.
-- For a workload that scans a lot and wants MORE protection:
-- SET GLOBAL innodb_old_blocks_time = 2000;
-- SET GLOBAL innodb_old_blocks_pct = 20; -- smaller probation areaFlush list, page cleaners, adaptive flushing
Dirty pages must reach disk eventually. Page cleaner threads
(innodb_page_cleaners) do that work in the background, so that
user threads never have to wait for a write.
They flush from two ends for two different reasons:
- From the flush list — oldest modifications first, to advance the checkpoint and free redo log space.
- From the LRU tail — so that evictable clean pages are always available and a read never has to wait for a write first.
Adaptive flushing
Flushing at a constant rate is wrong in both directions: too slow and the redo log fills, too fast and you waste IO writing pages that would have been modified again. Adaptive flushing adjusts the rate continuously based on two inputs — how fast redo is being generated, and how full the redo log is (checkpoint age).
innodb_io_capacity tells InnoDB roughly how many IOPS it may use
in steady state; innodb_io_capacity_max is the emergency ceiling
when it is falling behind. These are the most commonly mis-set values on modern
hardware — defaults of 200/2000 were chosen for spinning disks, and an NVMe
drive doing 50,000 IOPS is being throttled to a fraction of its capability.
innodb_io_capacity is a budget for background work. Set it to the
drive's full capability and background flushing will compete with user queries
for the same IO. A reasonable starting point is 50–75% of measured random write
IOPS, with _max at roughly double. Then watch checkpoint age: if
it stays low and stable, you have enough.Read-ahead
Two mechanisms for fetching pages before they are asked for.
| Type | Trigger | Action | Status |
|---|---|---|---|
| Linear | innodb_read_ahead_threshold pages (default 56) of a 64-page extent have been read in order | Fetch the whole next extent | On. Genuinely useful for scans. |
| Random | 13 pages of one extent are in the pool, regardless of order | Fetch the rest of that extent | Off by default. Rarely helps; often wastes IO. |
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_read_ahead%'; -- ..._read_ahead pages prefetched -- ..._read_ahead_evicted prefetched and thrown away UNUSED -- If evicted is a large fraction of prefetched, read-ahead is -- doing harm: it is reading pages nobody wanted, evicting pages -- someone did. Consider raising the threshold or disabling: -- SET GLOBAL innodb_read_ahead_threshold = 64; -- max, least eager -- SET GLOBAL innodb_random_read_ahead = OFF; -- already default
Checkpoint age and the stall
From chapter 61: checkpoint age is the distance between the last checkpoint LSN and the current LSN — the amount of redo that has not yet been made safe by flushing its pages.
As that age approaches the redo log's capacity, InnoDB escalates:
-- as a fraction of innodb_redo_log_capacity < 50% → relaxed. background flushing at io_capacity. ~ 75% → adaptive flushing ramps up aggressively. ~ 87.5% → async flush: heavy, user threads start to feel it. ~ 95% → SYNC FLUSH. all writes BLOCK until space is reclaimed.
That last state is the classic "MySQL froze for 30 seconds" incident. No slow query, no lock, no deadlock — every write simply stops, then everything resumes at once.
innodb_io_capacity. Since 8.0.30 you can raise
innodb_redo_log_capacity without a restart, which makes this one
of the rare production problems you can fix live.SELECT
MAX(IF(name='log_lsn_checkpoint_age', count, NULL)) AS checkpoint_age,
@@innodb_redo_log_capacity AS capacity,
ROUND(MAX(IF(name='log_lsn_checkpoint_age', count, NULL))
/ @@innodb_redo_log_capacity * 100, 1) AS pct_used
FROM information_schema.innodb_metrics
WHERE name = 'log_lsn_checkpoint_age';
-- Enable the metric first if it returns NULL:
-- SET GLOBAL innodb_monitor_enable = 'log_lsn_checkpoint_age';
-- pct_used consistently above 75 → enlarge the redo log.
-- Any Innodb_log_waits at all → you are already stalling.
SHOW GLOBAL STATUS LIKE 'Innodb_log_waits';Dump and load on restart
A restarted server with an empty buffer pool is dramatically slower until the working set is re-cached — a period that can last hours on a large database. Every query hits disk; the server looks broken.
So InnoDB saves the page identifiers (not the pages) at shutdown into
ib_buffer_pool, and reloads those pages at startup. The file is
tiny — 8 bytes per page — and warming happens in the background while the
server accepts connections.
SELECT @@innodb_buffer_pool_dump_at_shutdown,
@@innodb_buffer_pool_load_at_startup,
@@innodb_buffer_pool_dump_pct; -- % of each LRU, default 25
-- Dump now, without restarting (do this before planned maintenance):
SET GLOBAL innodb_buffer_pool_dump_now = ON;
-- Watch warming progress after a restart:
SHOW STATUS LIKE 'Innodb_buffer_pool_dump_status';
SHOW STATUS LIKE 'Innodb_buffer_pool_load_status';
-- For a big pool, raise dump_pct to 50-75 so more of the
-- working set is restored. The cost is a slower shutdown.Per-connection memory, and the OOM arithmetic
The buffer pool is global. Almost everything else is per connection, and sometimes per join or per sort within a connection. This is where servers die.
| Buffer | Allocated | Default | Note |
|---|---|---|---|
sort_buffer_size | Per sort | 256KB | A query with two sorts allocates twice. |
join_buffer_size | Per join | 256KB | A five-table join without indexes can allocate four. |
read_buffer_size | Per scan | 128KB | Sequential scans. |
read_rnd_buffer_size | Per connection | 256KB | Used by MRR and some sorts. |
tmp_table_size | Per temp table | 16MB | Can be several per query. |
binlog_cache_size | Per connection | 32KB | Grows for large transactions; spills to disk. |
The arithmetic people get wrong
-- a single bad query can use several of these at once per connection, worst case ≈ 84 MB × 1000 connections ≈ 84 GB + innodb_buffer_pool_size 48 GB
total possible ≈ 132 GB on a 64 GB machine -- works fine at low concurrency. OOM-killed during the traffic spike.
SET SESSION sort_buffer_size = 64*1024*1024; before a big report,
then let the session end. A large global sort buffer is not a performance
setting; it is a latent outage multiplied by max_connections.-- What MySQL is ACTUALLY using, by allocation site.
SELECT event_name,
ROUND(current_alloc/1024/1024) AS mb
FROM sys.memory_global_by_current_bytes
WHERE current_alloc > 1024*1024*10
ORDER BY current_alloc DESC LIMIT 15;
-- Per-connection memory, worst offenders first.
SELECT user, current_count_used, current_allocated,
current_max_alloc
FROM sys.memory_by_user_by_current_bytes
ORDER BY current_allocated DESC LIMIT 10;
-- Enable memory instrumentation if these are empty:
-- UPDATE performance_schema.setup_instruments
-- SET enabled='YES' WHERE name LIKE 'memory/%';
-- Is anything spilling to disk for lack of buffer?
SHOW GLOBAL STATUS WHERE Variable_name IN
('Sort_merge_passes', -- sorts that needed disk
'Created_tmp_disk_tables', -- temp tables on disk
'Created_tmp_tables');Temporary tables
The optimizer creates internal temporary tables for GROUP BY,
DISTINCT, UNION, derived tables and some subqueries.
Where they live has changed significantly in 8.0.
| Engine | Location | Note |
|---|---|---|
| TempTable | Memory, up to temptable_max_ram (1GB) | Default since 8.0. Handles VARCHAR/BLOB efficiently, unlike MEMORY. |
| TempTable overflow | Memory-mapped files, or InnoDB from 8.0.16 | Controlled by temptable_use_mmap / internal_tmp_mem_storage_engine. |
| MEMORY (legacy) | Memory | Pre-8.0 default. Converts any VARCHAR to a fixed-width CHAR — so one VARCHAR(255) column made tables enormous and forced them to disk. |
| InnoDB on disk | ibtmp1 | The slow path. Every Created_tmp_disk_tables increment is a query that got much slower. |
VARCHAR was padded to full width — so a GROUP BY over
a VARCHAR(255) column allocated 255 bytes per row regardless of
actual content, blew past tmp_table_size, and spilled to disk.
TempTable stores variable-length data properly, which quietly fixed a large
class of "why is this GROUP BY so slow" problems on upgrade to 8.0.SELECT @@internal_tmp_mem_storage_engine, @@temptable_max_ram,
@@tmp_table_size, @@max_heap_table_size;
-- The ratio that matters. High disk_tables = queries hitting disk.
SELECT
MAX(IF(Variable_name='Created_tmp_tables', Variable_value, 0)) AS tmp_total,
MAX(IF(Variable_name='Created_tmp_disk_tables', Variable_value, 0)) AS tmp_on_disk
FROM performance_schema.global_status
WHERE Variable_name LIKE 'Created_tmp%';
-- WHICH queries are doing it? This is the actionable one.
SELECT LEFT(digest_text, 70) AS query,
count_star, sum_created_tmp_disk_tables AS disk_tmp,
ROUND(avg_timer_wait/1e9, 1) AS avg_ms
FROM performance_schema.events_statements_summary_by_digest
WHERE sum_created_tmp_disk_tables > 0
ORDER BY sum_created_tmp_disk_tables DESC LIMIT 10;
-- Is ibtmp1 growing without bound? It only shrinks on restart.
SELECT file_name, ROUND(file_size/1024/1024) AS mb
FROM information_schema.files WHERE file_name LIKE '%ibtmp%';You now know how MySQL stores, protects and caches data. Part 8 is about the component that decides how to get at it — the optimizer, and the twelve chapters it deserves.