Part 3 · 1 chapters · ~8 min

Lucene Segments and Merges

Lucene's architecture (immutable segments, term dictionaries, postings, doc values, stored fields), indexing buffers and refreshes for near-real-time search, deletes and updates via bitmaps, merge policies, the translog and commits, and the parallels with LSM trees.

4

Immutable segments

code
GET merchants/_segments                            # segments per shard, sizes, deleted docs
PUT merchants/_settings { "index.refresh_interval": "30s" }   # during bulk loads: fewer, larger segments
POST merchants/_forcemerge?max_num_segments=1     # only for read-only indexes (e.g. last month's logs)

Besides the inverted index, each segment stores doc values (columnar per-field data used for sorting, aggregations and scripting) and stored fields (the original document). Sorting and aggregating read doc values, not the inverted index.

LUCENE SEGMENTS
immutable mini-indexes, refreshed for visibility and merged in the background
indexing buffernew docs in memorysegment 1immutablesegment 2immutablerefresh (1 s)new segment searchablemergecombine, drop deleted docstranslogdurability before commit
swipe the figure sideways, or tap expand for full screen
1/5
immutable segments
Lucene never modifies an index file in place. New documents are buffered in memory, then written as a new immutable segment: a complete mini inverted index.
each segment is a small immutable indexlike LSM SSTables (NoSQL P1)