Part 3 · 8 chapters · ~20 min

Aggregation & grouping

Aggregation collapses many rows into one. The mechanics are simple; the subtleties are all about what happens to NULLs, what the engine will let you select alongside a group, and the difference between filtering rows and filtering groups.

22

Aggregates and NULL

One rule covers almost everything: aggregate functions ignore NULLs. They are removed before the function runs. The consequences are larger than that sentence suggests.

ExpressionWith NULLs presentNote
COUNT(*)Counts rowsThe only one that does not skip NULLs — it counts rows, not values.
COUNT(col)Counts non-NULL valuesOften silently different from COUNT(*).
SUM(col)Sums non-NULLsNULL if every value is NULL — not 0.
AVG(col)SUM/COUNT(col)Divides by non-NULL count. Not the row count.
MIN/MAXIgnores NULLsNULL only if all values are NULL.
No rows at allCOUNT→0, everything else →NULLThe empty-set asymmetry.
the two that cause real bugs
SUM of an empty set is NULL, not zero. So total + SUM(x) becomes NULL when there is nothing to sum, and the whole expression vanishes. Wrap it: COALESCE(SUM(x), 0).

AVG divides by the non-NULL count. If half your rows have NULL, the average is over the other half — which may be what you want, or may be badly wrong. If NULL means zero in your domain, write AVG(COALESCE(x, 0)) and make the intent explicit.
run it
CREATE TABLE r (score INT);
INSERT INTO r VALUES (10),(20),(NULL),(NULL);

SELECT COUNT(*)                 AS rows_,       -- 4
       COUNT(score)             AS scored,      -- 2
       SUM(score)               AS total,       -- 30
       AVG(score)               AS avg_,        -- 15.0
       AVG(COALESCE(score, 0))  AS avg_zero     -- 7.5
FROM r;

-- the empty-set asymmetry
SELECT COUNT(*), SUM(score), AVG(score)
  FROM r WHERE score > 1000;
--   0, NULL, NULL   ← COUNT gives 0, the others give NULL

-- so this silently produces NULL, not 100:
SELECT 100 + SUM(score) FROM r WHERE score > 1000;   -- NULL
SELECT 100 + COALESCE(SUM(score), 0) FROM r WHERE score > 1000;  -- 100
23

GROUP BY and functional dependency

After GROUP BY, each group is one output row. So every expression in SELECT must produce exactly one value per group. It can do that two ways: by being a grouping column, or by being an aggregate.

Anything else is ambiguous — which of the group's several values should it return? — and the standard rejects it.

code
-- ambiguous: which name? the group may contain many.
SELECT dept, name, COUNT(*) FROM emp GROUP BY dept;   -- ERROR
-- unambiguous: pick one explicitly
SELECT dept, MAX(name), COUNT(*) FROM emp GROUP BY dept;
SELECT dept, ANY_VALUE(name), COUNT(*) FROM emp GROUP BY dept;

Functional dependency, and why some "invalid" queries are fine

If you group by a primary key, every other column of that table is functionally determined — there is only one possible value per group. Modern engines recognize this:

code
-- legal: grouping by the PK determines name and email
SELECT u.id, u.name, u.email, COUNT(o.id)
  FROM users u LEFT JOIN orders o ON o.user_id = u.id
 GROUP BY u.id;
--   Postgres and MySQL 8 both accept this.
--   Grouping by a non-key column would not be accepted.
ONLY_FULL_GROUP_BY, and the legacy it replaced
Before 5.7, MySQL permitted the ambiguous form and returned an arbitrary value from the group — no error, no warning, and a different row potentially on every execution. Enormous amounts of code was written against that behaviour, and some of it was wrong in ways nobody noticed.

ONLY_FULL_GROUP_BY is in the default sql_mode since 5.7. If you inherit a codebase that breaks when you enable it, the queries were always ambiguous — the mode is reporting a pre-existing bug, not creating one. Fix them with ANY_VALUE() where the ambiguity is genuinely harmless, and with a proper aggregate where it is not.
24

HAVING versus WHERE

Chapter 6 already answered this: WHERE is step 2 and filters rows; HAVING is step 4 and filters groups. Two practical corollaries follow.

  • WHERE cannot use an aggregate, because groups do not exist yet.
  • HAVING can use non-aggregates, but should not: a non-aggregate condition in HAVING is evaluated after grouping, so it filters far more rows than necessary.
code
-- WRONG PLACE: groups every department, then throws most away
SELECT dept, COUNT(*) FROM emp
 GROUP BY dept
HAVING dept <> 'temp';          -- filters AFTER aggregating
-- RIGHT: excludes the rows before any grouping work happens
SELECT dept, COUNT(*) FROM emp
 WHERE dept <> 'temp'
GROUP BY dept;

-- both together: the normal shape
SELECT dept, COUNT(*) FROM emp
 WHERE hired_at >= '2026-01-01' -- which rows to consider
GROUP BY dept
HAVING COUNT(*) > 5;               -- which groups to keep
the optimizer often saves you
Most engines will push a non-aggregate HAVING condition down into WHERE automatically. Writing it correctly still matters: it states intent, it is guaranteed rather than dependent on an optimization, and it makes the query's shape obvious to the next reader.
25

GROUPING SETS, ROLLUP, CUBE

One pass, several levels of aggregation. The alternative is several queries combined with UNION ALL, each re-scanning the table.

code
-- GROUPING SETS: name exactly the groupings you want
SELECT region, product, SUM(amount)
  FROM sales
 GROUP BY GROUPING SETS (
   (region, product),   -- detail
   (region),            -- subtotal per region
   ()                   -- grand total
 );

-- ROLLUP: the hierarchical shortcut for exactly that pattern
SELECT region, product, SUM(amount)
  FROM sales GROUP BY ROLLUP (region, product);
--   (region, product), (region), ()
-- MySQL's older spelling of ROLLUP:
--   GROUP BY region, product WITH ROLLUP
-- CUBE: every combination (Postgres; MySQL has no CUBE)
SELECT region, product, SUM(amount)
  FROM sales GROUP BY CUBE (region, product);
--   (region,product), (region), (product), ()

Telling a subtotal from a real NULL

Subtotal rows carry NULL in the columns being rolled up — which is indistinguishable from a genuine NULL in the data. GROUPING() resolves it, returning 1 when the NULL is a subtotal marker:

code
SELECT
CASE WHEN GROUPING(region) = 1 THEN 'ALL REGIONS'
ELSE COALESCE(region, '(unknown)') END AS region,
  SUM(amount)
FROM sales GROUP BY ROLLUP (region);
26

FILTER, and its workaround

SQL:2003's FILTER clause restricts one aggregate to a subset of rows, so several differently-filtered aggregates can share a single pass.

code
-- Postgres, SQLite, standard
SELECT
COUNT(*)                                    AS total,
  COUNT(*) FILTER (WHERE status = 'paid')     AS paid,
  COUNT(*) FILTER (WHERE status = 'refunded') AS refunded,
  SUM(amount) FILTER (WHERE status = 'paid')   AS revenue
FROM orders;

MySQL has no FILTER. The equivalent exploits the fact that aggregates skip NULLs (chapter 22):

code
-- MySQL: CASE returning NULL for non-matching rows
SELECT
COUNT(*)                                              AS total,
  COUNT(CASE WHEN status = 'paid' THEN 1 END)        AS paid,
  SUM(CASE WHEN status = 'paid' THEN amount END)     AS revenue,
  SUM(status = 'paid')                                AS paid_short
FROM orders;
--   note: no ELSE, so non-matching rows yield NULL and are skipped.
--   SUM(boolean) works in MySQL because TRUE is 1.
why this beats several queries
The obvious alternative is one query per status, each scanning the table. Conditional aggregation computes all of them in one pass. On a large table that is the difference between four scans and one — the single most useful aggregation technique in this part.
27

Conditional aggregation and pivoting

The same technique turns rows into columns. Standard SQL has no PIVOT; this is how everyone does it.

code
-- from: one row per (month, region)
-- to:   one row per month, one COLUMN per region
SELECT
  month,
  SUM(CASE WHEN region = 'EU' THEN amount ELSE 0 END) AS eu,
  SUM(CASE WHEN region = 'US' THEN amount ELSE 0 END) AS us,
  SUM(CASE WHEN region = 'AF' THEN amount ELSE 0 END) AS af,
  SUM(amount) AS total
FROM sales
GROUP BY month
ORDER BY month;
the structural limitation
The column list must be known when the query is written. A pivot over dynamic values requires generating the SQL, which is usually a sign the reshaping belongs in the application or the BI layer rather than in SQL. Pivot in SQL when the categories are fixed and few; do it elsewhere when they are not.
28

COUNT(*) vs COUNT(col) vs COUNT(DISTINCT col)

FormCountsCost
COUNT(*)RowsCheapest. Can use any index, or the smallest one.
COUNT(1)RowsIdentical to COUNT(*). The folklore that one is faster is false in every modern engine.
COUNT(col)Non-NULL valuesMust examine the column.
COUNT(DISTINCT col)Distinct non-NULL valuesExpensive — needs a sort or hash of all values.
COUNT(*) on a large InnoDB table is not free
MyISAM stored a row count in the table header, so COUNT(*) was instant. InnoDB cannot: with MVCC, the count depends on which transaction is asking (MySQL module, ch 42), so it must actually scan an index.

On a hundred-million-row table that is seconds. If you need an approximate count for a UI, use information_schema.tables.table_rows — it is an estimate from the same sampled statistics the optimizer uses, and it is instant. If you need an exact count frequently, maintain a counter table.
run it
-- exact, and it scans
SELECT COUNT(*) FROM big_table;

-- approximate, and instant
SELECT table_rows FROM information_schema.tables
 WHERE table_schema = DATABASE() AND table_name = 'big_table';

-- Postgres equivalent
--   SELECT reltuples::bigint FROM pg_class WHERE relname = 'big_table';

-- the three counts differing on the same column
SELECT COUNT(*), COUNT(region), COUNT(DISTINCT region) FROM sales;
29

Ordered-set aggregates and percentiles

Some aggregates need their input in order — medians and percentiles. SQL:2003 gives them a distinct syntax, WITHIN GROUP.

code
-- Postgres / Oracle / standard
SELECT
PERCENTILE_CONT(0.5)  WITHIN GROUP (ORDER BY ms) AS p50,
  PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY ms) AS p95,
  PERCENTILE_DISC(0.95) WITHIN GROUP (ORDER BY ms) AS p95_actual
FROM requests;
--   CONT interpolates between values; DISC returns an actual
--   observed value. For latency SLOs, DISC is usually what you want.

MySQL has neither. The portable approach uses a window function (Part 4) to rank rows and pick the one at the right position:

code
-- MySQL: p95 latency
SELECT ms AS p95 FROM (
  SELECT ms,
         PERCENT_RANK() OVER (ORDER BY ms) AS pr
    FROM requests
) t
WHERE pr >= 0.95
ORDER BY pr
LIMIT 1;
why the average is the wrong metric
AVG(response_ms) is dominated by the bulk of fast requests and hides the tail entirely. A service with a 40ms mean can have a 4-second p99, and the p99 is what users experience as "the site is broken". Report percentiles; use the mean only alongside them.

Part 4 is window functions — the same aggregate machinery, but computed per-row instead of collapsing the rows. It is the single largest capability gap between engineers who learned SQL before 2010 and after.