← All digests

DBMS Weekly — 2026-10-19 (week of Oct 19–25)

Incremental backup had a bad week. Four separate bugs make pg_combinebackup restore stale or already-deleted rows while pg_verifybackup passes every input, and a fifth sends files replaced by rename truncated or zero-padded in any base backup. Five more 19 regressions surfaced against RC1, among them wrong results from sorted GROUPING SETS and hash joins that spill NULL-keyed rows to disk; last week's CLUSTER/REPACK row-loss bug was fixed just after the window closed. A large machine-generated ROW IDENTITY patch drew a blunt policy answer from three committers, and John Naylor posted a proof of concept for true in-place major upgrades, which Tom Lane doubts. On the blogs, Jeremy Schneider shows recovery_prefetch turning a standby's catch-up into a 40-minute IOPS wall that hangs every write on a synchronous primary, Robert Haas puts numbers on why 19 is late, and a PolarDB-for-PostgreSQL fork's port review found three silent visibility bugs in CSN snapshots. pgrust was pulled from ClickBench for per-query kernels, then readmitted with its source published. DuckDB 2.0 shipped, and the CommitFest grew by a net +10.

PostgreSQL

  • Postgres recovery_prefetch considered harmful and causing outages — an outage seen in production, now reproducible with a published repro. (Jeremy Schneider · ardentperf.com) [unverified: the 16× figure is his estimate; the charts show IOPS, not bytes]
    • setup: a synchronous-replication primary on a 500-IOPS disk, 2 vCPU / 8 GB; pgbench scale 640 (~10 GB) held at 5 TPS. Update 5% of pgbench_accounts, then kill the walreceiver
    • the standby falls back to replaying WAL from pg_wal and doesn't return to streaming until it has caught up. Meanwhile recovery_prefetch turns the primary's large sequential I/O into 8 kB random reads — up to 16× the IOPS for the same bytes. The standby sat at its 500-IOPS limit for about 40 minutes (the primary had used about 300 IOPS for 5), and every write on the primary hung for more than 30 minutes
    • with recovery_prefetch=off (full-page writes still on) the standby stayed caught up and applied the update in under 10 minutes
    • he traces it to a 2021 objection that was waved off and a 2025 field report from Postgres Pro; Postgres Weekly links a new patch to skip redundant prefetches after full-page images. He leans to turning the setting off everywhere and to quorum commit (ANY 1) with two standbys. A separate prefetcher correctness bug is in the mailing lists below
  • Postgres 19 is late, and it's AI's fault — a committer puts numbers on the delay, comparing April 8 – September 25 in 2025 and 2026: commits to master 1,037 → 1,443; to the stable branch being prepared 724 → 1,169 (+61%); to the previous stable branch 248 (REL_17) → 433 (REL_18), +74%. His reading: the back-patch flood comes from LLM-found bugs reported from outside the project, which also explains the late reverts (FOR PORTION OF, SQL/PGQ, MERGE/SPLIT PARTITION) and the November date. He asks how the project catches up on "forty years of bugs". (Robert Haas · neon.com) [by committer]
  • Awide Polar: an open-source PolarDB-for-PostgreSQL fork on PG17 — the release plus three deep-dives. (Awide Labs · awide.tech) [vendor blog — substantive] [unverified: ">150k TPS" pgbench and the fio gains give no hardware or harness]
    • CSN MVCC: a snapshot becomes one commit sequence number, kept in a new CSNLOG SLRU and read from atomic counters without ProcArrayLock, ported from PolarDB 11 onto PG17's GlobalVis/FullTransactionId code. The port review found silent wrong-results bugs: no barrier between the reads of xmin and snapshotcsn, so a snapshot could "see the future"; a truncated running-XID list (suboverflowed) read as "not running"; and on a standby the CSN snapshot never set takenDuringRecovery, so scans trusted PD_ALL_VISIBLE bits arriving in full-page images outside conflict resolution
    • read-your-writes at the proxy: a patched libpq lets the backend append its LSN to ReadyForQuery, so the proxy learns write and replay positions with no extra round trip — compare CF #7415 (extensible ReadyForQuery) from the 10-12 issue
    • PolarFS write leases: upstream only stopped a second read-write mount on the same host; the fix adds per-host 512-byte disk-Paxos records with ballots, a 30 s lease and a watchdog
  • Building a Table Access Method, Part 6: TRUNCATE — a series building pg_memtable, a TAM that keeps each table in its own DSM segment with the handle stored in pg_class.relfilenode. relation_set_new_filelocator runs before RelationSetNewRelfilenumber writes the new pg_class row, so the callback can read the old handle but anything it writes there is overwritten. The old segment must survive until commit and the new one be freed on abort, subtransactions included; tables created in the same (sub)transaction go through relation_nontransactional_truncate, which needs a relfilenode fallback lookup after a rolled-back savepoint drops the cached entry. A precise map of what the TAM API leaves an extension to do for transactional storage. (Mats Kindahl · dbmsdrops.kindahl.net)
  • pghacker-mcp: PostgreSQL indexing its own history — a public MCP server over the pgsql-hackers archive (645k messages, 58,734 threads) and all 105,000 commits on 39 branches. Each commit gets a 64-bit address: master's commits are spaced 2³² apart and each branch numbers its commits in the gap after its fork point, so with no merge commits "is this line in commit X" becomes lo <= X AND X < hi, plus an int8multirange for lines deleted on one branch but kept on another — 24M line versions, 6.5 GB with indexes. pg_trgm GIN finds every line that ever contained fsync_fname in 50 ms vs 54 s for git log -S; 7,254 replayed agent grep commands matched GNU grep exactly. Apache AGE holds ~1M vertices / 4M edges; LLM-judged thread outcomes must quote messages verbatim or are rejected. (Maxim Lukiyanov · Microsoft Tech Community) [vendor blog — substantive]
  • Huge pages and Postgres, measured — r7i.4xlarge, PG16, shared_buffers=32GB, 15.6 GB prewarmed, each connection scanning it once. Kernel PageTables grow 31.1 MB per connection with 4 kB pages (exactly 15.6 GB / 4 kB × 8 B), reaching 6.12 GB at 200 connections vs 111 MB with 2 MB pages; select-only pgbench at 100 clients 373,083 → 418,087 TPS (+12%). Use huge_pages=on (try falls back silently), reserve the pool before memory fragments, and size it from postgres -C shared_memory_size — the segment was ~750 MB larger than shared_buffers. (Kaushik Iska · clickhouse.com) [vendor blog — substantive]
  • Postgres FDW: query pushdown is a negotiation — one analytics query traced through each pushdown pg_clickhouse shipped. Grouped pushdown is all-or-nothing for the upper relation, so one unmapped expression keeps the whole aggregate local; mappings added step by step: percentile_cont → quantile, ->> → sub-columns, FILTER → countIf, and window functions, which postgres_fdw doesn't push. Joins go out as ALL INNER JOIN to keep Postgres semantics. Some pushdowns get revoked for wrong results — array_dims here, and in postgres_fdw 8cfbac1492b (WITH TIES) and 5c571a34d0e (LIMIT with backward scans). (Kaushik Iska, David Wheeler, Philip Dubé · clickhouse.com) [vendor blog — substantive] [unverified: speed-up figures self-reported]
  • hstore vs jsonb, measured and when (select auth.uid()) helps — PG 17.11, repeated on 18.6, scripts included. (Chris van Eijk · now-next.nl)
    • 1M flat 10-key documents: hstore read a key in 58.7 vs 67.9 ms, changed a key in 100k rows in 407 vs 458 ms (jsonb_set; || took 556 ms) and loaded in half the time; jsonb with all-string values is byte-for-byte hstore's size
    • for a 1%-selective containment query on 17, hstore+GIN (67.2 ms) lost to a parallel seqscan (48.1 ms); on 18.6 every GIN variant won (jsonb_path_ops 39.5 ms), and every GIN index built faster but came out 24–28% larger
    • auth.uid() gets inlined, so without an index on user_id the RLS filter ran 1,000,001 times: 923 ms vs 42 ms when wrapped in (select …) as an InitPlan; with an index both took 0.1–0.3 ms
  • My PostgreSQL is faster than yours — extensions only: ClickHouse's MergeTree as a table AM, driven by his vectorized executor pg_vexec. Profiling showed 51–86% of GROUP BY cycles deforming wide rows, because the TAM API hands out one slot at a time and scan_begin() gets no column list. ClickBench on 10M rows: 715 → 49.9 ms geomean, vs 16.8 ms for ClickHouse on the same laptop. Also iheap (in-place updates, undo in the table's own main fork, no MultiXacts) and pg_knowledge_base, Oxigraph's SPARQL engine over btree quad tables (990/990 W3C tests). (Igor Suhorukov · igor-suhorukov.github.io) [unverified: single laptop; code mostly LLM-written]
  • OAuth in Postgres 19 (and beyond) — from the feature's author. In 19, validator modules can register pg_hba.conf options, libpq's all-or-nothing OAuth debug mode is split, PGOAUTHCAFILE becomes its own parameter and the libpq-oauth plugin ABI firms up. For 20: a client-side token cache that must defend against session hijacking inside libpq. On sender-constrained tokens, DPoP doesn't fit OAUTHBEARER SASL and would need a new mechanism, mTLS binding needs client certificates, and neither forces an unaware server to reject bound tokens. (Jacob Champion · enterprisedb.com) [by committer]
  • I don't trust ON DELETE CASCADE — two logical-replication traps: the apply worker runs with session_replication_role=replica, so cascades don't fire on the subscriber and subscriber-only child rows are silently orphaned; and if a published cascade child has no replica identity, every parent DELETE fails even when no child rows match. (Tom Pang · planetscale.com) [vendor blog]
  • Data mirroring: how Snowflake Postgres does CDC — a GA post that explains the design: an extension pushes per-table Iceberg change logs to object storage, each batch covering all tables in one Postgres transaction and applied in one Snowflake transaction, so foreign keys hold on the target; a "meta log" drives schema changes and automatic re-snapshots after slot invalidation; $live views union the target with pending changes. (Marco Slot · snowflake.com) [vendor blog] [unverified: throughput figures]
  • pg_duckdb quietly ignored force_execution on my numeric column — with duckdb.force_execution=true, an aggregate over an unconstrained numeric (typmod −1) falls back to the Postgres executor with only a planning-time WARNING, because DuckDB's DECIMAL needs a fixed precision. Declare numeric(p,s), or accept duckdb.convert_unsupported_numeric_to_double's loss of exactness. (Franck Pachot · dev.to)
  • pgCluu v5.0 — major release of the cluster-audit tool (the announcement body says "6.0", title and tag say 5.0). (Gilles Darold · HexaCluster · pgsql-announce)
    • new reports: replication slots, pg_hba_file_rules, pg_stat_io (16+) per backend type incl. evictions, ring buffers, WAL and temp I/O, and AIO (18+)
    • postmaster limits (oom_score_adj, rlimits, cgroup v1/v2 limits and OOM counters), kernel PSI, memory events and vmstat counters sar doesn't report; THP shmem_enabled and per-device readahead collected
  • pgrecon 0.7.3 — first public release of an Apache-2.0 Oracle→PostgreSQL assessor/converter that works fully offline from a read-only SQL*Plus dump. (DevCrafterAI LLC · pgsql-announce) [unverified: self-reported benchmark]
    • 80 rules with line-level evidence; PL/SQL parsed with a full grammar; converts only what it can prove (CONNECT BY → WITH RECURSIVE, ROWNUM top-N → LIMIT, MERGE, partitions) and sends the rest to a "residue report" instead of emitting DDL
    • extraction tested nightly against Oracle XE 11g/21c and Free 23ai; output applied to PG 16–18 on every commit

PostgreSQL mailing lists

  • [bugs] BUG #19774: Incremental backup loses re-extended blocks when WAL summaries are merged out of LSN order — a week of reports that incremental backups restore stale or deleted data while pg_verifybackup says every input backup is valid. (Ke; Fujii Masao, Manu, Ayush Tiwari, Shihao Zhong, Raja Sai Pranav, David Steele, Michael Paquier · pgsql-bugs/-hackers) [patch posted]
    • #19774 (17+): WAL summaries are merged in readdir() order, so blocks a table regained after a VACUUM truncation can be left out of the incremental, and pg_combinebackup takes them from the prior backup. In the repro every live row is lost and the 20,000 rows deleted before the backup come back. Fujii's fix sorts the summaries; he'll commit it back to 17 barring objections
    • #19775: with data checksums and wal_log_hints both off, VACUUM truncation's change to the last visibility-map page isn't WAL-logged, so the restore brings back a stale VM page. Heap pages added later inherit all-visible/all-frozen bits, VACUUM moves relfrozenxid past tuples it never froze, and index-only scans return deleted rows (17.11, 18.6). Two fixes posted: summarizer-side (no new WAL, suits back branches) and writer-side (always WAL-log the page)
    • #19776: in a full → incremental → incremental chain, pg_combinebackup ignores the middle backup's truncation_block_length; deleted rows reappear in the table but not its PK index, and bt_index_check() fails
    • across a promotion: a standby's summarizer runs ahead of replay because GetLatestLSN() returns the walreceiver's flush position, so a timeline-1 summary can extend past the switch point and drop blocks — rows come back with pre-update contents — and it trips an assertion in the WAL summarizer. Separately, pg_basebackup --incremental accepts a prior backup whose WAL runs past the new primary's switch point after an ordinary async failover
    • plain base backups too: files replaced by rename (postgresql.auto.conf, pg_logical/replorigin_checkpoint) between stat and open are sent truncated or zero-padded: the restored cluster fails its CRC check, or silently loses ALTER SYSTEM settings. A pg_twophase file caught mid-write comes back empty, also a crash-safety bug, which Paquier will fix now (CF #7509)
  • [bugs] BUG #19783: Sorted GROUPING SETS return a grouping column's value in a set that leaves it out — 19 regressions found against RC1 this week, ahead of RC2 on 12 November. (Alperen Keles; David Rowley; Shihao Zhong, Tom Lane, Fujii Masao, Jeff Davis, Bhuvan Chilukuti, Vaibhav Jain · pgsql-bugs/-hackers)
    • wrong results: with GroupAggregate, a column outside the current grouping set shows its real value instead of NULL whenever the row has a NULL. Since c456e391138 ("Optimize tuple deformation") deforming further rewrites tts_isnull[] from the null bitmap and loses the NULLs prepare_projection_slot() forced. Added to open items [open]
    • hash join spills: since 1811f1af98f, outer rows with a NULL join key went to a tuplestore and to disk even in a single-batch join — a 1M-row LEFT JOIN took 22 ms on 18 and 110 ms plus 106 MB of temp files on 19. Tom pushed a fix that emits them immediately in non-parallel and single-batch joins [committed]
    • ILIKE index scan misses rows under a libc UTF-8 collation on macOS (1 row vs 2 from a seqscan); Jeff Davis plans to commit the fix [patch posted]
    • NOT IN → anti-join keeps an outer row when a user-defined implicit cast on the inner side returns NULL for non-NULL input; the reporter attributes it to 383eb21eb [patch posted]
    • VACUUM sets reltuples to 0 if a read-only query runs between a load and the first VACUUM: on-access VM setting has already marked the full pages all-visible, so VACUUM scans only the tail pages and extrapolates their density. No replies yet [open]
    • pg_upgrade --check passes, then the upgrade fails on a role setting standard_conforming_strings = off, or a user-defined uuid→bytea cast that 19 now has built in [patch posted]
  • [hackers] Wrong results from a parameterized Append and three more wrong-results bugs in released branches. (Richard Guo, Tom Lane; Venu Kummari; Xinlei Zhang · pgsql-hackers/-bugs)
    • the 10-05 issue's bug: a Materialize over a parameterized Append lost its join filter — 17+ via ANY-sublink pull-up (bisected to 9f133763961), 16 via plain LATERAL. Pushed and back-patched on 26 October [committed]
    • mergejoin: prepare_sort_from_pathkeys() copied a lateral reference into a sort key without a nestloop Param, returning 0 rows instead of 1. Fixed back to 14 [committed]
    • ranges, 17+: elem <@ range folds to a constant for a NULL element when the range is empty or (NULL, NULL), so WHERE booked_on <@ daterange(NULL, NULL) also returns NULL rows [patch posted]
    • plan cache: an invalidation arriving while a plan is built (for a sub-partition, or an SQL function that only appears after inlining) leaves a cached plan stale across transactions, returning wrong results until something unrelated invalidates it. Reproduced on 17.8 and master; TAP test posted, no committer reply yet [open]
  • [bugs] BUG #19725: pg_restore read failure with io_uring, not observed with worker — the "Operation canceled" reads on 18.6 turn out to be a Linux bug: since 5.18 (6bf9c47a3989, also marked for 5.15+ stable) buffered reads submitted with IOSQE_ASYNC go to io_uring's unbound worker pool, limited only by RLIMIT_NPROC; once it's used up fork() fails and queued reads return -ECANCELED. A fix is queued for 7.4 (98bc044d5a53); one user moved production to io_method = worker. (Dmytro Astapov; Tomas Barton, Shihao Zhong; Alexander Lakhin · pgsql-bugs) [open]
    • #19792 (18.6): an INSERT splitting a btree leaf and autovacuum's read-ahead hung each other, immune to cancel and terminate — WaitIO() sleeps before the IO is registered and nobody wakes it. Draft patches build on an earlier fix from Thomas Munro [patch posted]
    • #19796 (19rc1): PANIC "io_uring submit submitted only 0 of 1" with io_max_concurrency = 1024 and small shared_buffers, under plain pgbench [open]
  • [hackers] [PATCH v1 1/1] Add GENERATED ALWAYS AS ROW IDENTITY columns — a large PoC for a zero-byte row identity built on a 64-bit "expanded TID" got a policy answer instead of a review. Álvaro Herrera: "stop flooding this list with code that nobody wrote and nobody will read", pointing at an earlier pgbench refactor and the review backlog; Heikki Linnakangas called it "dead on arrival", and Michael Paquier won't spend committer time on code its author visibly didn't write. Future PoCs of this kind will live on an external branch, with the idea posted first. (Hannu Krosing; Álvaro Herrera, Heikki Linnakangas, Michael Paquier · pgsql-hackers) [rejected]
  • [hackers] PoC: true in-place major version upgrades — a "tracer bullet" series aimed at pg_upgrade's two limits: downtime with very large schemas, and no PITR across major versions. pg_inplace_upgrade prepare freezes the catalogs and marks the shutdown checkpoint as an upgrade boundary; swap starts the new binary, which converts pg_control and migrates catalogs in a new "upgrade" mode, WAL-logged but not archived or sent; rollback is possible until golive, and a standby simply replays new-version WAL. Tom Lane is "really, really skeptical" beyond trivial catalog changes — information moving between catalogs (pg_constraint for NOT NULL), new pg_depend links, platform-dependent pg_collation: "We built pg_upgrade the way we did precisely to avoid such problems." Bruce Momjian thinks it may now be feasible. Naylor presents a related talk at PGConf.Asia next month. (John Naylor; Tom Lane, Bruce Momjian · pgsql-hackers) [patch posted]
  • [committers] Fix logical replication with lossy identity indexes — committed fixes worth knowing. (Paul A. Jungwirth, Zhijie Hou, Amit Kapila; Andrey Borodin, Álvaro Herrera; John Naylor; Rui Zhao, Tom Lane · pgsql-committers) [committed]
    • 18+: a GiST index on a multirange (a WITHOUT OVERLAPS key used as replica identity) stores only the bounding range, so the apply worker could UPDATE or DELETE the wrong row; it now rechecks the key columns
    • 14+: since pg11 validatePartitionedIndex() released locks early, so FK restores in a parallel pg_restore could fail; locks are now kept until commit
    • 14+: jsonb_set_lax() with a constant NULL new_value wrote a per-tuple pointer into a constant argument; later rows read freed memory
    • 14+: one more GIN jsonb_ops wrong result, strict $ == "a scalar" missing root strings; fixed on the search side, no reindex needed
  • [hackers] Parallel vacuum: workers together with leader could overrun cost budget — each parallel VACUUM process sleeps only for its own cost, so the leader and N workers consume N+1 times the budget — 19's parallel autovacuum included. Against a 100k units/s budget: 152k at PARALLEL 2 and 243k at PARALLEL 8 on master; v2 stays under. Open: capping the longer sleeps without pg_usleep() over 1 s, and how far to back-patch. (Bharath Rupireddy; Masahiko Sawada, Chao Li, Surya Poondla · pgsql-hackers) [patch posted]
  • [committers] Mark GUC-dependent type output functions as STABLE — master only, so a 20 upgrade issue: byteaout, float4out/float8out, the geometric output functions, pg_mcv_list_out and BRIN minmax-multi's summary output lose IMMUTABLE, so expression indexes, generated columns and partition keys using them will be rejected ("Do not backpatch, since this … can break existing schemas"). Next in the thread: cube_out, hstore(record)/populate_record, xpath(), quote_ident(); age(timestamptz, timestamptz) depends on TimeZone too, but nobody proposes changing it. (Fujii Masao; Peter Eisentraut, Tom Lane, Ilya Staroverov · pgsql-committers) [committed]
  • [hackers] A bug in recovery prefetcher, triggering asserts / FATAL: checkpoint request failed — since 15, XLogPrefetcherBeginRead() doesn't reset prefetcher->record; with a full I/O queue (seen with reduced maintenance_io_concurrency) the prefetcher works from a stale record: an assertion on dev builds, "xlog flush request … is not satisfied" then "FATAL: checkpoint request failed" on release builds. One-line fix plus a TAP test. (Tomas Vondra; Chao Li · pgsql-hackers) [patch posted]
  • [bugs] FORCE ROW LEVEL SECURITY makes ADD/VALIDATE FOREIGN KEY skip rows hidden from the owner — under ENABLE + FORCE RLS, RI_Initial_Check sees only rows the owner's policies expose: a hidden orphan goes unchecked and the constraint is still marked validated, while a hidden referenced row makes a valid row fail. Reproduced on 18.6 and 19rc1; RI triggers on later inserts are correct. AI-assisted report, no developer reply yet. (Ludwig Kannicht · pgsql-bugs) [open] [unverified]
  • [hackers] Fix concurrent-abort detection for subtransactions of prepared transactions — when decoding a prepared transaction, ReorderBufferProcessTXN() sets CheckXidAlive only for top-level changes (RBTXN_IS_PREPARED is set only there), so subtransaction changes skip the concurrent-abort check; in the report a prepared transaction rolled back during decoding still had all its changes sent downstream. (Vaijayanti Bharadwaj · pgsql-hackers) [patch posted]
  • Follow-ups from last week — (pgsql-hackers/-bugs/-committers)
    • CLUSTER/REPACK row loss (#19760): fixed on 26 October, back-patched to 14, by setting the missing hint in HeapTupleSatisfiesVacuumHorizon(). A second case is still open — VACUUM FULL on a catalog can lose a committed table's pg_class row; the candidate patch made VACUUM 1.98× slower on deletes and Paquier called that unacceptable [committed] in part
    • promotion hang (thread): after last week's fix, promotion still waits on a standby with a logical slot — and a slot kept in sync for failover, the typical HA setup, is enough. 19 gets only a documentation note (tcp_user_timeout); for master, Manu's v5 lets backends absorb the barrier during sync rep or a paused COPY FROM STDIN, which also fixes a DROP DATABASE hang reproducible on 17.11 and 18.6 [patch posted]
    • indexes from 18 that miss rows after pg_upgrade (thread): Andreas Karlsson argues a release note is enough; Shihao points out 18 added a pg_upgrade check for lower()/upper() indexes after a Unicode update and asks Jeff Davis to decide. Unresolved
    • from last week's open [PG19] list: the stuck NOTIFY queue is fixed, and CREATE SCHEMA elements are schema-qualified again (19 had let a CREATE INDEX inside CREATE SCHEMA land on a same-named pg_temp table) [committed]
    • RI cascade-delete reentrancy (thread): a self-referencing table with BEFORE DELETE triggers can segfault if the plan cache is reset mid-cascade (use-after-free, 14+); v10 passes on 14 through 19 and Amit Langote is about to commit it [patch posted]

CommitFest (open: PG20-3, #62)

  • Balance (Oct 19–25): 30 new · 20 closed (16 committed / 3 withdrawn / 1 rejected) → net +10. (vs +31 last week, 21 lower. Both weeks are complete counts: new entries from sequential patch IDs #7480–#7510, closures from the history of every patch in #61 and #62.) 11 entries moved to Ready for Committer. Queue at scan: 383 needs review · 45 waiting on author · 107 ready for committer (535 active, up from 511). Last week's burst is being worked off: three of the four ruleutils deparse fixes and three of the six [PG19] bug entries were committed, and a fourth [PG19] entry was withdrawn. PG20-4 opens on 1 November.
  • New this week: Reducing relcache memory usage 2: shrink sizeof(RelationData) (David Geier) — every relcache entry is a 488-byte RelationData, rounded by AllocSet to a 512-byte chunk plus an 8-byte header, in every backend. The series puts table-only and index-only fields in a union, computes the lock tag's dbId only when needed, moves partitioning fields to a separate struct and allocates entries from a slab context: 496 → 248 bytes on current master (v4). Loading 38,966 tables and 148,468 indexes into one backend cut CacheMemoryContext from 589 to 536 MiB (~10%); a 1000-partition pgbench -S showed no TPS change. Review found the union's sharp edges — a SERIALIZABLE TRUNCATE on a table with a rule hit an assertion because rd_rules now sat where predicate locking looked for rd_index, and pg_relation_is_updatable() crashed on an index; v6 fixes both with a relkind-based RelationIsIndex(). Matthias van de Meent wants an explicit variant field instead of the extra rd_rel dereference, and Álvaro Herrera objected to moving RelationData into a new header just to get past gen_node_support.pl (thread). Also new: Use-after-free of a shared Path in add_path()/add_partial_path() (#7498, Jeevan Chalke; Richard Guo found a reproducer on 19 and master), Fix rescan costing of single-batch hash joins (#7502; a 43–60 ms hash join beat a ~1.5 ms nested loop + Memoize), Improve UNION de-duplication (#7496, partial HashAggregate below Gather), Check XID wraparound failsafe between indexes in parallel vacuum (#7490), Report WAL replay progress before a standby reaches consistency (#7488, already Ready for Committer), Stale hot_standby_feedback xmin when wal_receiver_status_interval = 0 (#7480), Avoid truncated or zero-padded copies of replaced files in backups (#7509, David Steele), Add -fstrict-flex-arrays=1 (#7510, Peter Eisentraut). Last week's standout Specialize ExecIndexOnlyScan() (#7468) hasn't moved; Batch TIDBitmap insertions in index scans (#7422) is still Ready for Committer.
  • Closed: committed: Fix stale allocChunkLimit when recycling AllocSet (#7473), Report no unpinned buffers as insufficient resources (#7342), Reject COPY TO of unlogged relations during recovery (#7469), Fix WITHOUT OVERLAPS multirange with logical replication (#7463), [PG19] Stale state after the join-removal restart gives wrong results (#7450), 035_standby_logical_decoding can fail when libpq loses a FATAL message (#7247), three ruleutils deparse fixes (#7441, #7442, #7451), EXPLAIN prints another ModifyTable's RETURNING aliases in a SubPlan (#7486) and Fix leaked hash scans in pg_listening_channels() (#7481), both filed and committed the same week, and 5 smaller ones. Withdrawn: pgbench: PL/pgSQL versions of tests (#4105, carried since the 2023-03 CF), pgbench functions available server-side (#7225), a [PG19] pg_dump --statistics fix (#7446). Rejected: Check all addresses for a matching target_session_attrs (#5396).

Community pulse

  • pgrust pulled from ClickBench for "cheating", then readmitted with its source published — the thread is about the maintainers' removal PR #2165 (merged in September). Evidence from the v0.4-preview binary: one source file per benchmark query (q00.rs … q36.rs) and plan matchers whose errors read "clickbench group count: filter is not k <> 0"; "hot" runs replaying cached verdicts (Q20 10.8 s cold, 11 ms on the second run, 283 ms with a new constant); 20 of 67 near-benchmark probes refused. The author replied that the per-query kernels were IR benchmarking probes reachable only through an environment variable, that the cache stores which dictionary strings match a predicate rather than results, and conceded that submitting an incomplete build was a mistake. A v0.4-beta with public source (a "Gravity" engine with a Lisp-based JIT IR, MVCC on pgrcolumnar, automatic work_mem sizing thanks to its thread-per-connection design) was submitted on 22 October and merged on 25 October with every cache documented and switchable. The most useful comment came from someone who read the beta: no single trick — dictionary encoding, zone maps, late materialisation and vectorisation all contribute; disabling its dictionary optimisations took a run from 3.8 s to 20.4 s, against ClickHouse's 15.0 s. An engine aimed at one benchmark can apply data-dependent optimisations far more aggressively than a planner serving millions of users. (Hacker News · 193 pts, 160 comments) [unverified: author's claims about dead code and the beta's speed]
  • COPY vs batched JDBC, rerun after a reader's catch — a 1M-row CSV import benchmark (COPY ~1.7 s, batched JDBC ~13.6 s, MariaDB ahead of Postgres on batching) drew the reply that pgJDBC defaults to reWriteBatchedInserts=false while MariaDB's driver batches by default. With rewriting on, JDBC went 11.3 → 7.3 s and the MariaDB gap shrank from 7.3 s to 0.7 s; raising the batch from 1,000 to 5,000 then made JDBC ~18% slower. Consensus: COPY into an unlogged staging table and transform with INSERT … SELECT, or stream through pgJDBC's CopyManager. PG 17.11 vs MariaDB 11.8.8, 4 vCPU, three runs each. (r/PostgreSQL · 15 + 10 comments across two threads)
  • A JSON column round-trip changed 11% of float values on MySQL 8.4 — a hash-chained audit log verified on SQLite and broke on MySQL at event 34: MySQL parses JSON into a binary format and re-renders it, and some doubles change their last digit. 1,075 of 10,000 random doubles changed on 8.4.10, none on MariaDB (which stores JSON as checked LONGTEXT). Replies disagreed about whether Postgres jsonb preserves long floats (it stores numeric), but agreed it drops key order and whitespace while plain json keeps the text. Fix everyone landed on: hash and store the exact serialised string (or canonicalise per RFC 8785) and keep jsonb as a second column for querying. (r/mysql · 19 comments) [unverified: one user's measurements]

Wider DBMS & distributed data

  • HPTS 2026, day 1 and days 2–3 — notes from the invite-only workshop, which publishes no proceedings. (Murat Demirbas · muratbuffalo.blogspot.com)
    • "Learned systems are dead" (Todd Lipcon, Google): learned B-trees, ML cache eviction and Kepler query optimisation never shipped; RL-tuned compaction gained 2% and didn't either. His rule: "don't ship the model, ship what the model wrote" — have the LLM generate code, review it, ship that
    • Spanner Omni: the downloadable Spanner had to replace Colossus, Chubby and TrueTime; commit wait only has to cover clock uncertainty, so uncertainty below one replication round trip suffices — one centrally synced reference with a per-region fallback replaces GPS and atomic clocks
    • "S3 is the past" (Viktor Leis): S3 behaves like a million hard disks, ~10 ms and 80 MB/s per request; Iceberg on S3 manages about 1 transaction/s, and every OLTP-on-object-storage system ends up building its own flash layer
    • Cassandra witness replicas (Apple): journal-only witnesses force repair to move from Merkle trees to mutation tracking, since a witness holds no data to hash; storage overhead 6.2× → 4.2×
    • Alex Miller: 20 leaderless quorum protocols judged against "survive contention, consistent local snapshot, incremental catch-up" — Accord/CEP-15 passes
    • Differential Change Discovery (Databricks): OLTP engines should expose the net change since an LSN, not CDC's forty events for forty updates of one row
  • Torin: lazy evaluation in a GPU-native SQL engine — how to keep column-at-a-time evaluation without breaking SQL guards like qty > 0 AND price/qty > 1000. Each expression is classed at compile time by how it can fail: never (evaluated eagerly), only on non-NULL input (NULL-masking: decided rows' inputs set to NULL), or on any row (filter-and-scatter into a smaller batch); coalesce(x,0)/coalesce(y,0) is where NULL-masking breaks. AND/OR/COALESCE become IF plus a LET binding, and adjacent ANDs split into segments at each fallible operand, so comparison-only filters still compile to one eager AND. Relevant to any vectorised or JIT expression evaluator. (Piotr Rzysko, Piotr Findeisen · starburst.io) [vendor blog — substantive]
  • Single-row lookups in DataFusion — point reads into one 100M-row Parquet file on a 4-vCPU VM: a sorted file is 6,000× faster than an unsorted one (1 page read vs 67 row groups and 122 pages); the best layout is ~128k rows per row group and 1k per page, vs defaults of 1M and 20k; dictionary encoding and bloom filters only hurt on unique sorted keys. Planning dominates — 12% of CPU in the logical plan and 50% in the physical — and an unmerged prototype reusing physical plans in prepared statements gives 2–3× throughput. (Nuno Faria · datafusion.apache.org)
  • Birth and death of a row in Turso's MVCC — Hekaton-style MVCC in a SQLite rewrite, including the durability design Hekaton's paper left out: versions in a SkipMap<RowId, RwLock<Vec<RowVersion>>>, commit rewriting begin/end from TxId to commit timestamp, first-committer-wins over rowids and UNIQUE/PK keys, speculative reads from Preparing transactions. Commits go to a CRC-framed logical log (a torn tail frame is ignored on replay); at ~4 MB a checkpoint moves rows through the SQLite WAL into the B-tree, and the default Truncate mode blocks new transactions with busy errors. Reads merge the in-memory store and the B-tree with a two-pointer scan. (Pere Diaz Bou · turso.tech) [vendor blog — substantive]
  • Jepsen: Orkes Conductor 5.3.2 — a workflow engine that keeps state in Postgres and queues work in Redis, with no outbox between them. A crash between the Postgres insert and the Redis enqueue left workflows RUNNING forever (the sweeper reads only the queue); when the enqueue failed, the server deleted the already-committed row, so clients saw a workflow for 74 s and then a 404; 1,752 of 25,920 "failed to connect" errors had in fact started the workflow. Most issues are fixed in 5.5.x. (Kyle Kingsbury · jepsen.io)
  • DuckDB 2.0.0 LTS — out on 21 October, ~20,000 commits after 1.5 (the 10-12 issue had a measured preview). (DuckDB team · duckdb.org)
    • client-server "Quack" protocol polished; CONNECT attaches a remote database; a remote-pushdown optimiser sends queries to Postgres and MySQL instead of pulling tables
    • async I/O for Parquet, CSV and native files; VARIANT shredded end to end through Parquet
    • a PEG parser replaces the Postgres-derived Bison grammar and is extensible (enable_legacy_parser brings the old one back)
    • triggers with transition tables — enough to maintain views incrementally (BEFORE … FOR EACH ROW not yet)
    • breaking: 2.0 files can't be read by older versions; C++17 required; LTS until at least October 2027
    • on HN (462 pts, 63 comments) users reported 2.0 keeping all cores busy up to 128–192 vCPUs where earlier versions stopped at about 64, and a developer confirmed async I/O also covers the native format, contrary to the post
  • Twelve years of distributed SQLite — rqlite's author on how the field split into consensus (rqlite, dqlite), WAL streaming (Litestream, LiteFS) and merging (cr-sqlite), and how the money went to hosted platforms; Turso's site now says it is joining Supabase. On HN (385 pts, 66 comments) a reply noted rqlite doesn't officially support BEGIN/COMMIT while dqlite has full transactions, and the single-box camp reported in-process SQLite at 80k writes/s and 200k reads/s. (Philip O'Toole · philipotoole.com)
  • TiDB Cloud security incident — stolen service-account credentials gave access from 9 September to 7 October; exported records included user names and emails, SQL editor text and error messages, billing data and a table of encrypted API keys. PingCAP reports no evidence of commands run in the production systems reviewed; four vulnerabilities found during the response were fixed by 14 October. (PingCAP · pingcap.com) [unverified: attacker's broader claims are disputed]
  • Animations for system builders: metastability — animations replayed from traces of real systems. On Postgres 16 in a MicroVM, a load spike in which each query takes 12 MB evicts the cached working set, and the database stays disk-bound after the spike ends; in etcd, a read spike from 500 to 1,000/s locks in GC overload. (Marc Brooker · brooker.co.za)
  • ClickHouse on NVIDIA Vera — 1 TB of in-memory strings on 176 cores: LIKE '%ab%' stops scaling at 64 threads at 813–900 GB/s, the memory bandwidth, while a regex filter, an aggregation and a hash join scale at ≥93% efficiency to 176 threads; SMT adds little. (Robert Schulze · clickhouse.com) [vendor blog] [unverified: early-access hardware]
  • Iceberg tables over a CDN — one DuckDB scan is 1,147 ranged GETs (221 s from Sydney direct vs 6 s through a CDN). The useful finding is a trap: GCS caches an overwritten public version-hint.text for an hour, so readers saw a stale snapshot for 36 minutes and one query failed when two reads of the same file returned different versions. (Javier de la Torre · lakehousebox.com) [vendor blog]
  • DocumentDB 1.0 — the MIT-licensed MongoDB-compatible layer storing BSON in Postgres reaches GA (PG 15–18). (DocumentDB team · documentdb.io)
    • fixes a permission check skipped with batchSize: 0, requires write permission for $merge/$out, fixes a dropIndexes deadlock and RUM-index vacuum bugs
    • upgrade trap: a 1.0 container started on an existing volume does not upgrade the schema; the image now sends usage telemetry by default (DO_NOT_TRACK=1)

Commercial engines (SQL Server, Oracle, MySQL, …)

  • MySQL 26.10 Innovation — released with 9.7.4 and 8.4.13 LTS; the engine changes are in the release notes. (Mike Frank · blogs.oracle.com/mysql)
    • READ COMMITTED on NUMA: closing a read view no longer takes a global mutex and copies the active-transaction set
    • recovery flushes only pages with changes after the last redo batch, not the whole buffer pool; a page-reorganisation size mismatch that could corrupt a page after a crash is now fatal
    • stale Adaptive Hash Index entries after a tablespace import could return rows from the discarded tablespace
    • EXPLAIN TREE / JSON v2 now show selected partitions
  • Benchmarking vector indexes in MySQL — a survey of MariaDB's MHNSW, Google's ScaNN pull request to MySQL, Percona's HNSW inside InnoDB and VillageSQL's extension. Fairness note: pgvector loaded with COPY while the MySQL engines used INSERT, so build times compare different statements; HNSW reads dropped 40% with four concurrent writers. (Adam Levin · villagesql.com) [vendor blog] [unverified: laptop-scale numbers]

Migration experience

  • 119 TB from Oracle Exadata to RDS for Oracle — four 19c databases moved with incremental cross-platform transportable tablespaces (XTTS) over a link capped at 0.5–1 Gbps: the baseline took 7 days at 216 MB/s average, cutovers 2–6 h each. (aws.amazon.com/blogs/database) [vendor blog] [unverified: customer-reported figures]
    • Exadata Hybrid Columnar Compression data grows 3–5× when recompressed with Advanced Compression
    • ext4 caps a bigfile tablespace at 16 TB, so oversized tablespaces were reorganised first
    • up to 4 TB of changes a day meant incremental syncs could never fall more than a day behind; a few huge tablespaces left transfer agents idle, while an evenly sized 38 TB database ran fastest at 339 MB/s

Research & cutting edge

  • SWatcher+: deploying learned cardinality estimators in a changing database — tested inside PostgreSQL with MSCN, JGMP and Naru: instead of retraining on a schedule, it monitors only the subqueries whose misestimates change plans and applies correction factors (joins included) during optimisation. End-to-end time 44–101% of a tuned periodic refresh; wins 14 of 16 templates with MSCN and beats a reimplementation of LEO. (Liu, Capol, Hu, Agarwal, Yang) [paper]
  • The case for TPU-accelerated analytical query processing — all 22 TPC-H queries in JAX/XLA: 7.0× over DuckDB on one TPU at SF100, 68.6× with 16 TPUs at SF1000. Buffer sizes must be fixed at compile time, so catalog statistics set guarded static capacities and hash joins become sort plus vectorised search. Warm runs only — loading, host-to-device transfer and JIT compilation excluded. (Chung, Ha, Caminal, Ozcan, Kamath, Krishnamurthy, Levy) [paper]
  • SQLWitness and EquiTrace — two sides of SQL equivalence. SQLWitness (SIGMOD) uses the real engine as the oracle when generating counterexample data, so dialect semantics come free (3 * 2 || '1' gives TRUE, 63 or 61 depending on the engine), refuting 5–88% more pairs at 0.12 s each (An, Rigger); EquiTrace proves equivalence by chaining optimiser rewrite rules into a replayable derivation, 97 pairs proved vs 35 for QED and 38 for SQLSolver (Chen, Huang, Miao). [paper]
  • WaRP and RemoteANN — vector search over object storage, where cost is per request. WaRP merges probed byte ranges into requests with a DP within (1+ε) of optimal in under 10 ms per query (1.09–3.48× QPS, 2.1–5.2× fewer GETs on up to 100M vectors); RemoteANN combines IVF, RaBitQ, reordered layouts and an NVMe cache for up to 16× fewer requests. (Wan, Zeng, Qiao, Zhang; Gou et al.) [paper]
  • PrairieFire: high-throughput updates for SSD-resident ANN — buffered updates plus batched local rebuilds avoid both per-insert SSD writes and repeated background relocation: 3–35× update throughput at 3–5× search throughput, two months in production. (Huang et al.) [paper]
  • KatoANN — batching queries onto Intel AMX turns IVF's memory-bound distance scan compute-bound; vectors stored once as 16-bit codes split into high/low byte planes, rank on the high byte and refine survivors on both. 15,635 QPS at recall@10 ≥ 0.99 on 3,072-d vectors, 3.1× CABANA. (Cho, Kim) [paper]
  • Jaro-Winkler similarity joins on large string collections — SIGMOD 2027: linear-time Jaro-Winkler for bounded alphabets, cost-model-driven bitmap filters and prefix grouping; up to 1,000× faster at low thresholds where trie-based joins degrade. (Lankes, Kocher, … Augsten) [paper]

International (non-English sources)

  • Reading 100 bytes out of a 10 MiB TOAST value: EXTERNAL vs PGLZ vs LZ4 on PostgreSQL 18 — measured on 18.6, confirming the source: a partial read of an uncompressed EXTERNAL value fetches only the chunks it needs, PGLZ fetches compressed data from the start up to the offset, and LZ4 fetches the whole compressed value first. Median of five 30-s pgbench runs per point; complements PlanetScale's TOAST post from the 10-12 issue. (Signorino · Программный Продукт on Habr) [ru] (orig: 100 байт из 10 MiB: как PostgreSQL 18 читает TOAST с EXTERNAL, PGLZ и LZ4)
    • start / 50% / 90% offset: EXTERNAL 0.040 ms throughout; PGLZ 0.040 / 7.064 / 10.422 ms; LZ4 2.016 / 3.669 / 5.269 ms
    • buffer hits: PGLZ 5 at the start, ~750 at 50%; LZ4 ~747 even for the first 100 bytes; at 1 MiB the gap disappears
  • YooMoney moves its core banking system from MS SQL/Windows to PostgreSQL/Linux — more than 32,000 lines of T-SQL rewritten in PL/pgSQL by hand (automatic and AI converters weren't good enough). Of the 5 TB source only the last five years moved (just over 1 TB), converted in 64 threads, with index builds taking at least a third of the migration time. Bulk payment processing ran 2–5× slower until rewritten; the signing module ran 10× slower because of the Windows→Linux move, not the database. Each iteration shadow-ran on live input for at least a week; cutover took a few hours. (YooMoney · Habr) [ru] (orig: Большая миграция АБС: c MS SQL и Windows к PostgreSQL и Linux)
  • In a MySQL multi-table UPDATE, whether SET sees the new value depends on the plan — the same UPDATE t1,t2,t3 SET t1.a=t1.b+1, t1.c=t1.a … gave c=100 (old value) or c=11 (new) after row counts changed, on 5.7 and 8.0. From sql_update.cc: only the first table in the join order (best_ref[0]) is updated in place, where fill_record writes left to right; every other target goes through a temp table, so later expressions read the old row. The manual says assignment order isn't guaranteed; this shows statistics alone can silently flip the result. (China Mobile Cloud HaishanDB team · modb.pro) [zh] (orig: 移动云海山数据库 HaishanDB：关于MySQL多表UPDATE顺序赋值的不确定性分析)
  • Five ways to store a hierarchy on PG 18: reads tie, subtree moves differ ~780× — one skewed 5-level org tree built as adjacency list + recursive CTE, ancestry, nested set, closure_tree and ltree/GiST (setup in part 1). Descendants at 10k nodes took 3.2–8.4 ms for every method; moving the largest subtree took 0.97 ms (adjacency), 11.68 (ltree), 48.43 (nested set), 155.92 (ancestry) and 758.19 ms (closure_tree, holding locks throughout). The authors caught a "move" that ActiveRecord's dirty tracking had turned into a no-op. (m-nishikawa · TOKIUM on Zenn) [ja] (orig: 多階層の部署データ設計【第2回】結果編)
  • Two PostgreSQL settings lost in the noise of a 1C benchmark — five unchanged runs of a single-threaded 1C test scored 16.39–26.04 (1.6×) on PG 18.4-1.1C beside ten busy neighbouring bases; alternating A/B runs put shared_buffers 6 GB vs 128 MB and synchronous_commit off vs on inside that spread, and the same test scored 31–35 at night vs 22–26 by day. A model of measuring before tuning. (zhogov1985 · Habr) [ru] (orig: Два параметра PostgreSQL утонули в разбросе теста 1С)
  • OceanBase's DBA_INDEX_USAGE shows no rows for an index that is being used — on 4.3.5.6 the default _iut_stat_collection_type=SAMPLED records index use at a fixed 10% sample, so a used index can be missing from the view — a trap when dropping "unused" indexes; 'ALL' records everything at some overhead, flushed every ten-plus minutes. (OceanBase community) [zh] (orig: OceanBase 索引明明走了，DBA_INDEX_USAGE 却查不到使用记录？)
  • pg_walex: an offline tool that turns WAL segments back into SQL (PG 11–18) — a new AGPL-3.0 C tool (repo) that reads WAL segment files with no running server and no libpq, rebuilds table DDL from catalog data in the WAL, and emits SQL or JSON per database, table or transaction. Walkthrough only, no benchmark. (modb.pro) [zh] [unverified: load-speed claim] (orig: pg_walex最强WAL日志解析神器)

Upcoming events

  • PgConf.Nepal 2026 — Kathmandu, 18–21 November — two days of training, then two days of talks at Tribhuvan University. Program not posted yet; keynotes by Bruce Momjian, Marco Slot and Oleg Bartunov.
  • ACM SoCC 2026 — Singapore, 18–20 November — SIGMOD/SIGOPS cloud-computing symposium; only round-1 accepted papers are posted. Internals picks:
    • Spanner-RSS+: Diagnosing and Reducing Safe-Time Waiting of Read-only Transactions (Sejong Kim, Hena Cho, Yon Dohn Chung) — where read-only transactions stall on safe time, the same cost as serving snapshot reads from replicas.
    • TRAM: Tree-based atomic multicast on RDMA (Lorenzo Martignetti, Eliã Batista, Gianpaolo Cugola, Fernando Pedone) — the ordering primitive under partitioned, replicated transaction stores.
  • PGConf.PL 2026 — Gdańsk, 24 November — first edition, one day, seven talks. Internals picks:
    • Pushing the PostgreSQL Limits in a 1TB TPC-H Lab Exercise (Piotr Kołodziej) — almost 10× faster on the 1 TB query stream without changing query text, revisiting assumptions about statistics and indexing.
    • Multi-tenant Postgres: Where Each Design Hits a Wall (Aditya Surishetti, Avinash Vallarapu) — catalog, planning-time and autovacuum pressure as schema-per-tenant counts grow, and the RLS planner behaviour that surprises people.

New sources added this week

  • dbmsdrops.kindahl.net — a step-by-step series building a table access method, down to the call order inside RelationSetNewRelfilenumber. (Mats Kindahl)
  • brooker.co.za — metastability and distributed-database internals from first-hand systems work. (Marc Brooker)
  • jepsen.io — primary-source correctness analyses. (Kyle Kingsbury)
  • datafusion.apache.org/blog — engine-level measurement posts by contributors.
  • awide.tech/blog — source-level deep-dives on a PG17 PolarDB fork (CSN snapshots, PolarFS leases).
  • planet.mysql.com — aggregator for the MySQL ecosystem, filling a gap on the commercial-engines side.

72 items · yield — mailing lists: 1,001 messages in window (581 hackers / 165 bugs / 0 performance / 46 general, plus 202 pgsql-committers and 7 pgsql-announce) → 61 shortlisted → 40 published · blogs: ~143 posts in window → 92 shortlisted → 35 published · community: ~220 threads viewed → 21 shortlisted → 6 published · research: 37 cs.DB announcements in window → 14 shortlisted → 9 published · international: ru 18→10→3, zh ~370→22→3, ja ~180→12→2, fr 2→0→0, de 1→0→0 (polyglot search unavailable this run; native feeds and site APIs, native-language web search and the browser used instead).