Parallel tuplesort (for parallel B-Tree index creation)

git remote add commitfest https://github.com/postgresql-cfbot/postgresql.git
git fetch commitfest cf/690
git checkout commitfest/cf/690

Parallel tuplesort (for parallel B-Tree index creation) ×

First at 2016-08-01 22:18:22 by Peter Geoghegan <pg at heroku.com>
Latest at 2018-02-08 12:45:52 by Robert Haas <robertmhaas at gmail.com>
Latest attachment (0001-Mark-logtape.c-buffer-s-tail-as-defined-to-Valgrind.patch) at 2018-02-06 18:33:45 from Peter Geoghegan <pg at bowt.ie>

    Attachment (0001-Mark-logtape.c-buffer-s-tail-as-defined-to-Valgrind.patch) at 2018-02-06 18:33:45 from Peter Geoghegan <pg at bowt.ie> (Patch: Yes)
    Attachment (0001-Add-logtape.c-Valgrind-suppression.patch) at 2018-02-03 03:26:59 from Peter Geoghegan <pg at bowt.ie> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting.patch) at 2018-02-02 16:16:50 from Peter Geoghegan <pg at bowt.ie> (Patch: No)
    Attachment (0001-Chaos-monkey-fork-failure.patch) at 2018-01-25 01:31:19 from Thomas Munro <thomas.munro at enterprisedb.com> (Patch: Yes)
    Attachment (pessimistic-fork-failure-detector-v2.patch) at 2018-01-24 20:13:10 from Thomas Munro <thomas.munro at enterprisedb.com> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting.patch) at 2018-01-20 01:33:28 from Peter Geoghegan <pg at bowt.ie> (Patch: No)
    Attachment (propagate-reindex-state-v1.patch) at 2018-01-17 18:40:14 from Robert Haas <robertmhaas at gmail.com> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting.patch) at 2018-01-16 00:54:40 from Peter Geoghegan <pg at bowt.ie> (Patch: No)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting.patch) at 2018-01-05 22:17:51 from Peter Geoghegan <pg at bowt.ie> (Patch: No)
    Attachment (0004-Calculate-parallel-worker-inside-index_create.patch) at 2018-01-02 09:38:14 from Rushabh Lathia <rushabh.lathia at gmail.com> (Patch: Yes)
    Attachment (0003-Adjust-planner-code-to-consider-parallel_leader_part.patch) at 2017-12-12 10:09:29 from Rushabh Lathia <rushabh.lathia at gmail.com> (Patch: Yes)
    Attachment (0002-Adjust-planner-code-to-consider-parallel_leader_part.patch) at 2017-12-08 07:28:59 from Rushabh Lathia <rushabh.lathia at gmail.com> (Patch: Yes)
    Attachment (barrier-forget-participants-v1.patch) at 2017-12-08 02:03:50 from Thomas Munro <thomas.munro at enterprisedb.com> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting_v14.patch) at 2017-12-07 08:25:06 from Rushabh Lathia <rushabh.lathia at gmail.com> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting_v13.patch) at 2017-11-14 09:41:39 from Rushabh Lathia <rushabh.lathia at gmail.com> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting_v12_rebase.patch) at 2017-10-26 11:22:16 from Rushabh Lathia <rushabh.lathia at gmail.com> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting_v12.patch) at 2017-10-10 09:23:41 from Rushabh Lathia <rushabh.lathia at gmail.com> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting_v11.patch) at 2017-09-19 10:21:29 from Rushabh Lathia <rushabh.lathia at gmail.com> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting.patch.gz) at 2017-03-20 01:03:51 from Peter Geoghegan <pg at bowt.ie> (Patch: No)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting.patch.gz) at 2017-03-12 22:05:40 from Peter Geoghegan <pg at bowt.ie> (Patch: No)
    Attachment (speedup-100.txt) at 2017-02-16 02:05:14 from Thomas Munro <thomas.munro at enterprisedb.com> (Patch: No)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting.patch.gz) at 2017-02-10 00:10:14 from Peter Geoghegan <pg at bowt.ie> (Patch: No)
    Attachment (speedup.txt) at 2017-02-03 13:04:55 from Thomas Munro <thomas.munro at enterprisedb.com> (Patch: No)
    Attachment (0002-Add-temporary-testing-tools.patch.gz) at 2017-01-03 23:53:59 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (0002-Add-temporary-testing-tools.patch.gz) at 2016-12-04 01:29:01 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (max-sort-tapes.patch) at 2016-11-10 00:01:30 from Robert Haas <robertmhaas at gmail.com> (Patch: Yes)
    Attachment (0001-Cap-the-number-of-tapes-used-by-external-sorts.patch.gz) at 2016-11-08 04:28:35 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (0003-Add-force_btree_randomaccess-GUC-for-testing.patch.gz) at 2016-10-25 01:17:41 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (0001-Cap-the-number-of-tapes-used-by-external-sorts.patch.gz) at 2016-10-08 00:47:13 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (0001-Refactor-LogicalTapeSet-LogicalTape-interface.patch) at 2016-09-21 16:52:00 from Heikki Linnakangas <hlinnaka at iki.fi> (Patch: Yes)
    Attachment (0003-Rearrange-header-file-include-directives.patch.gz) at 2016-09-11 18:05:07 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (root_displace_patched_timings.log) at 2016-09-10 00:22:01 from Claudio Freire <klaussfreire at gmail.com> (Patch: No)
    Attachment (0003-Displace-heap-s-root-during-tuplesort-merge.patch) at 2016-08-16 00:33:38 from Peter Geoghegan <pg at heroku.com> (Patch: Yes)
    Attachment (0001-Cap-the-number-of-tapes-used-by-external-sorts.patch.gz) at 2016-08-01 22:18:22 from Peter Geoghegan <pg at heroku.com> (Patch: No)

Annotations

When	Who	Mail	Annotation
2016-09-11 18:07:19	Peter Geoghegan (pgeoghegan)	From Peter Geoghegan <pg@heroku.com> at 2016-09-11 18:05:07	V2 -- uses maintenance_work_mem as budget for entire CREATE INDEX, regardless of number of workers ×
2016-10-08 00:49:11	Peter Geoghegan (pgeoghegan)	From Peter Geoghegan <pg@heroku.com> at 2016-10-08 00:47:13	V3 -- rebased on top of Heikki's logtape.c preload memory patch ×
2016-10-25 02:06:25	Peter Geoghegan (pgeoghegan)	From Peter Geoghegan <pg@heroku.com> at 2016-10-25 01:17:41	V4 -- mechanical rebase on top of refactoring commit b75f467b6eec0678452fd8d7f8d306e6df3a1076 ×
2016-10-26 16:42:26	Peter Geoghegan (pgeoghegan)	From Peter Geoghegan <pg@heroku.com> at 2016-10-26 16:02:21	Do-over of original benchmark from August 1, showing how preloading work from Heikki (and other improvements) have increased scalability of parallel CREATE INDEX ×
2016-11-08 04:35:03	Peter Geoghegan (pgeoghegan)	From Peter Geoghegan <pg@heroku.com> at 2016-11-08 04:28:35	V5 -- Overhaul to cost model, new testing tool ×
2016-12-04 01:30:15	Peter Geoghegan (pgeoghegan)	From Peter Geoghegan <pg@heroku.com> at 2016-12-04 01:29:01	V6 -- tuplesort.c now uses condition variables ×
2017-01-03 23:57:44	Peter Geoghegan (pgeoghegan)	From Peter Geoghegan <pg@heroku.com> at 2017-01-03 23:53:59	V7 -- rebased on top of logtape.c simplifications, pg_restore avoids using parallel CREATE INDEX unless storage parameter was used, refactoring ×
2017-02-10 00:12:17	Peter Geoghegan (pgeoghegan)	From Peter Geoghegan <pg@bowt.ie> at 2017-02-10 00:10:14	V8 -- rebased on top of logtape.c bugfix, implements BufFile refcount mechanism ×
2017-03-12 22:11:30	Peter Geoghegan (pgeoghegan)	From Peter Geoghegan <pg@bowt.ie> at 2017-03-12 22:05:40	V9 -- Bulletproof resource management, significantly simplified cost model ×
2017-03-20 01:06:55	Peter Geoghegan (pgeoghegan)	From Peter Geoghegan <pg@bowt.ie> at 2017-03-20 01:03:51	V10 -- Fixes bug caused by commit 2609e91fc, bitrot ×
2017-11-14 22:55:27	Peter Geoghegan (pgeoghegan)	From Rushabh Lathia <rushabh.lathia@gmail.com> at 2017-11-14 09:41:39	v13 -- Contains fixes requested by Peter following adopting patch to parallel hash join BufFile infrastructure ×
2018-01-06 00:16:32	Peter Geoghegan (pgeoghegan)	From Peter Geoghegan <pg@bowt.ie> at 2018-01-05 22:17:51	Revision of Rushabh's most recent patch -- mostly just polishing. ×

Parallel tuplesort, partitioning, merging, and the future ×

First at 2016-08-08 19:44:02 by Peter Geoghegan <pg at heroku.com>
Latest at 2016-08-12 19:22:24 by Robert Haas <robertmhaas at gmail.com>

Annotations

When	Who	Mail	Annotation
2016-08-08 20:54:30	Peter Geoghegan (pgeoghegan)	From Peter Geoghegan <pg@heroku.com> at 2016-08-08 19:44:02	Related design discussion -- how tuplesort.c may in the future support partitioning ×

Cost model for parallel CREATE INDEX ×

First at 2017-02-28 19:28:15 by Peter Geoghegan <pg at bowt.ie>
Latest at 2017-03-09 21:11:21 by Peter Geoghegan <pg at bowt.ie>