Parallel tuplesort (for parallel B-Tree index creation)

Edit
Title Parallel tuplesort (for parallel B-Tree index creation)
Topic Performance
Created 2016-08-01 22:19:22
Last modified 2017-03-23 02:32:23 (1 month, 1 week ago)
Latest email 2017-03-22 16:26:00 (1 month, 1 week ago)
Status
2017-03: Returned with feedback
2017-01: Moved to next CF
2016-11: Moved to next CF
2016-09: Moved to next CF
Authors Peter Geoghegan (pgeoghegan)
Reviewers Thomas Munro (macdice)Become reviewer
Committer
Links Wiki
Emails
Parallel tuplesort (for parallel B-Tree index creation)
First at 2016-08-01 22:18:22 by Peter Geoghegan <pg at heroku.com>
Latest at 2017-03-22 16:26:00 by Peter Geoghegan <pg at bowt.ie>
Latest attachment (0001-Add-parallel-B-tree-index-build-sorting.patch.gz) at 2017-03-20 01:03:51 from Peter Geoghegan <pg at bowt.ie>
    Attachment (0001-Add-parallel-B-tree-index-build-sorting.patch.gz) at 2017-03-20 01:03:51 from Peter Geoghegan <pg at bowt.ie> (Patch: No)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting.patch.gz) at 2017-03-12 22:05:40 from Peter Geoghegan <pg at bowt.ie> (Patch: No)
    Attachment (speedup-100.txt) at 2017-02-16 02:05:14 from Thomas Munro <thomas.munro at enterprisedb.com> (Patch: No)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting.patch.gz) at 2017-02-10 00:10:14 from Peter Geoghegan <pg at bowt.ie> (Patch: No)
    Attachment (speedup.txt) at 2017-02-03 13:04:55 from Thomas Munro <thomas.munro at enterprisedb.com> (Patch: No)
    Attachment (0002-Add-temporary-testing-tools.patch.gz) at 2017-01-03 23:53:59 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (0002-Add-temporary-testing-tools.patch.gz) at 2016-12-04 01:29:01 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (max-sort-tapes.patch) at 2016-11-10 00:01:30 from Robert Haas <robertmhaas at gmail.com> (Patch: Yes)
    Attachment (0001-Cap-the-number-of-tapes-used-by-external-sorts.patch.gz) at 2016-11-08 04:28:35 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (0003-Add-force_btree_randomaccess-GUC-for-testing.patch.gz) at 2016-10-25 01:17:41 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (0001-Cap-the-number-of-tapes-used-by-external-sorts.patch.gz) at 2016-10-08 00:47:13 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (0001-Refactor-LogicalTapeSet-LogicalTape-interface.patch) at 2016-09-21 16:52:00 from Heikki Linnakangas <hlinnaka at iki.fi> (Patch: Yes)
    Attachment (0003-Rearrange-header-file-include-directives.patch.gz) at 2016-09-11 18:05:07 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (root_displace_patched_timings.log) at 2016-09-10 00:22:01 from Claudio Freire <klaussfreire at gmail.com> (Patch: No)
    Attachment (0003-Displace-heap-s-root-during-tuplesort-merge.patch) at 2016-08-16 00:33:38 from Peter Geoghegan <pg at heroku.com> (Patch: Yes)
    Attachment (0001-Cap-the-number-of-tapes-used-by-external-sorts.patch.gz) at 2016-08-01 22:18:22 from Peter Geoghegan <pg at heroku.com> (Patch: No)

Annotations

When Who Mail Annotation
2016-09-11 18:07:19 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@heroku.com>
at 2016-09-11 18:05:07
V2 -- uses maintenance_work_mem as budget for entire CREATE INDEX, regardless of number of workers
2016-10-08 00:49:11 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@heroku.com>
at 2016-10-08 00:47:13
V3 -- rebased on top of Heikki's logtape.c preload memory patch
2016-10-25 02:06:25 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@heroku.com>
at 2016-10-25 01:17:41
V4 -- mechanical rebase on top of refactoring commit b75f467b6eec0678452fd8d7f8d306e6df3a1076
2016-10-26 16:42:26 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@heroku.com>
at 2016-10-26 16:02:21
Do-over of original benchmark from August 1, showing how preloading work from Heikki (and other improvements) have increased scalability of parallel CREATE INDEX
2016-11-08 04:35:03 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@heroku.com>
at 2016-11-08 04:28:35
V5 -- Overhaul to cost model, new testing tool
2016-12-04 01:30:15 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@heroku.com>
at 2016-12-04 01:29:01
V6 -- tuplesort.c now uses condition variables
2017-01-03 23:57:44 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@heroku.com>
at 2017-01-03 23:53:59
V7 -- rebased on top of logtape.c simplifications, pg_restore avoids using parallel CREATE INDEX unless storage parameter was used, refactoring
2017-02-10 00:12:17 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@bowt.ie>
at 2017-02-10 00:10:14
V8 -- rebased on top of logtape.c bugfix, implements BufFile refcount mechanism
2017-03-12 22:11:30 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@bowt.ie>
at 2017-03-12 22:05:40
V9 -- Bulletproof resource management, significantly simplified cost model
2017-03-20 01:06:55 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@bowt.ie>
at 2017-03-20 01:03:51
V10 -- Fixes bug caused by commit 2609e91fc, bitrot
Parallel tuplesort, partitioning, merging, and the future
First at 2016-08-08 19:44:02 by Peter Geoghegan <pg at heroku.com>
Latest at 2016-08-12 19:22:24 by Robert Haas <robertmhaas at gmail.com>

Annotations

When Who Mail Annotation
2016-08-08 20:54:30 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@heroku.com>
at 2016-08-08 19:44:02
Related design discussion -- how tuplesort.c may in the future support partitioning
Cost model for parallel CREATE INDEX
First at 2017-02-28 19:28:15 by Peter Geoghegan <pg at bowt.ie>
Latest at 2017-03-09 21:11:21 by Peter Geoghegan <pg at bowt.ie>
History
When Who What
2017-03-23 02:32:23 Peter Geoghegan (pgeoghegan) Closed in commitfest 2017-03 with status: Returned with feedback
2017-03-20 01:06:55 Peter Geoghegan (pgeoghegan) Added annotation "V10 -- Fixes bug caused by commit 2609e91fc, bitrot" to CAH2-WznqgFJVb0OpbrS445TnyS8ib8kSxAASvLqLCaSnQeyd0Q@mail.gmail.com
2017-03-12 22:11:30 Peter Geoghegan (pgeoghegan) Added annotation "V9 -- Bulletproof resource management, significantly simplified cost model" to CAH2-Wznuf2nwsr6=4n8aRQtqCu3PaFYuvDEoYeEenYK-bXYX4w@mail.gmail.com
2017-03-12 22:06:03 Peter Geoghegan (pgeoghegan) New status: Needs review
2017-02-28 19:32:44 Peter Geoghegan (pgeoghegan) Attached mail thread CAH2-WzmjVMCUviDnUmrJnvhfPpzODtCM71NEHx7_QYCtz+=8ng@mail.gmail.com
2017-02-27 22:35:14 Peter Geoghegan (pgeoghegan) New status: Waiting on Author
2017-02-10 00:12:17 Peter Geoghegan (pgeoghegan) Added annotation "V8 -- rebased on top of logtape.c bugfix, implements BufFile refcount mechanism" to CAH2-WzmWtorLU0qi63dTgNbBJPds1wRLDtoZSDRwkRWdvBnMww@mail.gmail.com
2017-02-01 04:38:04 Michael Paquier (michael-kun) Closed in commitfest 2017-01 with status: Moved to next CF
2017-01-03 23:57:44 Peter Geoghegan (pgeoghegan) Added annotation "V7 -- rebased on top of logtape.c simplifications, pg_restore avoids using parallel CREATE INDEX unless storage parameter was used, refactoring" to CAM3SWZRkQ64nhFiWeGTx4CUuvCaL8m+GSg8PhssHY=E-_wNpBA@mail.gmail.com
2016-12-07 00:25:00 Thomas Munro (macdice) Added macdice as reviewer
2016-12-05 04:26:29 Haribabu Kommi (haribabu) Closed in commitfest 2016-11 with status: Moved to next CF
2016-12-04 01:30:15 Peter Geoghegan (pgeoghegan) Added annotation "V6 -- tuplesort.c now uses condition variables" to CAM3SWZQ0EgCxs2NoEUwB6BACi-pp5qVmp432i+Wh_XyUrg1FbQ@mail.gmail.com
2016-11-08 05:52:54 Peter Geoghegan (pgeoghegan) Deleted annotation ""shift down" merge heap patch committed" from 19f05353-9514-9139-619a-dbaa7a0e5e78@iki.fi
2016-11-08 05:52:37 Peter Geoghegan (pgeoghegan) Deleted annotation "This is a spin-off of the parallel CREATE INDEX stuff, intended to represent 3 patches that are helpful with merging. Merge performance is particularly important for parallel tuplesort, but is independently useful." from CAM3SWZRhBhiknTF_=NjDSnNZ11hx=U_SEYwbc5vd=x7M4mMiCw@mail.gmail.com
2016-11-08 05:52:12 Peter Geoghegan (pgeoghegan) Deleted annotation "Patch has bitrot. New version pending." from CAM3SWZTjULhhHPMYb9KYKD48rs_RgnMQPMZMdf0dqkRpZoY_ZA@mail.gmail.com
2016-11-08 04:35:03 Peter Geoghegan (pgeoghegan) Added annotation "V5 -- Overhaul to cost model, new testing tool" to CAM3SWZSA_+CBsaYyb22cdqwkfNgD9vQL74NVYnbu62ZqhLD4Og@mail.gmail.com
2016-10-27 20:21:59 Peter Geoghegan (pgeoghegan) Changed wikilink to https://wiki.postgresql.org/wiki/Parallel_External_Sort
2016-10-26 20:16:04 Peter Geoghegan (pgeoghegan) Detached mail thread f298f77a-cf06-e70c-d5a4-a20b472b4627@iki.fi
2016-10-26 16:42:26 Peter Geoghegan (pgeoghegan) Added annotation "Do-over of original benchmark from August 1, showing how preloading work from Heikki (and other improvements) have increased scalability of parallel CREATE INDEX" to CAM3SWZSsvig8eZtxCG1L9bZZw+-uWdJ48yxSM41QmNLtJeQSyA@mail.gmail.com
2016-10-25 02:06:25 Peter Geoghegan (pgeoghegan) Added annotation "V4 -- mechanical rebase on top of refactoring commit b75f467b6eec0678452fd8d7f8d306e6df3a1076" to CAM3SWZStcX5DNxRvjGTj4KRvjg+sV2CdxJvL9S4cksbRumVmhA@mail.gmail.com
2016-10-08 00:49:11 Peter Geoghegan (pgeoghegan) Added annotation "V3 -- rebased on top of Heikki's logtape.c preload memory patch" to CAM3SWZTmkOFEiCDpUNaO4n9-1xcmWP-1NXmT7h0Pu3gM2YuHvg@mail.gmail.com
2016-10-03 03:37:07 Michael Paquier (michael-kun) Closed in commitfest 2016-09 with status: Moved to next CF
2016-09-12 04:17:42 Peter Geoghegan (pgeoghegan) Added annotation ""shift down" merge heap patch committed" to 19f05353-9514-9139-619a-dbaa7a0e5e78@iki.fi
2016-09-12 04:10:02 Peter Geoghegan (pgeoghegan) Added annotation "Review" to CAM3SWZQrU12mog=TG9jvNOpeUL5Krk9N1Y8GmsLGTGRBJui-aQ@mail.gmail.com
2016-09-12 04:09:30 Peter Geoghegan (pgeoghegan) Added annotation "Latest version from Heikki -- applies on top master branch on 2016-09-11 (with Peter's 0003-* merge heap patch committed)" to 49817174-c2b6-1378-3d3e-285d270f8fcc@iki.fi
2016-09-12 04:07:11 Peter Geoghegan (pgeoghegan) Added annotation "Heikki's proposed alternative to patch 0002-* from original patch set (batch memory patch)" to f298f77a-cf06-e70c-d5a4-a20b472b4627@iki.fi
2016-09-12 04:01:13 Peter Geoghegan (pgeoghegan) Attached mail thread f298f77a-cf06-e70c-d5a4-a20b472b4627@iki.fi
2016-09-11 18:07:19 Peter Geoghegan (pgeoghegan) Added annotation "V2 -- uses maintenance_work_mem as budget for entire CREATE INDEX, regardless of number of workers" to CAM3SWZTW+3vmZugxhy=_jdm5YxNXGdAEHv5Sz6jBpdyfsJVkRg@mail.gmail.com
2016-08-22 21:50:52 Peter Geoghegan (pgeoghegan) Added annotation "Patch has bitrot. New version pending." to CAM3SWZTjULhhHPMYb9KYKD48rs_RgnMQPMZMdf0dqkRpZoY_ZA@mail.gmail.com
2016-08-16 00:47:06 Peter Geoghegan (pgeoghegan) Added annotation "This is a spin-off of the parallel CREATE INDEX stuff, intended to represent 3 patches that are helpful with merging. Merge performance is particularly important for parallel tuplesort, but is independently useful." to CAM3SWZRhBhiknTF_=NjDSnNZ11hx=U_SEYwbc5vd=x7M4mMiCw@mail.gmail.com
2016-08-08 20:54:30 Peter Geoghegan (pgeoghegan) Added annotation "Related design discussion -- how tuplesort.c may in the future support partitioning" to CAM3SWZR+ATYAzyMT+hm-Bo=1L1smtJbNDtibwBTKtYqS0dYZVg@mail.gmail.com
2016-08-08 20:53:40 Peter Geoghegan (pgeoghegan) Attached mail thread CAM3SWZR+ATYAzyMT+hm-Bo=1L1smtJbNDtibwBTKtYqS0dYZVg@mail.gmail.com
2016-08-01 22:19:31 Peter Geoghegan (pgeoghegan) Changed authors to Peter Geoghegan (pgeoghegan)
2016-08-01 22:19:22 Peter Geoghegan (pgeoghegan) Attached mail thread CAM3SWZQKM=Pzc=CAHzRixKjp2eO5Q0Jg1SoFQqeXFQ647JiwqQ@mail.gmail.com
2016-08-01 22:19:22 Peter Geoghegan (pgeoghegan) Created patch record
Edit