https://bonesmoses.org/tags/partitioning/ Recent content in Partitioning on BonesMoses.org Hugo -- gohugo.io en-us Fri, 21 Oct 2016 11:48:49 +0000 https://bonesmoses.org/2016/pg-phriday-broken-parts/ Fri, 21 Oct 2016 11:48:49 +0000 https://bonesmoses.org/2016/pg-phriday-broken-parts/ Partitioning tables in Postgres can be an extremely risky endeavor. Unfortunately on many larger systems, it’s also essentially a requirement; the maximum size of a Postgres table is 32TB. This isn’t just because converting an existing table to a series of partitions is expensive or time consuming. We must consider how the query planner will react to the partitioned version of a table. There’s also the very real risk we will (or already have) implement flaws in the trigger or constraint logic. https://bonesmoses.org/2016/pg-phriday-perfectly-logical/ Fri, 14 Oct 2016 13:29:06 +0000 https://bonesmoses.org/2016/pg-phriday-perfectly-logical/ One of the things Postgres has been “missing” for a while is logical replication based on activity replay. Until fairly recently, in order to replicate single tables from one database to another, we had to encumber the table with performance-robbing triggers coupled to a third party daemon to manage transport. Those days might finally be behind us thanks to pglogical. But is it easy to use? Let’s give it a try on our trusty sensor_log table. https://bonesmoses.org/2016/pg-phriday-pesky-partition-plans/ Fri, 07 Oct 2016 14:05:51 +0000 https://bonesmoses.org/2016/pg-phriday-pesky-partition-plans/ For all of those warehouse queries that never seem to complete before the heat death of the universe, there’s often a faster version. Sometimes this is due to a fundamental misunderstanding of how queries work, or how Postgres specifically functions. The trick is knowing when to back away slowly from an ugly but efficient query, and when to inject a flurry of predicates to fully illustrate the original intent of the query so the planner makes better decisions. https://bonesmoses.org/2016/pg-phriday-trusty-table-tiers/ Fri, 20 May 2016 11:00:38 +0000 https://bonesmoses.org/2016/pg-phriday-trusty-table-tiers/ I always advocate breaking up large Postgres tables for a few reasons. Beyond query performance concerns, maintaining one monolithic structure is always more time consuming and consequentially more dangerous. The time required to create a dozen small indexes may be slightly longer than a single larger one, but we can treat the smaller indexes as incremental. If we want to rebuild, add more indexes, or fix any corruption, why advocate an all-or-nothing proposition? https://bonesmoses.org/2015/pg-phriday-parallel-o-postgres/ Fri, 16 Oct 2015 11:58:11 +0000 https://bonesmoses.org/2015/pg-phriday-parallel-o-postgres/ I wasn’t able to write an article last week due to an unexpected complication regarding tests I was running to verify its contents. So this week, it’s going to be extra special! Also long. What’s the fastest way to load a Postgres table? If you believe the documentation, the COPY command is the best way to unceremoniously heave data into a table. Fortunately after all of our talk about partitions, our minds are primed and ready to think in chunks. https://bonesmoses.org/2015/pg-phriday-when-partitioning-goes-wrong/ Fri, 02 Oct 2015 16:33:10 +0000 https://bonesmoses.org/2015/pg-phriday-when-partitioning-goes-wrong/ I’ve been talking about partitions a lot recently, and I’ve painted them in a very positive light. Postgres partitions are a great way to distribute data along a logical grouping and work best when data is addressed in a fairly isloated manner. But what happens if we direct a basic query at a partitioned table in such a way that we ignore the allocation scheme? Well, what happens isn’t pretty. Let’s explore in more detail. https://bonesmoses.org/2015/pg-phriday-fancy-partitioning/ Fri, 28 Aug 2015 12:00:23 +0000 https://bonesmoses.org/2015/pg-phriday-fancy-partitioning/ This week we’ll be covering another method of Postgres partitioning. This is a technique I personally prefer and try to use and advocate at every opportunity. It’s designed to straddle the line between traditional partitioning and standard monolithic table structure by using table inheritance as a convenience factor. The assumption here is that end-user applications either: Know that partitioning is in use. Only load “current” data and don’t care about partitions. https://bonesmoses.org/2015/pg-phriday-basic-partitioning/ Fri, 21 Aug 2015 11:32:15 +0000 https://bonesmoses.org/2015/pg-phriday-basic-partitioning/ Most Postgres (PostgreSQL) users who are familiar with partitioning use the method described in the partitioning documentation. This architecture comes in a fairly standard stack: One empty base table for structure. At least one child table that inherits the base design. A trigger to redirect inserts based on the partitioning scheme. A constraint on each child table to enforce the partition scheme, and help the planner exclude child partitions from inapplicable queries. https://bonesmoses.org/2015/pg-phriday-partitioning-candidates/ Fri, 14 Aug 2015 11:36:39 +0000 https://bonesmoses.org/2015/pg-phriday-partitioning-candidates/ What’s a good table to partition? It’s not always a question with an obvious answer. Most often, size and volume determine whether or not a table should be broken into several chunks. However, there’s also cases where business or architecture considerations might use partitioning to preserve a shared table structure, or drive aggregate analysis over a common core. In Postgres (PostgreSQL), this is even more relevant because of how partitioning is handled through inheritance. https://bonesmoses.org/2015/pg-phriday-the-case-for-partitioning/ Fri, 07 Aug 2015 16:37:32 +0000 https://bonesmoses.org/2015/pg-phriday-the-case-for-partitioning/ In the next few weeks, I’m going to be pushing a long discussion regarding Postgres (PostgreSQL) table partitioning. I’ve covered it in previous articles, but only regarding basic performance considerations. That’s a very limited view of what partitioning can offer; there’s a lot more variance and background that deserves elucidation. So for the next few articles, the topic of discussion will be partitioning. There’s not really enough of it, and a lot of the techniques used in the field are effectively pulled straight from the documentation. https://bonesmoses.org/2015/pg-phriday-date-based-partition-constraints/ Fri, 20 Mar 2015 12:02:58 +0000 https://bonesmoses.org/2015/pg-phriday-date-based-partition-constraints/ PostgreSQL has provided table partitions for a long time. In fact, one might say it has always had partitioning. The functionality and performance of table inheritance has increased over the years, and there are innumerable arguments for using it, especially for larger tables consisting of hundreds of millions of rows. So I want to discuss a quirk that often catches developers off guard. In fact, it can render partitioning almost useless or counter-productive.