Database on BonesMoses.org

PG Phriday: Kubernetes Killed the High Availability Star

Fri, 20 Dec 2024 12:26:38 -0600

Postgres Conference Seattle 2024 partnered up with PASS this year to present a united database front. They accepted my “Kubernetes Killed the High Availability Star” talk, which I graciously gave on the last day of the conference. The next talk in that room wasn’t for another hour, so I had plenty of time to talk shop with attendees, about the future of Postgres, high availability, and Kubernetes in general.

If you weren’t there and missed out on the fun, this is your chance to catch up and enjoy a few of my notorious bad puns along the way. Let me tell you why the concept of Postgres HA is dead.

PG Phriday: My Postgres is Rusty

Fri, 09 Aug 2024 12:00:00 +0000

Postgres and Rust go together like peanut butter and chocolate, or ice-cream and root beer, or Batman and Robin, or mice and cheese, or sand on a beach, or crabs and elephants! Err, maybe scratch that last one.

Hmmmmm…

Well regardless, there’s a whole lot of Rust going on in the Postgres world these days, especially thanks to contributions from the PGRX project. As a relative novice to Rust, I figured it was time to see what all the fuss was about and tentatively dip a foot into those turbulent and unforgiving waters.

PG Phriday: Taking Postgres for GRANTed

Fri, 31 May 2024 12:00:00 +0000

Not every database-backed application needs to be locked down like Fort Knox. Sometimes there are even roles that leverage blanket access to large swathes of available data, if not every table, simply for auditing or monitoring purposes. Normally this would require quite a bit of preparation or ongoing privilege management, but Postgres came up with a unique solution starting with version 14: predefined roles. This topic comes up relatively frequently in Postgres chats like Discord, Slack, and IRC.

PG Phriday: On the Move

Fri, 10 May 2024 20:13:36 +0000

Sometimes when we have an existing database and schema full of tables, there comes a time when we need to perform a migration. Maybe it’s because we want to move to or from a cloud service. Perhaps we have a small database and want to do a quick version upgrade via dump / restore. Whatever the reason, we may decide to clean up some technical debt while we’re making the transition.

Many Postgres experts recommend against creating objects in the public schema. This is the default schema that exists in nearly all Postgres databases, and there are often implicit grants that could make our objects available in unexpected scenarios. It’s also a cluttered namespace if all tables, views, functions, etc., are created there by default. Using it is sloppy and makes future data or operation segregation much more difficult.

So how can we move a bunch of existing stuff out of the public schema safely?

PG Phriday: Papa's Got a Brand New RAG

Fri, 26 Apr 2024 12:00:00 +0000

Remember this guy? Robo-Postgres returns! AI is all anyone talks about these days, isn’t it? Even when it comes to databases like Postgres, devs are finding new and innovative ways to leverage LLMs in everyday use cases. Can you really blame them though? This is an exciting new technology that will transform how we work and potentially society as a whole once it finally matures. We even covered building a crude RAG app a few short weeks ago.

PG Phriday: Under Observation

Fri, 12 Apr 2024 16:30:04 +0000

Have you ever wanted to use a non-superuser role in a Postgres database to perform actions that are normally restricted? Even something as simple as reading from the pg_stat_activity view requires special permissions to view the query column because it could contain sensitive information.

PG Phriday: Wanton Animal Cruelty

Thu, 04 Apr 2024 19:11:54 +0000

The last few PG Phriday articles have been somewhat dense content, so how about something a bit more irreverent? Rather than wax on about AI, HA architectures, or conceptual advancements to Postgres clusters, why not write a game instead? To keep things simple, let’s just build a no-frills Tamagotchi virtual pet for bored database professionals. There’s a lot of SQL in this article, so check out the git page for this blog if you want to follow along.

PG Phriday: A Dirty Postgres RAG

Fri, 29 Mar 2024 14:34:28 +0000

Has it really come to this? AI is everywhere these days Postgres and AI go together like elephants and chocolate. At first glance, it seems like a silly combination. Postgres is an RDBMS for storing data with ACID compliance, functions, views, and maybe some extensions or foreign data wrappers. Where is room for AI in that? It may be trendy to take something, rub some AI on it, and then declare it a breakthrough technology, but that isn’t necessarily reality.

PG Phriday: Why Postgres is the Best Database Engine

Fri, 22 Mar 2024 15:57:43 +0000

Last Phriday we explored just where Postgres could end up in the future. One possible question which may have occurred to a reader was probably something along the lines of “That doesn’t even really sound like Postgres anymore. Why not just write another database?” Let’s just be outright about it: Postgres is the best RDBMS engine currently available. It’s certainly bold to claim that any database engine is “the best”, and as the saying goes, “Extraordinary claims require extraordinary evidence.

PG Phriday: Redefining Postgres High Availability

Fri, 15 Mar 2024 01:27:19 +0000

What is High Availability to Postgres? I’ve staked my career on the answer to that question since I first presented an HA stack to Postgres Open in 2012, and I still don’t feel like there’s an acceptable answer. No matter how the HA techniques have advanced since then, there’s always been a nagging suspicion in my mind that something is missing. But I’m here to say that a bit of research has uncovered an approach that many different Postgres cloud vendors appear to be converging upon.

PG Phriday: Getting It Sorted

Fri, 08 Mar 2024 20:36:40 +0000

When it comes to reordering the items in a list, databases have long had a kind of Faustian Bargain to accomplish the task. Nobody really liked any of the more common solutions, least of all the poor database tasked with serving up the inevitable resulting hack.

Postgres is no different in this regard. Consider a list_item table like this, demonstrating five items in a to-do list:

PG Phriday: Community Edition

Fri, 28 Apr 2017 11:50:16 +0000

Postgres is one of those database engines that carves out a niche and garners adherents with various levels of religious zeal. The community, while relatively small when compared to that of something like MongoDB, is helpful almost to a fault. Members from the freshest minted newb to the most battle tested veteran will often trip over themselves to answer questions found in the various dedicated forums, mailing lists, and chat rooms.

PG Phriday: Who Died and Made You Boss?! (The Investigatining!)

Fri, 21 Apr 2017 11:22:43 +0000

The Postgres system catalog is a voluminous tome of intriguing metadata both obvious and stupendously esoteric. When inheriting a Postgres database infrastructure from another DBA, sometimes it falls upon us to dig into the writhing confines to derive a working knowledge of its lurking denizens. The trick is to do this before they burst forth and douse us with the database’s sticky innards and it experiences a horrible untimely demise.

PG Phriday: Design Pattern Workshop

Fri, 31 Mar 2017 12:20:46 +0000

Recently on the pgsql-performance mailing list, a question popped up regarding Postgres RAM usage. In this instance Pietro wondered why Postgres wasn’t using more RAM, and why his process was taking so long. There were a few insightful replies, and they’re each interesting for reasons that aren’t immediately obvious. Let’s see what is really going on here, and perhaps answer a question while we’re at it. Pietro presents several postgresql.conf settings, but here are the ones that matter:

PG Phriday: CONFLICT of Interests

Fri, 24 Mar 2017 12:20:10 +0000

MySQL has had a REPLACE INTO syntax to perform “UPSERT” logic since practically the very beginning. For the longest time, users who wanted to switch to Postgres, but for whatever reason relied on this functionality, were essentially trapped. Postgres 9.5 changed all that, but why did it take so long? As with much of Postgres history, it’s a long story. To really understand where Postgres started, we need to look at the “old” way of handling a row merge.

PG Phriday: RESTing in the Corn

Fri, 17 Mar 2017 11:33:24 +0000

Last week we explored using Postgres as a central communication nexus between several data sources. At the time, I made a fairly hand-wavy allusion to REST interfaces. Since I hadn’t really explored further, I had assumed PLV8 could use core node.js or other similar libraries to invoke HTTP APIs. Of course as a trusted language, PLV8 isn’t allowed to do that. It’s more of a language for easily manipulating JSON and JSONB objects within Postgres.

PG Phriday: Stuck in the Middle with Postgres

Fri, 10 Mar 2017 12:18:41 +0000

Earlier this year, I implied Postgres was some kind of super middleware for dragging data out of every external resource it could locate. But that example only used the Postgres foreign data wrapper to contact another Postgres server. Why be so unimaginative? The future is as unlimited as it is terrifying. Meet the new Postgres mascot Let’s start with a few prerequisites. We want to see the combined functionality of a few different Postgres capabilities, so let’s try and combine data from MySQL, Postgres, and Javascript using V8.

PG Phriday: Through the Window

Fri, 03 Mar 2017 13:40:27 +0000

Now that we know how Postgres window functions work, why not play with them a bit to get a better understanding of their capabilities? So long as we understand window functions are applied after data gathering and aggregation steps, much of their mystery and complexity is defanged. Let’s start actually using them for stuff! Captain Murphy is tired of your nonsense (Note: I’m a bit under the weather today, so this Phriday will probably be a bit truncated and potentially incoherent thanks to the drugs.

PG Phriday: In the Window

Fri, 24 Feb 2017 13:32:35 +0000

I’ll be the first to admit that I found Postgres window functions fantastically confusing when I first encountered them. They’re a powerful and versatile tool for building reports and summaries, but that functionality hides behind a fairly steep learning curve. One of the ways to combat their inherent complexity is to fully explore how they work, instead of just trying to wrap our heads around the expected results. Window doggies have gotten decidedly smug

PG Phriday: Getting Assertive

Fri, 17 Feb 2017 12:57:08 +0000

There are a lot of languages available for authoring Postgres functions, but there’s nothing quite like the the classic PL/pgSQL. It’s SQL! It’s not SQL! It’s a kind of horrifying mutant crossbreed suitable only for terrifying small children and generating complex reports from a cavalcade of dubious sources! And deep within its twisted entrails is an often overlooked feature usually only available in far more mature entities. That’s right, it’s obvious we’re referring to the ASSERT statement.

PG Phriday: Down in the Dumps

Fri, 10 Feb 2017 14:14:28 +0000

These days with multiple Postgres database environments a commonality, it’s not unheard of to copy data from one to another. Perhaps a production extract is necessary to properly vet data in a staging environment. Maybe the development environment needs to update its data to reflect an especially pernicious and intractable edge case someone observed. In any of these scenarios, we are likely to extract data from multiple tables to import it elsewhere.

PG Phriday: Getting Back Up

Fri, 03 Feb 2017 12:16:10 +0000

In light of recent events where GitLab suffered a massive database loss, this is a great opportunity to examine what happened from a Postgres perspective. Since Simon Riggs over at 2ndQuadrant has already chimed in on improvements Gitlib might consider in their procedures, maybe we should walk the conversation back slightly. This isn’t the first time Postgres backup tooling has been misused or misunderstood. The topic of backups hits forums and mailing lists rather frequently.

PG Phriday: Alien Incursion

Fri, 27 Jan 2017 12:17:13 +0000

Foreign tables have been a headline feature of Postgres ever since the release of version 9.2. Combined with extensions, they’re the secret sauce that allows Postgres to pull data from other database engines, flat files, REST interfaces, and possibly every gas station, residence, warehouse, farmhouse, hen house, outhouse, and doghouse in the area. But that kind of power comes at a significant cost. Since the remote data comes from what is essentially a black box, there are a lot of performance optimizations Postgres can’t apply.

PG Phriday: Everything in Common

Fri, 20 Jan 2017 14:06:32 +0000

Not a lot of people remember what Postgres was like before version 8.4. In many ways, this was the first “modern” release of the database engine. CTEs, Window Functions, column level permissions, in-place upgrade compatible with subsequent versions, collation support, continuous query statistic collection; it was just a smorgasbord of functionality. Of these, CTEs or Common Table Expressions, probably enjoy the most user-level exposure; for good reason. Before this, there was no way to perform a recursive query in Postgres, which really hurts in certain situations.

PG Phriday: Why Postgres

Fri, 13 Jan 2017 14:59:00 +0000

There are a smorgasbord of database engines out there. From an outside perspective, Postgres is just another on a steadily growing pile of structured data storage mechanisms. Similarly to programming languages like Rust and Go, it’s the new and shiny database systems like MongoDB that tend to garner the most attention. On the other hand, more established engines like Oracle or MySQL have a vastly larger lead that seems insurmountable. In either case, enthusiasm and support is likely to be better represented in exciting or established installations.

PG Phriday: Who Died and Made You Boss?!

Fri, 16 Dec 2016 11:18:02 +0000

Postgres is great, but it can’t run itself in all cases. Things come up. Queries go awry. Hardware fails, and users leave transactions open for interminable lengths of time. What happens if one of these things occur while the DBA themselves has a hardware fault? While they’re down for maintenance, someone still has to keep an eye on things. For the last PG Phriday of the year completely unrelated to my upcoming surgery, let’s talk about what happens when your DBA becomes inoperative due to medical complications.

PG Phriday: Planner Pitfalls

Fri, 09 Dec 2016 12:07:14 +0000

Recently a coworker asked me this question: Should I expect variance between minutes and hours for the same query? And I was forced to give him this answer: Potentially, but not commonly. Query planning is an inexact science, and regardless of the query being the “same query,” the data is not the “same data.” This isn’t generally the case, but on occasion, changes in data can affect the query execution path.

PG Phriday: Ambling Architecture

Fri, 02 Dec 2016 14:26:35 +0000

It’s about the time for year-end performance reviews. While I’m always afraid I’ll narrowly avoid being fired for gross incompetence, that’s not usually how it goes. But that meeting did remind me about a bit of restructuring I plan to impose for 2017 that should vastly improve database availability across our organization. Many of the techniques to accomplish that—while Postgres tools in our case—are not Postgres-specific concepts. Much of database fabric design comes down to compromise.

PG Phriday: Extended Elections

Fri, 25 Nov 2016 11:33:27 +0000

One of the best features Postgres boasts is the ability to adapt. Any schmo off the street can write an extension and bolt it onto Postgres with nary a second glance. As proof, I’m going to whip one up really quick. That should be enough to convince anyone that it takes no skill at all to add functionality to Postgres. Just so our extension actually does something, let’s start off with the instant-runoff code we wrote a few weeks ago.

PG Phriday: Primal Planner Prep

Fri, 18 Nov 2016 14:22:16 +0000

The Postgres query planner is house of cards built upon the ever-shifting sand of our data. It has the utterly impossible mission of converting our ridiculous and inane requests into a logical series of fetch, filter, sort, join, and other instructions. Then the resulting steps must be ruthlessly efficient or the execution phase could very well saturate every hardware resource available; Set Theory isn’t very forgiving. Forewarned is forearmed is very apt when applied to database query planners.

PG Phriday: Instant Runoff Through SQL

Fri, 11 Nov 2016 14:47:41 +0000

The United States held an election recently, and there has been some … mild controversy regarding the results. Many raised issues about this before the election itself, but what if we had used instant-runoff voting instead? More importantly, can we implement it with Postgres? Well, the answer to the last question is a strong affirmative. So long as we don’t break the results down into voting districts, and make wild unsupported assumptions regarding rankings, that is.

PG Phriday: MySQL Mingle

Fri, 28 Oct 2016 14:05:13 +0000

Through the wonderful magic of corporate agreements, I’ve been pulled back into (hopefully temporarily) managing a small army of MySQL servers. No! Why can’t this just be a terrible nightmare?! Does anyone deserve such debasement? Side effects of using MySQL may include… Hyperbole? Maybe a little. If MySQL was really that terrible, it wouldn’t be in such widespread use. However, as a Postgres DBA for so many years, I’ve come to appreciate what really sets it apart from engines and development approaches like those showcased in MySQL.

PG Phriday: Broken Parts

Fri, 21 Oct 2016 11:48:49 +0000

Partitioning tables in Postgres can be an extremely risky endeavor. Unfortunately on many larger systems, it’s also essentially a requirement; the maximum size of a Postgres table is 32TB. This isn’t just because converting an existing table to a series of partitions is expensive or time consuming. We must consider how the query planner will react to the partitioned version of a table. There’s also the very real risk we will (or already have) implement flaws in the trigger or constraint logic.

PG Phriday: Perfectly Logical

Fri, 14 Oct 2016 13:29:06 +0000

One of the things Postgres has been “missing” for a while is logical replication based on activity replay. Until fairly recently, in order to replicate single tables from one database to another, we had to encumber the table with performance-robbing triggers coupled to a third party daemon to manage transport. Those days might finally be behind us thanks to pglogical. But is it easy to use? Let’s give it a try on our trusty sensor_log table.

PG Phriday: Pesky Partition Plans

Fri, 07 Oct 2016 14:05:51 +0000

For all of those warehouse queries that never seem to complete before the heat death of the universe, there’s often a faster version. Sometimes this is due to a fundamental misunderstanding of how queries work, or how Postgres specifically functions. The trick is knowing when to back away slowly from an ugly but efficient query, and when to inject a flurry of predicates to fully illustrate the original intent of the query so the planner makes better decisions.

PG Phriday: Postgres 9.6 Pluses

Fri, 30 Sep 2016 15:00:55 +0000

Timing can often be extremely fortuitous. Yesterday marked the official release of Postgres 9.6! Yaaaaayy… I’ve covered 9.6 previously, but that was a beta and clearly doesn’t count. Besides, while the beta was undoubtedly high quality, the frequency of patch turnover is enough to produce a significantly different final release. So let’s skim through the release notes a bit for stuff that really stands out or seems different from last time.

PG Phriday: Bodacious Benchmarks

Fri, 23 Sep 2016 14:56:45 +0000

When it comes to putting Postgres through its paces, we often turn to benchmarks to absolutely bury it under a torrent of oppressive activity. It’s a great way to obtain maximum performance metrics and also observe how Postgres reacts and breaks down under such pressure. But these kinds of tests aren’t really practical, are they? After all, many such simulated workloads are nothing but bragging rights measured against previous Postgres releases, or for hardware comparisons.

PG Phriday: Working Together

Fri, 16 Sep 2016 16:00:49 +0000

There seem to be quite a few popular Postgres conferences peppering the globe these days. This year, Simon Riggs of 2ndQuadrant gave the sponsored keynote at Postgres Open. I’m not entirely sure it was intentional since it wasn’t the title of his presentation, but he uttered the words “working together to make Postgres better for everyone” at one point. The phrase “Working Together” really stood out, because that’s a significant part of what makes Postgres so great.

PG Phriday: Irrelevant Inclinations

Fri, 09 Sep 2016 11:32:15 +0000

Say hi to Princess Kittybutt. She’ll be our mascot (and subject) for today. We’ll get to her in a minute. When the only tool you have is a hammer, everything becomes a nail, right? With Postgres becoming more of an environment than simply a database engine, this colloquialism is starting to resemble reality. Of course, that’s not necessarily a bad thing! As Postgres accumulates copious and varied extensions, its role as an adaptive middleware solidifies.

PG Phriday: Cult of Functionality

Fri, 02 Sep 2016 14:00:25 +0000

It’s no surprise Postgres does more than merely store data; you can’t do everything with SQL. Often, it’s often more beneficial to process data locally without transmitting it to a client interface. Local manipulation can save hours in network traffic alone, let alone client-side allocation and per-row processing. Databases like Postgres are specifically for bulk data operations, so why not take advantage? But doing that requires functions—anonymous or otherwise—and a Turing-complete language to write them with.

PG Phriday: Forensic Fundamentals

Fri, 19 Aug 2016 13:42:45 +0000

All database engines, even Postgres, occasionally present a seemingly intractable problem that will drive everyone insane while attempting to isolate it. All it takes is the perfect storm of situational circumstances, and even a perfectly running stack of software will grind to a screeching halt. It’s situations like this that we must turn to various Postgres forensic tools to track down the issue before management starts firing people in frustration.

PG Phriday: Inevitable Interdiction

Fri, 12 Aug 2016 11:48:24 +0000

“Hey! That row shouldn’t be in that table! How the heck did that get there!? Alright, who wrote the application client filters, because you’re fired!” Good application developers know never to trust client input, but not all realize that a single app is rarely the only vector into a database. Databases don’t just preserve data with various levels of paranoia, they’re also the central nexus of a constellation of apps, scripts, APIs, GUIs, BMIs, HMOs, and STDs.

PG Phriday: Elephantary, My Dear

Fri, 05 Aug 2016 12:38:19 +0000

Occasionally with a lot of similar-sounding terminology, there’s ample room for misunderstandings. This is especially relevant with overloaded terms like ‘index’, which can be anything from a data lookup to a term used in mutual funds. This is further complicated if a developer’s first experience with databases is with another variant with sufficiently divergent syntax, methodology, or assumptions. To prevent future code refactors born of misunderstandings, let’s build a basic Postgres glossary to act as an executive summary for app devs.

PG Phriday: Constipated Connections

Fri, 29 Jul 2016 11:53:49 +0000

We’ve all had it happen. One day, we run a query or function and we wait for the result. And we wait. And we wait. Eventually, we realize something is wrong and find a DBA and yell at them. “Hey Postgres dude!” we scream. “The database is slow!” Or maybe we can’t even get a connection. Postgres just keeps saying something about too many clients. This application isn’t launching and there are ten managers breathing down our neck and we panic.

PG Phriday: The Audacity of NoSQL

Fri, 22 Jul 2016 12:55:44 +0000

The pure, unadulterated, presumptuous impudence of NoSQL. Engines like MongoDB recklessly discard concepts like ACID in some futile quest to achieve “web scale”, and end up accomplishing neither. RDBMS systems have literally decades of history to draw upon, and have long since conquered the pitfalls NoSQL platforms are just now encountering. There may be something to a couple of them, but by and large, they’re nothing we really need. At least, that’s something I might have said a couple of weeks ago.

PG Phriday: A Postgres Perspective on MongoDB

Fri, 15 Jul 2016 15:06:24 +0000

I’ve been almost exclusively a Postgres DBA for a seemingly interminable length of time. While this is good for specializing, nobody wants to be a One-Trick Pony. And aside from the occasional bit of Python to write more advanced tools when Bash isn’t up to the job, it’s All Postgres All The Time. While few things would make me happier, it pays to branch out occasionally. When NoSQL databases hit the scene a few years ago, I pretty much ignored them wholesale.

PG Phriday: All in a Name

Fri, 08 Jul 2016 16:29:40 +0000

Naming objects in a database can sometimes be an exercise in frustration. What kind of guidelines should we follow for schemas and tables? What about columns or variables? Should the same rules apply to indexes, constraints, and sequences? Functions and triggers are much different than all of those elements, yet still exist within the same namespace. Then we have to contend with Postgres reserved words, many of which are probably only obvious to someone well versed in database lingo.

PG Phriday: EXCEPTIONal Performance

Fri, 01 Jul 2016 11:41:45 +0000

Like any programming language, the PL/pgSQL Postgres procedural language has plenty of handy control structures. Among those thankfully modern accoutrements is the humble EXCEPTION block. While not the more prevalent try/catch methodology, we can use BEGIN anywhere to start an embedded section for the same effect. Knowing this is the case, what are the costs of actually using them? Postgres is fairly streamlined, and as such, can take several shortcuts when there are no exception blocks within a chunk of PL/pgSQL.

PG Phriday: Let There Be Jank

Fri, 17 Jun 2016 13:26:37 +0000

One way the Postgres project is subtly misleading, is that it becomes easy to forget that not all other projects are nearly as well managed. This becomes more relevant when delving into niches that lack sufficient visibility to expose the more obvious deficiencies. As much as we like Postgres, it’s not quite as popular as it could be. This makes some of the side projects infrequently used, and as a direct consequence, they can often resemble jerky automatons cobbled together out of spit and bailing wire.

PG Phriday: Moving to 9.5

Fri, 10 Jun 2016 14:01:15 +0000

There comes a day in every young database’s life that it’s time to move on. I’m sorry 9.4, but the day has come that we must say goodbye. It’s not like we haven’t had our good times. While I truly appreciate everything you’ve done for me, we must part ways. I’m far too needy, and I can’t demand so much of you in good conscience. May your future patches make you and your other suitors happy!

PG Phriday: Rapid Prototyping

Fri, 03 Jun 2016 14:16:22 +0000

Ah, source control. From Subversion to git and everything in between, we all love to manage our code. The ability to quickly branch from an existing base is incredibly important to exploring and potentially abandoning divergent code paths. One often overlooked Postgres feature is the template database. At first glance, it’s just a way to ensure newly created databases contain some base functionality without having to bootstrap every time, but it’s so much more than that.

PG Phriday: Converting to Horizontal Distribution

Fri, 27 May 2016 13:28:04 +0000

Now that we’ve decided to really start embracing horizontal scaling builds, there is a critically important engine-agnostic element we need to examine. Given an existing table, how exactly should we split up the contents across our various nodes during the conversion process? Generally this is done by selecting a specific column and applying some kind of hash or custom distribution mechanism to ensure all node contents are reasonably balanced. But how do we go about figuring that out?

PG Phriday: Trusty Table Tiers

Fri, 20 May 2016 11:00:38 +0000

I always advocate breaking up large Postgres tables for a few reasons. Beyond query performance concerns, maintaining one monolithic structure is always more time consuming and consequentially more dangerous. The time required to create a dozen small indexes may be slightly longer than a single larger one, but we can treat the smaller indexes as incremental. If we want to rebuild, add more indexes, or fix any corruption, why advocate an all-or-nothing proposition?

PG Phriday: Bountiful Beta Benefits

Fri, 13 May 2016 15:41:52 +0000

The Postgres developers recently announced the availability of the first public beta for Postgres 9.6. I would be highly remiss to ignore such an opportunity to dig into any interesting functionality listed in the 9.6 release notes. All in all, it’s a pretty exciting series of advancements, and assuming this is a glimpse of what we see when 9.6 drops, I’d say we’re on the right track. Plentiful Parallelism Probably the most high-profile addition for 9.

PG Phriday: Big Data is Hard

Fri, 06 May 2016 12:43:43 +0000

Let’s just get the obvious out of the way early: dealing with multiple Terabytes or Petabytes in a database context is something of a nightmare. Distributing it, retrieving it, processing it, aggregating and reporting on it, are all complicated—and perhaps worst of all—non-intuitive. Everything from tooling and maintenance, to usage and input, are either ad-hoc or obfuscated by several special-purpose APIs and wrappers. One of the reasons a self-scaling database is such a killer app, derives from the failure rate from having so many moving parts.

PG Phriday: Derivation Deluge

Fri, 29 Apr 2016 14:46:14 +0000

Having run into a bit of a snag with Postgres-XL, and not wanting to be dead in the water with our project, I went on a bit of a knowledge quest. Database scaling is hard, so I expected a bunch of either abandoned or proprietary approaches. In addition, as a huge fans of Postgres, compatibility or outright use of the Postgres core was a strict prerequisite. So, what options are out there?

PG Phriday: Growing Pains

Fri, 22 Apr 2016 14:36:18 +0000

Postgres is a great tool for most databases. Larger installations however, pretty much require horizontal scaling; addressing multi-TB tables relies on multiple parallel storage streams thanks to the laws of physics. It’s how all immense data stores work, and for a long time, Postgres really had no equivalent that wasn’t a home-grown shard management wrapper. To that end, we’ve been considering Postgres-XL as a way to fill that role. At first, everything was going well.

PG Phriday: Postgres Password Practices

Fri, 15 Apr 2016 12:49:59 +0000

Connecting to a Postgres database can be a headache for end users and DBAs alike. Not because of the work involved, but the general irritation of managing passwords—an issue common to any system that requires authentication controls. The user wants to say, “Who am I? None of your business!” While the database is more comfortable with, “Go away, before I taunt you a second time!” Well, there’s some middle ground everyone can enjoy, and a few Postgres-specific caveats which add a bit of flavor to the whole interaction.

PG Phriday: JOIN the Club

Fri, 08 Apr 2016 12:13:14 +0000

I’ve been going on and on about esoteric Postgres features for so long, sometimes I forget my own neophyte beginnings. So let’s back way off and talk about how JOINs work. What are they? What do they do? Why would you use one? What can go wrong? It’s easier than you might imagine, but there are definitely some subtleties to account for. To make this more relatable, let’s start with a club of pet owners that likes to keep track of everything:

PG Phriday: 5 Reasons Postgres Sucks! (You Won't Believe Number 3!)

Fri, 01 Apr 2016 12:09:56 +0000

I’ve been a Postgres DBA since 2005. After all that time, I’ve come to a conclusion that I’m embarrassed I didn’t reach much earlier: Postgres is awful. This isn’t a “straw that broke the camel’s back” kind of situation; there is a litany of ridiculous idiocy in the project that’s, frankly, more than enough to stave off any DBA, end user, or developer. But I’ll limit my list to five, because clickbait.

PG Phriday: Mining for Metadata

Fri, 18 Mar 2016 12:26:02 +0000

Every good database engine has a system catalog that describes the myriad of structures that model and preserve our data. Of course this is expected, as it would be somewhat silly for a database system not to use tables to represent its internal mechanisms. But that doesn’t mean they have to be humanly readable, or even make sense without a series of views or esoteric functions to decipher them. The information_schema standard serves a necessary role in that regard, and the Postgres implementation is extremely comprehensive.

PG Phriday: Secret of the Ooze

Fri, 11 Mar 2016 11:54:36 +0000

A few days ago, a developer came to me with that inevitable scenario that every DBA secretly dreads: a need for a dynamic table structure. After I’d finished dying inside, I explained the various architectures that could give him what he needed, and then I excused myself to another room so I could weep silently without disturbing my coworkers. But was it really that bad? Databases have come a long way since the Bad Old Days when there were really only two viable approaches to table polymorphism.

PG Phriday: Being A Tattletale

Fri, 04 Mar 2016 13:26:22 +0000

In a heterogeneous database environment, it’s not uncommon for object creation and modification to occur haphazardly. Unless permissions are locked down to prevent it, users and applications will create tables, modify views, or otherwise invoke DDL without the DBA’s knowledge. Or perhaps permissions are exceptionally draconian, yet they’ve been circumvented or a superuser account has gone rogue. Maybe we just need to audit database modifications to fulfill oversight obligations. Whatever the reason, Postgres has it covered with event triggers.

PG Phriday: Corralling the Hordes

Fri, 26 Feb 2016 12:28:10 +0000

Ah, users. They log in, query things, accidentally delete critical data, and drop tables for giggles. Bane or boon, user accounts are a necessary evil in all databases for obvious reasons. And what would any stash of data be if nobody had access? Someone needs to own the objects, at the very least. So how can we be responsible with access vectors while hosting a Postgres database? We already covered automating grants, so let’s progress to the next step: building a “best practice” access stack.

PG Phriday: Replacing Business Logic

Fri, 19 Feb 2016 14:01:53 +0000

Back in 2005, I started a new job with a company that did work for other businesses. Their data model was designed by developers and they had no DBA, as is pretty common in smaller organizations. A critical part of our main application relied on an event log that captured customer activity and relayed their final decision to the client for reconciliation. One day someone noticed that there was a bug in the system that resolved the final decision from the event stack, and panic ensued.

PG Phriday: Database and Schema Basics

Fri, 12 Feb 2016 12:58:01 +0000

Sure, managing a Postgres cluster is great. We’ve got the software installed, more tool wrappers than we can wave several sticks at, and a running cluster that’s ready to do our bidding. But what do we do with it? In fact, what is a Postgres cluster anyway? How are databases involved? What about schemas? There’s critically important nomenclature that new users may find somewhat non-intuitive, so let’s demystify things a bit.

PG Phriday: Tackling Intractable ACLs

Fri, 05 Feb 2016 11:11:35 +0000

Say that three times fast! Joking aside, managing database object access is a headache for users and DBAs alike. Postgres isn’t really an exception in that regard, but it does provide a few tools to greatly reduce the pain involved. The crazy thing is that few people even know this feature exists. I’ve known about it for a while myself, but it always slips my mind because it feels so wrong not to explicitly grant permissions.

PG Phriday: COPY and Alternative Import Methods

Fri, 29 Jan 2016 12:16:11 +0000

I recently noted that the COPY command in Postgres doesn’t have syntax to skip columns in source data when importing it into a table. This necessitates using one or more junk columns to capture data we’ll just be throwing away. During that, I completely forgot that friendly devs had contributed alternative file handling methods as Foreign Data Wrappers. Most people think of foreign wrappers as a method for interacting with remote databases.

PG Phriday: Joining the Big Leagues

Fri, 22 Jan 2016 11:05:49 +0000

I’ve maintained since about 2011, that the problem with scaling Postgres to truly immense installations could be solved by a query coordinator. Why? Most sharding systems utilize an application-level distribution mechanism, which may or may not leverage an inherent hashing algorithm. This means each Postgres instance can be treated independently of all the others if the distribution process is known. On a cleverly architected system, the application is algorithm aware, and can address individual shards through a driver proxy or accessor class.

PG Phriday: Database Creation Workshop

Fri, 15 Jan 2016 13:15:52 +0000

Postgres theory, feature discussion, and advocacy are fun. But even I’ll admit it’s nice to have some practical application every once in a while. This week, we’re going to build an actual database. But what would be small enough for a proof of concept, yet somewhat interesting? Well, I’m a fan of Hearthstone. It’s a silly card game much like Magic: The Gathering, but has the distinct aura of “eSports!” Regardless, it’s a fun little time waster, and has a few hundred data points we can manipulate.

PG Phriday: How Far We've Come

Fri, 08 Jan 2016 12:10:02 +0000

With extremely fortuitous timing for my first article following the holidays, Postgres 9.5 was officially been released into the wild just yesterday. I tend to think about past releases when new versions come out, and consider everything that has changed since the early days. How early? I’ve personally been using Postgres since 2001, when my new employer bellyached about their Postgres 6.5 database crashing frequently and generally making their lives more difficult.

PG Phriday: JSON and JSONB Revisited

Fri, 18 Dec 2015 12:25:36 +0000

With Postgres 9.5 on the horizon, I figured it’s a good time to see how things have advanced since my last dive into that particular ocean. This is probably particularly relevant since even MongoDB, a JSON-driven NoSQL database, is now partially powered by Postgres. A lot of people found that particular revelation quite shocking, but maybe they shouldn’t, given the advancements embedded within the last couple of Postgres releases. As it turns out, there are quite a few advancements that really make JSONB a powerful addition to Postgres.

PG Phriday: Postgres-XL and Horizontal Scaling

Fri, 11 Dec 2015 15:51:07 +0000

With all of the upheaval in the Postgres world thanks to advancements in extensions, foreign data wrappers, and background workers, it’s getting pretty difficult to keep track of everything! One of these rapidly moving targets is Postgres-XL and its role in helping Postgres scale outward. Large warehouses have a critical need for horizontal scaling, as the very laws of physics make it effectively impossible to perform aggregate queries on tables consisting of several billion rows.

PG Phriday: Displaced Durability

Fri, 04 Dec 2015 11:41:49 +0000

A lot of DBAs are quite adamant regarding ACID compliance. I count myself among them. But unlike the other parts of the acronym, there are times when data durability isn’t actually a high priority. Data staging holding areas, temporary tables that need visibility across sessions, and other transient information do not require zealous protection. As a DBA it feels weird saying it, but there’s just some data we simply don’t care about losing.

PG Phriday: Cluster Control

Fri, 20 Nov 2015 10:45:06 +0000

It has occurred to me that I may have been spending a bit too much time being excited about new Postgres features and developments in the community. One of the intents of this weekly article was for educational purposes, so this week, let’s get back to basics. To that end, the topic for this week boils down to the tools available for managing Postgres instances, and how to use them. Surprisingly, it’s not as straight-forward as you might think.

PG Phriday: Parallel Sequence Scans

Fri, 13 Nov 2015 11:49:43 +0000

A couple days ago, Robert Haas announced that he checked in the first iteration of parallel sequence scans in the Postgres 9.6 branch. And no, that’s not a typo. One of the great things about the Postgres devs is that they have a very regimented system of feature freezes to help ensure timely releases. Thus even though 9.5 just released its second beta, they’re already working on 9.6. So what is a sequence scan, and why does this matter?

PG Phriday: Sidewinder

Fri, 06 Nov 2015 14:18:00 +0000

Maintaining a Postgres database can involve a lot of busywork. This is especially true for more robust architectures that allocate at least one replica for failover purposes. It’s still fairly common for a DBA to create a replica to accommodate emergency or upgrade scenarios, only to have to repeat the process when it came time to revert to the original master system. It’s not safe to simply subscribe the original primary to the newly promoted secondary, so this leaves either creating a new clone, or using rsync to synchronize all of the files first.

PG Phriday: Massively Distributed Operation

Fri, 30 Oct 2015 12:12:46 +0000

Postgres has been lacking something for quite a while, and more than a few people have attempted to alleviate the missing functionality multiple times. I’m speaking of course, about parallel queries. There are several reasons for this, and among them include various distribution and sharding needs for large data sets. When tables start to reach hundreds of millions, or even billions of rows, even high cardinality indexes produce results very slowly.

PG Phriday: Parallel-O-Postgres

Fri, 16 Oct 2015 11:58:11 +0000

I wasn’t able to write an article last week due to an unexpected complication regarding tests I was running to verify its contents. So this week, it’s going to be extra special! Also long. What’s the fastest way to load a Postgres table? If you believe the documentation, the COPY command is the best way to unceremoniously heave data into a table. Fortunately after all of our talk about partitions, our minds are primed and ready to think in chunks.

PG Phriday: When Partitioning Goes Wrong

Fri, 02 Oct 2015 16:33:10 +0000

I’ve been talking about partitions a lot recently, and I’ve painted them in a very positive light. Postgres partitions are a great way to distribute data along a logical grouping and work best when data is addressed in a fairly isloated manner. But what happens if we direct a basic query at a partitioned table in such a way that we ignore the allocation scheme? Well, what happens isn’t pretty. Let’s explore in more detail.

PG Phriday: Database Infrastructure

Fri, 25 Sep 2015 12:07:41 +0000

This PG Phriday is going to be a bit different. During my trip to Postgres Open this year, I attended a talk I had originally written off as “some Red Hat stuff.” But I saw the word “containers” in the PostgreSQL in Containers at Scale talk and became intrigued. A few days later, I had something of an epiphany: I’ve been wrong about servers for years; we all have. That’s a pretty bold claim, so it needs some background.

PG Phriday: The Bones of High Availability

Fri, 18 Sep 2015 18:07:53 +0000

Well, the bell has tolled, the day is over, and at the end of it all, Postgres Open has ended its fifth year in service of the community. I will say it was certainly an honor to speak again this year, though now that it’s not conveniently in Chicago, I’ll have to work harder to justify hauling myself across the country next year. Of course at this point, I’d feel guilty if I didn’t at least try, assuming any of my submissions are accepted.

PG Phriday: Dealing With Table Bloating

Fri, 11 Sep 2015 11:51:47 +0000

Most Postgres operators and informed users are aware that it uses MVCC for storage. One of the main drawbacks of this versioning mechanism is related to tuple reuse. In order to reuse the space, VACUUM must complete a cycle on the table. Unfortunately this isn’t always possible to “optimize” for larger tables. How so? If a large table needs to have a calculated column added, or some other bulk query updates a large portion of its content, a large fragment of the table is now empty space.

PG Phriday: Postgres as Middleware

Fri, 04 Sep 2015 12:32:39 +0000

One of the cool things I like most about Postgres, is that it’s probably the most inclusive database software I’ve ever encountered. It’s so full of features and functionality these days, it’s practically middleware. Almost anything plugs into it, and if it doesn’t, there’s usually a way to make it happen. Want a demonstration? SciDB is often used for large analytical data warehouses. They even use Postgres for metadata storage. Despite this, they still haven’t written a foreign data wrapper for back-and-forth interaction.

PG Phriday: Fancy Partitioning

Fri, 28 Aug 2015 12:00:23 +0000

This week we’ll be covering another method of Postgres partitioning. This is a technique I personally prefer and try to use and advocate at every opportunity. It’s designed to straddle the line between traditional partitioning and standard monolithic table structure by using table inheritance as a convenience factor. The assumption here is that end-user applications either: Know that partitioning is in use. Only load “current” data and don’t care about partitions.

PG Phriday: Basic Partitioning

Fri, 21 Aug 2015 11:32:15 +0000

Most Postgres (PostgreSQL) users who are familiar with partitioning use the method described in the partitioning documentation. This architecture comes in a fairly standard stack: One empty base table for structure. At least one child table that inherits the base design. A trigger to redirect inserts based on the partitioning scheme. A constraint on each child table to enforce the partition scheme, and help the planner exclude child partitions from inapplicable queries.

PG Phriday: Partitioning Candidates

Fri, 14 Aug 2015 11:36:39 +0000

What’s a good table to partition? It’s not always a question with an obvious answer. Most often, size and volume determine whether or not a table should be broken into several chunks. However, there’s also cases where business or architecture considerations might use partitioning to preserve a shared table structure, or drive aggregate analysis over a common core. In Postgres (PostgreSQL), this is even more relevant because of how partitioning is handled through inheritance.

PG Phriday: The Case for Partitioning

Fri, 07 Aug 2015 16:37:32 +0000

In the next few weeks, I’m going to be pushing a long discussion regarding Postgres (PostgreSQL) table partitioning. I’ve covered it in previous articles, but only regarding basic performance considerations. That’s a very limited view of what partitioning can offer; there’s a lot more variance and background that deserves elucidation. So for the next few articles, the topic of discussion will be partitioning. There’s not really enough of it, and a lot of the techniques used in the field are effectively pulled straight from the documentation.

PG Phriday: 10 Ways to Ruin Performance: Sex Offenders

Fri, 24 Jul 2015 12:20:16 +0000

We’re finally at the end of the 10-part Postgres (PostgreSQL) performance series I use to initiate new developers into the database world. To that end, we’re going to discuss something that affects everyone at one point or another: index criteria. Or to put it another way: Why isn’t the database using an index? It’s a fairly innocuous question, but one that may have a surprising answer: the index was created using erroneous assumptions.

PG Phriday: 10 Ways to Ruin Performance: Indexing the World

Fri, 17 Jul 2015 11:42:28 +0000

An easy way to give Postgres (PostgreSQL) a performance boost is to judiciously use indexes based on queries observed in the system. For most situations, this is as simple as indexing columns that are referenced frequently in WHERE clauses. Postgres is one of the few database engines that takes this idea even further with partial indexes. Unfortunately as a consequence of insufficient exposure, most DBAs and users are unfamiliar with this extremely powerful functionality.

PG Phriday: 10 Ways to Ruin Performance: MAXimized Value

Fri, 10 Jul 2015 12:46:10 +0000

I apologize for putting this series on a short hiatus last week for the 4th of July. But worry not, for this week is something special for all the developers out there! I’m going to try to make your life easier for a change. Screw the database! As a Postgres (PostgreSQL) DBA, it’s easy to get tied up in performance hints, obscure syntax, and mangled queries, but it’s really all about the people.

PG Phriday: 10 Ways to Ruin Performance: In The Loop

Fri, 26 Jun 2015 11:18:29 +0000

As a database, Postgres (PostgreSQL) is fairly standard in its use of SQL. Developers of all colors however, might have trouble switching gears and thinking in set operations, since so many language constructs focus on conditionals and looping. Last week in the performance pitfalls series, we discussed a bit of Set Theory, and how ignorance of its implications can be disastrous. But what about the more mundane? What happens, for instance, when we treat a database like a programming language?

PGCon 2015 Unconference: A Community

Tue, 23 Jun 2015 15:30:24 +0000

Well, I’ve returned from PGCon 2015 in Canada, and after a couple days to decompress, it’s time to share. I wrote about the PGCon 2014 unconference after returning to Chicago last year, so I felt it was only fitting that I start there. I feel as strongly now as I did a year ago, that directly interacting with the PostgreSQL maintainers at this level helps the community thrive. Even though PGCon is generally a more developer-focused conference, being able to brainstorm with the bigwigs, even if nothing comes of it, means the ideas have been given a fair shake.

PG Phriday: 10 Ways to Ruin Performance: Functionally Bankrupt

Fri, 19 Jun 2015 12:01:23 +0000

Functions are great. Having cut my teeth on a database that didn’t even provide the ability to define functions, I’ve come to almost take them for granted in Postgres (PostgreSQL). However, with this kind of ubiquity, sometimes they can be overused in ways that don’t seem to be part of the common programmer lexicon. In this week’s PG Phriday series on performance-killing missteps, I’m going to talk a bit about set theory, and how a certain amount of familiarity is necessary to properly interact with a database.

PG Phriday: 10 Ways to Ruin Performance: Out Of Order

Fri, 12 Jun 2015 12:14:34 +0000

There are a lot of database engines out there. As such, a developer or DBA will naturally have varying levels of experience with each, and some of this might conflict with how Postgres (PostgreSQL) operates. These kinds of embedded misunderstandings can cause potential issues by themselves, but in this particular case, corrective action is fairly simple. So this week, I’d like to talk about indexes. Many people treat them as a “make query faster” button, and this often results in egregious misuse.

PG Phriday: 10 Ways to Ruin Performance: Cast Away

Fri, 05 Jun 2015 12:31:32 +0000

Unlike a lot of programming languages that have loose typing, databases are extremely serious about data types. So serious in fact, many patently refuse to perform automatic type conversions in case the data being compared is not exactly equivalent afterwards. This is the case with Postgres (PostgreSQL) and surprisingly often, queries will suffer as a result. Fortunately this is something we can address! There are easier ways to demonstrate this, but in the interests of making this more of an adventure, let’s use a join between two tables:

PG Phriday: 10 Ways to Ruin Performance: IN-Sanity

Fri, 29 May 2015 12:17:32 +0000

When working with a database, sometimes performance problems are both far more subtle, and much worse than a query itself might suggest. The topic of this week’s Postgres (PostgreSQL) performance killers article concerns the use of the IN clause, and how misusing it can catastrophically obliterate the database in mysterious ways. To that end, we’ll use a slightly revised single-table test case since it’s served us pretty well so far:

PG Phriday: 10 Ways to Ruin Performance: Forgetting it EXISTS

Fri, 22 May 2015 18:41:33 +0000

For the second of my ten part series on hidden Postgres (PostgreSQL) performance killers, I’m going to talk about something called an anti-join. It’s not a well-known approach outside of the database world, but due to how it works, it can impart almost magical plan revisions that drastically improve query performance in the right scenario. Developers can add it to a growing bag of tricks when working on database-driven content, since it comes in handy more often than you might expect.

PG Phriday: 10 Ways to Ruin Performance: Forgetting to EXPLAIN

Fri, 15 May 2015 14:47:12 +0000

Yesterday I gave the developers at my company what I call a DBA Chat. It’s something I try to do every month to keep them apprised on various features, caveats, performance considerations, and so on. I find that educating the folks who regularly work with the database does wonders for application performance and my sanity. The benefit of this long format is that I can go over more information than a time constrained set of slides.

PG Phriday: Functions and Performance Attributes

Fri, 08 May 2015 15:56:42 +0000

Users both old and new are likely aware that Postgres has functions. Some lucky readers may have even written one or two. For those without much experience in this area, or are thinking about contributing a few functions to your database for the first time, there are a couple things you should know. This week, I want to cover a few of the function declaration decorators. If you have no idea what I’m talking about, then this is probably something you’ll want to read regardless of your experience level.

PG Phriday: Reducing Writes With Unlogged Tables

Fri, 01 May 2015 11:16:02 +0000

Last week, I covered how MVCC, Postgres’s storage system, works on a very high level. Near the end, I also mentioned that it doesn’t quite lend itself well to certain use cases, such as rapidly mutating session storage. Well, there is one caveat to that statement that I’d forgotten about because of its relatively limited utility: unlogged tables. Here’s what the Postgres documentation has to say about unlogged tables: Data written to unlogged tables is not written to the write-ahead log, which makes them considerably faster than ordinary tables.

PG Phriday: Let's Talk About Data Storage

Fri, 24 Apr 2015 11:48:06 +0000

As a DBA, I strive not to live in an isolated ivory tower, away from the developers that are destined to fill our databases with volumes of data from a myriad of applications. It’s important, I think, to occasionally come back to the basics. So I’d like to discuss one of the core elements that Postgres DBAs might be intimately familiar with, but comes up often enough that some clarification is warranted.

PG Phriday: Anonymous Blocks and Object Manipulation

Fri, 17 Apr 2015 10:45:19 +0000

Postgres has had anonymous blocks since the release of 9.0 in late 2010. But it must either be one of those features that got lost in the shuffle, or is otherwise considered too advanced, because I rarely see it used in the wild. If that’s the case, it’s a great shame considering the raw power it conveys. Without committing to a function, we can essentially execute any code in the database, with or without SQL input.

PG Phriday: Functions and Addressing JSON Data

Fri, 10 Apr 2015 11:30:14 +0000

Fairly recently, a friend of mine presented a problem he wanted to solve with some JSON he had in a table. After he presented the end result he was trying to reach, I made the assumption that this would be pretty easy to do. But then I looked at the JSON Functions to try and find that quick fix. Though I read extensively and used rather liberal interpretations of the functions, there’s no way to directly manipulate JSON object contents with PostgreSQL.

PG Phriday: High Availability Through Delayed Replication

Fri, 27 Mar 2015 11:53:02 +0000

High availability of PostgreSQL databases is incredibly important to me. You might even say it’s a special interest of mine. It’s one reason I’m both excited and saddened by a feature introduced in 9.4. I’m Excited because it’s a feature I plan to make extensive use of, and saddened because it has flown under the radar thus far. It’s not even listed in the What’s new in PostgreSQL 9.4 Wiki page.

PG Phriday: Date Based Partition Constraints

Fri, 20 Mar 2015 12:02:58 +0000

PostgreSQL has provided table partitions for a long time. In fact, one might say it has always had partitioning. The functionality and performance of table inheritance has increased over the years, and there are innumerable arguments for using it, especially for larger tables consisting of hundreds of millions of rows. So I want to discuss a quirk that often catches developers off guard. In fact, it can render partitioning almost useless or counter-productive.

PG Phriday: Interacting with JSON and JSONB

Fri, 13 Mar 2015 11:33:36 +0000

With the release of PostgreSQL 9.4, comes the ability to use binary JSON objects. This internal representation is faster and more capable than the original JSON included in 9.3. But how do we actually interact with JSON and JSONB in a database connection context? The answer is actually a little complicated and somewhat surprising. Casting. Casting Everywhere. Despite its inclusion as an internal type, PostgreSQL maintains its position as encouraging explicit casting to avoid bugs inherent in magic type conversions.

A Short Examination of pg_shard

Thu, 12 Mar 2015 16:28:01 +0000

For part of today, I’ve been experimenting with the new-ish pg_shard extension contributed by CitusData. I had pretty high hopes for this module and was extremely excited to try it out. After screwing around with it for a while, I can say it has a lot of potential. Yet I can’t reasonably recommend it in its current form. The README file suggests quite a few understandable caveats, but it’s the ones they don’t mention that hurt a lot more.

PG Phriday: Materialized Views, Revisited

Fri, 06 Mar 2015 11:32:08 +0000

Materialized views are a great improvement to performance in many cases. Introduced in PostgreSQL 9.3, they finally added an easy method for turning a view into a transient table that could be indexed, mined for statistics for better planner performance, and easily rebuilt. Unfortunately, refreshing a materialized view in PostgreSQL 9.3 caused a full exclusive lock, blocking any use until the process was complete. In 9.4, this can finally be done concurrently, though there are still a couple caveats.

PG Phriday: PostgreSQL Select Filters

Fri, 27 Feb 2015 11:54:29 +0000

Long have CASE statements been a double-edged sword in the database world. They’re functional, diverse, adaptive, and simple. Unfortunately they’re also somewhat bulky, and when it comes to using them to categorize aggregates, something of a hack. This is why I wanted to cry with joy when I found out that PostgreSQL 9.4 introduced a feature I’ve always wanted, but found difficult to express as a need. I mean, CASE statements are fine, right?

On PostgreSQL View Dependencies

Wed, 05 Nov 2014 17:20:41 +0000

As many seasoned DBAs might know, there’s one area that PostgreSQL still manages to be highly aggravating. By this, I mean the role views have in mucking up PostgreSQL dependencies. The part that annoys me personally, is that it doesn’t have to be this way. Take, for example, what happens if you try to modify a VARCHAR column so that the column length is higher. We’re not changing the type, or dropping the column, or anything overly complicated.

On PostgreSQL Logging Verbosity

Tue, 05 Aug 2014 14:35:05 +0000

Recently I stumbled across a question on Reddit regarding the performance impact of using pgBadger on an active database server. The only real answer to this question is: do not use pgBadger. Before anyone asks—no, you shouldn’t use pgFouine either. This is not an indictment on the quality of either project, but a statement of their obsolescence in the face of recent PostgreSQL features. One of the recommended postgresql.conf changes for both of these tools is to set log_min_duration_statement to 0.

Finally Done With High Availability

Tue, 29 Jul 2014 15:23:42 +0000

Well, my publisher recently informed me that the book I’ve long been slaving over for almost a year, is finally finished. I must admit that PostgreSQL 9 High Availability Cookbook is somewhat awkward as a title, but that doesn’t detract from the contents. I’d like to discuss primarily why I wrote it. When Packt first approached me in October of 2013, I was skeptical. I have to admit that I’m not a huge fan of the “cookbook” style they’ve been pushing lately.

Friends Don't Let Friends Use Loops

Fri, 25 Jul 2014 12:12:06 +0000

Programming is fun. I love programming! Ever since I changed my career from programming to database work, I’ve still occasionally dabbled in my former craft. As such, I believe I can say this with a fair amount of accuracy: programmers don’t understand databases. This isn’t something small, either; there’s a fundamental misunderstanding at play. Unless the coder happens to work primarily with graphics, bulk set-based transformations are not something they’ll generally work with.

Foreign Tables are not as Useful as I Hoped

Wed, 11 Jun 2014 16:14:01 +0000

When I heard about foreign tables using the new postgres_fdw foreign data wrapper in PostgreSQL 9.3, I was pretty excited. We hadn’t upgraded to 9.3 so I waited until we did before I did any serious testing. Having done more experimentation with it, I have to say I’m somewhat disappointed. Why? Because of how authentication was implemented. I’m going to get this out of the way now: The postgres_fdw foreign data wrapper only works with hard-coded plain-text passwords, forever the bane of security-conscious IT teams everywhere.

PGCon 2014 Unconference: A Community

Tue, 27 May 2014 09:02:42 +0000

This May, I attended my first international conference: PGCon 2014. Though the schedule spanned from May 20th to May 23rd, I came primarily for the talks. Then there was the Unconference on the 24th. I’d never heard of such a thing, but it was billed as a good way to network and find out what community members want from PostgreSQL. After attending the Unconference, I must admit I’m exceptionally glad it exists; it’s something I believe every strong Open Source project needs.

Foreign Keys are Not Free

Wed, 14 May 2014 16:40:23 +0000

PostgreSQL is a pretty good database, and I enjoy working with it. However, there is an implementation detail that not everyone knows about, which can drastically affect table performance. What is this mysterious feature? I am, of course, referring to foreign keys. Foreign keys are normally a part of good database design, and for good reason. They inform about entity relationships, and they verify, enforce, and maintain those relationships. Yet all of this comes at a cost that might surprise you.

Trumping the PostgreSQL Query Planner

Thu, 08 May 2014 12:24:45 +0000

With the release of PostgreSQL 8.4, the community gained the ability to use CTE syntax. As such, this is a fairly old feature, yet it’s still misunderstood in a lot of ways. At the same time, the query planner has been advancing incrementally since that time. Most recently, PostgreSQL has gained the ability to perform index-only scans, making it possible to fetch results straight from the index, without confirming rows with the table data.

Announcing walctl for PostgreSQL Transaction Log Management

Thu, 27 Mar 2014 10:23:31 +0000

I’ve managed to convince my employer to open source one of the tools I recently wrote. That tool goes by the name of walctl, and I believe the importance of this kind of tool can not be overstated. The PostgreSQL Write Ahead Log (WAL) files are key to crash recovery, point in time recovery, and all standby use not derived from streaming replication. WAL files are extremely critical to proper database operation.

Free Copy of Instant PostgreSQL Starter

Wed, 05 Jun 2013 15:18:16 +0000

Another free book giveaway? What, am I running a library here? Well, it turns out Packt liked my review of Instant PostgreSQL Starter so much, they want me host a short period where you can obtain your very own copy for the low, low price of $0. To those ends, I have a few brand new digital copies comprised of shiny premium electrons ready to dispense to three lucky commenters. Does that sound good?

Winning (Free eBooks) is Everything

Tue, 21 May 2013 09:43:40 +0000

It occurs to me I forgot to congratulate the winners of the free ebooks. So without further ado: SAB, who seems to host a nice blog geared toward server administration. Stephan, who’s looking to improve existing strategies. Jeff and his growing PostgreSQL cluster. Pierre, who apparently has an experimental PostgreSQL backend for MySQL. Interesting. Congrats to the winners. But more, I call upon them to pay it forward by contributing to the community, either by corresponding with the excellent PostgreSQL mailing lists, or maybe submitting a patch or two to the code.

Make pg_hba.conf Redundant by Using pg_hba.conf

Fri, 10 May 2013 10:02:07 +0000

Let’s face it, the pg_hba.conf file is a pain in the ass to use regularly. Sure, reloading the database will cause it to re-read this file, but with a lot of active users and frequent changes, this isn’t really tenable. Luckily lurking deep within its bowels, PostgreSQL has a little-known feature that can easily be overlooked because it’s so humbly stated. Here’s the manual entry for pg_hba.conf for the user section:

Free PostgreSQL Backup Book? Yes Please!

Fri, 26 Apr 2013 10:53:04 +0000

A little while ago, I wrote to the PostgreSQL general mailing list that I’d been approached by Packt Publishing to contribute a quick manual on doing PostgreSQL backups: Instant PostgreSQL Backup and Restore How-to. They’re the same guys who published Greg Smith’s PostgreSQL 9.0 High Performance book which everyone seems to swear by. The goal of the backup book was to distill the PostgreSQL documentation, tools, and Wiki down to a collection of short step-by-step guides much like the O’Reilly nutshell series.

2011 PGOpen is now Closed

Tue, 20 Sep 2011 22:25:09 +0000

Well, that was an interesting couple of days. Unfortunately my birthday came right after and I didn’t feel like writing anything for the duration. Now though? Why not! In the end, I think my presentation, NVRam for Fun and Profit went over okay. Not a ton of people showed up, but I did get ambushed afterwards with questions after I got off the stage. Why those people didn’t ask while I was in Presentation Mode, I can’t quite understand.

Dragged into the (Postgres) Open

Tue, 19 Jul 2011 20:38:51 +0000

A few months ago, Greg Smith of PostgreSQL fame suggested I submit a proposal to the new Postgres Open conference here in Chicago. Some of us residents of the Midwest have long waited for a PostgreSQL-related conference of our very own, and now the glorious day has finally arrived. I was asked to submit proposals to other conferences, but the travel involved quickly put me off; now I can be lazy and still help spread The Word.

When Query Hints Attack

Tue, 01 Mar 2011 21:22:08 +0000

Ah, query hints. For all those times when the database isn’t doing what you want, they’re a useful tool for forcing the query optimizer to perform your bidding. But in this case, not only is the road to Hell paved with good intentions, it’s paved with a frictionless slide directly into a wood chipper that empties handily into an active volcano. With query hints, be careful what you wish for, because—to the detriment of all you hold dear—you just might get it.

MySQL isn't YourSQL

Mon, 28 Feb 2011 22:12:45 +0000

Ok, so I’ve already corrected gaudy and horrible behavior part and parcel with default PostgreSQL installs, but what about that… other open-source SQL database? Is it wrong too? Sure is! Fixing Your Damn Server Again My fake system still has 8GB of RAM, but we’ll be using more of it for a MySQL install. Why? Because MySQL’s planner makes different assumptions about memory allocation than PostgreSQL. Remember sysctl.conf? Put this in there:

PostgreSQL for Newbs

Sun, 13 Feb 2011 21:57:49 +0000

So you’ve decided to use PostgreSQL as the database for your sparkly new website running some variant of a LAMP stack. Or maybe you just got a new job and must now administer a PostgreSQL install so you excitedly did your research. You’ve read the install docs, tinkered on a VM, and you think you’ve got everything ready. You’re wrong. You’re going to run out of disk space, your website will be slow, and you’ll go running to the PostgreSQL mailing lists in abject terror because you have no clue what is wrong.

Why I Married pg_migrator

Thu, 03 Jun 2010 15:42:08 +0000

With the introduction of PostgreSQL 8.4, Bruce Momjian, a significant core developer, contributed a tool that can actually upgrade an entire database cluster in place. The time required is essentially only that necessary to copy the data files from the old installation to the new one. On a quick RAID system, this can be an order of magnitude faster than a dump/restore. The main drawback is similar to Slony: disk space must effectively be doubled for this upgrade method.

LIMITed Performance

Fri, 16 Apr 2010 15:50:18 +0000

PostgreSQL can be both a beautiful thing, and an infuriating mess. Occasionally I look through the logs on one of our database servers to see if I can’t optimize some queries. It’s good practice, and is an excellent way to monitor basic system performance by watching for queries that take longer than might otherwise be possible. Sometimes performance can be fixed by tweaking an index, or manually rewriting the query and convincing a developer to integrate the changes.

On Upgrading and Databases

Mon, 15 Mar 2010 11:53:25 +0000

Postgresql hates itself. I’m convinced of this, and have considered the idea frequently over the years. I roll it around in my mouth just to savor the taste, only to hope the flavor changes eventually. A couple things have advanced, though not quite what one might hope. Parallel Restore With the introduction of PostgreSQL 8.4, the core developers have finally succumbed to the availability of multiple-CPU systems, and added parallel restores.

DBA Angry!

Wed, 27 Aug 2008 10:24:38 +0000

There’s a lot of debate in application development circles over various sundries such as column naming schemes and framework implementation details. Well, I’d like to clear all that up. See, there are more application frameworks than grains of sand on the entire planet Earth, and each one of them has a different philosophy and API for creating database objects. Rails, Django, Turbogears, Zope, Zoop, Drupal, Catalyst, Nitro, Nuke… holy fucking Christ, stop it already!