Query Airtable with all the power of SQL.

The Complete Developer's Guide to Airtable

What is RDS Proxy? Exploring with benchmarks


I was surprised when a customer wrote in to explain that he noticed - for simple queries - the Airtable API was about as performant as querying his [Sync Inc](https://syncinc.so/?utm_source=blog&utm_medium=post&utm_campaign=rds)-provisioned Airtable database. We'd expect a database query to be at least an order of magnitude faster than an API query.

So I had to investigate: What was the explanation for this?

But these outlier queries still bothered me. What was the explanation for them?

Airtable's API is pretty consistent and responsive, with a mean and median request time of 350ms:

<img src="./rds-proxy/airtable-api.png" style={{maxHeight: "500px", margin: "auto"}} />

This is about what you'd expect from a standard list request against a third-party API with 100 items returned per request.

So how could a database query get up to 350ms? A little digging, and we discovered that the customer was querying his database via a Lambda function. This meant opening a connection to his Sync Inc db before each query. A solid lead.

I wanted to see how much overhead the connection times were adding. So I wrote some simple benchmark functions in Go.

The timing for the benchmarking is handled by this simple function:

```go
func timeTrack(start time.Time, name string) {
	elapsed := time.Since(start)
	log.Printf("%s took %s", name, elapsed)
}
```

You can log how long a function execution took in Go using this function in combination with `defer` like so:

```go
func benchmarkQuery(count int) {
	defer timeTrack(time.Now(), "benchmarkQuery")
	for i := 0; i < count; i++ {
		funcUnderTest()
	}
}
```

Above, the arguments to `timeTrack()` are evaluated immediately, meaning the first argument is set to `time.Now()` - the time `benchmarkQuery()` was invoked.

But `defer` waits to _invoke_ `timeTrack()` until right before the function `benchmarkQuery()` returns. Which, in this case, will be after the for loop executing `funcUnderTest()` `count` times has completed. Neat trick.

With a timing function in place, I wrote out some benchmarks. We want to benchmark two different kinds of behavior:

- A function that opens a connection _on every loop_ and then makes a request
- A function that opens a connection _then_ loops and makes requests

This should help us understand how much time is spent establishing the connection and how much time is spent executing the query.

Here's what the function that opens a connection every loop looks like:

```go
func benchmarkOpen(count int) {
  defer timeTrack(time.Now(), "benchmarkOpen")
	for i := 0; i < count; i++ {
		db := openDb()
		defer db.Close()
		rows, err := db.Query("SELECT id from purchase_orders limit(100);")
    rows.Close()
		if err != nil {
			log.Printf("Got error: %v", err)
		}
	}
}
```

And the one that opens a connection then makes a bunch of queries on that connection:

```go
func benchmarkQuery(count int) {
  defer timeTrack(time.Now(), "benchmarkQuery")
	db := openDb()
	defer db.Close()

	for i := 0; i < count; i++ {
		rows, err := db.Query("SELECT id from purchase_orders limit(100);")
    rows.Close()
		if err != nil {
			log.Printf("Got error: %v", err)
		}
	}
}
```

(Note: It's important to run `rows.Close()`, otherwise the connection is not immediately released - which means Go's `database/sql` will default to just opening a new connection for the next request, defeating this test.)

Using these two functions, we're prepared to find out:

1. What kind of impact does re-creating a db connection on each run have on the wall time of a 1-off execution?
2. Does using RDS proxy have an effect on that wall time?
3. What are the basic load characteristics of a db.r5.large getting hammered with new connection requests? How does RDS Proxy help it perform?

This inquiry is all motivated by the nature of Lambda functions, which the customer uses. And indeed Amazon recently released RDS proxy to address this precise architecture:

> Many applications, including those built on modern serverless architectures, can have a large number of open connections to the database server, and may open and close database connections at a high rate, exhausting database memory and compute resources. Amazon RDS Proxy allows applications to pool and share connections established with the database, improving database efficiency and application scalability.

### RDS Proxy gotchas

Before we get into the benchmarks, I want to touch on the gotchas that sucked up a lot of my time. (I guess after all these years I haven't learned my lesson: The AWS UX is just not going to help you use their products. You have to read the user guide from end-to-end.)

Here are the two that tripped me up big time:

**1. RDS Proxy only supports specific database engine versions** – and the latest RDS Postgres is not one of them.

The first time I went to setup a new RDS proxy, I didn't see any available databases in the dropdown:

![](./rds-proxy/empty-dropdown.png)

After much trial and error, I ended up spinning up a new Aurora db. At last, I saw _something_ populate the dropdown.

It wasn't until much later I learned that Postgres 12 - the engine I use everywhere - just isn't supported yet.

(What would have been nice: Showing me all my RDS databases in that drop-down, but just _greying out the ones that are not supported by Proxy_.)

**2. You can only connect to your RDS Proxy from inside _the same VPC_**

This one was a time sink. There are few things I dread more than trying to connect to something on AWS and getting a timeout. There are about a half-dozen layers (security groups, IGWs, subnets) that all have to be lined up _just so_ to get a connection online. The bummer is that there's no easy way to debug _where_ a given connection has failed.

So, the first wrench was when I discovered - after much trial and error - that I couldn't connect to my RDS Proxy from my laptop.

(How about a small bone, right next to "Proxy endpoint," that lets me know the limitations associated with this endpoint?)

### Now that we're up and connected

**RDS vs RDS Proxy**

With my RDS Proxy 101 hard-won, we're ready to run some benchmarks.

I copied the benchmark binary up to an EC2 server co-located with the RDS database and RDS proxy.

Per above:

- `benchmarkOpen` opens a connection and then runs a query on each loop (sequential)
- `benchmarkQuery` opens a connection first, then each loop is a query on that connection (also sequential)

Here's what I got running this against the RDS database directly, 1000 times per, with a 10s pause between:

```bash
# RDS direct, same datacenter
$ ./main
2020/12/25 00:08:14 Starting benchmarks...
2020/12/25 00:08:19 benchmarkOpen took 5.846447324s
2020/12/25 00:08:30 benchmarkQuery took 504.32703ms
```

So:

- `benchmarkOpen` - ~5.85ms per loop
- `benchmarkQuery` - ~0.500ms per loop

Fair to assume the connection handshake takes 5ms, which isn't bad.

This benchmark runs too fast to register on any RDS graphs. But if I loop the benchmark to sustain it over a couple minutes, I can see a healthy connection spike for the first test:

![](./rds-proxy/db-direct-graphs.png)

I recognize that in `benchmarkOpen`, instead of deferring `db.Close()` I could be nicer to my database and close it immediately. But I kind of like the DB slow boil.

Let's move on to the RDS Proxy. Remember, RDS Proxy has to be in the same VPC as the entity calling it as well as the database. So all three are co-located:

```bash
# RDS Proxy, same datacenter
$ ./main
2020/12/25 00:15:21 Starting benchmarks...
2020/12/25 00:15:28 benchmarkOpen took 7.094809555s
2020/12/25 00:15:39 benchmarkQuery took 1.239892867s
```

Comparing to the first benchmark: It appears the Proxy adds an overhead to each request of 0.7ms (1.2ms - 0.5ms). As we'd expect, connection openings take a bit longer, about 1.15ms longer.

And, sure enough, I can't get the graph to budge. It sustains about 100 connections, with no real CPU spike, with the benchmark sustained over several minutes:

![](./rds-proxy/rds-with-proxy.png)

We'll see soon how all this performance translates to Lambda, but the motivation for RDS Proxy is clear: **It's less about the client, more about the database**. RDS Proxy is just a layer of indirection for our database connections. It doesn't speed up establishing new connections - in fact, we pay a small tax on connection opening. It just saves our database from the thundering herd.

**Cross-region**

Because we're here, I'm curious: What happens if we run this benchmark cross-region, from US-East-1 to US-West-2?

```bash
# RDS - different datacenter
$ ./main
2020/12/25 00:11:12 Starting benchmarks...
2020/12/25 00:15:53 benchmarkOpen took 4m41.607514892s
2020/12/25 00:17:15 benchmarkQuery took 1m11.767803425s
```

Different story:

- `benchmarkOpen` - ~281.6ms per loop
- `benchmarkQuery` - ~71.77ms per loop

Connections take about 200ms, presumably because they involve a lot of back and forth that require coast-to-coast travel.

The difference is over a full order of magnitude. At this point, opening a database connection and making a query starts to have performance characteristics similar to querying Airtable's API.

Indeed, the customer that opened this inquiry operates out of US-East-1. Bingo.

Opening connections is certainly a factor here. But RDS Proxy won't help, because as we've learned it's not intended to reduce client-side connection opening costs. And even if it did, we couldn't use it cross-region!

So co-locating the database in the customer's data center is the move, and will give their team performance an API can only dream of.

### Benchmarking Lambda

Lambda got us into this mess, so let's do an end-to-end benchmark with it. Our direct-to-RDS test will route like this:

```bash
API Gateway -> Lambda -> RDS
```

And our RDS Proxy test will look like this:

```bash
API Gateway -> Lambda -> RDS Proxy -> RDS
```

Each lambda function call will open a new database connection and issue a single query. I just adapted the function above to make one open/query as opposed to doing so inside a `for` loop.

I'll run a benchmark locally on my computer that will call the API Gateway endpoint. Because I'm running locally, I'm able to save a little work by using Go's native benchmarking facilities:

```go
func BenchmarkRequest(b *testing.B) {
	for i := 0; i < b.N; i++ {
		resp, err := http.Post("https://[REDACTED]/run", "application/json", nil)
		if err != nil {
			panic(err)
		}
		defer resp.Body.Close()
		if resp.StatusCode != 200 {
			fmt.Println("Received non-200 response. Continuing")
		}
	}
}
```

The first test routes to RDS directly:

```bash
$ go test -bench=. -benchtime 1000x
goos: darwin
goarch: amd64
pkg: github.com/acco/rds-proxy-bench
BenchmarkRequest-8   	    1000	 128684974 ns/op
PASS
ok  	github.com/acco/rds-proxy-bench	130.351s
```

So, 128ms per request.

Now using RDS Proxy:

```bash
$ go test -bench=. -benchtime 1000x
goos: darwin
goarch: amd64
pkg: github.com/acco/rds-proxy-bench
BenchmarkRequest-8   	    1000	 118576056 ns/op
PASS
ok  	github.com/acco/rds-proxy-bench	120.529s
```

118ms per request.

We're not impressed until we look at the graphs:

![](./rds-proxy/proxy-lambda.png)

The first big spike is the first benchmark, hitting RDS directly. The second non-spike is the second benchmark, routed through the proxy.

The end-to-end Lambda benchmark takes it all home: The purpose of RDS Proxy is foremost to tame connections (and related load) on the database. Any happy gains on the _client_ side will be a result of a relieved database.


How to go further with Airtable using Postgres Arrays


While Airtable looks and feels like a database, you can't query it like a database.

That's why we built [Sync Inc](https://syncinc.so/).

In this guide, you'll first replicate Airtable to Postgres using [Sync Inc](https://syncinc.so/). Then, you'll unlock your Airtable data with some killer PostgreSQL array methods.

## Airtable and Postgres Arrays

Airtable's power lies in its flexibility. Thanks to Airtable's friendly UI, I can add new tables, new columns, or change the format of a column at will. It's great.

For example, let's say I have a Products table with a column that contains the product's image. But what happens later when I decide to keep more than one image per product? I just drag another image into the row! No need to migrate the column or my previous images.

To reflect this flexibility, when Airtable data is pulled into Postgres via Sync Inc, that data is often represented as [Postgres Arrays](https://www.postgresql.org/docs/9.1/arrays.html). So when a new image pops into the Products table, your Postgres table can support it immediately, without migrations or changes to existing queries. (The new image is simply another value in the array.)

Many field types in Airtable can contain multiple values: attachments, multiple select, lookups, and collaborators. With Sync Inc, these field types will all end up in Postgres as arrays. These might not be as familiar to you, but arrays are first-class in Postgres.

In a couple minutes you'll get the hang of Postgres Arrays and never look at Airtable the same way.

## Wait...a growing small business needs your help?

To make this guide more interesting, imagine you are helping a small business owner grow their business.

They've been using Airtable to catalog products, purchase orders, and sales orders. The business owner wants their Airtable base to remain simple and easy to use - but she needs more insight to grow.

To play along with this story, take a look at the [small business's inventory tracker base](https://airtable.com/shrV2tLjq2FbrRtO2) and add it to your workspace by clicking **Copy base** in the top right corner:

![Copy base](./postgres-arrays/copy_base.png)

> I’m going to assume that if you’ve made it this far, you already know Airtable is the bees knees and know how to set it up.

## Connect Airtable to Postgres

Now, let’s get your Airtable base into Postgres using [Sync Inc](https://syncinc.so/):

1. Go to https://app.syncinc.so/signup and create an account.
2. Connect your base to Sync Inc using the tutorial or check out the [Quickstart guide](https://docs.syncinc.so/get-started).
3. In a couple seconds, you’ll be provisioned a Postgres database and Sync Inc will begin filling it with all your Airtable data.
4. Now [connect to your new Postgres database](https://docs.syncinc.so/reference#tools-and-applications) with any standard SQL client. Tools like [TablePlus](https://tableplus.com/) and [PopSQL](https://popsql.com/) are great options (I’ll be using TablePlus).

## How Airtable looks in Postgres

Take a look at your Postgres schema.

First, you’ll see that all your Airtable table names have been converted to snake_case:

![Schema](./postgres-arrays/schema.png)

Second, you’ll notice that any field type in Airtable that could hold multiple values (like `lookup`, `multiple select` , and `attachment`) have been stored in columns of type `text[]`. `text[]` is an array of text fields:

![arrays](./postgres-arrays/product_array.png)

Arrays in Postgres are delimited with curly braces `{}`, instead of the common square bracket `[]` you are probably familiar with. (It will keep you on your toes.)

## Airtable and the PostgreSQL `ARRAY`

Before you start solving real problems for the small business you are assisting, let's just get comfortable with Postgres arrays.

For example, in your Inventory Tracker base, you can see that the Warehouse Locations table contains a `lookup` field where you can select which products are associated to a given warehouse location:

![product inventory](./postgres-arrays/warehouse_product_inventory.png)

You can select multiple products for each location. So in your synced Postgres database, the `warehouse_locations` table shows the `product_inventory` column as an array type containing multiple `record_id`s representing each selected product.

![tableplus](./postgres-arrays/tableplus.png)

Let's play with the `product_inventory` arrays.

Selecting the `product_inventory` column from the `warehouse_locations` table is simple:

```sql
SELECT product_inventory FROM warehouse_locations;
```

This will return all the `product_inventory` as a column full of arrays:

<img src="./postgres-arrays/product_inventory_arrays.png" alt="product inventory" style={{width: "400px"}} />

This is okay, but let’s say you want to select the first product listed on every warehouse shelf.

To do so, you need to access the values in the array. Postgres arrays are base-1 indexed (another fun way to keep you on your toes) - so to access the first value in the array we use `[1]` with square brackets. Like so:

```sql
SELECT product_inventory[1] FROM warehouse_locations;
```

Now you'll see the `record_id` for the first product on each shelf.

<img src="./postgres-arrays/product_inventory.png" alt="locations" style={{width: "300px"}} />

Again, when using Postgres with Airtable, any field that can contain multiple values will be an array. So this simple approach of extracting the first value of the array comes in handy.

As a practical example, in the Product Inventory table in your Airtable base, you can see that the Product Name column is a multi-select field. This is probably better represented as a text field - but then you don't get the pretty colors 😌:

![product name](./postgres-arrays/product_name.png)

To query for just the text of the product name, you'll use the array syntax again:

```SQL
SELECT product_name[1] FROM product_inventory;
```

This will return a clean list of product names:

<img src="./postgres-arrays/product_name_output.png" alt="product names" style={{width: "300px"}} />

## `unnest()`

With a grasp of how Postgres arrays work, let's begin to use SQL to extract some insights.

For example, let’s say you want to see which product colors are most appealing to customers. So you want to pull the list of colors for products that sell well. Let's say "sell well" for this small business means more than 5 sales.

To make get this data, you need to get a list of `colors` from the `product_inventory` table where the units sold are greater than 5:

```sql
SELECT colors FROM product_inventory WHERE units_sold[1]::integer > 5;
```

> Tip: The double colon next to `WHERE units_sold` is short-hand for `CAST`. Casting allows you to change the data type of a field - in this case from `text` to `integer` - so that you can evaluate the `> 5` statement. You'll use `CAST` often with your Airtable data as Sync Inc works to support more data types beyond `text`, `text[]`, and `bool`;

The results from this query are a little convoluted because all the colors return in arrays. Some of the colors are duplicated across rows so you can't really tell what the most popular colors are:

<img src="./postgres-arrays/color_arrays.png" alt="color arrays" style={{width: "400px"}} />

It would be nice to see each unique color in a clear list.

`unnest()` allows you to _un-nest_ each item of an array onto its own row:

```sql
SELECT
    DISTINCT UNNEST(colors) AS "Color",
    count(colors) AS "Frequency"
FROM product_inventory
WHERE units_sold[1]::integer > 5
GROUP BY "Color"
ORDER BY "Frequency" DESC;
```

Here, you use `DISTINCT` in conjunction with `unnest()` to get a clean list of the most popular colors. You are then using `AS` to give the column an easy to understand name - in this case "Color".

Then you use `count(colors)` (again, using `AS` to alias the column to the name "Frequency") to calculate how frequently each unique color is appearing. `count(colors)` works with `GROUP BY` so that `count(colors)` is tabulated against each unique Color.

Lastly, you use `ORDER BY` to sort the table so that the most popular color is at the top.

It looks like "Desert Brown" is a common color amongst the most popular products:

<img src="./postgres-arrays/colors.png" alt="colors" style={{width: "300px"}} />

## `ANY`

Building on your new insights around the popularity of "Desert Brown", you now want to understand which products are available in this desirable color.

We can use `ANY` to quickly check if a product's color array contains our desired color - in this case "Desert Brown"

```sql
SELECT product_name[1] FROM product_inventory WHERE 'Desert Brown' = ANY(colors);
```

Okay, some ideas are beginning to come together. Let’s unpack (dare I say unnest) this query.

First, because `product_name` is represented in your database as an array (you'll recall that product name is a `multiple select` in our Airtable base) we use `product_name[1]` to select the first item in the array. In this case, the name of the product.

Then, the `WHERE` clause is checking that the product colors array (`colors`) contains a string matching `Desert Brown`.

This produces a list of all the products with a "Desert Brown" color option:

<img src="./postgres-arrays/product_color.png" alt="Desert Brown Products" style={{width: "300px"}} />

## Airtable and PostgreSQL `JOIN`

Now, you're ready for some heavy lifting. Warehouse optimization.

It would be nice to see how much inventory is on each shelf in the warehouse. This might help the business owners use their inventory space more efficiently.

The first tool you might reach for is an Airtable lookup. But it won't render exactly what you want:

<img src="./postgres-arrays/airtable_mess.png" alt="Airtable mess" style={{width: "300px"}} />

In Airtable, you'd need to resort to writing complex functions. But now that your data is in Postgres, you can just use SQL to express what you want in a few lines.

To do so, you can use `ANY` with a `JOIN` to see how much inventory is on each shelf:

```sql
SELECT
    warehouse_locations.name AS "Shelf",
    SUM((product_inventory.units_ordered[1]::INTEGER) - (product_inventory.units_sold[1]::INTEGER)) AS "Inventory"
FROM warehouse_locations
JOIN product_inventory
    ON product_inventory.id = ANY (warehouse_locations.product_inventory)
GROUP BY warehouse_locations.name
ORDER BY warehouse_locations.name ASC;
```

You've already worked with most of the elements in this query, so let's highlight how this is all working together.

First, you use `CAST` to turn `units_ordered` and `units_sold` into `INTEGERS` so you can calculate the inventory of each product. You use `AS` to give each returned column a nice name.

Then, you use `ANY` to `JOIN` the available product inventory to each shelf.

Lastly, as with the prior query, you use `GROUP BY` and `ORDER BY` to organize and sort the output.

<img src="./postgres-arrays/shelf_inventory.png" alt="shelf inventory" style={{width: "300px"}} />

## Airtable and PostgreSQL `UNION`

There's one more SQL operator I find myself reaching for all the time: `UNION`. (It's not strictly related to Postgres arrays, so let's call this a bonus.)

`UNION` is a particularly helpful tool when working with Airtable because it can help you pull together similar data from across different tables.

As an example, perhaps you want to see all the store's product images in one place to prepare for a new marketing campaign. To do so, you'll need to pull product images from both the Product Inventory table as well as from the Purchase Orders table.

In SQL this is no big deal:

```sql
SELECT product_inventory.images[1] FROM product_inventory
UNION
SELECT purchase_orders.image[1] FROM purchase_orders;
```

Now, in one list you have access to all the product images appearing across the system. To do this natively in Airtable you would have needed to create a unique `Images` table with lookups...and restructure the entire base.

<img src="./postgres-arrays/product_images.png" alt="SQL union" style={{width: "300px"}} />

## Best of both worlds

When you combine Airtable with a Postgres database, you get all the amazing capabilities of Airtable plus the flexibility and reliability of PostgreSQL.

Now you can query, integrate and build on your data with the ease of Airtable and the power of SQL.


Posts

The Complete Developer's Guide to Airtable

What is RDS Proxy? Exploring with benchmarks

How to go further with Airtable using Postgres Arrays