JoT
Note
coding

Auto-increment Keys: IndexedDB vs SQLite

2023-05

Huidong Yang

This is a short piece. So in IDB, you can enable auto-incrementing (primary) keys when creating an object store:

// using Jake's "idb" lib, below is part of an `openDB` call
upgrade(idb) {
    const store = idb.createObjectStore(STORE_NAME, { keyPath: "sn", autoIncrement: true });
    // store.createIndex(...);
}

The question is, when you add (insert) an object to the store, should the object already contain the key path (property)? If it does, will its value get overwritten? What if it's set to null?

This detail got confusing esp. after I used Android Jetpack's Room, which is an ORM for SQLite. As an ORM, you get to define the schema using Kotlin classes, that's why you have to include the PK property, and typically what you do is set the default value to e.g. 0, and when you do insert, it automatically overwrites the value with the generated one.

@Entity
data class Entry(
    @PrimaryKey(autoGenerate = true)
    val id: Long = 0,
)

That's pretty ergonomic, because 1) you don't have to specify a PK value when you're constructing a particular object, thanks to Kotlin's default value feature, and 2) the object you try to insert has exactly the same shape as what's stored in the DB.

What do I mean by (2)? Well, with IDB, you do it quite differently: if you want to make use of this auto-increment functionality, then you must NOT include the PK property in the object to be inserted! And if you do, that value you specify will actually trump the generated one when it gets stored! Pretty counter-intuitive if you ask me. (But at least this behavior is well-documented in the spec.)

The consequence of this behavior? In Elm, when I encode a record into JSON value to be sent over to JS for storage, I have to exclude the PK field, using a different encode function, e.g. MyRecord.encodeNoPk; However, when I implement data export, i.e. dumping all the records into a single JSON file, I decide to retain the PK values. Why? Because when I need to import the data back into the IDB, I can actually take advantage of this weird behavior of IDB as long as I do include those PK values in the objects. It's a bit subtle, but note that we typically have to clear the object store before the batch insert, right? But that doesn't reset the key generator state (unless you delete the entire database, which is obviously overkill), so if you rely on the key generator when importing the data, the objects will then have different PK values, also leaving a gap of unused values, because, again, the key generator doesn't auto-shrink when the store is cleared. Now instead, since I'm able to dictate what the PK values should be when the objects are imported back, the end result will be an identical PK state. For instance, if I'm just testing data export & import locally, on a set of example objects, then I get to always preserve the state no matter how many times I run the import.

So you see, this specific behavior of IDB gives us more control over the PK value. Of course, you're then responsible not to cause troubles like PK clashing, so data import is a good use case here, assuming the exported data is good.


Again, Room's design choice (key generator trumps user-specified value) is very reasonable (and probably more familiar to people), because it wants you to work with DB data in mostly the same way as with Kotlin types, so the PK property can never be omitted (that would create a different type); unlike in Elm, where you're dealing with JS interop (and JS is way more dynamic than Kotlin, perhaps like how IDB is way more flexible than SQLite, even though SQLite is actually pro-flexibility); yet, by thorough control of how JSON encoding/decoding is done, you still get to be safe in terms of data integrity, but the drawback is you need to write more code.

But that's fine, Elm can be verbose (which is subjective), but as long as the verbosity is on things that are important (and often error-prone in JS world), then I think it's well worth it.