Posted on by & filed under javascript, mobile, programming.

At Safari, we’re working on some web products that allow offline usage. For various reasons, using the Application Cache isn’t appropriate for our use case; we need a “real” database. The browser requests the data from the server and then inserts it into the relevant tables in WebSQL. (Yes, I know WebSQL is deprecated, but unfortunately iOS doesn’t yet support IndexedDB and this functionality is particularly important on mobile phones, so that’s what we’re focusing on for the moment.)

The Problem

Generally speaking this works well, but you have to navigate the landscape of device capabilities, and there are times when it is particularly frustrating. Take for instance this scenario: I chose some content that has a lot of images, adding up to 18MB of stuff, and decide to save it offline. On iOS, we get 50MB of storage per domain (though the data is actually encoded as UTF-16, so it’s about half that in reality), so this should be OK.

So the content is requested from the server, streamed to the client, and the client attempts to save it in the database. What we didn’t know is that there’s already  12MB worth of stuff stored in the database. So what happens? After making the user wait to download the entire 18MB blob and then wait to parse it and try to load it all in the database (which could amount to a fairly large amount of time depending on a variety of conditions) – the database throws an error and we tell the user they’re out of space. I think you’ll agree that this experience is sub-optimal.

A solution?

Can we do better? I think we can. I want to preface this by saying that this most likely isn’t for everyone. It may very well be faster to just throw your data at the database and see if it sticks, but in our situation most of the time it will be faster to detect and then decide what to do. The problem is that there’s no built-in way to determine how much space is left before you’ve hit whatever cap there may be. Believe me, I googled quite a lot looking for a solution. After that search and chatting with some co-workers, we came up with something and that’s what I want to share: a proof-of-concept I’m calling yardstick.js. I’ve created a gist of the code on Github.

Using it

Yardstick works by using a temporary table to measure how much space is left by filling up to a given targetSize. Originally, I wanted to create a separate database and exploit the per domain storage limitationbut there were some issues with the allocation of the two different databases. Say we are in that situation above, we have already prompted for the max size (see here for why this is important), and we know that the content we want to insert is 18MB, usage of yardstick.js would look like this:

  var ys = new Yardstick();
  ys.measure(18, function(avail, err) {
      // continue what you wanted to do here
      alert(avail + 'MB available for saving');

What’s happening

An in-memory string of 1024 * 1024 characters is used to fill the database using targetSize for the number of loops. So we can assume that each loop inserts 1MB. On each successful iteration we set the internal available attribute to equal the result.insertId of the SQLResult returned, letting the auto-incrementing table do the counting for us. The result of this setup is that if we get to targetSize loops, we know that amount can be inserted, if we error before getting there, we have the internal available attribute that will tell us how much we can safely insert. Upon completion, it deletes the in-memory fake data and drops all of the records that were created, and then calls the passed-in callback with the value of available and the error object, if an error occurred.

Now we can determine if we can save that content locally before we make the (even more) expensive call to the server. And if we can’t save the 18MB, we know how much we can save so that we might be able to suggest saving alternate content that’s smaller.

What do you think?

This is currently just an idea and I’d love to get some feedback on it because I think there are others who know more about this than me. There are obvious downsides to this approach (speed being the obvious one; I also killed the browser on my iPhone 3GS trying to use it, and it probably has battery life implications) and I’m sure there are improvements to be made as well. In addition to the gist, I wired up a little playground. The first input will create a database and fill it with that much dummy data, you can then input a size you’d like to save and it will use yardstick.js to determine if that’s possible. If you have any feedback or want to talk about this, please leave a comment here, on the Github gist, or contact me on twitter at @meirish.


One Response to “Detecting Available Free Space in WebSQL”

  1. Dan Smart (@danielsmart)

    Interesting idea – I’m in the same place that I need to measure the space remaining, to improve the user experience prior to adding a large amount of data. Has this idea gone any further, i.e. have you used it in the wild, or decided not to use it due to the issues you mention in the last paragraph?