Use Cases for DataCache

I discussed a set of use cases not well supported by HTML5 ApplicationCache with the HTML WG during a breakout session at TPAC. These use cases were the basis for AtomDB are being solved by the DataCache API.

Found out that OMA and AT&T had similar ideas too. Also learnt that these techniques could be merged with HTTP caching.

Hixie agreed these use cases covered the ones the GMail team had asked for but he favored waiting until after AppCache had stabilized and browser implemented other HTML5 stuff.

Read more ...

WebSimpleDB gets thumbs up from major browser vendors

Microsoft and Mozilla declared their intention to advance WebSimpleDB and their preference for it over WebDatabase (which itself is getting renamed to WebSQLDatabase) and WebStorage.

WebDatabase will be frozen in terms of its current API and likely stay at the SQL dialect of SQLite 3. WebSimpleDB has taken on the mandate of the WG to create a small and general set of primitives for building various rich JS libraries. WebSimpleDB is heavily influenced by Oracle's experience with Berkeley DB.

Stay tuned here for more on this topic.

Read more ...

DataCache API now rewritten and ready for FPWD

There was a wave of support for programmable HTTP caching when the first DataCache editor's draft was published. The main issue with the draft was the inability to derive the exact list of additional requirements on browsers compared with HTML5's AppCache. I went ahead and rewrote DataCache to make it easier to spot the differences. Here are the differences:

1. A data cache does not have fallback or online whitelist namespaces or foreign entries.
2. A data cache provides a monotonically increasing numeric version identifier.
3. A data cache may have an authorization cookie.
4. Every resource managed in a data cache may be configured to process certain (HTTP) protocol methods locally
5. Zero or more data caches are automatically associated with a cache host at its creation/load time. A secure data cache group is available to a cache host if its authorization cookie will be sent to an origin server for fetching that cache host [RFC2965].
6. The events checking and noupdate are not used. The events cached and updateready are merged in to a single event ready. Events downloading and progress are renamed as fetching and captured. There is no downloading update status for a data cache group.
7. No manifest is used. Instead an update transaction encapsulates a set of changes to the cache. An update transaction consists of any number of capture steps or release steps. An application can store a bag of bits independent of a network representation of that resource. This allows storing of off-line resources. The results are not visible until the transaction is committed.
8. Update transactions can be performed in workers but not in the background independently of applications.
9. It is possible to have only one online transaction but multiple off-line transactions are allowed.
10. It is possible to find out the resources added to or removed from cache starting at a certain version.
11. A data cache offers applications the ability to get the contents corresponding to a URI.
12. The navigator registers a scriptable HTTP interceptor that can off-line respond to arbitrary HTTP requests based on prior (data cache) configuration of the resources in those requests.
13. A new header is defined to by-pass data cache in request processing
14. A slightly different networking model is required to take into account interception.

I have a request to publish this new editor's draft as a FPWD and hopefully that will happen before TPAC with a view to discussing the draft during WebApps WG f2f meeting. Read this in conjunction with WebSimpleDB so you can create off-line Web apps that can serve HTTP resources locally, including even query on the off-line cache.

Read more ...

Now published - an alternative to SQL for database storage in Web browsers

Many of us have raised serious issues with the WebDatabase API that provides SQL query access to a browser embedded database. Some of these are:

1. Use of SQL means that it will be next to impossible to get agreement on the exact dialect supported by independent implementations.
2. The API is extraordinarily complex that an ordinary programmer would have a hugely difficult time using it. This means more bugs will remain hidden for longer in the spec, and more bugs would be found in code using the API. The complexity arises out of a need to prevent delays when accessing a database from affecting the user interface. As a result, practically every call to the API results in an asynchronous callback.
3. The API requires a lock acquisition scheme that does allows extremely limited parallelism in data access. As a result, superior implementations cannot do well. Even though the intention is noble - avoid producing exceptions as a result of deadlocks - the result is delays and timeouts as well as higher level deadlocks, which the programmer still has to deal with.
4. The API has chosen to use SQL due to its well developed relational model. However, it doesn't want to provide an API that the typical database developer will be able to program against. This mythical "Web database" programmer receives undue importance in the design of the API.
5. Overly reliant on SQL, when the data model in browsers is that of objects. This forces everyone to develop layers to translate between rowsets and objects.

There is currently a deadlock that has arisen out of a number of browser vendors refusing to go along with WebDatabase (notably Mozilla and Microsoft) and a lack of an alternative. To break this deadlock, we have developed an alternative that will provide a highly efficient and functionally rich (although less richer than SQL), called WebSimpleDatabase.

This editor's draft of WebSimpleDB is now available from W3C. I am inviting feedback on this draft so that I can incorporate it before it goes to FPWD.

Read more ...

Embed XML document inside another XML document

Ever considered the implications of embedding an XML document inside another XML document? Every new format developer, especially extension format developer has to consider the consequences of embedding an XML extension payload inside an XML envelope.

In plain terms, what are the consequences of doing the following:

<feed xmlns="http://www.w3.org/2005/Atom">
<id>...</id>
...
<entry>
 <id>...</id>
 ...
</entry>
...
</feed>

While we do it all the time with Atom, this is an example of a document embedded inside another - Atom entry inside Atom feed. There are a number of issues lurking under the surface that one has to consider before copying the entry bits in to the feed itself. In general, there are many issue to consider while embedding one XML document inside another.

These issues don't surface if you are managing all the bits yourself, i.e., you store the original bits of everything and normalize to a common set of encodings, namespaces, signatures, entities, and so on. However, if you are aggregating content from different publishers, then it's a different ball game. I started thinking about these issues as a result of Mark Nottingham's feedback about the Atom in-line I-D.

Here are some of the issues I could think about (and manifestations in the case of Atom):

1. Duplication of content (e.g., <atom:title>)
2. Inherited values for metadata (e.g., xml:lang, xml:base)
   
3. Character set and encoding (for XML and binary content)
   
4. Entity declarations
5. Namespace declarations
6. Digital signatures
7. Protocol metadata (ETag, Content-Length, Content-Type)
   

Did I miss some more? If so, please help me identify them. It will help in making the in-line I-D stronger and easier to implement and use.

Read more ...

Temper your 3G and iPhone expectations

"Why do we need offline capabilities when today's phones are so savvy about using the network and the network is so good."

In a PCWorld article, Bill Snyder, veteran mobile industry journo, basically admonishes us for expecting too much from mobile devices. Here are a few excerpts:

But my first few days with the iPhone [...] made me realize how far we are from the mobile, Web-based nirvana so often promised. The technology and the infrastructure are simply not there yet. It's time to lower our expectations.

...

Our national infrastructure lags behind demand for high-speed services, and given the costs and the difficult economic conditions, we just have to wait until it catches up.

...

Batteries have gotten much better. The problem, though, is the ever increasing demands made upon them by new features. Take my iPhone: That big, bright screen sucks power like a baby quaffs milk. Use a few other features, and suddenly you're running on empty. It's a serious problem for engineers, and it's only going to get worse.

...

Battery technology, unlike CPU technology, does not dance to Moore's Law -- and it won't for some time.

There are two engineering alternatives for poor network and low battery life:

1. Hunker down and make designs for offline apps
2. Design seamless online/off-line apps
   

#1 is good old mobile app design. There is no reason to go back to that since networks are indeed better and users want to use the network more often. However, #2 is a smart evolution path while we wait for 4G.

Read more ...

Requirements for and components needed in real Web Storage

Several times I have prodded people to provide requirements to WebStorage. Here's what I have gathered from use cases and scenarios people have suggested for WebStorage's scope:

Here are three sets of requirements as I see them:

Structured key-value pair storage and cursors

1. Store, remove, and retrieve individual data items
   1. items identified by a string key
   2. item values can be either null, object, or string
2. Allow duplicate values for the same key
3. Allow organization of items in to more than one database per application
4. Sequentially walk items in a cursor starting at a point identified by a key value
   1. items should be presented in an increasing key value as determined by JavaScript's String comparison operator '=='
   2. once the highest key value is reached, no more items are provided
   3. if the matching key value is empty, the first key in the database is the starting point
   4. can walk through items in reverse order
   5. items obtained in the cursor are deletable - IOW, cursor is modifiable
5. Provide a sequence object in each database to generate a monotonically increasing sequence of numbers
6. A database belongs to a single storage unit.
7. Applications are able to open or create a storage unit.
8. Allow storage of multiple items to be committed as part of an atomic transaction
9. Offer READ_COMMITTED and READ_UNCOMMITTED cursors
10. Transaction scope is limited to operations on databases within the same storage unit.
11. All APIs are synchronous

HTTP Cache programming:

1. capture a resource that is used for satisfying GET & HEAD requests
   
   1. given just its URL, fetch and durably store its representation
      
   2. as a locally produced representation
   3. over write an existing representation
2. retrieve or remove a resource's representation or check for its presence

HTTP interception:

1. Store the methods on resources that are locally intercepted
2. Choose among the following for a locally intercepted method
3. Pass the headers and body of the request, without jeopardizing security, to the programmed interceptor
4. Make the request to the server, relay the request and the response to the programmed interceptor, and return the response to the application

A JS library for synchronized Web storage would use all three components to process network requests locally using HTTP interception, respond to queries locally using the b-tree storage and the HTTP cache as well as store pending requests for forwarding to the server when it is reachable.

A pictorial representation of the layers involved in an implementation of CouchDB to run in a browser captures the contrast between the existing and the proposed breakdown of Web Storage.

The browser is then required to do the switching between off-line and on-line and between relay and intercept as I explain in the post on BITSY 0.5.

As I see it, these requirements subsume the requirements known to date for Ian Hickson's Web Storage draft. Were it not for the widespread use of LocalStorage, we could do away with it completely because the b-tree store is a more robust way of dealing with key-value pairs. However, there is no reason and, hopefully, little chance of the SQL portion making its way to the W3C hall of fame, I mean Rec.

Read more ...