There are a lot of situations where it’d be nice to know when the contents of a web page change, and the modern way to do that is the WebSub standard, formerly known as PubSubHubbub.

It’s an elegantly simple system: A page advertises its support by providing a link to a “hub” that handles sending out change notifications. A subscriber sends to that hub a URL where notifications should be delivered.

However, this requires the subscriber to have a web server that’s always reachable from the public internet, which means that desktop and mobile applications can’t take advantage of it without a dedicated server. Even for some web applications this might be the only aspect which requires a server; otherwise they could use much cheaper static-file hosting, or be delivered via something decentralized such as IPFS.

And that’s a shame, because there are only a few options available to developers and they’re all undesirable:

Ignore WebSub and just poll for changes, spamming web sites with “have you changed yet?” requests.
Require users to have the technical skills and resources to set up their own web server.
Build centralized infrastructure, which requires a different set of skills than app development, and gets expensive as it becomes popular.

Push notifications

Standards bodies have been working for years on specifications for how to deliver notifications to applications in a timely fashion without requiring the application to be associated with a dedicated server.

The web Push API makes this a problem for the browser vendors to solve, instead of pushing (hah) it onto application developers. JavaScript can ask the web browser for a push URL, then forward that URL off to an application server which can post to it any time there’s something new. The browser arranges that those messages will get back to the JavaScript service worker… somehow. (The Push API doesn’t need to specify how the browser communicates with the push server or even how it finds one to use; browser vendors just provide their own infrastructure and co-evolve it with their browser products.)

The Push API is built on RFC8030 which specifies how application servers should send messages to push servers. While the Push API is specific to browsers, RFC8030 should in principle be usable for native mobile and desktop apps as well. In practice, as far as I can tell, nobody is implementing it for those settings: mobile OSes only provide proprietary APIs for push notifications, and every desktop app developer has to just roll their own thing. So for the rest of this article, I’m going to focus on web apps.

Naturally, everything is harder than I’d like: the push URL expects to receive messages that are a little different than what a WebSub hub sends. The details are specified in RFC8030, but in particular:

WebSub sends the entire contents of the changed page, while a push server may reject messages larger than 4kB.
Current push servers require the sender to sign the message with a public key provided at registration (see RFC8292, “VAPID”) but there’s no provision for doing so in WebSub.
WebSub verifies a subscription request by immediately sending a challenge to the subscriber’s URL, which an RFC8030 server won’t respond to in the way the WebSub spec requires.
Because the Push API is designed for devices which may be offline, RFC8030 requires specifying a Time-To-Live (TTL) for how long the push server should keep trying to deliver the message. WebSub has no means to convey that or any of the other options that RFC8030 allows.

For all those reasons, using WebSub with the Push API still requires a dedicated server to match each specification’s expectations to the other. So that’s disappointing.

Stateless impedance matching

If we have to have a separate server, can we at least make it really cheap to operate? Here are the limits I wanted on this hypothetical service in order to keep costs down, allow scaling up to any amount of demand, and enable deployment at any cloud or shared hosting provider:

No persistent state: no database, no writable filesystems.
No background or scheduled processing: all work happens while responding to an HTTP request.

I think this is feasible, but I haven’t actually tried it, for reasons I’ll get into later. First I’ll just outline how I think a lightweight WebSub proxy would work.

The most complex part is the process for initially setting up subscriptions:

An application asks the WebSub proxy for its VAPID application server key, as defined in RFC8292.
The application passes the application server key to the Push API to get a new push URL.
The application posts the push URL in an HTTP request to the WebSub proxy, which doesn’t reply immediately.
The proxy signs or HMACs the push URL with its own key, and constructs a subscription URL containing the signed push URL.
The proxy sends the new subscription URL to the application via the push URL. (This confirms that the application can receive messages sent to that push URL.)
The proxy returns an error if the push URL returned an error; otherwise it returns an empty response with a “202 Accepted” status code.
Once the application receives the push notification containing its new subscription URL, it uses that URL in subscription requests to WebSub hubs.
When a hub sends a verification request to the proxy, the proxy checks that the wrapped push URL was signed by its own key.

Later, when the hub sends a change notification to the proxy, it again checks that the wrapped push URL was signed by its own key, and then passes the notification on to the push URL, signed with its application server key.

During initial registration, the application could also provide values for the TTL, Topic, and Urgency headers as defined in RFC8030. The proxy would encode those into its subscription URL alongside the push URL and use them when forwarding change notifications.

I think this satisfies all requirements of both specifications while being just as scalable to tiny single-person installations as it is to multi-data-center worldwide deployments.

Same-origin strikes again

There’s just one problem, which reveals itself in several different ways.

URLs that are of interest for use with WebSub are, almost by definition, not at the same origin site as any JavaScript application that would try to subscribe to them. If they were part of the same site, the application wouldn’t need WebSub to find out about changes.

JavaScript can’t access cross-origin URLs without the target site opting in using CORS, and I doubt any WebSub deployments opt in. That means the client web app can’t access pages to check whether they support WebSub, and it can’t reliably send subscription requests either.

We could work around those issues by making the WebSub proxy also be a CORS proxy, forwarding requests to arbitrary web sites. But that opens up a variety of security and abuse concerns, and also could make the proxy significantly more expensive to run.

Alternatively, there are existing CORS proxies out there, though as far as I can tell none of them support the POST requests necessary for establishing a subscription at a hub. I’m guessing that’s because of the same potential for abuse.

On top of that, once you have a subscription going, WebSub delivers the current contents of the page to you every time it changes… but we can’t count on being able to deliver that through a push server, which may have a length limit as low as 4kB. So the application needs to request a page that’s already been delivered to the proxy, which suggests that it would be more efficient for the proxy to cache the copy it received from the hub, and return that cached copy to the application on demand. But that means having state that persists across multiple requests, which again makes the proxy more expensive to operate.

And if the proxy is going to be a general CORS proxy or at least cache WebSub-delivered content, then it might as well be fully stateful and coalesce multiple subscription requests for the same page into a single subscription at the hub.

Conclusion

In short:

solving the small question of connecting WebSub with Push is straightforward,
but the same-origin policy effectively negates all the advantages,
so we’re probably better off spending our time building application-specific servers.

Or in other words, the web is a terrible place and makes me cry.

Coda

While I was digging into this, I did find one other direction the world could have gone:

In parallel with the early development of PubSubHubbub, RFC5989 described a way to use SIP for notification of web page changes. Since mobile phones already use SIP for voice call setup, I could imagine adopting it for mobile push notifications as well. If that was exposed to applications, standards like RFC5989 would have provided direct interoperability between mobile apps and the kinds of web sites that use WebSub today.

But SIP is quite different from HTTP, while WebSub and Push both use HTTP in mostly straightforward ways. So I suspect RFC5989 faces pretty significant barriers to adoption, comparatively speaking. At minimum, I can’t find any evidence of anyone having ever implemented RFC5989…