When the line breaks: how our application brings itself back

There is a moment everybody knows who works with an application that talks to a server. You type, click, save — and suddenly nothing happens any more. Somewhere in the background the connection is gone. Maybe the server was updated, maybe the network coughed. And then comes the sentence you read far too often in manuals: please restart the application.

We looked at that sentence more closely and decided it has no place in our estate agent software.

What really happens when a server restarts

From the workplace's point of view a server restart is a small apocalypse. Not because data would be lost — that sits safely. But because three things lose their validity at once.

The first is the login. Whoever logs in receives a pass for the session, and every further request is signed with that pass. If the server restarts, it no longer knows the passes it issued. So the workplace holds a pass in its hand that nobody recognises any more — and only notices when it shows it.

The second are the running loads. A workplace with several open documents constantly requests data. If the server fails, all these requests run into the void at the same time — and each one on its own would try to solve the problem. Twenty views, twenty error messages, twenty login attempts.

The third is the notification. Our application does not wait for someone to press “refresh”: if a colleague changes something, your own workplace learns of it immediately over a permanently open line. That line is the first thing gone in an outage.

The pass that renews itself

The actual cause of the forced restart was mundane: the pass was hard-wired in several places. It could not have been exchanged anywhere without rebuilding half the application — and all open documents would have been lost in the process.

So we turned it around. There is now exactly one place where the current pass lives, and every route to the server fetches it there fresh, on every single request. The benefit shows in the moment a new pass becomes necessary: it is entered, and from the next request it applies everywhere — without an open list, an exposé being edited or a dialog so much as flinching.

Building on that, the rest was almost self-evident. If the server rejects a request because it no longer knows the session, the application logs in again in the background and sends the request once more. The caller sees nothing of it.

One detail we paid particular attention to: when twenty requests are rejected at once after a restart, that may trigger one login, not twenty. The first renews the session, the others wait briefly and use the result. You can see it nicely in the log: many rejections, one login, then quiet.

Waiting beats failing

The second insight was that failing is rarely the right answer.

As long as the server does not answer, there is no point in twenty views requesting their data. Previously each of them would have tried, failed and written an error into the log — and the list would have gone on showing old data without anyone knowing. Today they simply do not load at all. The request stays noted and is honoured as soon as the server is back. Then everything refreshes together.

The same goes for the permanently open notification line. Connections like that usually try a few times and then give up — except a server restart takes longer than those few attempts. The result was an application that could load data again but heard nothing more until the program was closed. Now it keeps trying, with growing intervals, for as long as the application runs.

A dialog that closes itself again

That leaves the question of what the person in front of the screen should see in the meantime.

We deliberately decided against a small indicator in the status bar. An application that stays half operable invites you to keep working — filling in forms, pressing buttons whose effect arrives nowhere. That is not friendliness, that is a trap.

Instead a window lays itself over the application, explains what is going on and catches every input. It is not an error dialog: it does not list failures, it says that the connection is being restored and that the work will continue where it was interrupted. Whoever does not want to wait can trigger an immediate attempt at any time instead of watching for the next automatic one.

Three properties mattered to us here:

• It closes itself. Once the connection is back, the window disappears without anyone doing anything — and the cursor still sits where it sat before.
• It cannot be clicked away. Neither an accidental escape nor the close cross leads back into an application that could not do anything anyway. Whoever really wants to stop ends it through the button provided for that.
• It only asks when it must. If the password was changed in the meantime, it asks for it — otherwise it settles everything in the background.

And then everything comes back

The finest part is the moment of return, because that is where an order of operations sits that is easy to get wrong.

The obvious approach: the server answers again, so reload everything immediately. That is exactly what makes twenty loads set off with the old, long-invalid pass, be rejected and have to recover one by one. It works, but it is a detour you can see clearly in the log.

The right order is the other one: log in first, then let things load. The application obtains a valid pass, and only afterwards do the views and open documents get the signal to refresh. Every document knows best what it has to reload — it is only told that the time has come.

One special case remains, and it is deliberately hard: if the server was updated in the meantime and speaks a newer data version, the application does not reconnect. Then no amount of waiting helps, only a restart of the application — and that is exactly what it says, instead of writing on with an outdated understanding of the data. We are almost a little proud of that exception: it is the only case in which the unloved sentence about restarting still appears, and it stands there for good reason.

What we take away

Two things that reach far beyond this one feature.

First: an outage is rarely noticed where you expect it. Whoever listens only for the obvious errors misses the quieter variants — and notices nothing at all the second time round. We therefore built our application so that several independent places may report an outage, but only a single one decides what it means.

Second: we would never have found much of this without trying it out. We stopped the server repeatedly during operation, started it again, disconnected the network in between, pressed buttons impatiently — and looked into the log every time. The most revealing find came not from a test but from a line of log showing that something happened too early.

In everyday use you will hopefully not notice the result at all. A short window that explains what is going on and disappears again — and afterwards everything still stands where you left it. That is how software should feel when the technology behind it briefly falls out of step.