Add --interval-on-error flag for timed restart on failure

[hongkongkiwi]

Could you add a flag --interval-on-error which pauses xx seconds before restarting the app (if it dies with a non-zero exit code).

Right now, it just keeps immediately restarting the app on error which can hammer the system.

[aperezdc]

Currently there is a one second wait when the last start attempt was done less than one second ago, but I understand that indeed it may not be enough if many processes are being (re)started at the same time.

A fixed wait time may also result in a similar issue if there are many instances of dmon which are monitoring processes that died simultaneously. While a bit less likely, I can see that happening e.g. if dmon is used to monitor processes as part of system boot.

I think it would make sense to use exponential backoff with some random jitter added to avoid unlucky coincidences, and instead make the option configure the maximum amount of time for the backoff. Once the wait time reaches that value it would stop increasing.

@hongkongkiwi How does the exponential backoff idea sound like?

[hongkongkiwi]

The concern about many services dying at once are not necessarily an issue in my case as each restart time would be tuned to the app. E.g. one app would be set at 10s and the other might be 20s, so they wouldn't hit all at the same time (probably).

But I could also see the value in having exponential backoff for some apps, if implementing this I think being able to set the limits of that backoff would also be also be helpful, with some sensible defaults.

E.g. --exp-backoff --exp-backoff-max=360s --exp-backoff-min=10s