Automatic restart of HouseBot upon crash

[starfire]

Would like to see some sort of wrapper that detects if HB crashes and automatically restarts it.

Even better then would be to allow limited number of automatic restarts before its no longer restarted (to stop continuous crash/restart loop) and some sort of reporting mechanism if it does crash so you get notified (eg email, popup etc).

[roussell]

Not to negate your request, but to offer some help until it's implemented: Most of us here have wrapped HB in a batch file or a Windows Shell Script (vbscript) to enable autorestarts upon crashes. This has an additional advantage of allowing you to log/email/announce through the script when/if HB dies.

Here's an example of a quick StartHouseBot.cmd batch file that will loop forever:

Code: Select all

@echo off
cls
:Start
echo HouseBot started on %Date% at %Time%
start "HouseBot Server" /MIN /ABOVENORMAL /WAIT C:\Progra~1\HouseBot\HouseBotServer.exe
echo HouseBot Server Terminated on %Date% at %Time%
goto Start

This version will loop 3 times and then exit:

Code: Select all

@ECHO OFF
SET COUNT=0
SET MAX=3
:Loop
Echo Housebot Automation Server Started on %Date% at %Time% 
start "HouseBot Server" /MIN /ABOVENORMAL /WAIT C:\Progra~1\Housebot\HouseBotServer.exe 
Echo Housebot Automation Server Terminated on %Date% at %Time% 
REM **** Put something here to alert you that HB has restarted.****
SET /A COUNT=COUNT+1
If "%COUNT%"=="%MAX%" goto Exit
goto Loop
:Exit
REM ****Put something here to alert you that HB has looped it's max number of times and is dead.****

Lastly, here is a vbscript that will loop x times and send a configurable email after each restart and after the final crash. It will also write an error message to the local machine's Application Event Log (as WSH.exe) on each restart and a warning message after the final death. Save as StartHouseBot.vbs and place in your startup folder.

Code: Select all

Option Explicit
Dim objWSHShell, HbLoopMax, HbAppStart, HbLoopCount, EmailFrom, EmailNotify, objEmail, HbAppPath
Dim EmailServer, RestartWarnSub, RestartWarnMsg, MaxLoopWarnSub, MaxLoopWarnMsg, RsltEvtAdd

'************************************************************************
' Set Parameters for Maximum Loops, HB Location and Email
HbLoopMax      = 3
HbAppPath      = "C:\Progra~1\HouseBot\HouseBotServer.exe"
EmailFrom      = "HouseBot"  ' ***This may need to be a valid email address from your ISP***
EmailNotify    = "[email protected]"
EmailServer    = "smtp.YourMailServer.com"
RestartWarnSub = "Notice: HouseBot Error"
RestartWarnMsg = "HouseBot has restarted, you may want to check it."
MaxLoopWarnSub = "Warning: Critical HouseBot Error"
MaxLoopWarnMsg = "HouseBot has restarted " & HbLoopMax & " times and is now dead. Manual intervention required."
'************************************************************************

'Heres a list of event codes to use when calling the AddAppEvent function
'0   SUCCESS
'1   ERROR
'2   WARNING
'4   INFORMATION
'8   AUDIT_SUCCESS
'16  AUDIT_FAILURE

'Begin Code to loop through restarts
HbLoopCount=0
do until HbLoopCount = HbLoopMax
    HbLoopCount=HbLoopCount+1
    Set objWSHShell = WScript.CreateObject("WScript.Shell")
    objWSHShell.Run HbAppPath, 1, true
    AddAppEvent 1, RestartWarnMsg
    SendEmail RestartWarnSub, RestartWarnMsg
    Set objWSHShell = Nothing
loop
AddAppEvent 2, MaxLoopWarnMsg
SendEmail MaxLoopWarnSub, MaxLoopWarnMsg

'Function to Send Email
Public Function SendEmail(EmailSubject, EmailMessage)
    Set objEmail = CreateObject("CDO.Message")
    objEmail.From = EmailFrom
    objEmail.To = EmailNotify
    objEmail.Subject = EmailSubject
    objEmail.Textbody = EmailMessage
    objEmail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2
    objEmail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserver") = EmailServer
    objEmail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = 25
    objEmail.Configuration.Fields.Update
    objEmail.Send
    Set objEmail = Nothing
End Function

'Function to add restarts to Local Application Event log
Public Function AddAppEvent(EventLevel, EventMessage)
    set objWSHShell = Wscript.CreateObject("Wscript.Shell")
    RsltEvtAdd = objWSHShell.LogEvent(EventLevel, EventMessage)
    set objWSHShell = nothing
End Function

Terry

[starfire]

Hi Terry,

Thanks for the above scripts, very useful.

After posting my message I did do a quick search and also found a couple of programs which do similar things, windaemon and application monitor - of course ideally this should be built into HB

Cheers

[roussell]

I edited my post above to include a vbScript example that will also email on HB crashes.
Terry

[dlmorgan999]

That's a nice vbScript Terry. I think I'll replace my current batch file solution with that. Thanks for the post!

-- Dave

[ScottBot]

It definitely runs circles around my old simple batch file "loop forever" solution

[Steve Horn]

The proverbial elephant in the room: Why does HB crash? My HB server app crashes periodically (once every week or so) for no reason (apparent to me anyway).

[starfire]

Contact support if you are getting a crash every week and I'm sure they will help

[roussell]

The proverbial elephant in the room: Why does HB crash? My HB server app crashes periodically (once every week or so) for no reason (apparent to me anyway).

My install is very stable and typically only crashes if I've done something stupid like installing an untested app that eats memory, or more often a piece of bad script somewhere that does something stupid that I forgot about. My biggest problem is that if the HB box loses power the Access db is trashed when I try to restart. I make backups to restore upon restart if needed but the dead db usually requires manual intervention. I have HB on a UPS that is monitored y HB, but last week the UPS itself died, while the main power was still on... I'm moving my install to an old laptop but that brings up a new problem - laptops don't usually auto-restart after a power interruption which will also require manual intervention... I'll stop here before we get too off topic.

I second Starfire's suggestion, If you're seeing crashes every week then definitely work with Scott to figure it out.

Terry

[ScottBot]

I second Starfire's suggestion, If you're seeing crashes every week then definitely work with Scott to figure it out...

If you are seeing crashes, please send me the latest dump files (found in the \HouseBot\Dump directory).

[Steve Horn]

Will do, the next time it crashes. I guess I'm a little surprised though that several folks have found it necessary to build in mechanisms to recover from crashes. Discounting self-induced foot-shooting crashes like Terry described, is there a crash history pattern for the HB server that warrants these recovery efforts? (Of course, I'm always quick to blame Windows. )

[Richard Naninck]

If you look in the dump directory you should see a dump for each crash. If all dumps point in the same direction, you must definately report this since it may be a hard HB bug and could even be resolved. Many others might benifit from it. The restart mechanism is there to insure a running HB all the time. My setup controls many stuff, but the most important would be my HVAC control. Without HB all of this control would have to be done manually and it takes time for the GF to figure out the house is not getting warm because HB just crashed.

[roussell]

I changed the vbScript above to add an entry to the local machine's Event Log after each restart. It will add to the Application log in the event viewer under "WSH.exe". It adds an "Error" flag for each restart" and a "Warning" flag after the final death.

Terry

[sundodger]

hi all, have tried using this script but get a runtime error.
line 56 char 5 Variable is undefined 'objWSHShell'

any suggestions.
Thanks Mark..

[Richard Naninck]

Just add somewhere on top:

Dim objWSHShell and you should be good. Or delete the Option Explicit. Either one will work, but the first is how it should be... clean