Myth Backend Monitor

mythtv is very stable, but we've all had the occasional experience of a backend failure resulting in lost recordings. To make matters more complicated, I "administer" mythtv systems for my brother, my daughter and myself. I use "mbemon" ("Myth Backend Monitor") to automatically monitor and implement recovery procedures on all 3 of these systems, thus saving me from phone calls and wasted time.

"mbemon" automatically detects and recovered from the following types of problems:

• Errors that cause the backend to spontaneously exit.
• Runaway log file growth -- a problem that can result in the root partition filling up.
• "waiting for a thread" messages in the backend log file.
• Failure of backend to respond to status queries on port 6544.

If any of these conditions are detected mbemon will automatically:

1. Captures diagnostic information that can be used to determine the root-cause of the problem.
2. Restart the Myth backend & frontend processes.

Installation procedure

• Download the tar file mbemon.tgz
• untar the file into a directory that is on your path, ie
  
• cd /usr/local/bin
• tar xzf mbemon.tgz
• Add the following line to your bootup procedure. In Knoppmyth R5.5 this is done by editing the file /etc/init.d/bootmisc.sh and adding the following line:
  
• /usr/local/bin/minit.mbemon start

You can manually start mbemon at anytime as root by entering the command "minit.mbemon start" and you can halt it by entering the command "minit.mbemon stop". If is important that you halt mbemon before you run mythbackup or any other process that needs the myth backend stopped. If you don't do this, mbemon will detect that missing mythbackend process and automatically restart it, even though you deliberately halted mythbackend.

Diagnostic information

mbemon maintains the following files in the directory /var/log/mythtv/mbemon

1. mbemon.log: When mbemon is started this file is initialized with a single line that indicates the date and time mbemon was started. After that, each time mbemon detects the need to restart the backend, information is added to the end of this log file that indicates the time the restart happened and the reason that mbemon felt that a restart was necessary.
2. history.log: A single line is added to this file everytime that mbemon does a status check on the backend. The file is limited in size to 15 lines.
3. status.txt: The output from the last status query sent to the Myth backend.
4. mythbackend.log: When mbemon detects the need to restart the backend, it automatically moves /var/log/mythtv/mythbackend.log to location. This is helpful for diagnosing the root-cause of the backend problem. mbemon maintains a history of the last 4 backend log files in case multiple restarts happen. In chronological order from more recent to oldest they are named "mythbackend.log", "mythbackend.log-1", "mythbackend.log-2" and "mythbackend.log-3".

Finally, as a safety precaution mbemon will terminate itself if the backend process has required a restart on 4 of the last 15 checks, as this likely indicates that existence of a persistent error that must be addressed to achieve stability.