CopiaFacts provides the following means of detecting critical failures:
Operations Monitor Alert (OMA) Files
A number of CopiaFacts programs write to a control file (with extension .OMA) at regular intervals, typically every 15 seconds. The timestamp of this file can then be checked by the supplied OMACHECK program to ensure that it is recent: if it found to be more than a specified time period 'out-of-date', then OMACHECK will initiate a notification or recovery action.
To maximize the effectiveness of OMACHECK, it is necessary that:
•you have multiple machine nodes which can access the CopiaFacts Application Data folders, and run OMACHECK on the one which is most likely to remain operating in a failure situation. OMACHECK will only detect a failure if it is still running and can see the OMA files.
•your machines are fully time-synchronized.
•if backup or other external operations may prevent OMACHECK from seeing the OMA files, the OMACHECK 'out-of-date' tolerance must be set to long enough to avoid false activation.
The following CopiaFacts applications can write OMA files:
•COPIAFACTS writes an OMA file if the Operations Monitor Alert option is checked in Run-Time Options. This is not checked by default.
•FFEXTERN writes an OMA file if specified on the command line.
•JOBMON and JOBSUM write an OMA file if specified on the command line.
OMA files are always written in the FAXFACTS\LOG folder, unless this folder is overridden using $log_def.
For detection of failures in outbound operations, you can use the $kill_group command to specify an outcome code which, if repeated on a nominated group of channels, will either cause the CopiaFacts node to do a controlled shut down, or take some other specified action. A shutdown is then detected because OMA file updating will cease.
As soon as a kill group is triggered, the channels in the channel group are taken out of service, preventing further operations on these channels from commencing.
A configuration file variable, KILL_GROUP_ACTION can specify a filename which is to initiate the action to be taken on a kill group trigger:
•If the filename is that of a .FS file, it is copied into the TOSEND folder. This file can be used either to send an e-mail notifying the action, or to initiate a $worker_box action which starts an infobox script.
•If the filename is a .EXE, .BAT or .CMD file, it is executed to initiate any notification or recovery action which you have specified.
If there is no defined action, or the action fails, or if taking the channels out of service left no channels in the node operational, the node is shut down and OMA file updating will cease. This is a 'controlled shut down' in which no new tasks are started and all channels are idle, but OMA file updating ceases immediately so that OMACHECK will take action after the configured delay time.
It is not possible or sensible to list in advance all the possible outcome codes which you might wish to use to trigger a kill group. These depend too much on the hardware combinations you have in use, the type of telephone line or IP port, and the connected telephone service or IP provider. In practice, most times a kill group needs to be set up, it will be in reaction to a specific failure which has revealed the need to catch the next occurrence.
Logging Errors
Failures to write the daily DBF log file from COPIAFACTS can be caused to suppress OMA file writes by checking the Quit on Logging Errors checkbox in Run-Time Options. This is not checked by default. The shutdown will be detected because OMA file updating will cease.
For special purposes, a number of critical errors can be set to suppress OMA file writes by adding a code to the Special Kill Flags field in Run-Time Options. These codes may change from time to time. The shutdown will be detected because OMA file updating will cease. Triggered kill events matching a kill flag cause a 'controlled shut down' in which no new tasks are started and all channels become idle, but OMA file updating ceases immediately so that OMACHECK will take action after the configured delay time.
The current kill events are (multiple flags are specified by adding the codes):
| 1 | Add 1 to the flag value to cause an immediate shutdown of COPIAFACTS when on of the events below is triggered. This will leave transactions unfinished, but FS files with locks in ACTIVE or WORK folders will be set for retry either when the same COPIAFACTS node is restarted, or by OMACHECK. |
| 2 | If BladeWare or Copia VoIP port registration (when configured) fails after retries. |
| 8 | when certain Windows errors are reported on copying to a local folder files which are to be faxed. |
| 16 | When a monitored BladeWare process stops, when a BladeWare 'Bad Session' or 'System' error is returned, or an exception is detected in BladeWare. In the most serious of these cases the shutdown occurs whether or not this flag is specified. |
| 32 | When the NEXTMCF file cannot be accessed, after retries, to get a unique number for an incoming fax. |
| 64 | When command files cannot be written, updated or moved after processing, after retries. |
| 128 | When more than a quarter of the configured outbound e-mail channels have blocked on STARTTLS |
| 256 | When events have ceased to arrive from an active Diva board channel |
| 512 | When more than the number of fax/voice board channels have been taken out of service than is specified as CHANNEL_ERROR_MAX. |
Shutdown Code
If your infobox scripting detects a critical error which is non-recoverable and would require a restart, you can use $next_box to transfer to a pre-specified dummy box number which will initiate COPIAFACTS shutdown. The special code (which must be a number) is specified in the configuration command $shutdown_code. The shutdown will be detected because OMA file updating will cease.
FFEXTERN Failures
The environment variables FFEXTERN_CONSECUTIVE_ERRORS, FFEXTERN_QUEUE_ERRORS and FFEXTERN_TIMEOUTS can be used to specify the corresponding counts after which updating of the FFEXTERN OMA file will be suppressed, so that OMACHECK is activated.
Worker Box Actions
Some sites use a worker box action to create an FS file which sends a daily e-mail to indicate that the system is running, either to an automated or human recipient. The $worker_box command provides options to set the repeat interval. The failure detection of course relies on noticing that the e-mail has not arrived.
Third-Party Utilities
Many network-monitoring utilities are available which can monitor running processes, files or connections.
