I have configured upstart services on the acromag1 Ubuntu machine to automatically start python scripts. This will help keep the PID slow loops live whenever EPICS channels are alive and accessible. Scripts will committed to the /CTNLab/current/computing/scripts/service_jobs directory on the 40m svn.
We have the perennial problem of PID python slow control loops dying when modbus goes down or some other error or computer shut down happens. As more processes and autolockers get added, this will become a pain to manage. Booting scripts automatically can be tricky with crontab and its difficult to manually manage whether tasks are already running without spawning an Armageddon of run away processes.
Eric Q suggested that we use upstart services. I've finally gotten around to getting this working in the PSL lab. Its very simple just create a script (with sudo) in the /etc/init/ directory with the following setout
description "Description of the service that is being run"
author "Author Name"
start on runlevel 
stop on runlevel [!2345]
respawn limit 17280 86400
<insert export commands for necessary env variables ect>
exec <bash command to execute>
echo "Script run started [`date`]" >> /home/controls/logs/servicelog_PSL-SLC-north.log
echo "Service exited properly [`date`]" >> /home/controls/logs/servicelog_PSL-SLC-north.log
Here the file should be placed in the /etc/init/ directory with the .conf extension on the end. This particular service script will start on boot of the machine and, if interrupted, also try to respawn every five seconds for a day and then give up. The nice thing is that it can be configured with a preamble to do things before running the main script (such as make a log file entry). A post script can also be specified to run after a stop command has been issued (e.g. logging successful safe exit of service).
It is also possible to trigger a service based on the successful launch of another service. Importantly multiple services can be started asynchronously without a rigid boot order: this is good for starting various scripts that stop due to some error but restart when appropriate. This means that hierarchies can be establish: booting modbus EPICS server, then PIDs, then autolocker scripts. Any level can die and the dependent services will wait for that service/script to respawn before restarting. In the chaoic startup of a computer there is no need to speificy a boot order, just the start up condition of each service.
To start the script manually
Here servicename does not include the .conf of the file.
To stop gracefully use
$ sudo service servicename stop
I've now managed to get the laser slow controls to start as a upstart service. These are called PSL-SLC-north.conf and PSL-SLC-south.conf and are configured to start at reboot and also respawn for about a day. To start manually run
from anywhere in the command prompt on acromag1. Use PSL-SLC-south to boot south path service.
I'm having problems launching the modbus service for the acromag slow channels using upstart. It may be an issue with environment variables. I looked through the .bashrc and copied over any export commands that might help, but the service still seems to exit mysteriously without properly booting.
Up until now modbus has been started within tmux using
This works fine within terminal and tmux, but the log files output from within services show it going through the motions and then exiting without stopping at the modbusApp proprietary consol prompt. Some of the 2> output show some channels popping up errors, but its not clear that this is exiting the script. Its not uncommon for epics to return no response on a channel request every now and again, but that doesn't stop tmux finishing the launch. Going to leave this for now, but it would be good to have modbus acromag channels launched with services as well.