40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  PSL  Not logged in ELOG logo
Entry  Fri Aug 25 00:23:24 2017, awade, DailyProgress, Computers, Configuring upstart services to boot python scripts 
    Reply  Sun Sep 3 15:55:47 2017, awade, DailyProgress, Computers, Updates to python PID locker and new autolocker scripts 
Message ID: 1878     Entry time: Fri Aug 25 00:23:24 2017     Reply to this: 1894
Author: awade 
Type: DailyProgress 
Category: Computers 
Subject: Configuring upstart services to boot python scripts 

I have configured upstart services on the acromag1 Ubuntu machine to automatically start python scripts.  This will help keep the PID slow loops live whenever EPICS channels are alive and accessible. Scripts will committed to the /CTNLab/current/computing/scripts/service_jobs directory on the 40m svn.

We have the perennial problem of PID python slow control loops dying when modbus goes down or some other error or computer shut down happens.  As more processes and autolockers get added, this will become a pain to manage. Booting scripts automatically can be tricky with crontab and its difficult to manually manage whether tasks are already running without spawning an Armageddon of run away processes. 

Eric Q suggested that we use upstart services.  I've finally gotten around to getting this working in the PSL lab. Its very simple just create a script (with sudo) in the /etc/init/ directory with the following setout

description "Description of the service that is being run"
author "Author Name"

start on runlevel [2345]
stop on runlevel [!2345]

respawn limit 17280 86400 

    <insert export commands for necessary env variables ect>
    exec <bash command to execute>

end script

pre-start script
    echo "Script run started  [`date`]" >> /home/controls/logs/servicelog_PSL-SLC-north.log
end script

pre-stop script
    echo "Service exited properly [`date`]" >> /home/controls/logs/servicelog_PSL-SLC-north.log
end script

Here the file should be placed in the /etc/init/ directory with the .conf extension on the end. This particular service script will start on boot of the machine and, if interrupted, also try to respawn every five seconds for a day and then give up. The nice thing is that it can be configured with a preamble to do things before running the main script (such as make a log file entry).  A post script can also be specified to run after a stop command has been issued (e.g. logging successful safe exit of service).  

It is also possible to trigger a service based on the successful launch of another service.  Importantly multiple services can be started asynchronously without a rigid boot order: this is good for starting various scripts that stop due to some error but restart when appropriate. This means that hierarchies can be establish: booting modbus EPICS server, then PIDs, then autolocker scripts.  Any level can die and the dependent services will wait for that service/script to respawn before restarting.  In the chaoic startup of a computer there is no need to speificy a boot order, just the start up condition of each service.

To start the script manually

$ sudo service servicename start

Here servicename does not include the .conf of the file.

To stop gracefully use

$ sudo service servicename stop


I've now managed to get the laser slow controls to start as a upstart service. These are called PSL-SLC-north.conf and PSL-SLC-south.conf and are configured to start at reboot and also respawn for about a day. To start manually run

sudo service PSL-SLC-north start

from anywhere in the command prompt on acromag1.  Use PSL-SLC-south to boot south path service.


I'm having problems launching the modbus service for the acromag slow channels using upstart.  It may be an issue with environment variables. I looked through the .bashrc and copied over any export commands that might help, but the service still seems to exit mysteriously without properly booting.

Up until now modbus has been started within tmux using

/home/controls/modbus/bin/linux-x86_64/modbusApp  /home/controls/modbus/iocBoot/iocTest/acromag.cmd

This works fine within terminal and tmux, but the log files output from within services show it going through the motions and then exiting without stopping at the modbusApp proprietary consol prompt.  Some of the 2> output show some channels popping up errors, but its not clear that this is exiting the script.  Its not uncommon for epics to return no response on a channel request every now and again, but that doesn't stop tmux finishing the launch.  Going to leave this for now, but it would be good to have modbus acromag channels launched with services as well.




ELOG V3.1.3-