I've set up nodus to start the ELOG on boot, through /etc/init/elog.conf. Also, thanks to this, we don't need to use the start-elog.csh script any more. We can now just do:

controls@nodus:~  $ sudo initctl restart elog

I also tweaked some of the ELOG settings, so that image thumbnails are produced at higher resolution and quality. 

elog died b/c someone somewhere did something which may or may not have been innocuous. I ran the script in /cvs/cds/caltech/elog to restart the elog (thrice).

I have now banned Warren from clicking on the elog from home

Restarted elog 9:11PM 9/5/11

I deleted a bunch of categories from the elogd.cfg file. Not every kind of locking and activity gets its own activity. All of the things related to length sensing and control should be LSC. If there's a high pollen count its under PEM. etc., etc., etc.

I decided that the 'tetex' distribution which comes with CentOS is total BS and I removed it from rossa using 'yum erase tetex'. I installed TexLive using the instructions on the web; its much better, actually works, and also has RevTex 4.1 which allows Jenne and I to write our paper.

Eventually, we'll standardize our workstation setups.

Yuta has also successfully set up zita to run the projector, so we should clear our some of the boxes and bookshelves in that area so that the projection can be larger.

This is cool though the projector is flashing the blue screen alternately.

I gave the dual head video card (ATI RADEON something) to Yuta a month ago.
It is on the top of Zita. This would make the things more fun.

The elog was so dead this time that the restart script didn't work.  I followed the restart-by-hand instructions instead, with success.

 Just for added interest, I tried a different method when the restart script broke.  The "start-elog-nodus" script has a line "kill elogd".  This seems not to be actually killing anything anymore, which means the elog can't restart.  So this time I went for "kill <pid number>", and then ran the startup script.  This worked.  So it's the "kill elogd" which isn't working reliably.

I found the elog dead again this morning, and the script didn't kill again. I modified the script to use the following line instead of "pkill elogd":

kill `ps -ef | grep elogd | grep -v grep | grep -v start-elog-nodus | awk '{print $2}'`

Hopefully the script will be a bit more robust in the future.

like the subject says...

I found the elog got down around 7:30 am in this morning.

So I restarted it by running the script: "start-elog-nodus" as instructed on the wiki.

http://lhocds.ligo-wa.caltech.edu:8000/40m/How_To/Restart_the_elog

...I restarted it.

I found the elog down and I restarted it.

Then, after few seconds it was down again. Maybe someone else was messing with it. I restarted an other 5 times and eventually it came back up.

In nodus, I moved the elog from /export to /cvs/cds/caltech. So now it is in the cvs path instead of a local directory on nodus.

For a while, I'll leave a copy of the old directory containing the logbook subdirectory where it was. If everything works fine, I'll delete that.

I also updated the reboot instructions in the wiki. some of it also is now in the SVN.

Today I found the elog down, so I rebooted it following the instructions in the wiki.

 I have the impression that Nodus has been rebooted since last night, hasn't it?

nodus:log>dmesg

Sun Jun 21 00:06:43 PDT 2009
Mar  6 15:46:32 nodus sshd[26490]: [ID 800047 auth.crit] fatal: Timeout before authentication for 131.215.114.93
Mar 10 11:11:32 nodus sshd[22775]: [ID 800047 auth.crit] fatal: Timeout before authentication for 131.215.114.93
Mar 11 13:27:37 nodus sshd[7768]: [ID 800047 auth.error] error: connect_to 131.215.115.52 port 7000: Connection refused
Mar 11 13:27:37 nodus sshd[7768]: [ID 800047 auth.error] error: connect_to nodus port 7000: failed.
Mar 11 13:31:40 nodus sshd[7768]: [ID 800047 auth.error] error: connect_to 131.215.115.52 port 7000: Connection refused
Mar 11 13:31:40 nodus sshd[7768]: [ID 800047 auth.error] error: connect_to nodus port 7000: failed.
Mar 11 13:31:45 nodus sshd[7768]: [ID 800047 auth.error] error: connect_to 131.215.115.52 port 7000: Connection refused
Mar 11 13:31:45 nodus sshd[7768]: [ID 800047 auth.error] error: connect_to nodus port 7000: failed.
Mar 11 13:34:58 nodus sshd[7768]: [ID 800047 auth.error] error: connect_to 131.215.115.52 port 7000: Connection refused
Mar 11 13:34:58 nodus sshd[7768]: [ID 800047 auth.error] error: connect_to nodus port 7000: failed.
Mar 12 16:09:23 nodus sshd[22785]: [ID 800047 auth.crit] fatal: Timeout before authentication for 131.215.114.93
Mar 14 20:14:42 nodus sshd[13563]: [ID 800047 auth.crit] fatal: Timeout before authentication for 131.215.114.93
Mar 25 19:47:19 nodus sudo: [ID 702911 local2.alert] controls : 3 incorrect password attempts ; TTY=pts/2 ; PWD=/cvs/cds ; USER=root ; COMMAND=/usr/bin/rm -rf kamioka/
Mar 25 19:48:46 nodus su: [ID 810491 auth.crit] 'su root' failed for controls on /dev/pts/2
Mar 25 19:49:17 nodus last message repeated 2 times
Mar 25 19:51:14 nodus sudo: [ID 702911 local2.alert] controls : 1 incorrect password attempt ; TTY=pts/2 ; PWD=/cvs/cds ; USER=root ; COMMAND=/usr/bin/rm -rf kamioka/
Mar 25 19:51:22 nodus su: [ID 810491 auth.crit] 'su root' failed for controls on /dev/pts/2
Jun  8 16:12:17 nodus su: [ID 810491 auth.crit] 'su root' failed for controls on /dev/pts/4


nodus:log>uptime
 12:06am  up 150 day(s), 11:52,  1 user,  load average: 0.05, 0.07, 0.07

I found elog got crashed. I rebooted the elog daemon just 10minutes before.

I just restarted the elog.  It was crashed for unknown reasons.  The restarting instructions are in the wiki.

This morning I found the elog down. I restarted it using the procedure in the wiki.

Just now found it dead. Restarted it. Is our elog backed up in the daily backups?

In the last hour or so the elog crashed. I  have restarted it.

I just found the elog down and I restarted it.

This afternoon the elog crashed. We just restarted it.

 Again, this morning Zach told me that the elog had crashed while he was trying to post an entry.

I just restarted it.

 I restarted the ELOG on NODUS just now. Our attempt to set up error logging worked - it turns out ELOG was choking on the .ps file attachment.

So for the near future: NO MORE .PS files! Use PDF - move into the 20th century at least.

matlab can directly make either PNG or PDF files for you, you can also use various other conversion tools on the web.

Of course, it would be nice if nodus could handle .ps, but its a Solaris machine and I don't feel like debugging this. Eventually, we'll give him away and make the new nodus a Linux box, but that day is not today.

 The elog was down and I ran the restart script.

i just restarted the elog for the third time in the past 12 hours.

I checked the elog.log file to debug the problem. It doesn't contain eveidence of any particular cause, except for png/jpg file uploads happened last night.

I'm not sure we can blame Image Magic again because the last crash seems to be occurred just after an entry with e jpg picture was included in the body of the message. I think Image Magic is used only for previews of attachments like pdfs or ps.

Maybe we should totally disable image magic.

 The elog crashed when I was uploading a photo just now. I logged into nodus and restarted it.

 I restarted the elog with the restart script as it was down.

 Restarted the elog with the script as it was down.

And again.

 Elog was down. Restarted with the script.

You'll never guess what happened: the elog crashed!

I restarted it with the script. Yay.

The elog crashed, so we restarted it again.

with script

elog crashed on an upload. restarted and it worked fine with the same file.

 Again. Resubmitted an old entry with just text changes. Elog hung for 5 min +.

 Elog didn't respond, so I restarted it with the script.

Before killing the process somehow two elog daemons were running at the same time. I killed those two manually. 

 took two runs of the script as usual

I restarted the elog because i changed the configuration for the cryo-elog.

Used the "start-elog.csh" script in /cvs/cds/caltech/elog/

I restarted the elog with the script as it was not up when I tried to make a post. It was again unresponsive when I went to submit, but this time the script couldn't restart it. The log said it couldn't bind to 8080, which usually happens if the daemon is still running. I pkilled it, then reran the script, and it appears to be working.

elog was not responding for unknown reasons, since the elogd process on nodus was alive; anyway, I restarted it. 

 I have created an updated version of the "start-elog-nodus" script and put it in .../elog/elog-2.8.0. It seems to work fine.

Nancy notified me that the elog crashed. It was fixed.

I restarted elog, but it kept crashing. Some of the entries on Aug 6th caused the trouble.

I tried to refresh the pictures in entries 3376, 3377, 3378. Still it kept crashing.
I started to dig into the elog file itself. (/cvs/cds/caltech/elog/elog-2.7.5/logbooks/40m/100806a.log)

FInally I found that there was some invalid reply links in the entry 3379.

Date: Fri, 06 Aug 2010 19:29:59 -0700
Reply to: 3379
In reply to: 3379

The entry is refering this entry itself. That is weird. So I deleted the reply-to and in-reply-to lines.
Then elogd got happy.

In fact, 3379 was a dupulication of 3380, so I deleted this entry.

Restarted elog.

found it unresponsive. Restarted fine using procedure documented in wiki

Elog did not respond despite running the /cvs/cds/caltech/elog/start-elog.csh  script two times.  

It worked the after the third restart. 
 

The elog was not responding. I attempted to restart it by running .../start-elog.csh, but this didn't work (even after the usual ~2 times it needs).

Somehow pkill did not kill the daemon, so I kill -9'd it and that did the trick. I ran the start script once more and it came back online.