40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 338 of 344  Not logged in ELOG logo
ID Date Author Type Categoryup Subject
  3466   Tue Aug 24 22:26:16 2010 ZachUpdateelogrestarted

took two again

Quote:

 Restarted the elog using the script. I had to do it twice for it to work. This is not the first time this has happened---does anyone know why this might be?

 

  3467   Wed Aug 25 12:18:47 2010 josephbUpdateelogTrying new version of elog to see if it helps stability

So unfortunately, I made the start-elog-nodus script smart enough to kill the debugging run I had (although thats probably good since there might have been issues with continuing to run - just poor timing on part of the crash).

In related news, I have gotten the latest version of the elog code to actually compile on Nodus.  I had to hack the cryptic.c file (elog/src/cryptic.c) to get it to work though.

The following was copied from the #ifdef _MSC_VER section of the code into the #else directly following that section. 

#define MAX(x,y) ((x)>(y)?(x):(y))
#define MIN(x,y) ((x)<(y)?(x):(y))
#define __alignof__(x) sizeof(x)
#define alloca(x) malloc(x)
#define mempcpy(d, s, n) ((char *)memcpy(d,s,n)+n)
#define ERANGE 34


I also removed #include <stdint.h> as the functionality it provides is covered by inttypes.h on Solaris machines, which is automatically included.

This new code was released August 5th 2010, while the old elog code we were running was 2.7.5 and was released sometime in 2008.  There are several crash fixes mentioned in the version notes so I'm hoping this may improve stability. I'm in the process of making a copy of the elog logbooks into the elog-2.8.0 install (so as to have a backup with the original 2.7.5).  I'm also copying over all the configuration files.   In a few minutes I'm going to try switching over to the new elog.  If it doesn't work, or is worse, its easy enough to just start up the current version.

All files are located in /cvs/cds/caltech/elog/elog-2.8.0 (the old directory is elog-2.7.5).  I've made  a new startup script called start-elog-nodus-2.8.0.  To start the new one, just run that script.  To start the old one, just go to the elog-2.7.5 directory and run the old start-elog-nodus script.

  3468   Wed Aug 25 12:40:28 2010 josephbUpdateelogReverted back to 2.7.5 until further testing is done

So apparently the themes/configurations didn't work so nicely on some of the logbooks with 2.8.0, so I'm reverting to 2.7.5 until I can figure out (assuming I can) how to get them to display properly.

  3471   Wed Aug 25 15:55:33 2010 josephbUpdateelogStaying with 2.7.5 until passwords sorted out

Turns out the elog version 2.8.0 uses a different encryption method than 2.7.5.  This mean the encrypted passwords stored in the elogd.cfg don't work with the new code.  elogd includes functionality to generate encrypted passwords, but unfortunately I don't know the administration passwords for some of the logbooks.  So I'm going to leave 2.7.5 running until I can get those added properly to the 2.8.0 cfg file.

  3533   Tue Sep 7 14:42:02 2010 DmassConfigurationelogelog restarted

elog crashed on an upload. restarted and it worked fine with the same file.

  3538   Tue Sep 7 22:21:47 2010 DmassConfigurationelogelog restarted

Quote:

elog crashed on an upload. restarted and it worked fine with the same file.

 Again. Resubmitted an old entry with just text changes. Elog hung for 5 min +.

  3613   Mon Sep 27 21:57:59 2010 ZachUpdateelogelog restarted

 took two runs of the script as usual

  3704   Wed Oct 13 09:35:41 2010 ranaUpdateelogstart script edited

The existing elog restart script was running the kill process in the background using the '&' symbol before starting new elog process. This is a BAD idea since there's no way to make sure that the background process has actually worked before the new one tries to start.

That's why you sometimes had to run the script twice. I've removed all of the background "cleverness" so now it will take ~2s more for the script to run - however, it now actually works. We may also upgrade from v2.7.5 to 2.8 today.

  3772   Sun Oct 24 19:23:41 2010 ranaConfigurationelogELOG 2.8.0
I stopped the ELOG and restarted us on 2.8.0.

To make sure nothing got lost, I killed the old process, copied over the logbooks/, themes/, and elogd.cfg to the new 2.8.0/ directory before starting the new Daemon.

I encountered the same Administrator bug as Joe had before. I delete all the old Admin passwords to bypass the issue.

To restart the ELOGD on NODUS, you now type '/cvs/cds/caltech/elog/start-elog.csh'.
I also added ELOG to the man pages in /usr/local/man/ on nodus by putting the *.1 files in man1/ and the *.8 files into man8/.
  3775   Mon Oct 25 02:23:47 2010 KojiConfigurationelogELOG 2.8.0
When I push the reply button, the raw html shows up in the edit window and have to use HTML to write the entry.
Does this happen only to me???


Quote:
I stopped the ELOG and restarted us on 2.8.0.

To make sure nothing got lost, I killed the old process, copied over the logbooks/, themes/, and elogd.cfg to the new 2.8.0/ directory before starting the new Daemon.

I encountered the same Administrator bug as Joe had before. I delete all the old Admin passwords to bypass the issue.

To restart the ELOGD on NODUS, you now type '/cvs/cds/caltech/elog/start-elog.csh'.
I also added ELOG to the man pages in /usr/local/man/ on nodus by putting the *.1 files in man1/ and the *.8 files into man8/.
  3780   Mon Oct 25 23:59:37 2010 KojiConfigurationelogELOG 2.8.0 -> ELOG 2.7.5

ELOG reverted to 2.7.5 due to editing difficulties

- /cvs/cds/caltech/elog/start-elog.csh reconfigured to launch 2.7.5

- /cvs/cds/caltech/elog/elog is linked to ./elog-2.7.5

- logbook dir of 2.8.0 was copied in the dir of 2.7.5. The old and obsolete 2.7.5 was discarded.

  3783   Tue Oct 26 07:02:05 2010 AlbertoConfigurationelogELOG 2.8.0 -> ELOG 2.7.5

Quote:

ELOG reverted to 2.7.5 due to editing difficulties

- /cvs/cds/caltech/elog/start-elog.csh reconfigured to launch 2.7.5

- /cvs/cds/caltech/elog/elog is linked to ./elog-2.7.5

- logbook dir of 2.8.0 was copied in the dir of 2.7.5. The old and obsolete 2.7.5 was discarded.

I think I had the same problem when I switched to 2.75 from 2.65.

Then the problem was FCKeditor.

We should try the solution I put in the elog page of the wiki.

 

  3784   Tue Oct 26 10:50:08 2010 KojiConfigurationelogELOG 2.8.0 -> ELOG 2.7.5 -> ELOG 2.8.0

ELOG restarted with 2.8.0 again.

- moved elog-2.8.0/script dir to elog-2.8.0/script.orig

- copied elog-2.7.5/script to elog-2.8.0/script

- /cvs/cds/caltech/elog/start-elog.csh reconfigured to launch 2.8.0

- /cvs/cds/caltech/elog/elog is linked to ./elog-2.8.0

- logbooks on 25th and 26th were copied from 2.7.5 to 2.8.0.

 

  3795   Wed Oct 27 11:52:45 2010 josephbUpdateelogElog needed to be restarted

I had to restart the elog on Nodus because it was no longer responding.

  3989   Mon Nov 29 17:45:28 2010 ZachUpdateelogrestarted elog

 The elog was down so I restarted it. The instructions on the wiki do not work as the process has some complicated name (i.e. it is not just 'elogd'). I used kill and the pid number.

I will get around to updating the restart script to work with 2.8.0.

  3993   Tue Nov 30 11:44:36 2010 ZachUpdateelogelog restarted -- SCRIPT adapted for 2.8.0

 I have created an updated version of the "start-elog-nodus" script and put it in .../elog/elog-2.8.0. It seems to work fine.

  3994   Tue Nov 30 12:10:44 2010 josephbUpdateelogElog restarted again

The elog seemed to be down at around 12:05pm.  I waited a few minutes to see if the browser would connect, but it did not.

I used the existing script in /cvs/cds/caltech/elog/ (as opposed to Zach's new on in elog/elog-2.8.0/) which also seems to have worked fine.

  3998   Tue Nov 30 14:21:21 2010 ZachUpdateelogScript unnecessary
I didn't realize that the script in .../elog was configured to start 2.8.0 already. I will revert the instructions on the wiki to point to that 
one.
  4001   Tue Nov 30 19:35:09 2010 ZachUpdateelogrestarted again

 Restarted the elog again, this time using .../elog/start-elog.csh, which Joe pointed out works just fine. I have amended the wiki instructions to point to this script, instead.

  4017   Mon Dec 6 22:43:19 2010 FrankConfigurationelogelog restarted

I restarted the elog because i changed the configuration for the cryo-elog.

Used the "start-elog.csh" script in /cvs/cds/caltech/elog/

  4052   Tue Dec 14 09:26:17 2010 ZachUpdateelogrestarted

 Restarted the elog with the script

  4071   Sat Dec 18 06:24:49 2010 ranaUpdateelogrestarted

The process was taking up 100% of the CPU and not responding via web. The .log file showed the last action was somebody reading/editing one of Jenne's entries from August regarding TT ECD. The restart script didn't work, so I had to do a 'kill -9' to get it to die.

  4072   Sat Dec 18 23:33:06 2010 KojiUpdateelogrestarted

Did the same.

Quote:

The process was taking up 100% of the CPU and not responding via web. The .log file showed the last action was somebody reading/editing one of Jenne's entries from August regarding TT ECD. The restart script didn't work, so I had to do a 'kill -9' to get it to die.

 

  4073   Sun Dec 19 11:19:42 2010 KojiUpdateelogrestarted

Did it again. It seemed that Google bot came to the elog and tried to obtain "http://nodus.ligo.caltech.edu:8080/robots.txt". That was the last of the log.
Bot came from the AJW's homepage. Also Google FeedFecther came to the elog.

  4074   Sun Dec 19 22:45:28 2010 ranaUpdateelogrestarted

I deleted the yellow box which showed up by default when making an elog entry. Would be nice if we could make it so that you have to click a button to 'opt-in' for the yellow box rather than get it by default.

I added a 'robots.txt' file to the /users/public_html/ area using Google's instructions (it only works with robot compliant crawlers), but am not sure how to put robots.txt into the elog port.

  4081   Tue Dec 21 08:26:08 2010 ranaUpdateelogelogd is getting killed by Suresh

Suresh killed the elogd again from India. This was the log file:

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Referer: http://www.ligo.caltech.edu/~ajw/40m_upgrade.html
Cookie: elmode=threaded; __utma=65601905.411937803.1291369887.1291369887.1291369887.1; __utmz=65601905.1291369887.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SITESERVER=ID=4981c5fd42ae53c9c9e0980f2072be4f
Via: 1.1 sfire5.du.ac.in:3128 (squid/2.6.STABLE6)
X-Forwarded-For: 10.11.1.120
Cache-Control: max-age=259200
Connection: keep-alive

Stop killing our elog!

  4085   Wed Dec 22 06:50:19 2010 ranaUpdateelogelogd is getting killed by Suresh

After another elog crash, I've blacklisted the domain that Suresh is using by editing the apache httpd.conf. Let's see what happens now.

  4091   Thu Dec 23 03:15:11 2010 KojiUpdateelogelogd is getting killed by Suresh

ELOG has crashed and I restarted it.

Actually the filtering is not effective so far as elog is not using apache but has its own web server inside.
So this just block the access to port 30889 (=SVN, Dokuwiki, etc).

Quote:

After another elog crash, I've blacklisted the domain that Suresh is using by editing the apache httpd.conf. Let's see what happens now.

 

  4092   Thu Dec 23 08:54:32 2010 SureshUpdateelogthe delhi univ syndrome

Sorry folks!  I couldnt get to the elog and didnt know that the elog was crashing every time I tried to access it. 

But have found other means to access it and the elog is safe for now!

  4098   Wed Dec 29 18:53:11 2010 ranaSummaryelogfound hung - restarted

This was the error today:

GET /40m/ HTTP/1.1
Host: nodus.ligo.caltech.edu:8080
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: http://www.ligo.caltech.edu/~ajw/40m_upgrade.html
Cookie: elmode=threaded; __utma=65601905.411937803.1291369887.1291369887.1291369887.1; __utmz=65601905.1291369887.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SITESERVER=ID=4981c5fd42ae53c9c9e0980f2072be4f

  4102   Mon Jan 3 10:32:27 2011 kiwamuSummaryelogfound hung - restarted

Found exactly the same error messages at the end of the log file.

Quote: #4098

This was the error today:

GET /40m/ HTTP/1.1
Host: nodus.ligo.caltech.edu:8080
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: http://www.ligo.caltech.edu/~ajw/40m_upgrade.html
Cookie: elmode=threaded; __utma=65601905.411937803.1291369887.1291369887.1291369887.1; __utmz=65601905.1291369887.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SITESERVER=ID=4981c5fd42ae53c9c9e0980f2072be4f

 

  4111   Wed Jan 5 10:45:51 2011 KojiUpdateelogrestarted

Google bot  crashed the elog again. Then, I found that Google bot (and I) can crash elogd by trying to show the threaded view.
There looks similar issue reported to the elog forum, the author did not think this is a true bag.

Note: This happens only for the 40m elog. The other elogs (ATF/PSL/TCS/SUS/Cryo) are OK for the threaded view.

Quote:

Did it again. It seemed that Google bot came to the elog and tried to obtain "http://nodus.ligo.caltech.edu:8080/robots.txt". That was the last of the log.
Bot came from the AJW's homepage. Also Google FeedFecther came to the elog.

 

  4114   Wed Jan 5 21:19:29 2011 ZachUpdateelogrestarted

 Restarted the elog with the script

  4121   Thu Jan 6 10:47:11 2011 KojiUpdateelogELOG fixed (re: restarted)

Fixed the 40m elog crashing with the threaded display.

This morning I found that Google bot crashed the elog again. I started the investigation and found the threaded mode is fine if we use the recent 10 entries.

I gradually copied the old entries to a temporary elog and found that a deleted elog entry on August 6 had a corrupted remnant in the elog file. This kept crashed the threaded mode.

Once this entry has been eliminated again, the threaded mode got functional.

I hope this eliminates those frequent elog crashing.

Quote:

Google bot  crashed the elog again. Then, I found that Google bot (and I) can crash elogd by trying to show the threaded view.
There looks similar issue reported to the elog forum, the author did not think this is a true bag.

Note: This happens only for the 40m elog. The other elogs (ATF/PSL/TCS/SUS/Cryo) are OK for the threaded view.

Quote:

Did it again. It seemed that Google bot came to the elog and tried to obtain "http://nodus.ligo.caltech.edu:8080/robots.txt". That was the last of the log.
Bot came from the AJW's homepage. Also Google FeedFecther came to the elog.

 

 

  4168   Wed Jan 19 10:31:24 2011 josephbUpdateelogElog restarted again

Elog wasn't responding at around 10 am this morning.  I killed the elogd process, then used the restart script.

  4332   Mon Feb 21 11:05:51 2011 ZachUpdateelogrestarted

 I restarted the elog using the script.

  4334   Mon Feb 21 23:00:06 2011 ZachSummaryelogrestarted

 again

  4365   Tue Mar 1 08:42:18 2011 AidanUpdateelogRestarted the elog this morning

 The elog was dead this morning. I reanimated it. It is now undead.

Attachment 1: Zombie.gif
Zombie.gif
  4378   Fri Mar 4 13:25:04 2011 ZachUpdateelogrestarted

with script

  4438   Thu Mar 24 13:56:05 2011 josephbUpdateelogelog restarted at 1:55pm

Restarted elog.

  4556   Fri Apr 22 02:10:53 2011 ZachUpdateelogrestarted

Restarted the elog with the script.

  4571   Tue Apr 26 22:56:35 2011 ZachUpdateelogrestarted

 with script

  4575   Wed Apr 27 20:14:16 2011 AidanSummaryelogRestarted with script ...
  4741   Wed May 18 18:33:46 2011 AidanUpdateelogrestarted elog with script
  4744   Thu May 19 00:15:20 2011 JenneUpdateelogRestarted, Italian-style

Aka, from a hotel in Pisa.

  4745   Thu May 19 00:22:21 2011 ranaUpdateelogRestarted, Italian-style

Quote:

Aka, from a hotel in Pisa.

 Restarted Thu May 19 00:21:49 2011 to recover from Jenne's Italian terrorism.

  4978   Fri Jul 15 19:00:18 2011 dmassMetaphysicselogCrashes

Elog crashed a couple times, restarted it a couple times.

  4998   Wed Jul 20 11:13:59 2011 Larisa ThorneUpdateelogI restarted the ELOG as it seemed to have crashed

 

  5232   Sun Aug 14 19:06:50 2011 JenneUpdateelogelog dead. Brought back to life

like the subject says...

  5348   Tue Sep 6 21:00:48 2011 ZachUpdateelogelog restarted

I restarted the elog with the script as it was not up when I tried to make a post. It was again unresponsive when I went to submit, but this time the script couldn't restart it. The log said it couldn't bind to 8080, which usually happens if the daemon is still running. I pkilled it, then reran the script, and it appears to be working.

ELOG V3.1.3-