40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 242 of 344  Not logged in ELOG logo
ID Date Authorup Type Category Subject
  12202   Mon Jun 20 14:03:04 2016 jamieConfigurationCDSEndRun GPS receiver upgraded, fixed

I just upgraded the EndRun Technologies Tempus LX GPS receiver timing unit, and it seems to have fixed all the problems.  cool

Thanks to Steve for getting the info from EndRun.  There was indeed a bug in the firmware that was fixed with a firmware upgrade.

I upgraded both the system firmware and the firmware of the GPS subsystem:

Tempus LX GPS(root@Tempus:~)-> gntpversion 
Tempus LX GPS 6010-0044-000 v 5.70 - Wed Oct 1 04:28:34 UTC 2014
Tempus LX GPS(root@Tempus:~)-> gpsversion 
F/W 5.10 FPGA 0416
Tempus LX GPS(root@Tempus:~)->

After reboot the system is fully functional, displaying the correct time, and outputting the correct IRIG-B data, as confirmed by the VME timing unit.

I added a wiki page for the unit: https://wiki-40m.ligo.caltech.edu/NTP

 

Steve added this picture

Attachment 1: GPSreceiverWorking.jpg
GPSreceiverWorking.jpg
  12724   Mon Jan 16 22:03:30 2017 jamieConfigurationComputersMegatron update
Quote:
 

We should consider upgrading a few of our workstations to Ubuntu 14 LTS to see how painful it is to run our scripts and DTT and DV. Better to upgrade a bit before we are forced to by circumstance.

I would recommend upgrading the workstations to one of the reference operating systems, either SL7 or Debian squeeze, since that's what the sites are moving towards.  If you do that you can just install all the control room software from the supported repos, and not worry about having to compile things from source anymore.

  12763   Fri Jan 27 17:49:41 2017 jamieUpdateCDStest of new daqd code on fb1

Just FYI I'm running a test of updated daqd code on fb1. 

fb1 has it's own fiber to the daq network switch, so nothing had to be modified to do this test. This *should* not affect anything in the rest of the system, but as we all know these are famous last words....  If something is going haywire, and you can't get in touch with me and can't figure what else to do, you can just log on to fb1 and shut it down.  It's not writing any data to any of the network filesystems.

The daqd code under test is from the latest advLigoRTS 3.2.1 tag, which has daqd stability fixes that will hopefully address the problems we were seeing last time I tried this upgrade.  We'll see...

I'm going to let it run over the weekend, and will check in periodically.

  12769   Sat Jan 28 12:05:57 2017 jamieUpdateCDStest of new daqd code on fb1
Quote:

I'm not sure if this is related, but since today morning, I've noticed that the data concentrator errors have returned. Looking at daqd.log, there is a 1 second timing mismatch error that is being generated. Usually, manually running ntpdate on the front ends fixes this problem, but it did not work today.

If this problem started before ~4pm on Friday then it's probably unrelated, since I didn't start any of these tests until after that.  If unexplained problem persist then we can try shutting of the fb1 daqd and see if that helps.

  12770   Mon Jan 30 18:41:41 2017 jamieUpdateCDSTEST ABORTED of new daqd code on fb1

I just aborted the fb1 test and reverted everything to the nominal configuration.  Everything looks to be operating nominally.  Front ends are mostly green except for c1rfm and c1asx which are currently not being acquired by the DAQ, and an unknown IPC error with c1daf.  Please let me know if any unusual problems are encountered.

The behavior of daqd on fb1 with the latest release (3.2.1) was not improved.  After turning on the full pipe it was back to crashing every 10 minutes or so when the full and second trend frames were being written out.  lame.  back to the drawing board...

  12794   Fri Feb 3 11:03:06 2017 jamieUpdateCDSmore testing fb1; DAQ DOWN DURING TEST

More testing of fb1 today.  DAQ DOWN UNTIL FURTHER NOTICE.

Testing Wednesday did not resolve anything, but Jonathan Hanks is helping.

  12798   Sat Feb 4 12:20:39 2017 jamieSummaryCDS/cvs/cds/caltech/chans back on svn1.6
Quote:

True - its an issue. Koji and I are updating zita into Ubuntu16 LTS. If it looks like its OK with various tools we'll swap over the others into it. Until then I figure we're best off turning allegra back into Ubuntu12 to avoid a repeat of this kind of conflict. Once the workstations in the LLO control room are running smoothly on a new OS for a year, we can transfer into that. I don't think any of us wants to be the CDS beta tester for DV or DTT.

Just to be clear, since there seems to be some confusion, the SVN issue has nothing to do with Debian vs. Ubuntu.  SVN made non-backwards compatible changes to their working copy data format that breaks newer checkouts with older clients.  You will run into the exact same problem with newer Ubuntu versions.

I recommend the 40m start moving towards the reference operating systems (Debian 8 or SL7) as that's where CDS is moving.  By moving to newer Ubuntu versions you're moving away from CDS support, not towards it.

  12799   Sat Feb 4 12:29:20 2017 jamieSummaryCDS/cvs/cds/caltech/chans back on svn1.6

No, not confused on that point. We just will not be testing OS versions at the 40m or running multiple OS's on our workstations. As I've said before, we will only move to so-called 'reference' systems once they've been in use for a long time.

Quote:
Quote:

True - its an issue. Koji and I are updating zita into Ubuntu16 LTS. If it looks like its OK with various tools we'll swap over the others into it. Until then I figure we're best off turning allegra back into Ubuntu12 to avoid a repeat of this kind of conflict. Once the workstations in the LLO control room are running smoothly on a new OS for a year, we can transfer into that. I don't think any of us wants to be the CDS beta tester for DV or DTT.

Just to be clear, since there seems to be some confusion, the SVN issue has nothing to do with Debian vs. Ubuntu.  SVN made non-backwards compatible changes to their working copy data format that breaks newer checkouts with older clients.  You will run into the exact same problem with newer Ubuntu versions.

I recommend the 40m start moving towards the reference operating systems (Debian 8 or SL7) as that's where CDS is moving.  By moving to newer Ubuntu versions you're moving away from CDS support, not towards it.

 

  12800   Sat Feb 4 12:50:01 2017 jamieSummaryCDS/cvs/cds/caltech/chans back on svn1.6
Quote:

No, not confused on that point. We just will not be testing OS versions at the 40m or running multiple OS's on our workstations. As I've said before, we will only move to so-called 'reference' systems once they've been in use for a long time.

Ubuntu16 is not to my knowledge used for any CDS system anywhere.  I'm not sure how you expect to have better support for that.  There are no pre-compiled packages of any kind available for Ubuntu16.  Good luck, you big smelly doofuses. Nyah, nyah, nyah.

  13108   Mon Jul 10 21:03:48 2017 jamieUpdateGeneralAll FEs down

 

Quote:
 

However, FB still will not boot up. The error is identical to that discussed in this thread by Intel. It seems FB is having trouble finding its boot disk. I was under the impression that only the FE machines were diskless, and that FB had its own local boot disk - in which case I don't know why this error is showing up. According to the linked thread, it could also be a problem with the network card/cable, but I saw both lights on the network switch port FB is connected to turn green when I powered the machine on, so this seems unlikely. I tried following the steps listed in the linked thread but got nowhere, and I don't know enough about how FB is supposed to boot up, so I am leaving things in this state now. 

It's possible the fb bios got into a weird state.  fb definitely has it's own local boot disk (*not* diskless boot).  Try to get to the BIOS during boot and make sure it's pointing to it's local disk to boot from.

If that's not the problem, then it's also possible that fb's boot disk got fried in the power glitch.  That would suck, since we'd have to rebuild the disk.  If it does seem to be a problem with the boot disk then we can do some invasive poking to see if we can figure out what's up with the disk before rebuilding.

  13115   Wed Jul 12 14:52:32 2017 jamieUpdateGeneralAll FEs down

I just want to mention that the situation is actually much more dire than we originally thought.  The diskless NFS root filesystem for all the front-ends was on that fb disk.  If we can't recover it we'll have to rebuilt the front end OS as well.

As of right now none of the front ends are accessible, since obviously their root filesystem has disappeared.

  13151   Sat Jul 29 16:24:55 2017 jamieUpdateGeneralPSL StripTool flatlined
Quote:
Unrelated to this work: It looks like some/all of the FE models were re-started. The x3 gain on the coil outputs of the 2 ITMs and BS, which I had manually engaged when I re-aligned the IFO on Monday, were off, and in general, the IMC and IFO alignment seem much worse now than it was yesterday. I will do the re-alignment later as I'm not planning to use the IFO today.

This was me.  I restarted the front ends when I was getting the MX streams working yesterday.  I'll try to me more conscientious about logging front end restarts.

  13340   Thu Sep 28 11:13:32 2017 jamieUpdateCDS40m files backup situation

 

Quote:

After consulting with Jamie, we reached the conclusion that the reason why the root of FB1 is so huge is because of the way the RAID for /frames is setup. Based on my googling, I couldn't find a way to exclude the nfs stuff while doing a backup using dd, which isn't all that surprising because dd is supposed to make an exact replica of the disk being cloned, including any empty space. So we don't have that flexibility with dd. The advantage of using dd is that if it works, we have a plug-and-play clone of the boot disk and root filesystem which we can use in the event of a hard-disk failure.

  1. One option would be to stop all the daqd processes, unmount /frames, and then do a dd backup of the true boot disk and root filesystem.
  2. Another option would be to use rsync to do the backup - this way we can selectively copy the files we want and ignore the nfs stuff. I suspect this is what we will have to do for the second layer of backup we have planned, which will be run as a daily cron job. But I don't think this approach will give us a plug-and-play replacement disk in the event of a disk failure.
  3. Third option is to use one of the 2TB HGST drives, and just do a dd backup - some of this will be /frames, but that's okay I guess.

This is not quite right.  First of all, /frames is not NFS.  It's a mount of a local filesystem that happens to be on a RAID.  Second, the frames RAID is mounted at /frames.  If you do a dd of the underlying block device (in this case /dev/sda*, you're not going to copy anything that's mounted on top of it.

What i was saying about /frames is that I believe there is data in the underlying directory /frames that the frames RAID is mounted on top of.  In order to not get that in the copy of /dev/sda4 you would need to unmount the frames RAID from /frames, and delete everything from the /frames directory.  This would not harm the frames RAID at all.

But it doesn't really matter because the backup disk has space to cover the whole thing so just don't worry about it.  Just dd /dev/sda to the backup disk and you'll just be copying the root filesystem, which is what we want.

  13344   Fri Sep 29 09:43:52 2017 jamieHowToCDSpyawg

 

Quote:

I've modified the __init.py__ file located at /ligo/apps/linux-x86_64/cdsutils-480/lib/python2.7/site-packages/cdsutils/__init__.py so that you can now simply import pyawg from cdsutils. On the control room workstations, iPython is set up such that cdsutils is automatically imported as "cds". Now this import also includes the pyawg stuff. So to use some pyawg function, you would just do (for example):

exc=cds.awg.ArbitraryLoop(excChan,excit,rate=fs)

One could also explicitly do the import if cdsutils isn't automatically imported:

from cdsutils import awg

pyawg-away!


Linking this useful instructional elog from Chris here: https://nodus.ligo.caltech.edu:8081/Cryo_Lab/1748

?  Why aren't you able to just import 'awg' directly?  You shouldn't have to import it through cdsutils.  Something must be funny with the config.

  13350   Mon Oct 2 18:50:55 2017 jamieUpdateCDSc1ioo DC errors
Quote:

 

  • This time the model came back up but I saw a "0x2000" error in the GDS overview MEDM screen.
  • Since there are no DACs installed in the c1ioo expansion chassis, I thought perhaps the problem had to do with the fact that there was a "DAC_0" block in the c1x03 simulink diagram - so I deleted this block, recompiled c1x03, and for good measure, restarted all (three) models on c1ioo.
  • Now, however, I get the same 0x2000 error on both the c1x03 and c1als GDS overview MEDM screens (see Attachment #1).

From page 21 of T1100625, DAQ status "0x2000" means that the channel list is out of sync between the front end and the daqd.  This usually happens when you add channels to the model and don't restart the daqd processes, which sounds like it might be applicable here.

It looks like open-mx is loaded fine (via "rtcds lsmod"), even though the systemd unit is complaining.  I think this is because the open-mx service is old style and is not intended for module loading/unloading with the new style systemd stuff.

  13383   Tue Oct 17 17:53:25 2017 jamieSummaryLSCprep for tests of Gabriele's neural network cavity length reconstruction

I've been preparing for testing Gabriele's deep neural network MICH/PRCL reconstruction.  No changes to the front end have been made yet, this is all just prep/testing work.

Background:

We have been unable to get Gabriele's nn.c code running in kernel space for reasons unknown (see tests described in previous post).  However, Rolf recently added functionality to the RCG that allows front end models to be run in user space, without needing to be loaded into the kernel.  Surprisingly, this seems to work very well, and is much more stable for the overall system (starting/stopping the user space models will not ever crash the front end machine).  The nn.c code has been running fine on a test machine in this configuration.  The RCG version that supports user space models is not that much newer than what the 40m is running now, so we should be able to run user space models on the existing system without upgrading anything at the 40m.  Again, I've tested this on a test machine and it seems to work fine.

The new RCG with user space support compiles and installs both kernel and user-space versions of the model.

Work done:

  • Create 'c1dnn' model for the nn.c code.  This will run on the c1lsc front end machine (on core 6 which is currently empty), and will communicate with the c1lsc model via SHMEM IPC.  It lives at:
    • /opt/rtcds/userapps/release/isc/c1/models/c1dnn.mdl
  • Got latest copy of nn.c code from Gabriele's git, and put it at:
    • /opt/rtcds/userapps/release/isc/c1/src/nn/
  • Checked out the latest version of the RCG (currently SVN trunk r4532):
    • /opt/rtcds/rtscore/test/nn-test
  • Set up the appropriate build area:
    • /opt/rtcds/caltech/c1/rtbuild/test/nn-test
  • Built the model in the new nn-test build directory ("make c1dnn")
  • Installed the model from the nn-test build dir ("make install-c1dnn")

Test:

I tried a manual test of the new user space model.  Since this is a user space process running it should have no affect on the rest of the front end system (which it didn't):

  • Manually started the c1dnn EPICS IOC:
    • $ (cd /opt/rtcds/caltech/c1/target/c1dnn/c1dnnepics && ./startupC1)
  • Tried running the model user-space process directly:
    • $ taskset -c 6 /opt/rtcds/caltech/c1/target/c1dnn/bin/c1dnn -m  c1dnn

Unfortunately, the process died with an "ADC TIMEOUT" error.  I'm investigating why.

Once we confirm the model runs, we'll add the appropriate SHMEM IPC connections to connect it to the c1lsc model.

Attachment 1: c1dnn.png
c1dnn.png
  13386   Wed Oct 18 01:41:32 2017 jamieUpdateCDSFEs unresponsive
Quote:

While working on the IFO tonight, I noticed that the blinky status lights on c1iscex and c1iscey were frozen (but those on the other 3 FEs seemed fine). But all other lights on the CDS overview screen were green I couldn't access testpoints from these machines, and the EPICS readbacks for models on these FEs (e.g. Oplev servo inputs outputs etc) were frozen at some fixed value. This lasted for a good 5 minutes at least. But the blinky lights started blinking again without me doing anything. Not sure what to make of this. I am also not sure how to diagnose this problem, as trending the slow EPICS records of the CPU execution cycle time (for example) doesn't show any irregularity.

So this wasn't just an EPICS freeze?  I don't see how this had anything to do with any of the work I did earlier today.  I didn't modify any of the running front ends, didn't touch either of the end station machines or the DAQ, and didn't modify the network in any way.  I didn't leave anything running.

If you couldn't access test points then it sounds like it was more than just EPICS.  It sounds like maybe the end machines somehow fell of the network momentarily.  Was there anything else going on at the time?

  13388   Wed Oct 18 09:21:22 2017 jamieUpdateCDSFEs unresponsive
Quote:

I was looking at the ASDC channel on dataviewer, and toggling various settings like whitening gain. At some point, the signal just froze. So I quit dataviewer and tried restarting it, at which point it complained about not being able to connect to FB. This is when I brought up the CDS_OVERVIEW medm screen, and noticed the frozen 1pps indicator lights. There was certainly something going on with the end FEs, because I was able to ping the machine, but not ssh into it. Once the 1pps lights came back, I was able to ssh into c1iscex and c1iscey, no problems.

Could it be that some of the mx processes stalled, but the systemctl routine automatically restarted them after some time?

An mx_stream glitch would have interrupted data flowing from the front end to the DAQ, but it wouldn't have affected the heartbeat.  The heartbeat stop could mean either that the front end process froze, or the EPICS communication stopped.  The fact that everything came back fine after a couple of minutes indicates to me that the front end processes all kept running fine.  If they hadn't I'm sure the machines would have locked up.  The fact that you couldn't connect to the FE machine is also suspicious.

My best guess is that there was a network glitch on the martian network.  I don't know how to account for the fact that pings still worked, though.

  13390   Wed Oct 18 12:14:08 2017 jamieSummaryLSCprep for tests of Gabriele's neural network cavity length reconstruction
Quote:

I tried a manual test of the new user space model.  Since this is a user space process running it should have no affect on the rest of the front end system (which it didn't):

  • Manually started the c1dnn EPICS IOC:
    • $ (cd /opt/rtcds/caltech/c1/target/c1dnn/c1dnnepics && ./startupC1)
  • Tried running the model user-space process directly:
    • $ taskset -c 6 /opt/rtcds/caltech/c1/target/c1dnn/bin/c1dnn -m  c1dnn

Unfortunately, the process died with an "ADC TIMEOUT" error.  I'm investigating why.

Once we confirm the model runs, we'll add the appropriate SHMEM IPC connections to connect it to the c1lsc model.

I tried moving the model to c1ioo, where there are plenty of free cores sitting idle, and the model seems runs fine.  I think the problem was just CPU contention on the c1lsc machine, where there were only two free cores and the kernel was using both for all the rest of the normal user space processes.

So there are two options:

  • Use cpuset on c1lsc to tell the kernel to remove all other processes from CPU6 and save it just for the c1dnn model.  This should not have any impact on the running of c1lsc, since that's exactly what would be happening if we were running the model in kernel space (e.g. isolating the core for the front end model).  The auxilliary support user space processes (epics seq/ioc, awgtpman) should all run fine on CPU0, since that's what usually happens.  Linux is only using the additional core since it's there.  We don't have much experience with cpuset yet, though, so more offline testing will be required first.
  • Run the model on c1ioo and ship the needed signals to/from c1lsc via PCIe dolphin.  This is potentially slightly more invasive of a change, and would put more work on the dolphin network, but it should be able to handle it.

I'm going to start testing cpuset offline to figure out exactly what would need to be done.

  13395   Thu Oct 19 15:42:03 2017 jamieSummaryLSCMICH/PRCL reconstruction neural network running on c1lsc

Gabriele's PRCL/MICH reconstruction neural network is now running on c1lsc.  Summary:

  • front-end model is called c1dnn, and is running as an experimental user-space process
  • c1dnn is getting most of it's needed inputs from existing SHMEM IPC outputs from c1lsc
  • none of the output signals from the network are being sent anywhere yet (grounded)
  • c1dnn has not been integrated in any way, into the DAQ etc.  it is being run manually by hand, and will be completely shut down after this test

Simple MEDM screen I made to monitor the input/output signals:

The RTS process seems to run fine, but there is quite a bit of jitter in the CPU_METER, at the 50% level:

It's not running over the limit, but it is jumping around more than I think it should be.  Will look into that...

cpuset for cpu isolation for user-space model

The c1dnn model is running on CPU6 on c1lsc.  CPU6 was isolated from the rest of the system using cpuset.  The "cset" utility was used to create a "system" CPU set that was assigned to CPU0, and the kernel was instructed to move all running processes to that set:

controls@c1lsc:~ 2$ sudo cset set
cset:
         Name       CPUs-X    MEMs-X Tasks Subs Path
 ------------ ---------- - ------- - ----- ---- ----------
         root        0,6 y       0 y   343    0 /
controls@c1lsc:~ 0$ sudo cset set -c 0 -s system --cpu_exclusive
cset: --> created cpuset "system"
controls@c1lsc:~ 0$ sudo cset set
cset:
         Name       CPUs-X    MEMs-X Tasks Subs Path
 ------------ ---------- - ------- - ----- ---- ----------
         root        0,6 y       0 y   342    1 /
       system          0 y       0 n     0    0 /system
controls@c1lsc:~ 0$ sudo cset proc --move -f root -t system -k
cset: moving all tasks from root to /system
cset: moving 292 userspace tasks to /system
cset: moving 0 kernel threads to: /system
cset: --> not moving 50 threads (not unbound, use --force)
[==================================================]%
cset: done
controls@c1lsc:~ 0$ sudo cset set
cset:
         Name       CPUs-X    MEMs-X Tasks Subs Path
 ------------ ---------- - ------- - ----- ---- ----------
         root        0,6 y       0 y    50    1 /
       system          0 y       0 n   292    0 /system
controls@c1lsc:~ 0$ sudo cset proc --move -f root -t system -k --force
cset: moving all tasks from root to /system
cset: moving 50 kernel threads to: /system
[==================================================]%
cset: **> 29 tasks are not movable, impossible to move
cset: done
controls@c1lsc:~ 0$ sudo cset set
cset:
         Name       CPUs-X    MEMs-X Tasks Subs Path
 ------------ ---------- - ------- - ----- ---- ----------
         root        0,6 y       0 y    29    1 /
       system          0 y       0 n   313    0 /system
controls@c1lsc:~ 0$

I then created a set for the RTS process ("rts-c1dnn") on CPU6, and executed the c1dnn model in that set:

controls@c1lsc:~ 0$ sudo cset set -c 6 -s rts-c1dnn --cpu_exclusive
cset: --> created cpuset "rts-c1dnn"
controls@c1lsc:~ 0$ sudo cset set
cset:
         Name       CPUs-X    MEMs-X Tasks Subs Path
 ------------ ---------- - ------- - ----- ---- ----------
         root        0,6 y       0 y    24    2 /
    rts-c1dnn          6 y       0 n     0    0 /rts-c1dnn
       system          0 y       0 n   340    0 /system
controls@c1lsc:~ 0$ sudo cset proc -s rts-c1dnn --exec /opt/rtcds/caltech/c1/target/c1dnn/bin/c1dnn -- -m c1dnn
cset: --> last message, executed args into cpuset "/rts-c1dnn", new pid is: 27572
sysname = c1dnn
....

When done I just hit Ctrl-C.

I left the cpusets as they are, with all system processes in the "system" set.  This should not pose any problems since it's the identical configuration as would be if a normal kernel-level model was running in CPU6.

The c1dnn process and it's EPICS sequencer were shutdown after this test.

  13400   Tue Oct 24 20:14:21 2017 jamieSummaryLSCfurther testing of c1dnn integration; plugged in to DAQ

In order to try to isolate CPU6 for the c1dnn neural network reconstruction model, I set the CPUAffinity in /etc/systemd/system.conf to "0" for the front end machines.  This sets the cpu affinity for the init process, so that init and all child processes are run on CPU0.  Unfortunately, this does not affect the kernel threads.  So after reboot all user space processes where on CPU0, but the kernel threads were still spread around.  Will continue trying to isolate the kernel as well...

In any event, this amount of isolation was still good enough to get the c1dnn user space model running fairly stably.  It's been running for the last hour without issue.

I added the c1dnn channel and testpoint files to the daqd master file, and restarted daqd_dc on fb1, so now the c1dnn channels and test points are available through dataviewer etc.  We were then able to observe the reconstructed signals:

We'll need to set the phase rotation of the demodulated RF PD signals (REFL11, REFL55, AS55, POP22) to match them with what the NN expects...

  13411   Mon Nov 6 18:22:48 2017 jamieSummaryLSCcurrent procedure for running c1dnn code

This is the current procedure to start the c1dnn model:

$ ssh c1lsc
$ sudo systemctl start rts-epics@c1dnn
$ sudo systemctl start rts-awgtpman@c1dnn
$ sudo /usr/bin/cset proc -s rts-c1dnn --exec /opt/rtcds/caltech/c1/target/c1dnn/bin/c1dnn -- -m c1dnn
...

Then to shutdown:

...
Ctrl-C
$ sudo systemctl stop rts-awgtpman@c1dnn
$ sudo systemctl stop rts-epics@c1dnn

The daqd already knows about this model, so nothing should need to be done to the daqd to make the dnn channels available.

  13480   Fri Dec 15 01:53:37 2017 jamieUpdateCDSCDS recovery, NFS woes
Quote:

I would make a detailed post with how the problems were fixed, but unfortunately, most of what we did was not scientific/systematic/repeatable. Instead, I note here some general points (Jamie/Koji can addto /correct me):

  1. There is a "known" problem with unloading models on c1lsc. Sometimes, running rtcds stop <model> will kill the c1lsc frontend.
  2. Sometimes, when one machine on the dolphin network goes down, all 3 go down.
  3. The new FB/RCG means that some of the old commands now no longer work. Specifically, instead of telnet fb 8087 followed by shutdown (to fix DC errors) no longer works. Instead, ssh into fb1, and run sudo systemctl restart daqd_*.

This should still work, but the address has changed.  The daqd was split up into three separate binaries to get around the issue with the monolithic build that we could never figure out.  The address of the data concentrator (DC) (which is the thing that needs to be restarted) is now 8083.

Quote:

UPDATE 8:20pm:

Koji suggested trying to simply retsart the ASS model to see if that fixes the weird errors shown in Attachment #2. This did the trick. But we are now faced with more confusion - during the restart process, the various indicators on the CDS overview MEDM screen froze up, which is usually symptomatic of the machines being unresponsive and requiring a hard reboot. But we waited for a few minutes, and everything mysteriously came back. Over repeated observations and looking at the dmesg of the frontend, the problem seems to be connected with an unresponsive NFS connection. Jamie had noted sometime ago that the NFS seems unusually slow. How can we fix this problem? Is it feasible to have a dedicated machine that is not FB1 do the NFS serving for the FEs?

I don't think the problem is fb1.  The fb1 NFS is mostly only used during front end boot.  It's the rtcds mount that's the one that sees all the action, which is being served from chiara.

  16325   Tue Sep 14 15:57:05 2021 jamieFrogsCDSfb1 /var full after reboot, caused all sorts of problems

/var on fb1 filled up today, which caused all sorts of CDS issues.  I found out about the problem by reading the logs of the services that were having trouble running, in which they complained about not being able to write to disk.  I looked at the filesystem status with 'df' and noticed that /var was full, which is where applications write temporary data, and will always cause problems if it's full.

I tracked the issue down to multiple multi-gigabyte log files: /var/log/messages and /var/log/messages.1.  They were full of lines like this one:

Aug 29 06:25:21 fb1 kernel: l called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl called cmd = 1gpstime iotcl ca

Seems like something related to the gpstime kernel module?

Anyway, I deleted the log files for now, which cleared up the space on /var.  Things should be back to normal now, until the logs fill up again...

  16327   Tue Sep 14 16:44:54 2021 jamieFrogsCDSfb1 /var full after reboot, caused all sorts of problems

Jonathan Hanks pointed me to this fix to the gpstime kernel module that was unfortunately put in after the 3.4 release that we're currently using:

https://git.ligo.org/cds/advligorts/-/commit/6f6d6e2eb1d3355d0cbfe9fe31ea3b59af1e7348

I hacked the source in place (/usr/src/gpstime-3.4/drv/gpstime/gpstime.c) to get the fix, and then rebuilt the kernel module with dkms :

sudo dkms uninstall gpstime/3.4
sudo dkms install gpstime/3.4

I then stopped daqd_dc, unloaded gpstime, reloaded it, restarted daqd_dc.  The messages are no longer showing up in /var/log/messages, so I think we're ok for the moment.

NOTE: the fix will be undone if we for some reason reinstall the advligorts-gpstime-dkms package.  There shouldn't be a need to do that, but we should be aware.  I'm discussing with Jonathan if we want to try to push out a new debian package to fix this issue...

  9786   Mon Apr 7 15:26:32 2014 jamie, ericqUpdateCDSaborted attempt to update c1sus machine with second CPU

This morning we attempted to replace the c1sus front end machine with a spare that had been given a second CPU, and therefore 6 additional cores (for a total of 12).  The idea was to give c1sus more cores so that we could split up c1rfm into two separate models that would not be running on the hairy edge of their cycle time allocation.  Well, after struggling to get it working we eventually aborted and put the old machine back in.

The problem was that the c1sus model was running erratically, frequently jumping up to 100 usec of a 60 usec clock allocation.  We eventually tracked the problem down to the fact that the CPUs in the new machine are of an inferior and slower model, than what's in the old c1sus machine.  The CPU were running about 30% slower, which was enough to bump c1sus, which nominally runs at ~51 usec, over it's limit.

This is of course stupid, and I take the blame.  I skimped on the CPUs when I bought the spare machines in an attempt to keep the cost down, and didn't forgot that I had done that when we started discussing using one of the spares as a c1sus replacement.

I think we can salvage things, though, by just purchasing a better CPU, one that matches what's currently in c1sus.  I'll get Steve on it:

c1sus CPU: Intel(R) Xeon(R) CPU X5680 3.33GHz

In any event, the old c1sus is back in place, and everything is back as it was.

Attachment 1: Screenshot-Untitled_Window.png
Screenshot-Untitled_Window.png
  5248   Tue Aug 16 11:49:17 2011 jamie, jenneUpdateGeneraltoday's work to do

>If necessary steer ETMs and ITMs to make the X and Y green beam flashing.

Green is now flashing in both X and Y arms

>Open the IOO and OMC chamber and lock MC.

Open, and cover in place. MC is flashing and locking.

 

  7235   Mon Aug 20 13:10:31 2012 jamie, jenneUpdateSUSETMX is not happy

Quote:

ETMX has some periodic oscillation. It's damping was found tripped this morning. 

We tracked this down to the power normalization stuff that Yoichi added over the weekend.

With a non-zero normalization factor, and a small TRX transmission, the input the XARM controller gets really big.  When XARM is then triggered, a huge impulse is sent into the SUS_ETMX_LSC input, which causes the Vio2 filter in FM0 to ring like crazy.  This probably also explains why Yoichi was seeing trouble locking the arm when the normalization is on

The solution, as Yoichi also mentions, is probably to trigger the normalization like we trigger the rest of the boost filters.

  7671   Mon Nov 5 19:38:52 2012 jamie, jenne, ayaka, denUpdateAlignmentmore alignment woes

Earlier this morning we thought things were looking pretty good.  IPPOS, IPANG, and the AS and REFL spots looked like they hadn't moved too much over the weekend.  Our plan was to do a quick check of things, check clearances, etc., tweak up the oplevs, and then close up.  This is when I made the ill-fated decisions to check the table levelling.

The BS table was slightly off so I moved one of the thick disk weights off of the other disk weight that it was sitting on, and on to the table next to it.  This seemed to improve things enough so I left it there.  ITMY didn't need any adjustment, and I move a couple smaller weights around on ITMX.  Meanwhile Jenne was adjusting the output PSL power back into it's nominal range (<100mW), and re-tweaking up the mode cleaner.

When we then looked at the vertex situation again it was far off in yaw.  This was clearly evident on PZT2, where the beam was no longer centered on the PZT2 mirror and was near the edge.  This was causing us to clip at the BS aperture.

We took some deep breaths and tried to figure out what we did that could have messed things up.

Jenne noticed that we had moved slightly on the PSL QPDs, so she adjusted the PSL output pointing to re-aquire the previous pointing, and realigned the MC.  This had a very small positive affect, but not nearly enough to compensate for whatever happened.

We spent some more time trying to track down what might have changed, but were unable to come up with anything conclusive.  We then decided to see if we could recover things by just adjusting the PZT input steering mirrors.  We couldn't; recentering at PRM, BS, ITMY, and ETMY was moving us off of PR3.

Jenne suggested we look at the spot positions on the MMT mirrors.  I had checked MMT1 and it looked ok, but we hadn't looked at MMT2.  When we checked MMT2 we noticed that we were also off in yaw.  This made us consider the possibility that the BS table had twisted, most likely when I was securing the moved mass.  Sure enough, when I manually twisted BS table, by grabbing it with my hand, very little force would cause the input beam to walk much of the way across PZT2, more than accounting for the offset.  The effect was also very clearly hysteretic as well; I could twist the table a little and it would stay in the new position.

At this point we had fucked things up enough that we realized that we're basically going to have to walk through the whole alignment procedure again, for the third time this vent.  We were able to recover the PRM retro-reflection a bit, but the tip-tilts have drifted in pitch (likely again because of the table levelling).  So we're going to have to walk through the whole procedure systematically again.

Lessons learned:  Many things are MUCH more sensitive than I had been assuming.  The tip-tilts are of course ridiculous, in that lightly touching the top or bottom of the mirror mount will move it by quite a lot in pitch.  The tables are also much more sensitive than I had realized.  In particular, tightening screws can twist the table hystereticly by milliradians, which can of course completely loose the pointing.  We need to be a lot more careful.

Assuming the table hasn't moved too much we should be able to recover the alignment by just adjusting the PZTs and tweaking the pitch of the tip-tilts.  At least that's the hope.    No more touching the table.  No more leveling.  Hopefully we can get this mostly done tomorrow morning.

  5288   Tue Aug 23 14:49:14 2011 jamie, jenne, kiwamu, suresh, keikoUpdateSUSAdjustment of ETMY, issue with ITMY whitening

Before lunch we took a closer look at two of the suspensions that were most problematic: ITMY and ETMY.  Over lunch we took new free swinging data.  Results below:

  • For ITMY we discovered that the whitening on the UL sensor was not switching.  This was causing the UL sensor to have a different response, with a steeper roll of, which was causing all of the transfer function estimates to the other sensors to have large imaginary components.   We took new free swing data with all of the whitening turned OFF.  The result is a much improved matrix and diagnalization.  The input matrix elements are mostly the same, but the coupling is basically gone.  We'll fix the whitening after the pump down.
ITMY ITMY.png       pit     yaw     pos     side    butt
UL    0.157   1.311   1.213  -0.090   0.956 
UR    1.749  -0.490   0.886  -0.038  -1.042 
LR   -0.251  -2.000   0.787  -0.007   1.066 
LL   -1.843  -0.199   1.114  -0.059  -0.936 
SD   -0.973  -0.205   1.428   1.000   0.239 
4.34779
  • ETMY has a very problematic SIDE OSEM.  The magnet does not line up with the OSEM axis, and since there is no lateral adjustment in the side OSEMs, there's not much we can do about this.  We're using aluminum foil to wedge the OSEM over as far as possible, but it's not quite enough.  With the OSEM plates horizontal there is a lot of POS->SIDE coupling.  With the OSEM plates vertical, the magnetic sits a little too close to the rear face, which can cause the magnet to get stuck to the LED plate.  We're trying to decide where to leave it now, but the new diagnalization with the OSEM plates vertical is definitely better: 
ETMX ETMY.png        pit     yaw     pos     side    butt
UL   -0.138   1.224   1.463  -0.086   0.944  
UR    0.867  -0.776   1.501  -0.072  -1.051  
LR   -0.995  -0.896   0.537  -0.045   0.754  
LL   -2.000   1.104   0.499  -0.059  -1.251  
SD    0.011   0.220   1.917   1.000   0.224 
 4.42482
  5295   Wed Aug 24 11:30:27 2011 jamie, jenne, kiwamu, suresh, steveUpdateSUSETMX wiped, replaced, door on

We've closed up ETMX:

  • the optic was drag wiped
  • the suspension tower was put back in place
  • earthquake stops were backed off the appropriate number of turns, and de-ionized
  • chamber door was put on
  5296   Wed Aug 24 11:40:21 2011 jamie, jenne, kiwamu, suresh, steveUpdateSUSproblem with ITMX

ITMX was drag wiped, and the suspension was put back into place.  However, after removing all of the earthquake stops we found that the suspension was hanging in a very strange way.

The optic appears to heavily pitched forward in the suspension.  All of the rear face magnets are high in their OSEMs, while the SIDE OSEM appears fine.  When first inspected, some of the magnets appeared to be stuck to their top OSEM plates, which was definitely causing it to pitch forward severely.  After gently touching the top of the optic I could get the magnets to sit in a more reasonable position in the OSEMs.  However, they still seem to be sitting a little high.  All of the PDMon values are also too low:

  nominal now
UL 1.045 0.767
UR 0.855 0.718
LR

0.745

0.420

LL

0.780

0.415
SD

0.840

0.752

Taking a free swing measurement now.

  5259   Thu Aug 18 00:53:48 2011 jamie, kiwamu, suresh, jenneUpdateGeneralPUMP PLAN ABORTED; need to work more on IFO alignment

We have decided that the IFO alignment is bad enough that we're not ready to pump down.  PUMP ABORTED.

The IFO alignment is somewhat OK, in that the green and IR beams are flashing in the arms, and the return beams are overlapping at the BS.  However the beams appear to be not centered on any of the optics at the moment.  They are all displaced in yaw by ~0.5 to 1 cm or so in various directions.

From this we have decided that we need to step back and reattack the IFO alignment from square one.  Here is our current suggested procedure:

  1. check ETM positions relative to what we think they should be on the drawings.  This is to verify that the ETMs were not placed in the wrong places laterally.
  2. translate Y green axis north, centering green on ETMY and ITMY (by looking at cards).  North is the opposite direction from how the beams are displaced from the TM centers.
  3. steer input pointing to overlap IR on green beam at BS, ITMY, and ETMY.  IR should visibly overlap green at both BS and ITMY, and we should be able to see IR on target in front of ETMY with ETMY face camera, and in ETMY trans camera.
  4. center IR on ETMX by steering BS with DC bias.
  5. align Y arm cavity for green resonance by adjusting ITMY.
  6. adjust ITMX to achieve michelson fringes at AS
  7. adjust PRM lateral translation to center beam on PRM, if needed
  8. adjust SRM lateral translation to center beam on SRM, if needed
  9. align PRC to see fringes
  10. align SRC to see fringes
  11. extract AS (no clipping)

 Once this is done, we will need to check the following:

  • IPANG/IPPOS extraction
  • pick-off extraction
  • OPLEVs
  • OSEMs
  • green periscopes and green beam extraction at PSL

We've decided to stop for the night, get a good nights rest, and attack all of this tomorrow morning.

Beam_spot_shifts.png

  5258   Wed Aug 17 20:14:49 2011 jamie, kiwamu, suresh, jenne, keiko, anamariaUpdateGeneralin-vacuum work status, prep for pump

This afternoon's work:

  • OSEMs were adjusted on all suspended optics.
  • X and Y-arms were aligned to green.
  • Once that was done, the input pointing was adjusted with the PZTs to get the IR beam centered on ITMY and ETMY.
  • Once the input pointing was ok we extracted IPANG and IPPOS.
  • BS, ITMY, PRM, SRM optical levers (oplevs) were extracted.
  • Prepared rails and stops for TMs for morning drag wiping.

TODO before drag wiping:

  • Check full IFO alignment.
  • Readjust OSEMs if needed.
  • Extract ITMX oplev.
  7599   Tue Oct 23 17:30:33 2012 jamie, nic, jenne, raji, manasaUpdateAlignmentInitial attempts to fix IFO alignment

We went into the vertex today to see about fixing the alignment.  The in-air access connector is in place, and we took heavy doors off of BS, ITMY, and ETMY chambers.

We started by looking at the pointing from the PZTs.  Manasa and Raji hooked up HV power supplies to the PZTs and set them to the middle of their ranges (75 V).

We installed a target on the BS cage, and new "free standing" targets made special by Steve for the SOSs on ITMY and ETMY.

Using a free-standing aperture target we looked at the beam height before PZT2.  It was a little high, so we adjusted it with PZT1.  Once that was done we looked at the beam height at PR2, and adjusted that height with PZT1.

We then tried to use the hysteresis in PR2 to adjust the beam height at ITMY.  Pushing just a little bit at the top or bottom of PR2 would repoint the beam in pitch.  This sort of works, but it's stupid.  Using this method we got the beam more or less centered vertically at ITMY.

We moved on to ETMY with the idea that we would again use the hysteresis in PR3 to get the vertical pointing to the ETM correct.  This was a good demonstration of just how stupid the tip-tilts really are.  Just touching slightly at the top or bottom or PR3 we could completely change the pointing at ETMY, by mili-radians (~4 cm over 40m).

At this point I cried foul.  This is not an acceptable situation.  Very little stimulation to the tip-tilts can repoint the beam inside the PR cavity.

Steve says that the TT weights, which will attach to the base of the TT mirror mounts and should help keep the mirrors vertical and not hysteretic, are being baked now and should be available tomorrow.  We therefore decided to stop what we were doing today, since we'll have to just redo it all again tomorrow once the weights are installed.

 

  9276   Thu Oct 24 09:08:27 2013 jamie.UpdateLSCPRMI + 2 ALS arms

nice.

  1898   Thu Aug 13 11:20:43 2009 janoschHowToPEMthree-channel self-noise estimation

There are two new Matlab files on the svn in /mDV/extra/C1. 'mycsd.m' is to calculate the cross-spectral density between two channels, 'csd_40T_40T_SS1.m' calls this function with the available seismic channels and derives a self-noise spectrum for the vertical axis using all three seismometers. The method requires that there are no correlations between two instruments only which is a bad idealization for certain frequencies if you have seismometers of totally different types.

'mycsd.m' uses the high-gain, low-resolution Nuttall window (built-in Matlab function 'nuttallwin.m'). High-gain windows are used for broad-band spectra like seismic spectra, but it should be exchanged by another window if you plan to look at small-bandwidth features like peaks.

Since the three-channel analysis does not require knowledge of response functions, it could be used to evaluate the performance of the adaptive filter. For example, if three channels responding to the same signal are available, then the ratio of any two csds corresponds to one of the relative transfer functions. You can then compare this function with the result produced by the adaptive filter.

  7029   Wed Jul 25 15:33:55 2012 janoschUpdateLSCringdown measurement

We did our first ringdown measurement on the Y arm. First we tried to keep the arm locked in green during the ringdown, but for some reason it was not possible to get the cavity locked in green. So we decided to do the first measurement with infrared locked only.

For the measurement we had to change the LSC model to acquire the C1:LSC-TRY_OUT_DQ at higher sampling frequency. We changed the sampling frequencies of C1:LSC-{TRX,TRY}_OUT_DQ from 2048Hz to 16384Hz.

The measurement was done at GPS 1027289507. The ringdown curve looks very clean, but there seem to be two time constants involved. The first half of the curve is influenced by the shutter speed, then curvature is changing sign and the ringdown is likely taking over. We will try to fit a curve to the ringdown part, but it would certainly be better to have a faster shutter and record a more complete ringdown.

Rindown1.png

 

  7038   Thu Jul 26 13:10:51 2012 janoschUpdateLSCmodelled ringdown

We fitted shutter and ringdown functions to the ringdown data. It is not perfectly clear how the power change due to the shutter is handed over to the power change due to ringdown. The fit suggests that the ringdown starts at a later time, but this does not necessarily make sense. It could be that the slow power decrease when the shutter starts clipping the TEM00 beam is followed by the cavity, but then the power decrease becomes too fast when the shutter reaches the optical axis and the ringdown takes over. Also, the next measurement should be taken with adjusted DC offset.

Ringdown.png

  7301   Tue Aug 28 18:28:21 2012 janoschMetaphysicsRingdownripples

Let's see if the ripples observed in the MC ringdown can be due to tilt motion of the mirrors.

The time it takes to produce a phase shift corresponding to N multiples of 2*pi is given by:

t = sqrt(2*N*lambda/(L*omega_T^2*(alpha_1+alpha_2)))

L is the length of the MC (something like 13m), and alpha_1, alpha_2 are the DC tilt angles of the two mirrors "shooting into the long arms of the MC" produced by the MC control with respect to the mechanical equilibrium position. omega_T is the tilt eigenfrequency of the three mirrors (assumed to be identical). lambda = 1.064e-6m;

The time it takes from N=1 to N=2 (the first observable ripple) is given by: tau1 = 0.6/omega_T*sqrt(lambda/L/(alpha_1+alpha_2))

The time it takes from N=2 to N=3 is given by: tau2 = 0.77*tau1

etc

First, we also see in the measurement that later ripples are shorter than early ripples consistent with some accelerated effect. The predicted ripple durations tau seem to be a bit too high though. The measurements show something like a first 14us and a late 8us ripple. It depends somewhat on the initial tilt angles that I don't know really.

In any case, the short ripple times could also be explained if the tilt motions start a little earlier than the ringdown, or the tilt motion starts with some small initial velocity. The next step will be to program a little ringdown simulation that includes mirror tilts and see what kind of tilt motion would produce the ripples exactly as we observe them (or maybe tilt motion cannot produce ripples as observed).

  7303   Tue Aug 28 19:21:37 2012 janoschMetaphysicsRingdownripples

Hmm. I don't know what ringing really is. Ok, let's assume it has to do with the pump... I don't see how the pump laser could produce these ripples. They have large amplitudes and so I always suspected something happening to the intracavity field. Therefore I was looking for effects that would change resonance conditions of the intracavity field during ringdown. Tilt motion seemed to be one explanation to me, but it may be a bit too slow (not sure yet). Longitudinal mirror motion is certainly too slow. What else could there be?

Quote:

Isn't it just a ringing of the intracavity power as you shifted the laser frequency abruptly?

Quote:

Let's see if the ripples observed in the MC ringdown can be due to tilt motion of the mirrors.

The time it takes to produce a phase shift corresponding to N multiples of 2*pi is given by:

t = sqrt(2*N*lambda/(L*omega_T^2*(alpha_1+alpha_2)))

L is the length of the MC (something like 13m), and alpha_1, alpha_2 are the DC tilt angles of the two mirrors "shooting into the long arms of the MC" produced by the MC control with respect to the mechanical equilibrium position. omega_T is the tilt eigenfrequency of the three mirrors (assumed to be identical). lambda = 1.064e-6m;

The time it takes from N=1 to N=2 (the first observable ripple) is given by: tau1 = 0.6/omega_T*sqrt(lambda/L/(alpha_1+alpha_2))

The time it takes from N=2 to N=3 is given by: tau2 = 0.77*tau1

etc

First, we also see in the measurement that later ripples are shorter than early ripples consistent with some accelerated effect. The predicted ripple durations tau seem to be a bit too high though. The measurements show something like a first 14us and a late 8us ripple. It depends somewhat on the initial tilt angles that I don't know really.

In any case, the short ripple times could also be explained if the tilt motions start a little earlier than the ringdown, or the tilt motion starts with some small initial velocity. The next step will be to program a little ringdown simulation that includes mirror tilts and see what kind of tilt motion would produce the ripples exactly as we observe them (or maybe tilt motion cannot produce ripples as observed).

 

 

  7305   Wed Aug 29 09:35:03 2012 janoschMetaphysicsRingdownripples 2

Ok, so the whole idea that mirror motion can explain the ripples is nonsense. At least, when you think off the ringdown with "pump off". The phase shifts that I tried to estimate from longitudinal and tilt mirror motion are defined against a non-existing reference. So I guess that I have to click on the link that Koji posted...

Just to mention, for the tilt phase shift (yes, there is one, but the exact expression has two more factors in the equation I posted), it does not matter, which mirror tilts. So even for a lower bound on the ripple time, my equation was incorrect. It should have the sum over all three initial tilt angles not only the two "shooting into the long arms" of the MC.

Quote:

Laser frequency shift = longitudinal motion of the mirrors

Ringing: http://www.opticsinfobase.org/ol/abstract.cfm?uri=ol-20-24-2463

Quote:

Hmm. I don't know what ringing really is. Ok, let's assume it has to do with the pump... I don't see how the pump laser could produce these ripples. They have large amplitudes and so I always suspected something happening to the intracavity field. Therefore I was looking for effects that would change resonance conditions of the intracavity field during ringdown. Tilt motion seemed to be one explanation to me, but it may be a bit too slow (not sure yet). Longitudinal mirror motion is certainly too slow. What else could there be?

 

 

  7356   Fri Sep 7 00:08:10 2012 janoschMetaphysics baffle clipping loss

With a curvature radius of about 57m for the ETMs, flat ITMs at the beam waist, and using 39m for the arm lengths, one finds that the beam radius at the ETMs is about 5.3mm. The clipping power loss of a 5.3mm beam through a 20mm radius baffle hole would be less than a ppm of a ppm if the beam was perfectly centered. If the baffle hole had 15mm radius, the clipping loss would be 0.01ppm. If the baffle hole had 10mm radius, the loss would be 810ppm. The loss values are calculated using the formula of the  "Gaussian beam" Wikipedia article, "Power through an aperture" section. So I did not check if that one is ok.

  7480   Thu Oct 4 18:48:04 2012 janoschConfigurationPSLAOM installation

Quote:

Jenne and Jamie also find that the PMC is behaving very weird 

 Can someone detail what "weird" means? Is it singing old songs from Guns & Roses?

  7527   Thu Oct 11 11:20:05 2012 janoschUpdateGeneralbeam shape simulation, PRC

I started to create a Finesse model of the PRC cavity. We have the phase maps for the PRC and the two ITMs. I could not find anything for PR2,3 and BS. All files can be found in my SVN folder /janosch/PRC40m. I used the AutoCAD model to determine angles of incidence and distances. These numbers are largely inconsistent with numbers that you can find elsewhere on the 40m wiki, but this certainly depends on what accuracy is required for interferometer alignment and I don't understand anything about alignment.

The phase maps come in a format that needs to be modified before they can be used in Finesse. I have started with this work, but maybe someone else can take over. The phase maps show tilts and the PRC also has the curvature. These have to be subtracted out before the maps can be loaded into Finesse. I asked GariLynn for the code that they use. The Finesse model (MichPRC_40m.kat) does not load the phasemaps yet, and I just wrote some random parameter values for the TEM00 input beam to the PRC. So these Gauss parameters need to be corrected.

I will only go on with this work if Rana tells me that I should do so, otherwise it is on hold until we have a volunteer.

  7533   Thu Oct 11 21:26:40 2012 janoschUpdateGeneralPRC phase maps

Just some plots. There is nothing new here except for the fact that I learned how to analyze phase maps myself and how to prepare them for Finesse. In other words, everything is ready for a Finesse simulation.

These phase maps show the raw measurement of ITMY, ITMX and PRC:

ITM03_HR_raw.pngITM04_HR_raw.pngPRM02_HR_raw.png

Subtracting out the tilt from all phase maps, and the curvature from the PRC (I found the fit 121m consistent with previous fits), the one obtains the following residuals that can be used in Finesse (order is again ITMY, ITMX and PRC):

ITM03_HR_untilted.pngITM04_HR_untilted.pngPRM02_HR_untilted_uncurved.png

Attachment 3: PRC_40m.png
PRC_40m.png
  7615   Wed Oct 24 22:48:46 2012 janoschUpdateComputer Scripts / ProgramsPhase map summary of LaserOptik mirrors

Quote:

After a long search, I've found a way to finally read and analyze(?)  the Wyko opd format data using Image SXM, an image analysis software working only on mac osx.

I am attaching the images (in tiff) and profile plot of all the 6 mirrors.

 Great, however, unless you can save the images in FITS format, we still need another reader for the opd images.

  7627   Thu Oct 25 22:52:07 2012 janoschUpdateGeneraltip-tilt phase maps

Now that I read Koji's last elog about phase maps, I am not sure if these are still required, but here they are (the tilt-removed phase maps of the Laser Optik mirrors), first 1, 2, 3:

sn1Laseroptik_untilted.pngsn2Laseroptik_untilted.pngsn3Laseroptik_untilted.png

Then 4,5,6:

sn4Laseroptik_untilted.pngsn5Laseroptik_untilted.pngsn6Laseroptik_untilted.png

So they all have an elevated center. I am not sure why the phase maps of mirrors 5 and 6 are slightly smaller in dimension. Anyhow, all mirrors have quite strong aberrations. Also, there is no big difference between the mirrors. Check for yourself, but be careful with the colors since the scales are all different.

  7629   Thu Oct 25 23:14:42 2012 janoschUpdateGeneraltip-tilt phase maps

Quote:

Are these maps drawn from the data we extracted using Image SXM??

 Indeed. So the only manipulation that I did was to remove the tilt (since this should usually be seen as an artifact of the measurement, or better, we can assume that tilt is compensated by alignment). I did not remove the curvature.

  7639   Mon Oct 29 14:57:41 2012 janoschUpdateGeneraltip-tilt phase maps

Quote:

 [Jan, Manasa]

Below are phasemaps for the tip-tilts with both tilt and RoC removed. We have not used Koji's code; but tweaked the earlier code to remove curvature. 

 The posted residual phase maps show circular contours since the data came with relatively low resolution in height. This is ok for what we want to do with these phase maps (i.e. simulating higher-order mode content in the PRC using Finesse). Better resolution is only required if you want to understand in detail optical scattering out of the cavity. Anyhow, the circular artifacts can be removed by first interpolating the phase maps to a higher lateral resolution, and then performing tilt and curvature subtraction. So we will soon have better looking phase maps posted. Then we should think about what type of Finesse simulation we could run. Certainly one simulation is to look at the beam shape in the PRC, but more interesting could be how sensitive the shape is to mirror alignments. The current simulation shows a mode that resembles the TEM01, but I have not yet tried to find optimal alignment of the mirrors (in simulation) to search for the TEM00 mode.

ELOG V3.1.3-