40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 216 of 344  Not logged in ELOG logo
IDup Date Author Type Category Subject
  10817   Fri Dec 19 14:25:48 2014 diegoUpdateComputer Scripts / Programselog restarted

elog was not responding for unknown reasons, since the elogd process on nodus was alive; anyway, I restarted it. 

  10818   Fri Dec 19 15:59:49 2014 JenneUpdateLSCLockloss from Wed

I swapped out one of the channels on Q's lockloss plotter - we don't need POP22Q, but I do want the PC drive.  

So, we still need to look into why the PC drive goes crazy, and if it is related to the buildup in the arms or just something intrinsic in the current FSS setup, but it looks like that was the cause of the lockloss that Q and Diego had on Wednesday.

1102929772_PCdriveRailed.png

  10819   Fri Dec 19 16:39:25 2014 ericqUpdateComputer Scripts / Programselog autostart

I've set up nodus to start the ELOG on boot, through /etc/init/elog.conf. Also, thanks to this, we don't need to use the start-elog.csh script any more. We can now just do:

controls@nodus:~  $ sudo initctl restart elog

I also tweaked some of the ELOG settings, so that image thumbnails are produced at higher resolution and quality. 

  10820   Fri Dec 19 16:59:32 2014 ericqUpdateComputer Scripts / ProgramsFSS Slow servo moved to megatron

Given that op340m showed some undesired behavior, and that the FSS slow seems prone to railing lately, I've moved the FSS slow servo job over to megatron in the same way I did for the MC autolocker. 

Namely, there is an upstart configuration (megatron:/etc/init/FSSslow.conf), that invokes the slow servo. Log file is in the same old place (/cvs/cds/caltech/logs/scripts), and the servo can be (re)started by running: 

controls@megatron|~ > sudo initctl start FSSslow

Maybe this won't really change the behavior. We'll see

  10821   Fri Dec 19 18:08:46 2014 JenneUpdateCDSSOS!!! HELP!! EPICS freeze 45min+ so far!

[Jenne, Diego]

The EPICS freeze that we had noticed a few weeks ago (and several times since) has happened again, but this time it has not come back on its own.  It has been down for almost an hour so far. 

 So far, we have reset the Martian network's switch that is in the rack by the printer.  We have also power cycled the NAT router.  We have moved the NAT router from the old GC network switch to the new faster switch, and reset the Martian network's switch again after that.

We have reset the network switch that is in 1X6.

We have reset what we think is the DAQ network switch at the very top of 1X7.

So far, nothing is working.  EPICS is still frozen, we can't ping any computers from the control room, and new terminal windows won't give you the prompt (so perhaps we aren't able to mount the nfs, which is required for the bashrc).

We need help please!

  10822   Fri Dec 19 19:21:04 2014 diegoUpdateCDSSOS!!! HELP!! EPICS freeze 45min+ so far!

Quote:

[Jenne, Diego]

The EPICS freeze that we had noticed a few weeks ago (and several times since) has happened again, but this time it has not come back on its own.  It has been down for almost an hour so far. 

 So far, we have reset the Martian network's switch that is in the rack by the printer.  We have also power cycled the NAT router.  We have moved the NAT router from the old GC network switch to the new faster switch, and reset the Martian network's switch again after that.

We have reset the network switch that is in 1X6.

We have reset what we think is the DAQ network switch at the very top of 1X7.

So far, nothing is working.  EPICS is still frozen, we can't ping any computers from the control room, and new terminal windows won't give you the prompt (so perhaps we aren't able to mount the nfs, which is required for the bashrc).

We need help please!

[EricQ]

 

EricQ suggested it may be some NFS related issue: if something, maybe some computer in the control room, is asking too much to chiara, then all the other machines accessing chiara will slow down, and this could escalate and lead to the Big Bad Freeze. As a matter of fact, chiara's dmesg pointed out its eth0 interface being brought up constantly, as if something is making it go down repeatedly. Anyhow, after the shutdown of all the computers in the control room, a  reboot of chiara, megatron and the fb was performed.

 

[Diego]

Then I rebooted pianosa, and most of the issues seem gone so far; I had to "mxstream restart" all the frontends from medm and everyone of them but c1scy seems to behave properly. I will now bring the other machines back to life and see what happens next.

  10823   Fri Dec 19 20:32:11 2014 diegoUpdateCDSSOS!!! HELP!! EPICS freeze 45min+ so far!

[Diego, Jenne]

 

Everything seems reasonably back to normal:

Notes:

  • the machines in the control room have been rebooted;
  • the c1iscey frontend now behaves;
  • I saw on nodus, which remained up and running the whole time, a bunch of   nfs: server chiara is not responding, timed out  messages, belonging to the freezing time; it may be that the sync option for the nfs share is too resource demanding, or some other network issue;
  • the FSS was doing strange stuff and the MC couldn't recover the lock; the MCautolocker script wasn't running because of the lock loss of the MC and the lack of communication between the machines; so we did a sudo initctl start MCautolocker on megatron and recovered the MC too.
  10824   Fri Dec 19 20:44:23 2014 JenneUpdateComputer Scripts / ProgramsFSS Slow servo moved to megatron

Today Q moved the FSS slow servo over to some init thing on megatron, and some time ago he did the same thing to the MC auto locker script.  It isn't working though.

Even though megatron was rebooted, neither script started up automatically.  As Diego mentioned in elog 10823, we ran sudo initctl start MCautolocker and sudo initctl start FSSslow, and the blinky lights for both of the scripts started.  However, that seems to be the only thing that the scripts are doing.  The MC auto locker is not detecting lockloses, and is not resetting things to allow the MC to relock.  The MC is happy to lock if I do it by hand though.  Similarly, the blinky light for the FSS is on, but the PSL temperature is moving a lot faster than normal.  I expect that it will hit one of the rails in under an hour or so. 

The MC autolocker and the FSS loop were both running earlier today, so maybe Q had some magic that he used when he started them up, that he didn't include in the elog instructions?

  10825   Sat Dec 20 00:00:03 2014 ericqUpdateComputer Scripts / ProgramsFSS Slow servo moved to megatron

I ssh'd in, and was able to run each script manually successfully. I ran the initctl commands, and they started up fine too. 

We've seen this kind of behavior before, generally after reboots; see ELOGS 10247 and 10572

  10826   Sun Dec 21 18:46:06 2014 diegoUpdateIOOMC Error Spectra

The error spectra I took so far are not that informative, I'm afraid. The first three posted here refer to Wed 17 in the afternoon, where things were quiet, the LSC control was off and the MC was reliably locked. The last two plots refer to Wed night, while Q and I were doing some locking work; in particular, these were taken just after one of the locklosses described in elog 10814. Sadly, they aren't much different from the "quiet" ones.

I can add some considerations though: Q and I saw some weird effects during that night, using a live reading of such spectra, which couldn't be saved though; such effects were quite fast both in appearance and disapperance, therefore difficult to save using the snapshot measurement, which is the only one that can save the data as of now; moreover, these effects were certainly seen during the locklosses, but sometimes also in normal circumstances. What we saw was a broad peak in the range 5e4-1e5 Hz with peak value ~1e-5 V/rtHz, just after the main peak shown in the attached spectra.

Attachment 1: SPAG4395_17-12-2014_170951.pdf
SPAG4395_17-12-2014_170951.pdf
Attachment 2: SPAG4395_17-12-2014_172846.pdf
SPAG4395_17-12-2014_172846.pdf
Attachment 3: SPAG4395_17-12-2014_175147.pdf
SPAG4395_17-12-2014_175147.pdf
Attachment 4: SPAG4395_18-12-2014_003414.pdf
SPAG4395_18-12-2014_003414.pdf
Attachment 5: SPAG4395_18-12-2014_003506.pdf
SPAG4395_18-12-2014_003506.pdf
  10827   Mon Dec 22 13:34:34 2014 KojiUpdateelogStrange ELOG serach

I tried to find my own entry and faced with a strange behavior of the elog.

The search button invoked the following link and no real search has been done:

http://nodus.ligo.caltech.edu:8080/40m/?mode=summvry&reverse=0&reverse=1&npp=50&m&y&Authorthor=Koji

Summvry? Authorthor?

If I ran the following link, it returned correct search. So something must be wrong.

http://nodus.ligo.caltech.edu:8080/40m/?mode=summary&npp=50&Author=Koji

  10828   Mon Dec 22 15:11:08 2014 ranaUpdateIOOMC Error Spectra

che tristezza  

What we want is to have the high and low noise spectra on the same plot. The high noise one should be triggered by a high PC DRIVE signal.

  10829   Mon Dec 22 15:46:58 2014 KurosawaSummaryIOOSeven transfer functions

IMC OL TF has been measured from 10K to 10M

Attachment 1: MC_OLTF.pdf
MC_OLTF.pdf
  10830   Mon Dec 22 16:21:15 2014 ericqUpdateelogStrange ELOG search

In order to fix ELOG search, I have started running ELOG v2.9.2 on Nodus.

Sadly, due to changes in the software, we can no longer use one global write password. Instead, we must now operate with registered users.

Based on recent elog users, I'll be creating user accounts with the following names, using the same old ELOG write password. (These will be valid across all logbooks)

  • ericq
  • rana
  • koji
  • diego
  • jenne
  • manasa
  • Steve
  • Kate
  • Zach
  • Evan
  • Aidan
  • Chris
  • Dmass
  • nicolas
  • Gabriele
  • xiaoyue

All of these users will be "Admins" as well, meaning they can add new users and change settings, using the "Config" link.

Let me know if I neglected to add someone, and sorry for the inconvenience.

RXA: What Eric means to say, is that "upgrading" from Solaris to Linux broke the search and made us get a new elog software that;s worse than what we had.

  10831   Mon Dec 22 17:06:14 2014 manasaUpdateGeneralX end AUX laser fiber setup

Quote:

 I looked at the endtable for possible space to setup optics in order to couple the X end laser into a PM fiber.

Attached is the layout of where the setup will go and what are the existing stuff that will be moved.

ETMXtable.png

Since we will not be doing any major locking, I am taking this chance to move things on the X end table and install the fiber coupler.

The first steering mirror shown in the earlier elog will be a Y1 (HR mirror) and the second one will be a beam sampler (similar to the one installed at the Y endtable for the fiber setup). 

 

Configuration:

Doubler --> Y1 ---> Lens (f=12.5cm) ---> Beam sampler --->Fiber coupler

The fiber coupler mount will be installed in the green region to the right of the TRX camera. 

This work will involve moving around the TRX camera and the optic that brings the trans image on it.

Let me know if this work should not be done tomorrow morning for any reason.

  10832   Mon Dec 22 21:53:08 2014 rana, kojiUpdateIOOSeven transfer functions

Today we were looking at the MC TFs and pulled out the FSS box to measure it. We took photos and removed a capacitor with only one leg.

Still, we were unable to see the weird, flat TF from 0.1-1 MHz and the bump around 1 MHz. Its not in the FSS box or the IMC servo card. So we looked around for a rogue Pomona box and found one sneakily located between the IMC and FSS box, underneath some cables next to the Thorlabs HV driver for the NPRO.

It was meant to be a 14k:140k lead filter (with a high frequency gain of unity) to give us more phase margin (see elog 4366; its been there for 3.5 years).

From the comparison below, you can see what the effect of the filter was. Neither the red nor purple TFs are what we want, but at least we've tracked down where the bump comes from. Now we have to figure out why and what to do about it.

* all of the stuff above ~1-2 MHz seems to be some kind of pickup stuff.

** notice how the elog is able to make thumbnails of PDFs now that its not Solaris!

Attachment 1: MC_OLG.pdf
MC_OLG.pdf
  10833   Tue Dec 23 01:55:35 2014 rana, kojiUpdateIOOSeven transfer functions

Some TFs of the TTFSS box

Attachment 1: MC_FSS_TF.pdf
MC_FSS_TF.pdf
  10834   Tue Dec 23 13:18:37 2014 manasaUpdateGeneralX end AUX laser fiber setup

Quote:

 

Since we will not be doing any major locking, I am taking this chance to move things on the X end table and install the fiber coupler.

The first steering mirror shown in the earlier elog will be a Y1 (HR mirror) and the second one will be a beam sampler (similar to the one installed at the Y endtable for the fiber setup). 

Configuration:

Doubler --> Y1 ---> Lens (f=12.5cm) ---> Beam sampler --->Fiber coupler

The fiber coupler mount will be installed in the green region to the right of the TRX camera. 

This work will involve moving around the TRX camera and the optic that brings the trans image on it.

Let me know if this work should not be done tomorrow morning for any reason.

I was working around the X endtable and PSL table today.

1. Y1 mirror, beam sampler and the fiber coupler have been installed.

2. Removed TRX camera temporarily. The camera will be put back on the table once we have the filter for 532nm that can go with it.

3. Removed an old fiber mount that was not being used from the table. 

4. Lowered the current for X end NPRO while working and put it back up at 2A before closing.

5. The fibers running from the X end to the PSL table are connected at an FC/APC connector on the PSL table. 

6. Found the HEPA left on high (probably from yesterday's work around the PSL table). I have brought it back down and left it that way.

I have not installed the coupling lens as yet owing to the space restrictions - not enough space for footprint of the lens. I have to revisit the telescope design again.

  10835   Tue Dec 23 14:27:16 2014 ericqUpdateCDSChiara moved to UPS

 Steve and I switched chiara over to the UPS we bought for it, after ensuring the vacuum system was in a safe state. Everything went without a hitch. 

Also, Diego and I have been working on getting some of the new computers up and running. Zita (the striptool projecting machine) has been replaced. One think pad laptop is missing an HD and battery, but the other one is fine. Diego has been working on a dell laptop, too. I was having problems editing the MAC address rules on the martian wifi router, but the working thinkpad's MAC was already listed. 

  10836   Tue Dec 23 14:30:11 2014 ericqUpdateCDSChiara moved to UPS

Quote:

 Steve and I switched chiara over to the UPS we bought for it, after ensuring the vacuum system was in a safe state. Everything went without a hitch. 

Also, Diego and I have been working on getting some of the new computers up and running. Zita (the striptool projecting machine) has been replaced. One think pad laptop is missing an HD and battery, but the other one is fine. Diego has been working on a dell laptop, too. I was having problems editing the MAC address rules on the martian wifi router, but the working thinkpad's MAC was already listed. 

 Turns out that, as the martian wifi router is quite old, it doesn't like Chrome; using Firefox worked like a charm and now also giada (the Dell laptop) is on 40MARS.

  10837   Tue Dec 23 14:33:24 2014 SteveUpdateVACChiara gets UPS

Quote:

Quote:

We had an unexpected power shutdown for 5 sec at ~ 9:15 AM.

Chiara had to be powered up and am in the process of getting everything else back up again.

Steve checked the vacuum and everything looks fine with the vacuum system.

PSL Innolight laser and the 3 units of IFO air conditions turned on.

The vacuum system reaction to losing power: V1 closed and Maglev shut down. Maglev is running on 220VAC so it is not connected to VAC-UPS.  V1 interlock was triggered by Maglev "failure" message.

Maglev was reset and started. After Chiara was turned on manually I could bring up the vac control screen through Nodus and opened V1

"Vacuum Normal" valve configuration was recovered instantly.

 

Chiara needs UPS 

It is arriving Thursday

 EricQ and Steve,

Steve preset the vacuum for safe-reboot mode with C1vac1 and C1vac2 running normal: closed valves as shown, stopped Maglev & disconnected valves V1 plus valves with moving labels.

(The position indicator of the valves changes to " moving " when its cable disconnected )

Eric shut down Chiara, installed APC's UPS Pro 1000 and restarted it.

All went well. Nothing unexpected happened. So we can conclude that the vacuum system with running C1vac1 and C1vac2 is not effected by Chiara's losing AC power.

Attachment 1: prepUP.png
prepUP.png
  10838   Tue Dec 23 15:37:32 2014 SteveUpdateVACTP3 drypump replaced

Quote:

Quote:

 

 TP2's fore line - dry pump replaced at performance level 600 mTorr after 10,377 hrs of continuous operation.

Where are the foreline pressure gauges? These values are not on the vac.medm screen.

The new tip seal dry pump lowered the small turbo foreline pressure 10x

TP2fl after 2 day of pumping 65mTorr

 TP2 dry pump replaced at fore pump pressure 1 Torr,  TP2 50K_rpm 0.34A

 Top seal life 6,362 hrs

 New seal performance at 1 hr  36 mTorr, 

 Maglev at 560 Hz, cc1 6e-6 Torr

 

 TP3 dry pump  replaced at 540 mT as TP3 50K_rpm 0.3A with annulos load. It's top seal life time was 11,252 hrs

 

 

  10839   Tue Dec 23 16:49:32 2014 ericqUpdateelogStrange ELOG search

 So, despite having registered users, it turns out that the "Author" field is still open for editing when making posts. I.e. we don't really need to make new accounts for everyone. 

Thus, I've made a user named "elog" with the old write password that can write to all ELOGs. 

(Also, I've added a user called "jamie")

  10840   Tue Dec 23 18:43:33 2014 diegoUpdateComputer Scripts / ProgramsFSS Slow servo moved to megatron

Quote:

I ssh'd in, and was able to run each script manually successfully. I ran the initctl commands, and they started up fine too. 

We've seen this kind of behavior before, generally after reboots; see ELOGS 10247 and 10572

In the plot it is shown the behaviour of the PSL-FSS_SLOWDC signal during the last week; the blue rectangle marks an approximate estimate of the time when the scripts were moved to megatron. Apart from the bad things that happened on Friday during the big crash, and the work ongoing since yesterday, it seems that something is not working well. The scripts on megatron are actually running, but I'll try and have a look at it.

  10841   Tue Dec 23 20:50:39 2014 rana, kojiUpdateIOOSeven transfer functions

Today we decided to continue to modify the TTFSS board.

The modified schematic can be found here: https://dcc.ligo.org/D1400426-v1 as part of the 40m electronics DCC Tree.

What we did

1) Modify input elliptic filter (L1, C3, C4, C5) to give zero and pole at 30 kHz and 300 kHz, respectively. L1 was replaced with a 1 kOhm resistor.  C3 was replaced with 5600 pF. C4 and C5 were removed. So the expected locations of the zero and pole were at 28.4 kHz and 256 kHz, respectively. This lead filter replaces the Pomona box, and does so without causing the terrible resonance around 1 MHz.

2) Removed the notch filters for the PC and fast path. This was done by removing L2, L3, and C52.

At this point we tested the MC locking and measured the transfer function. We successfully turned up the UGF to 170kHz and two super-boosts on.

3) Now a peak at 1.7MHz was visible and probably causing noise. We decided to revert L2 and adjusted C50 to tune the notch filter in the PC path to suppress this possible PC resonance. Again the TF was measured. We confirmed that the peak at 1.7MHz is at -7dB and not causing an oscillation. The suppression of the peak is limited by the Q of the notch. Since its in a weird feedback loop, we're not sure how to make it deeper at the moment.

4) The connection from the MC board output now goes in through the switchable Test1 input, rather than the fixed 'IN1'. The high frequency gain of this input is now ~4x higher than it was. I'm not sure that the AD829 in the MC board can drive such a small load (125 Ohms + the ~20 Ohms ON resistance of the MAX333A) very well, so perhaps we ought to up the output resistor to ~100-200 Ohms?


Also, we modified the MC Servo board: mainly changed the corner frequencies of the Super Boost stages and some random cleanup and photo taking. I lost the connecting cable from the CM to the AO input (unlabeled).

  1.  The first two Super Boost stages were changed from 20k:1k to 10k:500 to give us back some phase margin and keep the same low freq gain. I don't really know what the gain requirement is for this servo here at the 40m. The poles and zeros were chosen for iLIGO so as to have the frequency noise be 10x less than the SRD at 7 kHz.
  2. The third Super Boost (which we never used) was changed from 10k:500 to ~3k:150 (?) just in case we want a little more low freq gain.
  3. There was some purple vestigial wiring on the back side of the board with a flying resistor; I think this was a way to put a DC offset in to the output of the board, but its not needed anymore so I removed it.

 

Attachment 1: MC_OLTF.pdf
MC_OLTF.pdf
Attachment 2: MC_OLTF2.pdf
MC_OLTF2.pdf
Attachment 3: matlab.zip
  10842   Wed Dec 24 08:25:05 2014 ranaConfigurationIOOnotes on MC locking

 I've updated the scripts for the MC auto locking. Due to some permissions issues or general SVN messiness, most of the scripts in there were not saved anywhere and so I've overwritten what we had before. 

After all of the electronics changes from Monday/Tuesday, the lock acquisition had to be changed a lot. The MC seems to catch on the HOM more often. So I lowered a bunch of the gains so that its less likely to hold the HOM locks.

A very nice feature of the Autolocker running on megatron is that the whole 'mcup' sequence now runs very fast and as soon as it catches the TEM00, it gets to the final state in less than 2 seconds.

I've also increased the amplitude of the MC2 tickle from 100 to 300 counts to move it through more fringes and to break the HOM locks more often. Using the 2009 MC2 Calibration of 6 nm/count, this is 1.8 microns-peak @ 0.03 Hz, which seems like a reasonable excitation.

Using this the MC has relocked several times, so its a good start. We'll have to work on tuning the settings to make things a little spicier as we move ahead.

 

That directory is still in a conflicted state and I leave it to Eric/Diego to figure out what's going on in there. Seems like more fallout from the nodus upgrade:

controls@chiara|MC > svn up

svn: REPORT of '/svn/!svn/vcc/default': Could not read chunk size: Secure connection truncated (https://nodus.ligo.caltech.edu:30889)

  10843   Fri Dec 26 17:45:21 2014 SteveUpdateVACvac pressure rose to 1.3 mTorr

We run out of N2 for the vacuum system. The pressure peaked at 1.3 mTorr with MC locked. V1 did not closed because the N2 pressure sensor failed.

We are back to vac normal. I will be here tomorrow to check on things.

Attachment 1: noN2.png
noN2.png
Attachment 2: backtoVacNormal.png
backtoVacNormal.png
  10844   Fri Dec 26 18:20:42 2014 ranaUpdateComputer Scripts / ProgramsFSS Slow servo thresh change

Quote:

 

In the plot it is shown the behaviour of the PSL-FSS_SLOWDC signal during the last week; the blue rectangle marks an approximate estimate of the time when the scripts were moved to megatron. Apart from the bad things that happened on Friday during the big crash, and the work ongoing since yesterday, it seems that something is not working well. The scripts on megatron are actually running, but I'll try and have a look at it.

I guessed that what was happening was that the SLOW servo settings were not restored to the right values after the code movements / reboots. The ON threshold for the servo was set at +6 counts and the channel is MC TRANS. Since the ADC noise on that channel is ~50 counts, this means that the servo keeps pushing the laser temperature off in some direction when the MC is unlocked.

I reset the threshold to +6666 counts (the aligned MC transmission is ~16000 for the  TEM00 mode) so that it only turns on when we're in a good locked state.

  10845   Sun Dec 28 07:29:16 2014 SteveUpdateVACvacuum is normal

Quote:

We run out of N2 for the vacuum system. The pressure peaked at 1.3 mTorr with MC locked. V1 did not closed because the N2 pressure sensor failed.

We are back to vac normal. I will be here tomorrow to check on things.

 ITMX damping restored.

Attachment 1: 3d.png
3d.png
  10846   Mon Dec 29 21:30:25 2014 ranaUpdateGeneralrecovery
  1. Control room is at +66 F. Brrrr.
  2. Alignment of input beam into the IMC was wacky; locked on HOM.
  3. Re-aligned beam into the PMC first.
  4. Restarted mxstream for c1sus.
  5. Power cycled Martian router; all laptops were lost. Now better.
  6. Aligned launch beam from PSL to get onto the MCWFS better, MC is locking OK now. Moved MC SUS a little to get back to OSEM values from 6 days ago.
  7. Fixed LOCK_MC screen quad displays to be cooler.
  8. changed many of the ezcawrite calls in the mcup / mcdown to be 'caput -l' for more robustness. Still need ezcawrite for the binbary calls.
  9.  I didn't touch the mirrors on the MC REFL path, so we can still use that as a reference once the temperature returns to normal; the PSL room temp is down to 20C from 22 C a couple days ago.
  10. TRX values coming in to the LSC were frozen and the TRY_OUT16 was going to huge values even though camera flashes were reasonable. Tried restarting c1lsc model. No luck.
  11. Also tried shutdown -r now on c1lsc. No luck. Probably needs a RFM boot.
  12. Increased the FSS SLOW servo threshold to 9999 counts to avoid it running on some misaligned TEM01 mode locks. Increased the PID's I gain from 0.05 to 0.356 by tuning on some step responses as usual.
  13. By midnight the control room temperature is back around 71 F.
  10847   Tue Dec 30 00:46:05 2014 ranaUpdateIOOInvestigations into the mad PCDRIVE

Koji and I noticed that there was a comb* of peaks in the MC and FSS at harmonics of ~37 kHz. Today I saw that this shows up (at a much reduced level) even when the input to the MC board is disconnected.

It also shows up in the PMC. At nominal gains, there is just the 37 kHz peak. After tweaking up the phase shifter settings, I was able to get PMC servo to oscillate; it then makes a comb, but the actual oscillation fundamental is 1/3 of 37 kHz (some info on Jenne from elog 978 back in 2008).

Not sure what, if anything, we do about this. It is curious that the peak shows up in the MC with a different harmonic ratio than in the PMC. Any theories?

 

Anyway, after some screwing around with phase and amplitude of the RF modulation for the PMC from the phase shifter screen**, I think the gain is higher in the loop and it looks like the comb is gone from the MC spectrum.

Another clue I notice is that the PCDRIVE mad times often are coincident with DC shifts in the SLOWDC. Does this mean that its a flakiness with the laser? While watching the PCDRIVE output from the TTFSS interface board on a scope, I also looked at MIXER mon. It looks like many of the high noise events are associated with a broadband noise increase from ~50-140 kHz, rather than some specific lines. Don't know if this is characteristic of all of the noisy times though.

 

* this 'comb' had several peaks, but seem not be precise harmonics of each other: (f3 - 3*f1)/f3 ~ 0.1%

** I think we never optimized this after changing the ERA-5 this summer, so we'd better do it next.

 *** UPDATE: the second plot show the comparison between the new quiet and noisy states. Its just a broad bump.

 

Attachment 1: MC_ERR.pdf
MC_ERR.pdf
Attachment 2: plotFSSerr.ipynb.xz
Attachment 3: MC_ERRcomp.pdf
MC_ERRcomp.pdf
  10848   Tue Dec 30 17:26:23 2014 ranaUpdatePSLRelaxation Osc and the NPRO Noise eater

I wonder if the variable bump around 100 kHz can be something about the NPRO and if the bump we see is the closed loop response due to the Noise Eater.

NE_Mephisto.png

This plot (from the Mephisto manual) shows the effect of the NE on the RIN, but not the frequency noise. I assume its similar since the laser frequency noise above 10 kHz probably just comes from the pump diode noise.

I went out to the PSL and turned off the NE at ~4:53 PM local time today to see what happened. Although the overall PCDRIVE signal looks more ratty, there is no difference in the spectra of ON/OFF when the PCDRIVE is low. When its noisy, I see a tiny peak around 1 MHz with NE OFF. Turned it back on after a few hours.

  10849   Tue Dec 30 20:35:59 2014 ranaSummaryPSLPMC Tune Up
  1. Calibrated the Phase Adjust slider for the PMC RF Modulation; did this by putting the LO and RF Mod out on the TDS 3034 oscope and triggering on the LO. This scope has a differential phase measurement feature for periodic signals.
  2. Calibrated the RF Amp Adj slider for the PMC RF Modulation (on the phase shifter screen)
  3. The PMC 35.5 MHz Frequency reference card is now in our 40m DCC Tree.
  4.  The LO and RF signals both look fairly sinusoidal !
  5. Took photos of our Osc board - they are on the DCC page. Our board is D980353-B-C, but there are no such modern version in any DCC.
  6. The PMC board's Mixer Out shows a few mV of RF at multiples of the 35.5 MHz mod freq. This comes in via the LO, and can't be gotten rid of by using a BALUN or BP filters.
  7. In installed the LARK 35.5 MHz BP filter that Valera sent us awhile ago (Steve has the datasheet to scan and upload to this entry). It is narrow and has a 2 dB insertion loss.

For tuning the phase and amplitude of the mod. drive:

- since we don't have access to both RF phases, I just maximized the gain using the RF phase slider. First, I flipped the sign using the 'phase flip' button so that we would be near the linear range of the slider. Then I put the servo close to oscillation and adjusted the phase to maximize the height of the ~13 kHz body mode. For the amplitude, I just cranked the modulation depth until it started to show up as a reduction in the transmission by ~0.2%, then reduced it by a factor of ~3. That makes it ~5x larger than before.

Attachment 1: 17.png
17.png
Attachment 2: PMCcal.ipynb.xz
Attachment 3: PMC_Osc_Cal.pdf
PMC_Osc_Cal.pdf
  10850   Sun Jan 4 12:49:18 2015 SteveUpdateSUSrecent earthquakes

All suspensions were tripped. Damping were restored. No obvious sign of damage. BS OSEM-UR may be sticking ?

Attachment 1: recentEQ.png
recentEQ.png
  10851   Sun Jan 4 22:08:46 2015 ranaUpdateIOOMC loop characterizations: PZT/EOM crossover

 * PMC + MC were unlocked when I came in.

* I fiddled around some more with the mcup/down scripts to make locking snappier. The locking was breaking the PMC lock often, so I re-enabled the MC servo board output limiter during acquisition. It is disabled in the MC UP script.

* Re-measured the MC OLG. Still OK.

* Measured the PZT / EOM crossover (aka the FAST / PC crossover) using the connectors on Koji's summing box. With the FAST gain at 18 dB, the crossover is ~10 kHz. Looks way to shallow. Plots to follow.

* I finally discovered today that the PMC PZT stroke is what's causing the main mis-alignment of the beam going to the IMC. By relocking at a few positions, I could see that the IOO QPDs have steps when the PMC relocks. So the IO beam wander is NOT due to temperature effects on the optics mounts of the PSL table. I wonder if we have a large amount of length to angle coupling or if this is the same as the OMC PZTs ?

P.S. I found that someone is using a temporary bench power supply to power the summing box between the TTFSS and the Thorlabs HV driver...whoever did this has ~48 hours to hook up the power in the right way or else Koji is going to find out and lose it and then you have to wear the Mickey Mouse hat.

http://www.arroyoinstruments.com/files/Arroyo-UsingBenchPowerSupplies_11Apr.pdf


The first attachment shows the OLG measurements with 2 different values of the fast gain (our nominal FG is 18 dB). You can see that the higher gains produce some crossover instability; when tuning the gain we notice this as an increase in the PCDRIVE rms channel.

The second attachment shows the measurement of the 'crossover'. Its really just the direct measurement of the IN1 / IN2 from the FAST summing box, so its the crossover measurement where the OLG is high.

Attachment 1: MC_OLGs.pdf
MC_OLGs.pdf
Attachment 2: MC_xover.pdf
MC_xover.pdf
  10852   Mon Jan 5 12:42:09 2015 ericqUpdateSUSBS misbehaving

The BS was showing some excess motion. I think I've fixed it. Order of operations:

  • The DC PIT bias from previous ASS runs was at ~500, I zeroed this and aligned the BS to be centered on its oplev QPD with DC alignment sliders
  • I squished the gold box cables. This changed the alignment slightly, and brought the UR voltage back to a normal value. Excess motion still existed
  • I found that the the C1:SUS-BS_LRSEN filter had HOLD OUTPUT enabled. I turned it off. All seems well. 

I'm not sure how this might have gotten switched on...

  10853   Mon Jan 5 18:15:18 2015 JenneUpdateCDSiscex reboot

Rana noted last week that TRX's value was stuck, not getting to the lsc from iscex.  I tried restarting the individual models scx, lsc and even scy (since scy had an extra red rfm light), to no avail.  I then did sudo shutdown -r now on iscex, and when it came back, the problem was gone.  Also, I then did a diag reset which cleared all of the unusual red rfm lights.

Things seem fine now, ready to lock all the things.

  10854   Mon Jan 5 20:17:26 2015 jamieConfigurationCDSGDS upgraded to 2.16.14

I upgraded the GDS and ROOT installations in /ligo/apps/ubuntu12 the control room workstations:

  • GDS 2.16.14
  • ROOT 5.34.18 (dependency of GDS)

My cursory tests indicate that they seem to be working:

2015-01-05-200025_1694x996_scrot.png

Now that the control room environment has become somewhat uniform at Ubuntu 12, I modified the /ligo/cdscfg/workstationrc.sh file to source the ubuntu12 configuration:

controls@nodus|apps > cat /ligo/cdscfg/workstationrc.sh
# CDS WORKSTATION ENVIRONMENT
source /ligo/apps/ligoapps-user-env.sh
source /ligo/apps/ubuntu12/ligoapps-user-env.sh
source /opt/rtcds/rtcds-user-env.sh
controls@nodus|apps > 

This should make all the newer versions available everywhere on login.

  10855   Mon Jan 5 23:36:47 2015 ericqUpdateIOOAO cable reconnected

Quote:

 I lost the connecting cable from the CM to the AO input (unlabeled). 

 This afternoon, I labelled both ends of this cable, and reconnected it to the MC servo board. 

  10856   Tue Jan 6 03:09:17 2015 diegoUpdateLSCPRFPMI status & IFO status

 [Jenne, Rana, EricQ, Diego]

Tonight we worked on getting the IFO back in a working status after the break, and then tried some locking.

  • the MC is behaving better, it could stay in a stable condition for hours, even if a couple of times it lost lock, and one of them persisted for a little time;
  • we managed to get to arm power of 20ish, before losing lock (this happened a couple of times);
  • the main thing seems to be that we have only ~ 20 degrees of phase margin at UGF for DARM, which is evidently too little;
  • one hypothesis is that DARM may change sign due to some weird length/angular interaction, and that this messes up the actuation causing the lockloss;
  • one other possibility is that maybe, when arm power rises, there are some weird flashes that go back to the MC and then cause the locklosses, but this has to be verified;
  • attached there is a plot of the last lockloss (and a zoom of it), which seems to point at DARM as the culprit;

 Lockloss_20150106_074552.png

 

Lockloss_20150106_074552_zoom.png

 

We left the IFO uncontrolled and in a "flashy" state so that tomorrow we can look into the "back-flashing to the MC" hypothesis.

  10857   Tue Jan 6 03:13:09 2015 ericqUpdateLSCPRFPMI status & IFO status

Two plots from tonight:

Lock loss. Based on the fact that it looked like the DARM servo was running away, Rana posited an effective sign flip in the DARM loop, perhaps due to a parasitic angular feedback mechanism.

Jan6lockloss1_zoom.png

 


While Jenne was probing the IFO at lower powers, we noticed a sudden jump in ASDC. Found the GPS time and fed it to the lockloss plotter. Seems fairly evident that some sudden ETMX motion was to blame. (~2urad kick in yaw)

Jan6_asdcjump.png

  10858   Tue Jan 6 10:04:39 2015 SteveUpdateIOOhappy IOO
Attachment 1: IOO.png
IOO.png
  10859   Tue Jan 6 17:41:20 2015 JenneConfigurationCDSDTT doesn't do envelopes??

[Jenne, Diego]

We are working on trying out the UGF servos, and wanted to take loop measurements with and without the servo to prove that it is working as expected.  However, it seems like new DTT is not following the envelopes that we are giving it. 

If we uncheck the "user" box, then it uses the amplitude that is given on the excitation tab.  But, if we check user and select envelope, the amplitude will always be whatever number is the first amplitude requested in the envelope.  If we change the first amplitude in the envelope, DTT will use that number for the new amplitude, so it is reading that file, but not doing the whole envelope thing correctly.

Thoughts?  Is this a bug in new DTT, or a pebkac issue?

  10860   Wed Jan 7 02:54:09 2015 JenneUpdateLSCFiddling with DARM filters

One of the things that we had talked about last night was the totally tiny amount of phase margin that we have in the CARM and DARM loops.  DARM seemed to be the most obnoxious loop last night, so I focused on that today, although the CARM and DARM loops are pretty much identical.

(Q tells me via email that the phase budget has the same ~14 degree discrepancy between what we expect and what we measure as his estimate last night.  However, the Caltech network issues prevented his posting an elog.)

So, we definitely need to figure out where this 14 deg is going, but for now, I wanted to see if I could recover a couple of extra degrees just by modifying the filters.

The original filters do seem to eat a lot of phase:

DARM_design_orig.pdf

The short version of the story is that I didn't leave the filters changed at all.  I reverted back to the last version of the filter file from Monday night, so currently everything is as it was.

I tried increasing the Q of the zeros on the cyan and brown filters, which would sacrifice some gain at ~20 Hz, but hopefully win us 10+ degrees of phase.  This gave me a dip of about a factor of 2 between the new and old filters (all servo filters combined added up to this factor of 2 in magnitude) between ~20Hz - 70Hz. 

When we were locked using DARM for just the Yarm (for the UGF servo commissioning), I took a spectra of the error signal (which was POY) as a reference, then loaded in my new filters.  For the most part, the spectra didn't change (which is good, since the magnitude of the filter didn't change much.).  The spectra was bigger though between 50-70Hz, in kind of a sharp bandpass-looking shape that I wasn't expecting.    I don't know exactly why that's happening.

Anyhow, we tried the new filters once or twice with the full IFO, but kept losing lock.  Since I clearly haven't put in enough thought yet for these (particularly, how much suppression do we really need? what are our requirements???), I reverted back to the filter file from last night.  We continued locking, and checking out the new UGF servo that Diego is elogging about.

  10861   Wed Jan 7 02:56:15 2015 diegoUpdateLSCUGF Servo for DARM

 [Jenne, Diego]

Today we began implementing the UGF Servos. Things we did:

  • we updated the LSC model with both DARM and CARM servos, and moved them from after the control system to before it, at the level of the error signal;
  • we updated the medm screens; new buttons are located in the main LSC screen;
  • we started commissioning the DARM servo, at first using DARM for the lock of the single Y arm, then we moved on to the PRFPMI lock and the usual transition from ALS to Transmission;
  • although we had several lock losses during the night, we managed to tweak the parameters of the DARM UGF servo (phases, excitation, gains), which now seems to work sufficiently fine;
  • the filters added to the I and Q filter banks are a single lowpass in each, while the only filter in the main servio is a standard integrator;
  • we don't have a step response at the moment, but we can say that the settling time of the servo is in the range of 10 seconds;
  • we updated the ALSdown.py and ALSwatch.py scripts with a call to a new UGFdown.py script; this script, located in the scripts/PRFPMI folder, takes care of disabling the servos and putting the excitation to zero in case of a lock loss; re-enablement of such things must be done manually;
  10862   Wed Jan 7 03:04:13 2015 JenneUpdateLSCTRY (thorlabs pd) weird noise

[Jenne, Diego, Rana]

This is a note about work done last night. 

We were starting to lock, and saw glitches in the Thorlabs TRY PD about once every 1/60th of a second.  It is not a sine wave, so it is not 60Hz line noise directly.  It looks like this:

TRY_60Hz_peaks_5Jan2015.pdf

Rana pointed out that this looks like it could be from a power supply that is converting AC to DC. 

We went down to the Yend, and noticed some weird symptoms.  So far, we do not know where the noise is coming from.  Rather, we are just using the QPD for locking.

* The noise comes and goes, particularly if someone is moving around at the end station.

* Moving the Thorlabs power supply farther from the HeNe power supply didn't do much.  Turning off and disconnecting the HeNe supply didn't make the noise go away, so we conclude that it is not the HeNe's fault.

* We suspected the loops of excess cable that were sitting on top of iscey, but moving the coils away from the computer did not make the noise go away.

* We removed a few disconnected BNC cables that were near or touching the end table, but that didn't fix things.

* We disconnected the PD's signal cable and pulled it out of the table enclosure, and then put it back.  Noise was gone when cable was disconnected (good), but it was back after plugging the cable back in.

* The noise still comes and goes, but we don't have to use the Thorlabs PD for locking, so we leave it for another day.

RXA: also moved the Thorlabs power supply to a different power strip and tried putting it closer/farther to the Uniblitz shutter controller. Another suspect is that its some PWM type noise from the doubler crystal temperature driver. Need to try turning off the heater and the Raspberry PI to if it effects the noise.

  10863   Wed Jan 7 03:09:15 2015 JenneUpdateLSCPRFPMI status & IFO status

As a warm-up after the holidays, before the real locking began, I installed 1064nm bandpass filters in front of the transmission QPDs to eliminate the stray green light that is there.

The Yend had threads epoxied to it, so that end should be good.  Steve is going to repeat that for the Xend QPD at some point.  Right now, the filter is just on a lens mount about 2cm away from the PD box aperture, since that's as close as I could get it.

Also, while I was at the Xend, I noticed that the transmission camera is gone.  I assume that it was in the way of Manasa's fiber work, and that it'll get put back somehow, sometime.  She elogged that she had removed it, but I mistakenly thought that it was already replaced.  We don't use that camera much, so I'm not worried.

  10864   Wed Jan 7 09:44:33 2015 ericqUpdateLSCDARM phase budget

As Jenne mentioned, I created a model of the DARM OLG to see why we have so little phase margin. However, it turns out I can explain the phase after all.

Chris sent me his work for the aLIGO DARM phase budget, which I adapted for our situation. Here's a stacked-area plot that shows the contributions of various filters and delays on our phase margin, and a real measurement from a few days ago . 

DarmPhaseBudget.pdf

This isn't so great! Informed by Chris's model, the digital delays look like: (Here I'm only listing pure delays, not phase lags from filters)

  • 64k cycle (End IOP)
  • 16k cycle (End isce[x/y])
  • 16k cycle x 2 (end to LSC through RFM) [See ELOG 10811]
  • 16k cycle (LSC)
  • 16k cycle (LSC to SUS through dolphin) [See ELOG 9881]
  • 16k cycle (SUS)
  • 16k cycle x2 (SUS to end through RFM)
  • 16k cycle (End isce[x/y])
  • 64k cycle (SUS IOP)
  • DAC zero order hold

This adds up to about 570usec, 20.5 degrees at 100Hz, largely due to the sheer number of computer hops the transmission loops involve. 

As a check, I divided the measured OLG by my model OLG, to see if there is any shape to the residual, that my model doesn't explain. It looks like it fits pretty well. Plot:

 DarmOLGresidual.pdf

So, unless we undertake a bunch of computer work, we can only improve our transmission loops through our control filter design. 

Everything I used to generate these plots is attached. 

Attachment 3: 2015-01-DARMphase.zip
  10865   Wed Jan 7 11:20:22 2015 SteveUpdateLSCX arm T-QPD gets SM1 thread adapter

C1:SUS-ETMX_QPD is removed and internal SM1 thread adapter epoxied into position as it is at the Y end

This adapter will take FL1064-10 line filter holder

Line filter is attached and qpd needs alignment.

  10867   Wed Jan 7 12:15:17 2015 manasaUpdateGeneralNew RF cables

I was working around the PSL table this morning.

1. I have fibers running from the Y end and the PSL table to the Optical Fiber Module for Frequency Offset Locking. 

Y+PSL out power is ~200uW. From the transimpedance and responsivity specs of the RFPD (ThorLabs FPD310), we expect ~100uW or -10dBm RF power. I have not hooked up the RF output to a spectrum analyser to confirm this as yet.

2. Also, Steve and I ran RF cables (LMR-195A) from the PSL table to the FC module on the IOO rack.

 

ELOG V3.1.3-