40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 238 of 344  Not logged in ELOG logo
ID Date Author Typeup Category Subject
  10757   Fri Dec 5 00:52:51 2014 JenneUpdateSUSETMX 2nd order violin

We looked at the spectra of POX and POY during IR lock, and Q saw a peak at 1285 in POX only.  We're actuating on the ETMs, so it must be an ETMX violin mode, although it doesn't match the others that are in the table.

Anyhow, I added it to FM9.  While I was doing that, I realized that yesterday I had forgotten to put back the 3rd order ETM violin notch, so that is also in FM9.

  10758   Fri Dec 5 02:44:43 2014 JenneUpdateGeneralIFO alignment shenanigans

[Jenne, Q, Diego]

OMG, today sucked alignment-wise.  Like, wow. 

I think that the problem with the ASS is with the input pointing part of the system.  I found that if I disable the TTs for the Yarm (iin practice, the outputs are held at zero), I could run the Yarm ASS at full gain of 1, and it would do an awesome job.  The first time I did this, I by-hand optimized the TTs by running the test mass loops to make them follow the input pointing.  After that, I haven't touched the TT pointing at all, and we've just been running the test mass loops for the Yarm ASS.  The Xarm seems to not have this problem (or at least not as drastically), so I left it as it was, touching BS as well as ITMX and ITMY, although the gain still needs to be about 0.3.

I feel pretty good about the IFO alignment now, although it is slightly different than it has been.  The transmitted arm powers are higher than they were before I changed the ASS procedure, and there seems to be a little less power fluctuation with alignment.  Q points out that I don't have concrete evidence that this is a good alignment, but it feels right. 

It was a significant enough change that I had to go down to the Yend to realign the green to the new arm axis.  Xgreen we did with the remote PZTs.  I also realigned both of the beatnotes on the PSL table. 

While I was on the PSL table, I quickly touched up the PMC alignment.

The biggest problem, the one that sucked up more hours and energy than I'd like to admit, is ETMX's jumping.  So frustrating.  Sometimes it is time-coincident with engaging the LSC, sometimes not.  I thought that it might be because there are too many violin filters, but even if I turn off all violin filters to ETMX, it jumped a few times while the cavity was locked.  Sometimes it moves when the cavity is just locked and seems happy, sometimes it moves when nothing is resonating except for the green.  It takes a few minutes to recover the alignment enough to lock, and then it'll jump again a few minutes later.  I haven't gone down to squish the cables today, although I did it yesterday and that didn't seem to do anything. 

We had a few hours of time when it wasn't jumping, so we tried a few times to lock the IFO.  The last several times we have lost lock because the PRC loop rang up.  We measured the loop at low-ish arm powers, but it kept losing lock at higher powers before we could measure.  At least 3 times, the PRC lockloss took out CARM and DARM too.

Anyhow, it has been a long day of not accomplishing anything interesting, but hopefully the IFO will feel better tomorrow.

  10759   Fri Dec 5 15:41:45 2014 JenneUpdateComputer Scripts / ProgramsNodus on fast GC network

Apparently, some time ago Larry Wallace installed a new, fast ethernet switch in the old nodus rack. Q and I have just now moved nodus' GC ethernet cable over to the new switch.  Dan Kozak is going to use this faster connection to make the data flow over to the cluster not so lag-y.

  10761   Mon Dec 8 09:03:49 2014 steveUpdatePEMair cond maintenance tomorrow morning

 

 AC maintenance is scheduled from 8am till 11am tomorrow morning.

  10768   Tue Dec 9 03:34:52 2014 JenneUpdateASCPOP yaw razor tuning

With the re-do of the IFO alignment last week, I think that the beam was no longer about halfway on the POP22 razor blade.  To fix this, I locked the PRMI on sideband, removed the razor blade, and then put it back in such that it occluded about half of the light.  

I'm not entirely sure why, but when I put the razor in, POP22 went from 104(ish) to 45(ish) but POPDC  went from 5200(ish) to 1600(ish).  [The 'ish'es are because the PRC wasn't angularly stabilized, so there was some motion changing the power levels that leaked out to the POP port].  The ETMs were misaligned, so this should not be a carrier vs. sideband effect, since they'll both share the cavity axis defined by the ITMs and the PRM.  It is possible, although I didn't check, that there is some oplev light scattered into the POP photodiode that is now blocked by the razor blade.  This light would only be at DC and not the 2f frequencies.  Since the signal levels for POP22 vs. POPDC didn't change with and without the table top on (and with and without room lights on), I don't think that it is an effect of ambient light getting into the diode.  To check if it is oplev light I should (a) just look, and (b) try to lock the PRMI without the ITMX oplev laser being on to see if there is a difference in the POPDC signal.

Anyhow, under the assumption that the POP22 signal level is correct, I tuned up the PRCL ASC a little bit.  These changes are now in the carm_cm_up script, and the carm_cm_down script resets things.  Before the PRC is locked, I have FM1 and FM7 (the basic servo shape and a 40Hz lowpass) on, the gain set to zero, and the input off.  After lock is acquired, the input is turned on, and the gain ramps from 0 -> 10 in 3 seconds.  Then FM2 and FM6 (boosts at 1 and 3Hz) are engaged.

In the plot below, the dark blue and red curves were taken when there was no angular control on the PRC.  Pink was taken last week with the old QPD yaw ASC on.  Light blue is today's version of the in-loop performance of the POP22 yaw ASC loop.  I didn't save the trace unfortunately, but the DC QPD saw out-of-loop improvement between about 0.8Hz - 4 Hz. 

Also, has anything happened with the LSC rack in the last few weeks that might be causing lots of 60Hz noise? I saw these large lines last week, but I don't think I remember them from the past.

PRC_YAW_QPDvs22_8Dec2014.pdf

After I got the PRCL ASC working, I tried several iterations of locking.  ETMX is still being annoying, although the last hour or so have been okay.  CARM keeps getting rung up right around the transition to the sqrtInv error signal.  Since CARM and DARM are kind of entangled, it took me a few iterations to figure out that it was CARM that is ringing up, and not DARM.  I'm a little worried about the phase loss from the 1kHz lowpass that we turn on just before the transition to sqrtInv.  I want to keep the lowpass off until after we have transitioned DARM also over to DC transmission.  I tried once, but I lost lock before starting the CARM transition.  Anyhow, the ETM alignment issue is annoying.

Also, Jamie, Q, Diego and I were discussing last Friday, but none of us elogged, that we think there might be something wrong with one of the Martian network switches.  I'll start a separate thread about that right now, but it slows things down when you can't trust EPICS channels to be current, and I (without evidence) am a little worried that this might also affect the fast signals.

  10769   Tue Dec 9 03:41:06 2014 JenneUpdateCDSEPICS running slow - network issue?

[Jamie, EricQ, Jenne, Diego]

This is something that we discussed late Friday afternoon, but none of us remembered to elog. 

We have been noticing that EPICS seems to run pretty slowly, and in fact twice last week froze for ~2 minutes or so (elog 10756). 

On Friday, we plotted several traces on StripTool, such as C1:SUS-ETMY_QPD_SUM_OUTPUT and C1:SUS-ETMY_TRY_OUTPUT and C1:LSC-TRY_OUTPUT to see if channels with (supposedly) the same signal were seeing the same sample-holding.  They were.  The issue seems to be pretty wide spread, over all of the fast front ends.  However, if we look at EPICS channels provided by the old analog computers, they do not seem to have this issue. 

So, Jamie posits that perhaps we have a network switch somewhere that is connected to all of the fast front end computers, but not the old slow machines, and that this switch is starting to fail. 

My understanding is that the boys are on top of this, and are going to figure it out and fix it.

  10772   Wed Dec 10 14:22:37 2014 diegoUpdateComputer Scripts / ProgramsStatus of the new nodus

[Diego, Steve]

We ran a Cat 6+ Ethernet cable from the 1X7 rack (where the new nodus is located) to the fast GC switch in the control room rack; now I will learn how to setup the 'outside world' network, iptables, and the like.

 

I remind that the current hardware/software status is posted in elog 10697 ; if additions or corrections are needed, let me know.

 

After I check a couple of things, we can use the new nodus (which is currently known in the martian network as rosalba) as a local test to see that everything is working. After that (and, mostly, after I'll have the network working), we will sync the data from the old nodus to the new one and make the switch.

  10773   Wed Dec 10 14:26:13 2014 steveUpdatePEMair cond maintenance and particle plot

Quote:

 

 AC maintenance is scheduled from 8am till 11am tomorrow morning.

 IFO air condition maintenance will continue tomorrow morning and it should be finished by 11:30AM

 We have new guys taking over this job: Sal and Chris so it takes longer. The units will be shut down for a bit.

They will not enter the IFO lab. The CES housed units will be worked on.

  10774   Wed Dec 10 15:05:32 2014 JenneUpdateElectronicsXend QPD whitening board modified already

In April, Koji logged that he had made some changes to the Yend QPD whitening board (elog 9854).  Today, I pulled the Xend board to see if it had the same modifications.  The filter shapes all seem to be the same (as in, the capacitors at the output filters were removed, etc.), and the final gain is the same.  I just realized that I didn't explicitly check if the whitening switches were pulled to ground to permanently turn on the whitenening, but hopefully I'll be able to see that in the photo. 

I have not made any changes today (yet) to the board, so the overall gain is still accessible via EPICS.  I wanted to do a quick check that we won't be saturating things at full power with the maximum gain, before I make a change.

IMG_1776.JPG

EDIT:  After comparing the photos here and in elog 9854, the X end board has the filter shape modifications that were done some time ago, but the whitening is not permanently enabled.  For the Yend board, Koji added a jumper wire connecting (for example) R97 and R106 to the grounded side of C69.  This jumper wire is not in place on the X qpd board.

Before I re-pull the board and modify it, I want to make sure I know what I'm going to do for the gain slider override.

  10776   Wed Dec 10 21:05:56 2014 KateUpdateSEIGuralp briefly powered down

 Kate & Jenne

About 2:30 this afternoon, we briefly powered off the Guralp (C1:PEM-SEIS_GUR1_{X,Y,Z}) in order to better align it with the other seismometers along its marked N/S direction. It had been visibly off by a few degrees. 

  10777   Thu Dec 11 09:11:18 2014 manasaUpdatePSLPSL FSS Slow actuator

I am not sure if people have been noticing it lately; but the slow actuator on the PSL FSS has been railing up quite often these days. I found it at >0.8 and as high as 1.5 on certain occasions before resetting it to nominal zero.

It could be because the PMC alignment needs to be tweaked. The night crew should consider doing this before starting to lock.

  10778   Thu Dec 11 10:08:10 2014 manasaUpdateGeneralIFO update

Status of IFO:

1. The X end slow computer is down. ETMX sliders and buttons on the ETMX suspension screens have gone white. I have disabled the ETMX oplev because it is largely misaligned in yaw. I am not poking it and leaving it as is for the time being.

2. The Y arm green alignment had drifted and GTRY was down to 0.15 . I tweaked the alignment using the last two steering mirrors and brought GTRY to 0.7 which gives a beat note of -14dBm.

  10779   Thu Dec 11 12:39:31 2014 diegoUpdateComputer Scripts / ProgramsFrequency Offset Locking scripts status

 I finished the polishing in the scripts/FOL directory, this is the current status and this post replaces my two previous posts on the subject:

  • the Raspberry Pi operates locally: everything is in its /opt/FOL directory, which is a mirror of /opt/rtcds/caltech/c1/scripts/FOL/ ; some backup/sync scripts should be set up, tell me what kind (sync direction, place to call the script from, etc..) is recommended and I'll set it up;

 

  • the /opt/FOL directory contains:
    • ADC_interface                 : Akhil's ADC interface software and dependencies;
    • akhilTestCodes                : Akhil's work directory with his programs and data;
    • backup                             : two zip files with a full backup of the FOL stuff for both chiara and domenica at 2014/12/08, before my work on the directory;
    • fcreadoutApp                   : the EPICS app compiled on domenica. I didn't modify anything in particular here, as I don't know much about EPICS Apps; I'm not even sure if it is used by now, as I launch EPICS manually by just giving him a .db file (see below).
    • armFC*                        : it is the single program that constantly fetches data for the channels: it takes as arguments the RPi device (/dev/hidraw0 for the X arm, and /dev/hidraw1 for the Y arm) and the value to write into the frequency counter (0x3 for initialization and 0x2 for actual use); hence, there is no more need for recompilation!
    • epics_channels.py :  this is the new version of the old codetorun.py script; it handles the initialization and the availability of the two beatnote channels;
    • fcreadout / freqCountIOC : these are the binaries of the EPICS apps that I found on chiara/domenica; they are not used as of now, but could be useful;
    • fcreadout.db           : it is the database file that is loaded by EPICS to handle the channels;
    • FOLPID.pl              : the Perl PID controller; it is still the old version, we will work on this one later on (see Manasa's schedule at elog 10760 for info)

 

  • Domenica's environment:
    • as I said, everything runs locally from /opt/FOL;
    • in particular, I added in /etc/inittab two lines that launch EPICS and the python script for the channels; respawn is supported so these processes should always be available. For this to happen, DO NOT MOVE armFC, epics_channels.py and fcreadout.db from the /opt/FOL directory on domenica!
    • I added a udev rule in /etc/udev/rules.d/98-hidraw-permissions.rules to let the controls user access the /dev/hidraw* devices without having to sudo all the time;
    • I updated the ~/.bashrc and /opt/epics/epics-user-env.sh files to fix syntax errors and add some aliases we usually use.
  10781   Thu Dec 11 15:40:04 2014 SteveUpdatesafetysafety training

 

 Katherine Dooley has received 40m specific basic safety training in the 40m lab

  10782   Thu Dec 11 16:42:12 2014 JenneUpdateElectronicsXend QPD whitening board plan

Here is a little PDF of what I plan to do to both of the transmission QPD whitening boards later today.  The idea is to take away the remote gain slider inputs, and force the gains to always be at +30dB.

The red and blue notes are from Koji's elog 9854, and the green are my plans for today. 

I will cut the traces from the gain slider inputs, and pull the negative input of the AD620 to ground.  The positive input will be connected to the +5 voltage line, with a divider so that the positive input to the AD620 is about 666mV. 

The AD602 will be maxed out at +30dB with anything over 625mV. 

QPDwhiteningModification_11Dec2014.pdf

Unless there are objections, I will start these modifications in an hour or so.  I will also make the Xarm whitening always-on, just like Koji has already done for the Yend.

EDIT, JCD, 12Dec2014:  These are not the modifications that were made.  Please see ____ for actual modifications.

  10783   Thu Dec 11 17:45:54 2014 ericqUpdateComputer Scripts / ProgramsLockloss plotting

 With some advice from Jamie, I've gotten the lock loss plotting script that is used at LHO working on our machines. The other night, I modified the ALSwatch.py script to log lockloss times. Tying it together, I've written a small wrapper script that grabs the last time from the lockloss log, and plots it. 

It is: scripts/LSC/LocklossData/lastlock.sh

Jamie's going to make an adjustment to the pydv codebase that will let me implement the auto y-scaling that we like. We also will need to get a feel for the right timing window, once we see what kind of delay in the ALSwatch script is typical. 

Here's an example of the output, with the window of [-10,+2] seconds from the logged GPS time:

figure.png

 

  10784   Thu Dec 11 18:00:43 2014 ericqUpdateComputer Scripts / ProgramsMC error signal monitoring

The other day, I hooked up the agilent analyzer to OUT2 of the MC board, which is currently set to output the MC refl error signal.  I've written a GPIB-based program that continuously polls the analyzer, and plots the live spectrum, an exponentially weighted running mean, and the first measured spectrum. 

The intended use case is to see if the FSS or MC loops are going crazy when we're locking. Sometimes the GPIB interface hangs/loses its connection, and the script needs a restart.

The script lives in scripts/MC/MCerrmon

 

 

  10785   Thu Dec 11 18:12:46 2014 ericqUpdateLSC(Fixed) Y end whitening board

Quote:

Gain mystery

- It was not sure how the whitening gains have been given.

- The corresponding database entry was found in /cvs/cds/caltech/target/c1auxey/ETMYaux.db as

grecord(ao,"C1:ASC-QPDY_S1WhiteGain")
grecord(ao,"C1:ASC-QPDY_S2WhiteGain")
grecord(ao,"C1:ASC-QPDY_S3WhiteGain")
grecord(ao,"C1:ASC-QPDY_S4WhiteGain")

- The gains for S2-S4 were set to be 30. However, C1:ASC-QPDY_S1WhiteGain was set to be 8.62068.
And it was not writable.

- After some investigation, it was found that the database was wrong. The DAC channel was changed from S100 to S0.
The corrected entry is shown here.

grecord(ao,"C1:ASC-QPDY_S1WhiteGain")
{
        field(DESC,"Whitening gain for QPDY Seg 1")
        field(DTYP,"VMIVME-4116")
        field(OUT,"#C0 S0 @")
        field(PREC,"1")
        field(EGUF,"42")
        field(EGUL,"-22")
        field(EGU,"dB")
        field(LINR,"LINEAR")
        field(DRVH,"30")
        field(DRVL,"-10")
        field(HOPR,"30")
        field(LOPR,"-10")
}

- Once c1auxey was rebooted, the S1 whitening gain became writable. Now all of the channels were set to be +30dB (max). 

This exact situation was happening at ETMX. I did the exact same change to the database, now I can read and write all four gain segments.

  10786   Thu Dec 11 22:44:23 2014 ranaUpdateLSCQPD screens

All of the QPDX matrix fields had a missing underscore in them. So I committed all of the c1asc screens to the SVN (because no one but me and Jamie seems to be able to remember to do this).

Then I did find/replace on the QPDY screen and saved it over the QPDX screen and committed the new thing to SVN as well. Values are now accessible.

  10789   Fri Dec 12 04:33:49 2014 JenneUpdateASCASS retuned

[Rana, Jenne]

We decided that tonight was the night for ASS tuning. 

We started from choosing new frequencies, by looking at the transmission and the servo control signals spectra to find areas that weren't too full of peaks.  We chose to be above the OpLev UGF by at least a factor of ~2, so our lowest frequency is about 18Hz.  This way, even if the oplevs are retuned, or the gains are increased, the ASS should still function. 

We set the peak heights for the lowest frequency of each arm to have good SNR, and then calculated what the amplitude of the higher frequencies ought to be, such that the mirrors are moving about the same amount in all directions. 

We re-did the low pass filters, and eliminated the band pass filters in the demodulation part of the servo.  The band passes aren't strictly necessary, as long as you have adequate lowpassing, so we have turned them off, which gives us the freedom to change excitation frequencies at will.  We modified the lowpass filter so that we had more attenuation at 2Hz, since we spaced our excitation frequencies at least ~2.5 Hz apart.

The same lowpass filter is in every single demodulator filter bank (I's and Q's, for both length and transmission demodulation).  We are getting the gain hierarchy just by setting the servo gains appropriately. 

We ran ezcaservos to set the demodulation phase of each lockin, to minimize the Q-phase signal. 

We then tuned up the gains of the servos.  Rana did the Y arm, but for the X arm I tried to find the gains where the servos went unstable, and then reduced the gain by a factor of 2.  The Xarm is having trouble getting good alignment if you start with something less than about 0.7, so there is room for improvement.

Rana wrote a little shell script that will save the burt snapshot, if the gains need adjusting and they should be re-saved. 

The scripts have been modified (just with the new oscillator amplitudes - everything else is in the burt snapshots), so you should be able to run the start from nothing and the start from frozen scripts for both arms.  However, please watch them just in case, to make sure they don't run away.

  10790   Fri Dec 12 10:38:22 2014 SteveUpdatePEMleaky roof

The first real rain of this year finds only one leak at the 40m

  10793   Fri Dec 12 19:38:49 2014 diegoUpdateComputer Scripts / ProgramsStatus of the new nodus

Quote:

[Diego, Steve]

We ran a Cat 6+ Ethernet cable from the 1X7 rack (where the new nodus is located) to the fast GC switch in the control room rack; now I will learn how to setup the 'outside world' network, iptables, and the like.

 

I remind that the current hardware/software status is posted in elog 10697 ; if additions or corrections are needed, let me know.

 

After I check a couple of things, we can use the new nodus (which is currently known in the martian network as rosalba) as a local test to see that everything is working. After that (and, mostly, after I'll have the network working), we will sync the data from the old nodus to the new one and make the switch.

[Diego, EricQ]

Update: work is almost completed; the old nodus is still online, as I don't feel confident to make the switch and leave it on its own for the weekend. However, the new nodus is online with the IP address 131.215.114.87, so everyone can check that everything works. From my tests I can say that:

After everything will be in place, I will save every reasonably important configuration file of nodus into the svn.

 

I remind that every change made while accessing the 131.215.114.87 machine will be purged during the sync&switch

 

  10794   Fri Dec 12 19:54:21 2014 JenneUpdateElectronicsXend QPD whitening board modified

Okay, I have finished modifying the Xend QPD whitening board, although I will likely need to change the gain on Monday.

Rather than following my plan in elog 10782, I removed the AD602's entirely, and just use the AD620's as the amplifiers.  We don't need remotely adjustable gains, and the AD620s are a less noisy part.

I set the gain to be 30dB using a 1.65k resistor for R_G, which turns out to be too high.  After I installed the board and realized that my counts were much higher than they used to be, I realized that what we had been calling +30dB was in fact +13.2dB. ( I am assuming that the ADC for the gain sliders were putting out a maximum of +10V.  The AD620 used to have a 1/10 voltage divider at the input, and an overall gain of 1, so the output of the AD620 was 100mV.  This goes into pin 16 of the AD602, which has gain of 32*V_set + 10.  Which gives 32*0.1+10=13.2dB.  Ooops.  We've been lying to ourselves. )

Anyhow, before I made the gain realization, I was happily going along, setting the AD620s' gains all to 30dB. I also copied Koji's modification from April of this year, and permanently enabled the whitening filters.

Here is the schematic of what ended up happening.  The red modifications were already in place, and the greens are what I did today.

QPDwhiteningModification_XtransCompleted_12Dec2014.pdf

You can see the "before" picture in my elog Wednesday, elog 10774.  Here is an "after" photo:

IMG_1779.JPG

Here is a spectrum comparing the dark noise of the Xend QPD after modification to the current Yend QPD (which is still using the AD602 as the main instrumentation amplifier).  I have given the Yend data an extra 16.8dB to make things match.

QPD_Xtrans_Ytrans_12Dec2014.pdf

And, here is a set of spectra comparing both ends, dark noise versus single arm lock.  While I'll have to sacrifice a lot of it, there's oodles more SNR in the Xend now.  The Yend data still has the "gain fixing" extra 16.8dB.

QPD_Xtrans_Ytrans_DarkVsLock_12Dec2014.pdf

The Xend quadrant input counts (before the de-whitening filters) now go up to peak values of about 1,000 at single arm lock.  If (optimistically) the we got full power recycling and the arms got to powers of 300, that would mean we would have 300,000 counts, which is obviously way more than we actually have ADC range for.  Currently, the Yend quadrant input counts go as high as 50, which with arm powers of 300 would give 15,000 counts.  I think I need to bring the Xend gain down to about the level of the Yend, so that we don't saturate at full arm powers.  I can't remember right now - are the ends 14-bit or 16-bit ADCs?  If they're 16-bit, then I can set the gain somewhere between the current X and Y values.

 Finally, I added a section of the 40m's DCC document tree for the QPD whitening:  E1400473, with a page for each end.  Xend = D1400414, Yend = D1400415.

  10795   Sat Dec 13 00:35:11 2014 ranaUpdateElectronicsXend QPD whitening board modified

 

 16 bit. There aren't any 14-bit ADCs anywhere in LIGO. The aLIGO suspensions have 18-bit DACs.

The Y-End gains seem reasonable to me. I think that we only use TRX/Y as error signals once we have arm powers of >5 so we should consider if the SNR is good enough at that point; i.e. what would be the actual arm motion if we are limited only by the dark noise?

I seem to remember that the estimate for the ultimate arm power is ~200, considering that we have such high losses in the arms.

  10796   Sat Dec 13 14:26:36 2014 ericqUpdateLSCMismatched gains on ETMY Transmon QPD

Yesterday, we were seeing anomalously high low frequency RIN in the y-arm (rms of 4% or so). I swung by the lab briefly to check this out. Turns out, despite TRY of 1.0, there was reasonable misalignment. ASS with the excitation lowered by a factor of two, and overall gain at 0.5 or so aligned things to TRY=1.2, and the RIN is back down to ~0.5% I reset the Thorlabs FM to make the power = 1.0

I then went to center the transmitted beam on the transmon QPD. Looking at the quadrant counts as I moved the beam around, things looked odd, and I poked around a little... 

I strongly suspect that we have significantly mismatched gains for the different quadrants on the ETMY QPD. 

Reasoning: With the y-arm POY locked, I used a lens to focus down the TRY beam, to illuminate the quadrants individually. Quadrants 2 and 3 would go up to 3 counts, while 1 and 4 would go up to 0.3 and 0.6, respectively. (These counts are in some arbitrary units that were set by setting the sum to 1.0 when pitch and yaw claimed to be centered, but mismatched gains makes that meaningless.)

I haven't looked more deeply into where the mismatch is occurring. The four individual whitening gain sliders did affect the signals, so the sliders don't seem sticky, however I didn't check the actual change in gains. Will the latest round of whitening board modifications help this?

Hopefully, once this is resolved, the DC transmission signals will be much more reliable when locking...

  10797   Mon Dec 15 12:53:13 2014 ericqUpdateComputer Scripts / ProgramsStatus of the new nodus

 Nodus (solaris) is dead, long live Nodus (ubuntu).

Diego and I are smoothing out the Kinks as they appear, but the ELOG is running smoothly on our new machine. 

SVN is working, but your checkouts may complain because they expect https, and we haven't turned SSL on yet...

  10798   Mon Dec 15 16:27:57 2014 diegoUpdateComputer Scripts / ProgramsStatus of the new nodus

Quote:

 Nodus (solaris) is dead, long live Nodus (ubuntu).

Diego and I are smoothing out the Kinks as they appear, but the ELOG is running smoothly on our new machine. 

SVN is working, but your checkouts may complain because they expect https, and we haven't turned SSL on yet...

 [Diego, EricQ]

SSL, https and backups are now working too!

A backup of nodus's configuration (with some explaining) will be done soon.

  10799   Mon Dec 15 22:30:50 2014 JenneUpdateElectronicsYend QPD modified

Details later - empty entry for a reply.

Short story - Yend is now same as Xend filters-wise and lack of gain sliders -wise.  Both ends have 13.7k resistors around the AD620 to give them gains of ~4.5.

Xend seems fine.

Yend seems not fine.  Even the dark noise spectrum sees giganto peaks.  See Diego's elog 10801 for details on this investigation.

  10801   Mon Dec 15 22:45:59 2014 JenneUpdateElectronicsYend QPD modified

 

 [Jenne, Rana, Diego]

We did some test on the modified QPD board for the Yend; we saw some weird oscillations at high frequencies, so we went and check more closely directly within the rack. The oscillations disappear when the cable from the QPD is disconnected, so it seems something is happening within the board itself; however, looking closely at the board with an oscilloscope in several locations, with the QPD cable connected or disconnected, there is nothing strange and definitely nothing changing if the cable is connected or not. In the plots there are the usual channels we monitor, and the 64kHz original channels before they are downsampled.

Overall it doesn't seem being a huge factor, as the RMS shows at high frequencies, maybe it could be some random noise coming up, but anyway this will be investigated further in the future.

  10802   Tue Dec 16 00:20:06 2014 diegoUpdateOptical LeversBS & PRM OL realignment

[Rana, Diego]

We manually realigned the BS and PRM optical levers on the optical table.

  10804   Tue Dec 16 03:43:09 2014 JenneUpdateLSCPRMI loops need help

[Jenne, Rana, Diego]

After deciding that the Yend QPD situation was not significant enough to prevent us from locking tonight, we got started.  However, the PRMI would not acquire lock with the arms held off resonance. 

This started some PRMI investigations.

With no arms, we can lock the PRMI with both REFL55 I&Q or REFL165 I&Q.  We checked the demod phase for both Refl 55 and 165.  REFL55 did not need changing, but REFL165 was off significantly (which probably contributed to the difficulty in using it to acquire lock).  I didn't write down what REFL165 was, but it is now -3 degrees.  To set the phase (this is also how Rana checked the 55 phase), I put in an oscillation using the sensing matrix oscillators.  For both REFL165I and 165Q, I set the sensing matrix demod phases such that all of the signal was in the I phase (so I_I and Q_I, and basically zero in I_Q and Q_Q).  Then, I set the main PD demod phase so that the REFL165Q phase (the Q_I phase) was about zero.

Here are the recipes for PRMI-only, REFL55 and REFL165:

Both cases, actuation was PRCL = 1*PRM and MICH = (0.5*BS - 0.2625*PRM).  Trigger thresholds for DoFs and FMs were always POP22I, 10 up and 0.5 down.

REFL55, demod phase = 31deg.

MICH = 2*R55Q, gain = 2.4, trig FMs 2, 6, 8.

PRCL = 12*R55I, gain = -0.022, trig FMs 2,6,9.

REFL165, demod phase = -3deg.

MICH = -1*R165Q, gain = 2.4, trig FMs 2,6,8.

PRCL = 2.2*R165I, gain = -0.022, trig FMs 2,6,9.

These recipes assume Rana's new resonant gain filter for MICH's FM6, with only 2 resonant gains at 16 and 24 Hz instead of a whole mess of them: elog 10803.  Also, we have turned down the waiting time between the MICH loop locking, and the filters coming on.  It used to be a 5 second delay, but now is 2 sec.  We have been using various delays for the PRCL filters, between 0.2s and 0.7s, with no particular preference in the end.

We compared the PRCL loop with both PDs, and note that the REFL 165 error signal has slightly more phase lag, although we do not yet know why.  This means that if we only have a marginally stable PRCL loop for REFL55, we will not be stable with REFL165. Also, both loops have a non-1/f shape at a few hundred Hz.  This bump is still there even if all filters except the acquisition ones (FM4,5 for both MICH and PRCL) are turned off, and all of the violin filters are turned off.  I will try to model this to see where it comes from.

PRMI_55vs165_15Dec2014.pdf

To Do list:

Go back to the QPDY situation during the daytime, to see if tapping various parts of the board makes the noise worse.  Since it goes up to such high frequencies, it might not be just acoustic.  Also, it's got to be in something common like the power or something, since we see the same spectra in all 4 quadrants. 

The ASS needs to be re-tuned. 

Rana was talking about perhaps opening up the ETMX chamber and wiggling the optic around in the wire.  Apparently it's not too unusual for the wire to get a bit twisted underneath, which creates a set of places that the optic likes to go to.

Diego is going to give us some spectra of the MC error point at various levels of pockel's cell drive.  Is it always the same frequencies that are popping up, or is it random?

  10805   Tue Dec 16 20:49:25 2014 diegoUpdateComputer Scripts / ProgramsStatus of the new nodus

Quote:

Quote:

 Nodus (solaris) is dead, long live Nodus (ubuntu).

Diego and I are smoothing out the Kinks as they appear, but the ELOG is running smoothly on our new machine. 

SVN is working, but your checkouts may complain because they expect https, and we haven't turned SSL on yet...

 [Diego, EricQ]

SSL, https and backups are now working too!

A backup of nodus's configuration (with some explaining) will be done soon.

 Nodus should be visible again from outside the Caltech Network; I added some basic configuration for postfix and smartmontools; configuration files and instructions for everything are in the svn in the nodus_config folder

  10806   Tue Dec 16 20:51:18 2014 diegoUpdateLSCPRMI loops need help

Quote:

[...]

Diego is going to give us some spectra of the MC error point at various levels of pockel's cell drive.  Is it always the same frequencies that are popping up, or is it random?

 I found out that the Spectrum Analyzer gives bogus data... Since now is locking time, tomorrow I'll go and figure out what is not working

  10807   Wed Dec 17 01:51:44 2014 rana, jenneUpdateASCASS retuned

Did a big reconfig to make the Y-arm work again since it was bad again.

  1. Undid Koji's topology change. The A2L loops now feedback to the arm mirrors to adjust the cavity axis. The cavity transmission signals now feedback to the input beam.
  2. The UGF of the Trans->Input beam servos is ~5-10x higher than the A2L servos.
  3. The Trans loops have a ~10-15 s settling time.
  4. The Input Matrix has been adjusted to fit with our intuition:The ETM tilt moves the beam equally on the ITM and ETM faces.
  5. The Output Matrix has also been adjusted to do like this: we're using an intuitive matrix inverse rather than one based on measurement. It turns out to be a reasonable guess and we can tune this later.
  6. Seems stable with many kinds of steps and misalignments. Seems not reliable if the arm power is less than ~0.5.
  7. Reducing the dither amplitudes to make the power fluctuation less than 5% made it much more stable.

With the arm aligned and the A2L signals all zeroed, we centered the beam on QPDY (after freezing the ASS outputs). I saw the beam going to the QPD on an IR card, along with a host of green spots. Seems bad to have green beams hitting the QPD alogn with the IR, so we are asking Steve to buy a bunch of the broad, dielectric, bandpass filters from Thorlabs (FL1064-10), so that we can also be immune to the EXIT sign. I wonder if its legal to make a baffle to block it on the bottom side?

P.S. Why is the Transmon QPD software different from the OL stuff? We should take the Kissel OL package and put it in place of our old OL junk as well as the Transmons.

  10809   Wed Dec 17 13:14:38 2014 ericqUpdateLSCPRMI loops need help

Quote:

However, the PRMI would not acquire lock with the arms held off resonance. 

This is entirely my fault. 

Last week, while doing some stuff with PRY, I put this filter in SUS_PRM_LSC, to stop some saturations from high frequency sensing noise

oops.pdf

After the discussion at today's meeting, it struck me that I might have left it on. Turns out I did. 

20 degree phase lag at 200Hz can explain the instability, some non-flat shape at few hundreds of Hz explains the non 1/f shape.

Sorry about all that...

  10810   Wed Dec 17 14:42:13 2014 JenneUpdateLSCPRMI loops need help

EricQ's crazy people filter has been deleted.  I'm trying to lock right now, to see if all is well in the world.

  10811   Wed Dec 17 18:14:36 2014 ericqUpdateASCTransmon QPD -> ASC servos ready for comissioning

 I have completed all of the model modifications and medm screen updates to allow for feedback from the transmon QPD pitch and yaw signals to the ITMs. Now, we can design and test actual loops...

newASCscreen.png

The signals come from c1sc[x/y] to c1rfm via RFM, and then go to c1ass via dolphin. 


Out of curiosity about the RFM+dolphin delay, I took a TF of an excitation at the end SUS model (C1:SUS-ETM[X/Y]_QPD_[PIT/YAW]_EXC) to the input FM in the ASC model (C1:ASC-ETM[X/Y]_QPD_[PIT/YAW]_IN1). All four signals exhibit the same delay of 122usec. I saved the dtt file in Templates/ASC/transmonQPDdelay.xml

This is less than a degree under 20Hz, so we don't have to worry about it. 

  10812   Wed Dec 17 19:04:12 2014 jenneUpdateASCASS retuned

I made the Xarm follow the new (old) topology of Length -> test masses, and Trans -> input pointing.

It takes a really long time to converge (2+ min), since the input pointing loops actuate on the BS, which has an optical lever, which is slow.  So, everything has to be super duper slow for the input pointing to be fast relative to the test mass motion.

Also, between last night and this afternoon, I moved the green ASX stuff from a long list of ezca commands to a burt file, so turning it on is much faster now.  Also, I chose new frequencies to avoid intermodulation issues, set the lockin demodulation phases, and tuned all 4 loops.  So, now the green ASX should work for all 4 mirrors, no hand tuning required.  While I was working on it, I also removed the band pass filters, and made the low pass filters the same as we are using for the IR ASS.  The servos converge in about 30 seconds.

  10813   Wed Dec 17 19:31:55 2014 KojiUpdateASCASS retuned

I wonder what to do with the X arm.

The primary purpose of the ASS is to align the arm (=transmission), and the secondary purpose is to adjust the input pointing.

As the BS is the only steering actuator, we can't adjust two dof out of 8 dof.
In the old (my) topology, the spot position on ITMX was left unadjusted.

If my understanding of the latest configuration, the alignment of the cavity (=matching of the input axis with the cavity axis)
is deteriorated in order to move the cavity axis at the center of the two test masses. This is not what we want as this causes
deterioration of the power recycling gain.

  10814   Thu Dec 18 02:23:34 2014 ericqUpdateLSCLocking update

 [ericq, Diego]

Some locking efforts tonight; many locklosses due to PRC angular motion. Furthest progress was arm powers of 15, and I've stared at the corresponding lockloss plot, with little insight into what went wrong. (BTW, lastlock.sh seems to catch the lock loss reliably in the window)

15lockloss.png

CARM and DARM loops were measured not long before this lock loss, and had nominal UGFs (~120Hz, ~20deg PM). However, there was a reasonably clear 01 mode shape at the AS camera, which I did nothing to correct. Here's a spectrum from *just* before the lockloss, recovered via nds. Nothing stands out to me, other than a possible loss of DARM optical gain. (I believe the references are the error signal spectra taken in ALS arms held away + PRMI on 3F configuration)

15spec.pdf

The shape in the DARM OLTF that we had previously observed and hypothesized as possible DARM optical spring was not ever observed tonight. I didn't induce a DARM offset to try and look for it either, though.

Looking into some of the times when I was measuring OLTFs, the AS55 signals do show coherence with the live DARM error signal at the excitation frequencies, but little to no coherence under 30Hz, which probably means we weren't close enough to swap DARM error signals yet. This arm power regime is where the AS55 sign flip has been modeled to be...


A fair amount of time was spent in pre-locking prep, including:

  • The usual X green beat alignment tweak, to make the X ALS control not terrible
  • Both ITM oplevs were centered
  • For some reason, I had to flip the signs of the REFL165 matrix elements for the PRMI...
  • "Restore PRMI SB" has ASC automatically enabled, which results in unexpected kicks even with LSC off, which caused a few minutes head-scratching
  10815   Thu Dec 18 15:41:30 2014 ericqUpdateComputer Scripts / ProgramsOffsite backups of /cvs/cds going again

Since the Nodus switch, the offsite backup scripts (scripts/backup/rsync.backup) had not been running successfully. I tracked it down to the weird NFS file ownership issues we've been seeing since making Chiara the fileserver. Since the backup script uses rsync's "archive" mode, which preserves ownership, permissions, modification dates, etc, not seeing the proper ownership made everything wacky. 

Despite 99% of the searches you do about this problem saying you just need to match your user's uid and gid on the NFS client and server, it turns out NFSv4 doesn't use this mechanism at all, opting instead for some ID mapping service (idmapd), which I have no inclination of figuring out at this time. 

Thus, I've configured /etc/fstab on Nodus (and the control room machines) to use NFSv3 when mounting /cvs/cds. Now, all the file ownerships show up correctly, and the offsite backup of /cvs/cds is churning along happily. 

  10816   Thu Dec 18 16:21:08 2014 ericqUpdateComputer Scripts / Programsscripts not being backed up!

 I just stumbled upon this while poking around:

Since the great crash of June 2014, the scripts backup script has not been workingon op340m. For some reason, it's only grabbing the PRFPMI folder, and nothing else. 

Megatron seems to be able to run it. I've moved the job to megatron's crontab for now.

  10817   Fri Dec 19 14:25:48 2014 diegoUpdateComputer Scripts / Programselog restarted

elog was not responding for unknown reasons, since the elogd process on nodus was alive; anyway, I restarted it. 

  10818   Fri Dec 19 15:59:49 2014 JenneUpdateLSCLockloss from Wed

I swapped out one of the channels on Q's lockloss plotter - we don't need POP22Q, but I do want the PC drive.  

So, we still need to look into why the PC drive goes crazy, and if it is related to the buildup in the arms or just something intrinsic in the current FSS setup, but it looks like that was the cause of the lockloss that Q and Diego had on Wednesday.

1102929772_PCdriveRailed.png

  10819   Fri Dec 19 16:39:25 2014 ericqUpdateComputer Scripts / Programselog autostart

I've set up nodus to start the ELOG on boot, through /etc/init/elog.conf. Also, thanks to this, we don't need to use the start-elog.csh script any more. We can now just do:

controls@nodus:~  $ sudo initctl restart elog

I also tweaked some of the ELOG settings, so that image thumbnails are produced at higher resolution and quality. 

  10820   Fri Dec 19 16:59:32 2014 ericqUpdateComputer Scripts / ProgramsFSS Slow servo moved to megatron

Given that op340m showed some undesired behavior, and that the FSS slow seems prone to railing lately, I've moved the FSS slow servo job over to megatron in the same way I did for the MC autolocker. 

Namely, there is an upstart configuration (megatron:/etc/init/FSSslow.conf), that invokes the slow servo. Log file is in the same old place (/cvs/cds/caltech/logs/scripts), and the servo can be (re)started by running: 

controls@megatron|~ > sudo initctl start FSSslow

Maybe this won't really change the behavior. We'll see

  10821   Fri Dec 19 18:08:46 2014 JenneUpdateCDSSOS!!! HELP!! EPICS freeze 45min+ so far!

[Jenne, Diego]

The EPICS freeze that we had noticed a few weeks ago (and several times since) has happened again, but this time it has not come back on its own.  It has been down for almost an hour so far. 

 So far, we have reset the Martian network's switch that is in the rack by the printer.  We have also power cycled the NAT router.  We have moved the NAT router from the old GC network switch to the new faster switch, and reset the Martian network's switch again after that.

We have reset the network switch that is in 1X6.

We have reset what we think is the DAQ network switch at the very top of 1X7.

So far, nothing is working.  EPICS is still frozen, we can't ping any computers from the control room, and new terminal windows won't give you the prompt (so perhaps we aren't able to mount the nfs, which is required for the bashrc).

We need help please!

  10822   Fri Dec 19 19:21:04 2014 diegoUpdateCDSSOS!!! HELP!! EPICS freeze 45min+ so far!

Quote:

[Jenne, Diego]

The EPICS freeze that we had noticed a few weeks ago (and several times since) has happened again, but this time it has not come back on its own.  It has been down for almost an hour so far. 

 So far, we have reset the Martian network's switch that is in the rack by the printer.  We have also power cycled the NAT router.  We have moved the NAT router from the old GC network switch to the new faster switch, and reset the Martian network's switch again after that.

We have reset the network switch that is in 1X6.

We have reset what we think is the DAQ network switch at the very top of 1X7.

So far, nothing is working.  EPICS is still frozen, we can't ping any computers from the control room, and new terminal windows won't give you the prompt (so perhaps we aren't able to mount the nfs, which is required for the bashrc).

We need help please!

[EricQ]

 

EricQ suggested it may be some NFS related issue: if something, maybe some computer in the control room, is asking too much to chiara, then all the other machines accessing chiara will slow down, and this could escalate and lead to the Big Bad Freeze. As a matter of fact, chiara's dmesg pointed out its eth0 interface being brought up constantly, as if something is making it go down repeatedly. Anyhow, after the shutdown of all the computers in the control room, a  reboot of chiara, megatron and the fb was performed.

 

[Diego]

Then I rebooted pianosa, and most of the issues seem gone so far; I had to "mxstream restart" all the frontends from medm and everyone of them but c1scy seems to behave properly. I will now bring the other machines back to life and see what happens next.

  10823   Fri Dec 19 20:32:11 2014 diegoUpdateCDSSOS!!! HELP!! EPICS freeze 45min+ so far!

[Diego, Jenne]

 

Everything seems reasonably back to normal:

Notes:

  • the machines in the control room have been rebooted;
  • the c1iscey frontend now behaves;
  • I saw on nodus, which remained up and running the whole time, a bunch of   nfs: server chiara is not responding, timed out  messages, belonging to the freezing time; it may be that the sync option for the nfs share is too resource demanding, or some other network issue;
  • the FSS was doing strange stuff and the MC couldn't recover the lock; the MCautolocker script wasn't running because of the lock loss of the MC and the lack of communication between the machines; so we did a sudo initctl start MCautolocker on megatron and recovered the MC too.
  10824   Fri Dec 19 20:44:23 2014 JenneUpdateComputer Scripts / ProgramsFSS Slow servo moved to megatron

Today Q moved the FSS slow servo over to some init thing on megatron, and some time ago he did the same thing to the MC auto locker script.  It isn't working though.

Even though megatron was rebooted, neither script started up automatically.  As Diego mentioned in elog 10823, we ran sudo initctl start MCautolocker and sudo initctl start FSSslow, and the blinky lights for both of the scripts started.  However, that seems to be the only thing that the scripts are doing.  The MC auto locker is not detecting lockloses, and is not resetting things to allow the MC to relock.  The MC is happy to lock if I do it by hand though.  Similarly, the blinky light for the FSS is on, but the PSL temperature is moving a lot faster than normal.  I expect that it will hit one of the rails in under an hour or so. 

The MC autolocker and the FSS loop were both running earlier today, so maybe Q had some magic that he used when he started them up, that he didn't include in the elog instructions?

ELOG V3.1.3-