40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 24 of 341  Not logged in ELOG logo
ID Date Authorup Type Category Subject
  9331   Sat Nov 2 22:49:44 2013 CharlesUpdateISSCTN ISS Noise Suppression Requirement - Updated 10/27

 Previously in elog 8959, I gave a very simple method for determining the noise suppression behavior of the ISS. Recently, I recalculated this requirement in a more correct fashion and again redesigned the ISS to be used in the CTN experiment.

  • Determining the Requirement

Just as before, the data from PSL elog 1270 is necessary to infer a noise suppression requirement. The data presented there by Evan consists of two noise spectra, 1) the unstabilized RIN presently observed in the CTN experiment readout and 2) the theoretical brownian noise produced by thermal processes in the mirror coating+substrate. The statement "TF_mag = (Unstabilized RIN) / (Calculated Brownian Noise Limit)", where TF_mag refers to the required open-loop gain of the ISS, is actually a first order approximation of the 'required' noise suppression. In fact if we wanted the laser noise to be suppressed below the calculated brownian noise level, it is more correct to say 

        Closed-loop ISS gain = (Calculated Brownian Noise Limit) / (Unstabilized RIN)

As this essentially gives a noise suppression spectrum i.e. a closed-loop gain in linear control theory. Below is a very simple block diagram showing how the ISS fits into the CTN experiment. The F(f) block represents my full servo board.

    ISS_path.png

Some of the relevant quantities involved:

            plant-quant_1.png

            plant-quant_2.png

So looking at the block diagram, our full closed-loop transfer function is given by,

cl-loop.png

So then to determine the required F(f), i.e. the required transfer function for my servo, we consider the expression 

               requirement.png

The plant transfer function is simply Plant = (C(f) * a * P * A) ~ 0.014 V/V, where I have ignored the cavity pole around 97 kHz as our open-loop transfer function ends up crossing unity gain around 10 kHz. In the above, I have included what I call a 'safety factor' of 10. Essentially, I want to design my servo such that it suppresses noise well beyond what is actually required so that we can be sure noise contributions to experiment readouts are not significantly influenced by the laser intensity noise.

  • Proposed Servo Design

Using the data Evan reported for the brownian noise and free-running RIN, I came up with an F(f) to the meet the requirement as shown below.

CTN_TF_req-vs-proposed.png

 Where the blue curve includes the safety factor mentioned before. This plot just demonstrates that using my modular ISS design, I can meet the given noise suppression requirements.

To be complete, I'll say a little more about the final design.  As usual, the servo consists of three stages. The first is the usual LP filter that is always 'on' when the ISS loop is closed. The boosts I have chosen to use consist of an integrator with a single zero and a filter that looks somewhat like a de-whitening filter. The simulated open-loop transfer functions are shown below.

switching-filters.png

 

 

 

 

 

 

 

 

  9332   Sun Nov 3 00:05:52 2013 CharlesSummaryISSISS Update - Bout' time

Right near the end of summer, I had an ISS board that was nominally working, but had a few problems I couldn't really sort out. Since I've been back, I've spent a lot of time just replacing parts, trying different circuit topologies and generally attempting to make the board function as I hoped it might in all those design stages. Below is a brief list of some of the problems I've been fixing as well as the first good characterization of the board transfer function that I've been able to get.

We'll start with some of the simple problems and proceed to more complicated ones.

  • The 5V reference I was using to obtain an error signal from some arbitrary DC photodiode readout was only producing ~2.5 V. 
    • Turns out I just need a FET type op-amp for the Sallen-Key Filter that I was using to clean up any noise in the reference output, as the leakage current in a AD829 was causing a significant voltage drop. I put in an OPA140 and everything worked marvelously.
  • The way I set up input grounding (i.e. send a ~0 amplitude signal through the board as an input) passed a few Amps through one of my chips causing it to burn out rather fantastically.
    • There isn't a good way to fix this on the current board (besides just getting rid of the functionality altogether) so my solution so far has just been to redesign that particular sub-system/feature and when we implement the second version of the ISS, the input grounding will be done correctly
  • One of the ICs I'm using, specifically the AD8436 RMS-to-DC converter, causes some super strange oscillations in -5V power line. When this chip is soldered onto the board, the -5V supply jumps between -3V and -10V rather sporadically and the DC power-supply used to provide that -5V says that board is drawing ~600 mA on that particular power line.
    • To date, I don't really have any idea what's going with this chip, and I've tried a lot of things to remedy the problem. My first thought was that I had some sort of short somewhere so I took the chip off the board, cleaned up all the excess solder and flux around the chip's footprint and then meticulously soldered a new chip on (when I say meticulously, it took over an hour to solder 20 little feet. I really really didn't want to short anything accidentally as the chip only comes in a package with ridicously small spacing between the leads). Lo and behold, nothing happened. I still saw the same oscillations in power supply and the board was still drawing between >500 mA on that line. Just to be sure, I soldered on a third chip taking the same amount of care and had the same problems.
    • I went over the schematic in Altium that we used to order the board, and unless the manufacturer made a mistake somewhere, there aren't any incorrectly routed signals would cause, say, two active devices to try setting the voltage of a particular node to different values.
    • I got some QSOP-to-DIP package converters so that I could mess around with the AD8436 on a breadboard to make sure it functioned correctly. I set up an identical circuit to the one on the PCB and didn't see any oscillations in the power supply, both for +-5V and +-15V as the chip can handle both supply voltages. I'm not really sure how to interpret this...
    • I'm still actively trying to figure this particular problem out, but I'm shooting in the dark at this point. 
  • Initial attempts to measure the transfer-function of the board were wrought with failure.
    • I figured out, with Nic's help, that the board needs the 'loop closed' with a significant broadband attenuator (to simulate the plant optics discussed in elog 9331) in order to not have constant railing of the high gain op-amp filter stages. Even after I did this, the measured transfer functions were not at all consistent with simulation. I wasn't sure if it was just a part issue, a design issue or a misunderstanding/bad data collection on my part so I just redesigned the whole servo and stuffed the board with entirely new components from around the 40m. Turns out the newly designed servo behaved more properly, as I will show below.

The above list encompasses all the issues I've had in making the ISS board function correctly. No other major problems exist to my knowledge.

I was able to measure both the open- and closed-loop transfer functions of the servo with the SR785. The results are shown below.

full-op-loop.png

The transfer function with the boosts on caps at a particular value set by op-amp railing, i.e. below 100 Hz, the op-amps are already putting out their max voltage. This is the usual physical limitation when measuring the transfer function of an integrator. We can also see that the measured phase follows the simulated phase above ~300 Hz. The 'phase matching' at low frequency is again do to the op-amp railing in the servo output..

The closed-loop gain is shown below,

full-cl-loop.png

The measured closed-loop gain with the boosts on again matches the LISO simulation quite well except at low frequency where we are limited by op-amp railing. We compare the measured closed-loop transfer function to the desired noise suppression stipulated in my previous elog 9331,

req-vs-meas.png

 And we might hopefully conclude that my servo functions as desired. One should note that the op-amp railing seen in these measurements is not indicative of limitations we might face in some application of the ISS for the following reason. These transfer functions were measured with a 100 mV excitation signal (it is necessary to keep this signal amplitude large enough so that the inherent signal-to-noise ratio of the excitation source is large enough for accurate measurement) which leads to somewhat prompt railing of the op-amps. When the ISS operates to actually stabilize a laser, the input error signal will be much smaller (on the order of a few 10's of mV or less) and will decrease significantly assuming correct operation of the ISS. This means we won't see the same type of gain limitations.

 

What now, you ask?

Aside from the problem with the AD8436 chip, the ISS board seems to be functioning correctly. The transfer functions we have measured are correct to within the component tolerances and all of the various subsystems are behaving as they were designed to. Moving toward the goal of having this system work in situ for the CTN experiment, I need to do the following things,

  • Design a housing for the board -> order said housing and the front panel previously designed
  • Make sure the power supply daughter PCB boards are compatible with the ISS board and can provide power correctly
  • Talk to Evan and Tara about integrating the ISS with their experiment and make sure my board can do everything it needs to in that context.

So close, or so I say all the time 

 

  9746   Mon Mar 24 19:42:12 2014 CharlesFrogsVACPower Failure

 The 40m experienced a building-wide power failure for ~30 seconds at ~7:38 pm today.

Thought that might be important...

  12232   Thu Jun 30 14:31:02 2016 ChemistryUpdateSUSNO

  4432   Wed Mar 23 12:40:22 2011 Chief Recycling OfficerHowToEnvironmentRecycle stuff!

The following is a message from the LIGO 40m Chief Recycling Officer:

Please get up off your (Alignment Stabilization Servo)es and recycle your bottles and cans!  There is a recycling bin in the control room.  Recent weeks have seen an increase in number of bottles/cans thrown away in the regular garbage.  This is not cool. 

Thank you,

---L4mCRO

  3161   Tue Jul 6 17:29:04 2010 ChipUpdatePEMThe Ranger is mine!
  8061   Mon Feb 11 18:39:10 2013 ChloeUpdateGeneralPictures of Circuitry in Photodiode

I am going to be making measurements to find the optical mounts with the least noise. I am using a quadrature photodiode to record intensity of laser light. These are pictures of the circuitry inside (both sides). I will be designing/making some circuitry on a breadboard in the next few days in order to add and subtract the signals to have pitch and yaw outputs.

Attachment 1: IMG_0327.JPG
IMG_0327.JPG
Attachment 2: IMG_0329.JPG
IMG_0329.JPG
  8067   Tue Feb 12 17:26:31 2013 ChloeUpdate QPD circuitry to test mount vibrations

 I spent awhile today reading about op-amps and understanding what would be necessary to design a circuit which would directly give pitch and yaw of the QPD I am using. After getting an idea of what signals would be summed or subtracted, I opened up the QPD to take better pictures than last time (sorry, the pictures were blurry last time and I didn't realize). It turns out some of the connections have been broken inside the QPD, which would explain why we saw an unchanging signal in Ch2 on the oscilloscope yesterday when trying to test the laser setup. 

I found a couple other QPDs, which I will be using to help understand the circuit (and what is going on). I will be trying to use the same QPD box since it has banana cable and BNC cable adapters, which is helpful to have in the lab. Once I have concluded what the circuitry is like and designed electronics to add and subtract signals, I will build and mount all the circuits within the box (more sturdily than last time) so as to have a quality way of measuring the mount vibrations when I get there. 

  8090   Fri Feb 15 17:11:13 2013 ChloeUpdate QPD circuit pictures

 I took better pictures of the circuits of the QPD and spent a couple of hours with a multimeter trying to figure out how all the connections worked. I will continue to do so and analyze the circuits over the weekend to try to understand what is going on. I also have an old SURF report that Eric sent me that is similar to the design I was planning to use to sum the pitch and yaw signals. I will try and look at this over the weekend. 

Attachment 1: IMG_0337.JPG
IMG_0337.JPG
Attachment 2: IMG_0338.JPG
IMG_0338.JPG
  8111   Tue Feb 19 17:14:56 2013 ChloeUpdate QPD Circuit

I finished working out the circuit and figured out where the broken connections were. This is diagrammed in my notebook (will draw up more nicely and include in future elog post). Within the QPD circuitry, it seems like there are already opamps which regulate the circuit. I need to discuss the final diagram further with Eric.

I rewired the circuit inside the QPD box, which took awhile because it was difficult to solder the wires to such small locations without having multiple wires touch. This is completed, and on Friday I will begin to make the circuit to add/subtract signals to give pitch and yaw.

  8139   Fri Feb 22 17:33:38 2013 ChloeUpdate QPD circuit diagram

Today I tested the circuitry within the QPD to make sure I had put it back together correctly. The output was able to detect when a laser pointer was shone on different quadrants of the QPD when hooked up to the oscilloscope, so fixing the QPD worked! 

Following this, I spent awhile understanding what was going on with the circuit reading from the QPD. I concluded that the opamp on the QPD circuit acted as a low pass filter, with corner frequency of about 17 kHz, which serves our purposes (we are only interested in frequencies below 1 kHz). I diagrammed the circuit that I plan to build to give pitch and yaw (attached). It will be necessary to make small modifications to the circuitry already built (removing some resistors), which I have started on. 

Next time, I will construct the circuit that adds/subtracts signals on a breadboard and test it with the QPD circuit that is already built. 

Attachment 1: IMG_0377.JPG
IMG_0377.JPG
  8173   Tue Feb 26 17:36:28 2013 ChloeUpdate QPD Circuit

I corrected the circuit diagram I constructed last time - it would read 0 output most of the time because I made a mistake with the op amps. This will be attached once I discuss it with Eric.

I searched the 40m lab and ended up finding a breadboard in the Bridge lab. I then spent awhile trying to remove the QPD from the circuit board it was on, which was difficult since it had 6 pins. After that, I soldered wires to the QPD legs and began constructing the circuit on the breadboard. I also spent time showing Annalisa the setup and explaining my project.

On Friday I will try and reconstruct all of the circuitry from before for the QPD.

  8233   Tue Mar 5 17:23:06 2013 ChloeUpdate QPD Adding/Subtracting Circuit

Today I finished building the adding/subtracting circuit for the QPD and tested that the QPD could see a laser moving across its visual field for both pitch and yaw. It didn't seem to behave weirdly (saturate) at the edges, but I need to test this more carefully to be sure.

However, this circuit uses many op amps, which will cause problems for building the actual circuit to fit into the QPD box. I am trying to figure out how to do this with fewer op amps (both with a quad op amp for amplifying the signals from the QPD and by summing/subtracting the signals with a single op amp instead of 3).

I finally got around to asking Steve to order more breadboards! Trying to determine what would be a good QPD to order for the final circuit, since we do not have any unmounted QPDs that aren't ancient. I'll read up on things I don't know enough about (namely op amps).

  8295   Thu Mar 14 16:56:16 2013 ChloeUpdate QPD Circuit Design

I have sketched out the circuit design for the QPD. However, it seems like even when using a different opamp configuration, which I talked to Eric about on Friday, space will be a problem. It may be possible to squeeze everything onto a single circuit board to fit in the QPD box but what I think is more likely is that I will need to have 2 separate circuit boards both mounted within the box, one which integrates the signal from the QPD and the other which adds/subtracts (this involves many resistors which will take up a lot of space). I will continue to think about the best design for this.

I will try to have the circuit built in the next week or so, which may be difficult since I just started finals which will take most of my time. I spent most of this week writing up an ECDL proposal for a SURF with Tara. I'll make up for whatever work I miss since I'll be here for my spring break and doing little besides working in lab.

  8338   Mon Mar 25 16:51:43 2013 ChloeUpdate Final QPD Circuit Design

This is the final version of the QPD circuit I'm going to build. After playing around with the spatial arrangement, this should fit into the box that I was planning to use, although it will be a rather tight fit. The pitch, yaw, and summing circuit will be handled with a quad op amp. Planning to meet with Eric tomorrow to figure out the logistics of building things.

In the meantime, I'm reading about designing the ECDL for my summer project with Tara. He sent me several papers to read so we can talk on Wednesday.

Attachment 1: IMG-20130325-00244.jpg
IMG-20130325-00244.jpg
  8346   Mon Mar 25 23:27:26 2013 ChloeUpdate QPD Circuit

In order to test the mount vibrations, I will likely try and make a different circuit work (with the summing/subtracting on an external breadboard) and designing an optimal circuit will be a side project. This is the circuit with the power supply Rana came up with, and the design I had in mind for the rest of the circuit. In my free time, I will try to figure out what parts to get that reduce noise and slowly work on building this, since it would be useful to have in the lab. 

 https://www.circuitlab.com/circuit/7sx995/qpd-circuit/#postsave_access_control_settings

  8358   Tue Mar 26 17:32:30 2013 ChloeUpdate QPD/ECDL Progress

I built the summing/subtracting circuit on the breadboard, and hooked this up with one of the other QPDs I found (image of setup attached). I wasn't able to get this to read the correct signals when testing with a laser pointer after a couple of hours of troubleshooting... I will hopefully get this working in the next day or 2...

I'm going to read up on ECDL stuff for Tara tonight and hopefully figure out what sort of laser diode we should purchase, since I'm meeting with Tara tomorrow. experimenting

Attachment 1: IMG-20130326-00245.jpg
IMG-20130326-00245.jpg
  8381   Mon Apr 1 16:13:50 2013 ChloeUpdate QPD

Because we would like to get started on testing mount vibrations as soon as possible, I've been trying to get one of the other QPDs we found to work with the summing/subtracting circuit on a breadboard. I've been using a power supply that I think Jamie built 15 years ago... which seems to be broken as of today, since I no longer read any signal from it with an oscilloscope.

I tried using a different power supply, but I still can't read any change in signal with the QPD for any of the quadrants when using a laser pointer to shine light on it. I'll be working with Eric on this later this week. In the meantime, I'll try and come up with a shopping list for the nicer QPD circuit that'll be a longer term side project.

  8403   Wed Apr 3 16:03:59 2013 ChloeUpdate QPD Voltage Regulators

The voltage regulator on the QPD breadboard seems to be having problems... yesterday Eric helped me debug my circuit and discovered that the +12V regulator was overheating, so we replaced it. Today, I found that the -12V regulator was also doing the same thing, so I replaced it. However, it's still overheating. We checked all of the setup for the power regulators yesterday, so I'm not sure what's wrong.

I've also noticed that not all the connections on the breadboard that I've been using seem to work - I may search for a new breadboard in this case. Need to check I'm not doing something stupid with that.

  8511   Tue Apr 30 12:25:23 2013 ChloeUpdate QPD

Annalisa and I met yesterday and fixed the voltage regulator on the breadboard so the QPD circuit is working. We will meet with Eric on Thursday to determine the course of action with measurements.

  1797   Mon Jul 27 14:43:34 2009 ChrisUpdate Photodetectors

I found two ThorLabs PDA55 Si photodetectors that says detect visible light from DC to 10MHz that I'm going to use from now on.  I don't know how low of a frequency they will actually be good to.

  1810   Wed Jul 29 19:41:58 2009 ChrisConfigurationGeneralEUCLID-setup configuration change

David and I were thinking about changing the non-polarizing beam splitter in the EUCLID setup from 50/50 to 33/66 (ref picture).  It serves as a) a pickoff to sample the input power and b) a splitter to send the returning beam to a photodetector 2 (it then hits a polarizer and half of this is lost.  By changing the reflectivity to 66% then less (1/3 instead of 1/2) of the power coming into it would be "lost" at the ref photodetector 1, and on the return trip less would be lost at the polarizer (1/6 instead of 1/4).

 

 

Attachment 1: EUCLID.png
EUCLID.png
  1845   Thu Aug 6 17:51:21 2009 ChrisUpdateGeneralDisplacement Sensor Update

For the past week Dmass and I have been ordering parts and getting ready to construct our own modified version of EUCLID (figure).  Changes to the EUCLID design could include the removal of the first lens, the replacement of the cat's eye retroreflector with a lens focusing the beam waist on a mirror in that arm of the Michelson, and the removal of the linear polarizers.  A beam dump was added above the first polarizing beam splitter and the beam at Photodetector 2 was attenuated with an additional polarizing beam splitter and beam dump.  Another proposed alteration is to change the non-polarizing beam splitter from 50/50 to 33/66.  By changing the reflectivity to 66\%, less power coming into the non-polarizing beam splitter would be ``lost" at the reference detector (1/3 instead of 1/2), and on the return trip less power would be lost at the polarizing beam splitter (1/6 instead of 1/4).  Also, here's a noise plot comparing a few displacement sensors that are used to the shot noise levels for the three designs I've been looking at.

Attachment 1: Actual_Sensor.png
Actual_Sensor.png
Attachment 2: Sensitivity.png
Sensitivity.png
  1846   Thu Aug 6 18:21:03 2009 ChrisUpdateGeneralDisplacement Sensor Update

Quote:

For the past week Dmass and I have been ordering parts and getting ready to construct our own modified version of EUCLID (figure).  Changes to the EUCLID design could include the removal of the first lens, the replacement of the cat's eye retroreflector with a lens focusing the beam waist on a mirror in that arm of the Michelson, and the removal of the linear polarizers.  A beam dump was added above the first polarizing beam splitter and the beam at Photodetector 2 was attenuated with an additional polarizing beam splitter and beam dump.  Another proposed alteration is to change the non-polarizing beam splitter from 50/50 to 33/66.  By changing the reflectivity to 66\%, less power coming into the non-polarizing beam splitter would be ``lost" at the reference detector (1/3 instead of 1/2), and on the return trip less power would be lost at the polarizing beam splitter (1/6 instead of 1/4).  Also, here's a noise plot comparing a few displacement sensors that are used to the shot noise levels for the three designs I've been looking at.

 I thought slightly harder and I think that the beamsplitter stays. We will lose too much power on the first PD if we do that:

33/66:  Pwr @ PD2 = 2/3*1/3*1/2 =  1/9 Pin

            Pwr @ PD3 = 2/3*2/3*1/2 = 2/9 Pin

 

50:50 Pwr @ PD2 = PWR @ PD3 = 1/8 Pin

balancing them is probably better.

  1894   Wed Aug 12 23:45:03 2009 ChrisUpdateGeneralLong range michelson

Today I set up the EUCLID long range michelson design on the SP table; It's the same as the setup posted earlier, but without the pickoff (at PD1), which can be added later, and a few other minor changes (moved lenses, mirrors, PDs - nothing major).  I hooked up the two PD's to the oscilliscope and got a readout that pointed to more power hitting PD2 than PD3.

Attachment 1: Actual_Sensor.png
Actual_Sensor.png
  1908   Fri Aug 14 23:45:14 2009 ChrisUpdateGeneralLong Range Readout

The EUCLID-style Michelson readout is on the SP table now and is aligned.  See image below.  I took several power spectra with the plotter attached to the HP3563 (not sure if there's another way to get the data out) and I'm still waiting to calibrate (since dP/dL isn't constant as it isn't locked, this is taking a bit longer).  When put into XY mode on the oscilliscope (plotting Voltage at PD2 on the x and Voltage at PD3 on the y), a Lissajous figure as in the first plot below.  It's offset and elliptical due to imperfections (noise, dc offset, etc) but can ideally be used to calculate the L_ target mirror movement.  By rotating the first quarter wave plate by ~80.5deg counter-clockwise (fast axis was originally at Pi/8, now at 103deg), I was able to turn the Lissajous figure from an ellipse into a more circular shape, which would ideally allow for us to use a circular approximation (much simpler) in our displacement calculations.

Attachment 1: Table_Setup.png
Table_Setup.png
Attachment 2: Ellipse.jpg
Ellipse.jpg
Attachment 3: Circle.jpg
Circle.jpg
  10189   Fri Jul 11 22:28:34 2014 ChrisUpdateCDSc1auxex "Unknown Host"

Quote:

c1auxex has forgotten who it is.  Slow sliders for the QPD head were not responding, so I did a soft reboot from telnet. The machine didn't come back, so I plugged the RJ45-DB9 cable into the machine and looked at it through a minicom session.  When I key the crate, it gives me an error that it can't load a file, with the error code 0x320001.  Looking that up on a List of VxWorks error codes, I see that it is:    S_hostLib_UNKNOWN_HOST (3276801 or 0x320001)

I'm not sure how this happened.  I unplugged and replugged in the ethernet cable on the computer, but that didn't help.  Rana is going in to wiggle the other end of the ethernet cable, in case that's the problem.  EDIT:  Replacing the ethernet cable did not help.

Former elogs that are useful:  10025, 10015

EDIT:  The actual error message is:

boot device          : ei                                                      
processor number     : 0                                                       
host name            : chiara                                                  
file name            : /cvs/cds/vw/mv162-262-16M/vxWorks                       
inet on ethernet (e) : 192.168.113.59:ffffff00                                 
host inet (h)        : 192.168.113.104                                         
user (u)             : controls                                                
flags (f)            : 0x0                                                     
target name (tn)     : c1auxex                                                 
startup script (s)   : /cvs/cds/caltech/target/c1auxex/startup.cmd             
                                                                               
Attaching network interface ei0... done.                                       
Attaching network interface lo0... done.                                       
Loading...                                                                     
Error loading file: errno = 0x320001.                                          
Can't load boot file!! 

 We fixed this problem (at least for now) by adding c1auxex to the /etc/hosts file on chiara (following a hint from this page). The DNS setup might be the culprit here.

  10309   Thu Jul 31 18:54:03 2014 ChrisFrogselogMicroSoft BingBot is attacking us

Quote:

 The ELOG was frozen, with this in the .log file:   

GET /40m/?id=1279&select=1&rsort=Type HTTP/1.1

Cache-Control: no-cache

Connection: Keep-Alive

Pragma: no-cache

Accept: */*

Accept-Encoding: gzip, deflate

From: bingbot(at)microsoft.com

Host: nodus.ligo.caltech.edu

User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

  (hopefully there's a way to hide from the Bing Bot like we did from the Google bot)

 

Yesterday elog was excruciatingly slow, and bingbot was the culprit. It was slurping down elog entries and attachments so fast that it brought nodus to its knees. So I created a robots.txt file disallowing all bots, and placed it in the elog's scripts directory (which gets served at the top level). Today the log feels a little snappier -- there's now much less bot traffic to compete with when using it.

We might be able to let selected bots back in with a crawl rate limit, if anyone misses searching the elog on bing.

  10632   Wed Oct 22 21:06:33 2014 ChrisUpdateDAQ40m frames onto the cluster

Quote:

 Dan Kozak is rsync transferring /frames from NODUS over to the LDAS grid. He's doing this without a BW limit, but even so its going to take a couple weeks. If nodus seems pokey or the net connection to the outside world is too tight, then please let me and him know so that he can throttle the pipe a little.

The recently observed daqd flakiness looks related to this transfer. It appears to still be ongoing:

nodus:~>ps -ef | grep rsync
controls 29089   382  5 13:39:20 pts/1   13:55 rsync -a --inplace --delete --exclude lost+found --exclude .*.gwf /frames/trend
controls 29100   382  2 13:39:43 pts/1    9:15 rsync -a --delete --exclude lost+found --exclude .*.gwf /frames/full/10975 131.
controls 29109   382  3 13:39:43 pts/1    9:10 rsync -a --delete --exclude lost+found --exclude .*.gwf /frames/full/10978 131.
controls 29103   382  3 13:39:43 pts/1    9:14 rsync -a --delete --exclude lost+found --exclude .*.gwf /frames/full/10976 131.
controls 29112   382  3 13:39:43 pts/1    9:18 rsync -a --delete --exclude lost+found --exclude .*.gwf /frames/full/10979 131.
controls 29099   382  2 13:39:43 pts/1    9:14 rsync -a --delete --exclude lost+found --exclude .*.gwf /frames/full/10974 131.
controls 29106   382  3 13:39:43 pts/1    9:13 rsync -a --delete --exclude lost+found --exclude .*.gwf /frames/full/10977 131.
controls 29620 29603  0 20:40:48 pts/3    0:00 grep rsync

Diagnosing the problem:

I logged into fb and ran "top". It said that fb was waiting for disk I/O ~60% of the time (according to the "%wa" number in the header). There were 8 nfsd (network file server) processes running with several of them listed in status "D" (waiting for disk). The daqd logs were ending with errors like the following suggesting that it couldn't keep up with the flow of data:

[Wed Oct 22 18:58:35 2014] main profiler warning: 1 empty blocks in the buffer
[Wed Oct 22 18:58:36 2014] main profiler warning: 0 empty blocks in the buffer
GPS time jumped from 1098064730 to 1098064731

This all pointed to the possibility that the file transfer load was too heavy.

Reducing the load:

The following configuration changes were applied on fb.

Edited /etc/conf.d/nfs to reduce the number of nfsd processes from 8 to 1:

OPTS_RPC_NFSD="1"

(was "8")

Ran "ionice" to raise the priority of the framebuilder process (daqd):

controls@fb /opt/rtcds/rtscore/trunk/src/daqd 0$ sudo ionice -c 1 -p 10964

And to reduce the priority of the nfsd process:

controls@fb /opt/rtcds/rtscore/trunk/src/daqd 0$ sudo ionice -c 2 -p 11198

I also tried punishing nfsd with an even lower priority ("-c 3"), but that was causing the workstations to lag noticeably.

After these changes the %wa value went from ~60% to ~20%, and daqd seems to die less often, but some further throttling may still be in order.

  10868   Wed Jan 7 13:39:42 2015 ChrisUpdateLSCDARM phase budget

I think the dolphin and RFM transit times are double-counted in this budget. As I understand it, all IPC transit times are already built in to the cycle time of the sending model. That is, the sending model is required to finish its computational work a little bit early, so there's time left to transmit data to the receivers before the start of the next cycle. Otherwise you get IPC errors. (This is why the LSC models at the sites can't use the last ~20 usec of their cycle without triggering IPC errors. They have to allow that much time for the RFM to get their control signals down the arms to the end stations.)

For instance, the delay measurement in elog 9881 (c1als to c1lsc via dolphin) shows only the c1lsc model's own 61 usec delay. If the dolphin transfer really took an additional cycle, you would expect 122 usec.

And in elog 10811 (c1scx to c1rfm to c1ass), the delay is 122 usec, not because the RFM itself adds delay, but because an extra model is traversed.

Bottom line: there may still be some DARM phase unaccounted for. And it would definitely help to bypass the c1rfm model, as suggested in 9881.

  10897   Tue Jan 13 18:47:20 2015 ChrisConfigurationComputer Scripts / Programsinstafoton setup

To use instafoton, right click an MEDM screen, open the Execute menu, and choose "Foton".  Then click on the EPICS channel of a filter module as displayed on the screen.

Here's how it was set up:

  • Install instafoton.py in /opt/rtcds/caltech/c1/scripts; edit paths to localize for the 40m
  • Add instafoton to the MEDM_EXEC_LIST environment variable, newly defined in /ligo/cdscfg/workstationrc.sh:
    export MEDM_EXEC_LIST="Edit this screen;medm &A &:Probe;probe &P &:Foton (Pick filter PV);/opt/rtcds/caltech/c1/scripts/instafoton.py &P &"
  10898   Tue Jan 13 23:17:57 2015 ChrisFrogsComputer Scripts / Programsmedm time machine

After recompiling medm with a patch for dumping screens (attached), I added a time machine to the right-click Execute menu.  It's installed under /cvs/cds/caltech/users/wipf/src/medm_time_machine. Dependencies include the python CA server module (pcaspy) and the latest nds2-client 0.11.2.  These were also installed under my users directory, to avoid interfering with other tools.

 

Attachment 1: tm.png
tm.png
Attachment 2: dump_medm_screen.patch
--- /ligo/apps/epics-3.14.12_long/extensions/src/medm/medm/utils.c.orig	2015-01-13 18:56:44.867720104 -0800
+++ /ligo/apps/epics-3.14.12_long/extensions/src/medm/medm/utils.c	2015-01-13 22:49:56.636820963 -0800
@@ -4156,6 +4156,37 @@
 	timeOffset = time900101 - time700101;
 }
 
+#if((2*MAX_TRACES)+2) > MAX_PENS
+#define MAX_COUNT 2*MAX_TRACES+2
+#else
+#define MAX_COUNT MAX_PENS
... 75 more lines ...
  11052   Thu Feb 19 23:23:52 2015 ChrisUpdateCDSBad CDS behavior

The frontends have some paths NFS-mounted from fb. fb is on the ragged edge of being I/O bound. I'd suggest moving those mounts to chiara. I tried increasing the number of NFS threads on fb (undoing the configuration change I'd previously made here) and it seems to help with EPICS smoothness -- although there are still occasional temporal anomalies in the time channels. The daqd flakiness (which was what led me to throttle NFS on fb in the first place) may now recur as well.

 

Quote:

At about 10AM, the C1LSC frontend stopped reporting any EPICS information. The arms were locked at the time, and remained so for some hours, until I noticed the totally whited-out MEDM screens. The machine would respond to pings, but did not respond to ssh, so we had to manually reboot.

Soon thereafter, we had a global 15min EPICS freeze, and have been in a weird state ever since. Epics has come back (and frozen again), but the fast frontends are still wonky, even when EPICS is not frozen. Intermittantly, the status blinkers and GPS time EPICS values will freeze for multiple seconds at a time, sporadically updating. Looking at a StripTool trace of an IOPs GPS time value shows a line with smooth portions for about 30 seconds, about 2 minutes apart. Between this is totally jagged step function behavior. C1LSC needed to be power cycled again; trying to restart the models is tough, because the EPICS slowdown makes it hard to hit the BURT button, as is needed for the model to start without crashing.

The DAQ network switch, and martian switch inside were power cycled, to little effect. I'm not sure how to diagnose network issues with the frontends. Using iperf, I am able to show hundreds of Mbit/s bandwidth betweem the control room machines and the frontends, but their EPICS is still totally wonky. 

What can we do??? indecision

 

  12487   Tue Sep 13 16:16:28 2016 ChrisOmnistructureVACvacuum interlock bypassed

(Steve, Chris)

The pumpdown had stalled because of some ancient vacuum interlock code that prevented opening the valve V1 between the turbo pump and the main volume.

This interlock [0] compares the channels C1:Vac-P1_pressure and C1:Vac-PTP1_pressure, neither of which is functioning at the moment. The P1 channel apparently stopped reading sometime during the vent, and contained a value of ~700 torr, while the PTP1 channel contained 0. So the interlock code saw this huge apparent pressure difference and refused to move the valve.

To bypass this check, we used caput to enter a pressure of 0 for P1.

[0] /cvs/cds/caltech/state/from_luna/VacInterlock.st

  13318   Mon Sep 18 17:30:54 2017 ChrisUpdateCDSFB wiper script

Attached is the version of the wiper script we use on the CryoLab cymac. It works with perl v5.20.2. Is this different from what you have?

Attachment 1: wiper.pl
#!/usr/bin/perl
use File::Basename;

print "\n" .  `date` . "\n";
# Dry run, do not delete anything
$dry_run = 1;

if ($ARGV[0] eq "--delete") { $dry_run = 0; }

print "Dry run, will not remove any files!!!\n" if $dry_run;
... 184 more lines ...
  13338   Thu Sep 28 06:35:44 2017 ChrisUpdateLSCDAC noise measurement (again)

Since you're monitoring two channels simultaneously, you could try subtracting them, as an alternative to carving out bandstops.

  1. Drive both channels with the same time series
  2. Tweak the filter module gains if needed to equalize the analog outputs
  3. Apply SR785 user math to subtract the two channels (or A-B mode of a single input, if you're not using that already?)
  4. Measure the residual, which should be the incoherent part containing DAC + coil driver noise

Subtraction can conceal certain annoying effects (like numerical noise or level crossing glitches) that remain coherent for two identical outputs. It might be worth experimenting with a differential offset or sinusoid, to try to break up that kind of coherence if it exists.

  16853   Sat May 14 08:36:03 2022 ChrisUpdateDAQDAQ troubleshooting

I heard a rumor about a DAQ problem at the 40m.

To investigate, I tried retrieving data from some channels under C1:SUS-AS1 on the c1sus2 front end. DQ channels worked fine, testpoint channels did not. This pointed to an issue involving the communication with awgtpman. However, AWG excitations did work. So the issue seemed to be specific to the communication between daqd and awgtpman.

daqd logs were complaining of an error in the tpRequest function: error code -3/couldn't create test point handle. (Confusingly, part of the error message was buffered somewhere, and would only print after a subsequent connection to daqd was made.) This message signifies some kind of failure in setting up the RPC connection to awgtpman. A further error string is available from the system to explain the cause of the failure, but daqd does not provide it. So we have to guess...

One of the reasons an RPC connection can fail is if the server name cannot be resolved. Indeed, address lookup for c1sus2 from fb1 was broken:

$ host c1sus2
Host c1sus2 not found: 3(NXDOMAIN)

In /etc/resolv.conf on fb1 there was the following line:

search martian.113.168.192.in-addr.arpa

Changing this to search martian got address lookup on fb1 working:

$ host c1sus2
c1sus2.martian has address 192.168.113.87

But testpoints still could not be retrieved from c1sus2, even after a daqd restart.

In /etc/hosts on fb1 I found the following:

192.168.113.92  c1sus2

Changing the hardcoded address to the value returned by the nameserver (192.168.113.87) fixed the problem.

It might be even better to remove the hardcoded addresses of front ends from the hosts file, letting DNS function as the sole source of truth. But a full system restart should be performed after such a change, to ensure nothing else is broken by it. I leave that for another time.

  16855   Mon May 16 12:59:27 2022 ChrisUpdateDAQDAQ troubleshooting

It looks like the RFM problem started a little after 2am on Saturday morning (attachment 1). It’s subsequent to what I did, but during a time of no apparent activity, either by me or others.

The pattern of errors on c1rfm (attachment 2) looks very much like this one previously reported by Gautam (errors on all IRFM0 ipcs). Maybe the fix described in Koji’s followup will work again (involving hard reboots).

Attachment 1: timeseries.png
timeseries.png
Attachment 2: err.png
err.png
  1694   Wed Jun 24 10:53:34 2009 Chris ZimmermanUpdateGeneralWeek 1/2 Update

I've spent most of the last week doing background reading; fourier transforms, shm, e&m, and other physics that I didn't cover at school.  I also read a few chapters in Saulson, especially the chapter on noise and shot noise.  To get a better grip on what I'm going to be doing I read through the polarization chapter in Hobbs' "Optics" text, mostly on wave plates since that's a large part of this readout.  Since then I've been working up to calculating the shot noise, starting with the electric field throughout the new interferometer readout.

  1710   Wed Jul 1 10:56:42 2009 Chris ZimmermanUpdateGeneralWeek 2/3 Update

I spent the last week working a lot with the differences between a basic Michelson readout and the new one as a displacement sensor.  The new one (w/ wave plates) ends with two differently polarized beams and should have better sensitivity; I've also been going through noise/sensitivity calculations for each, although that hit a road block when I had to start the 1st SURF progress report, which has taken up most of my time since Saturday.

  1720   Wed Jul 8 11:05:40 2009 Chris ZimmermanUpdateGeneralWeek 3/4 Update

The last week I've spent mostly working on calculating shot noise and other sensitivities in three michelson sensor setups, the standard michelson, the "long range" michelson (with wave plates), and the proposed EUCLID setup.  The goal is to show that there is some inherent advantage to the latter two setups as displacement sensors.  This involved looking into polarization and optics a lot more, so I've been spending a lot of time on that also.  For example, the displacement sensitivity/shot noise on the standard michelson is around 6:805*10^-17 m/rHz at L_=1*10^-7m, as shown in the graph.  NSD_Displacement.png

  1750   Wed Jul 15 12:44:28 2009 Chris ZimmermanUpdateGeneralWeek 4/5 Update

I've spent most of the last week working on finishing up the UCSD calculations, comparing it to the EUCLID design, and thinking about getting started with a prototype and modelling in MATLAB.  Attached is something on EUCLID/UCSD sensors.

Attachment 1: Comparison.pdf
Comparison.pdf Comparison.pdf
  1779   Wed Jul 22 16:15:52 2009 Chris ZimmermanUpdateGeneralWeek 5/6 Update

The last week I've started setting up the HeNe laser on the PSL table and doing some basic measurements (Beam waist, etc) with the beam scan, shown on the graph.  Today I moved a few steering mirrors that steve showed me from at table on the NW corner to the PSL table.  The goal setup is shown below, based on the UCSD setup.  Also, I found something that confused me in the EUCLID setup, a  pair of quarter wave plates in the arm of their interferometer, so I've been working out how they organized that to get the results that they did.  I also finished calculating the shot noise levels in the basic and UCSD models, and those are also shown below (at 633nm, 4mw) where the two phase-shifted elements (green/red) are the UCSD outputs, in quadrature (the legend is difficult to read).

 

 

Attachment 1: Beam_Scan.jpg
Beam_Scan.jpg
Attachment 2: Long_Range_Michelson_Setup_1_-_Actual.png
Long_Range_Michelson_Setup_1_-_Actual.png
Attachment 3: NSD_Displacement.png
NSD_Displacement.png
  14305   Mon Nov 19 14:59:48 2018 ChubUpdateVACVent 81

Vent 80 is nearly complete; the instrument is almost to atmosphere.  All four ion pump gate valves have been disconnected, though the position sensors are still connected,and all annulus valves are open.  The controllers of TP1 and TP3 have been disconnected from AC power. VC1 and VC2 have been disconnected and must remained closed. Currently, the RGA is being vented through the needle valve and the RGA had been shut off at the beginning of the vent preparations.  VM1 and VM3 could not be actuated.  The condition status is still listed as Unidentified because of the disconnected valves. 

  14350   Thu Dec 13 10:03:07 2018 ChubUpdateGeneralOMC chamber

Bob, Aaron, and I removed the door from the OMC chamber this morning.  Everything went well.

  14364   Tue Dec 18 11:42:40 2018 ChubUpdateGeneralAcromag box wired

The Auxiliary DAQ Chassis, or Acromag box, is now wired and ready for testing.  I will be sorting the cables at the vacuum rack to make connection to the box easier.

 

  14395   Thu Jan 10 11:32:40 2019 ChubUpdateVACManual valve interfaced with CDS

Connected the manual gate valve status indicator to the Acromag box this morning.  Labeled the temporary cable (a 50' 9p DSUB, will order a proper sized cable shortly) and the panel RV2.  

  14420   Tue Jan 29 16:12:21 2019 ChubUpdate  

The foam in the cable tray wall passage had been falling on the floor in little bite-sized pieces, so I investigated and found a fiber cable that had be chewed/clawed through.  I didn't find any droppings anywhere in the 40m, but I decided to bait an un-set trap and see if we'd find activity around it. There has been none so far.  If there is still none tomorrow, I will move the trap and keep looking for signs of rodentia.  At the moment, the trap is in a box in front of the double doors at the north end of the control room.  Next it will be place in the IFO room, up in the cable tray. 

gautam: the fiber that was damaged was the one from the LSC rack FiBox to the control room FiBox. So no DAFI action for a bit...

  14437   Wed Feb 6 10:07:23 2019 ChubUpdate pre-construction inspection

The Central Plant building will be undergoing seismic upgrades in the near future.  The adjoining north wall along the Y arm will be the first to have this work done, from inside the Central Plant.  Project manager Eugene Kim has explained the work to me and also noted our concerns.  He assured me that the seismic noise from the construction will be minimized and we will always be contacted when the heaviest construction is to be done.

Tomorrow at 11am, I will bring Mr. Kim and a few others from the construction team to look at the wall from inside the lab.  If you have any questions or concerns that you want to have addressed, please email them to me or contact Mr. Kim directly at x4860 or through email at eugene.kim@caltech.edu . 

  14572   Thu Apr 25 10:13:15 2019 ChubUpdateGeneralAir Handler Out of Commission

The air handler on the roof of the 40M that supplies the electronics shop and computer room is out of operation until next week.  Adding insult to injury, there is a strong odor of Liquid Wrench oil (a creeping oil for loosening stuck bolts that has a solvent additive) in the building.  If you don't truly need to be in the 40M, you may want to wait until the environment is back to being cool and "unscented".  On a positive note, we should have a quieter environment soon!

ELOG V3.1.3-