Cavity started out aligned pretty well, but not 100%.  Transmission was ~0.8 . Perhaps this was part of the problem.

I realize now that you mention it, it was totally amateur hour of me to only look at the lockin outputs on StripTool (plus POY and TRY on Dataviewer), and not look at TRY on DTT...or any spectra at all.  Not so intelligent.  I could see some fluctuation of TRY on Dataviewer that corresponded to me turning on the oscillators, as well as the spot wiggling on the camera view of ETMYT a teeny bit.

When applying a 10% misalignment to ETMY Pit (by adding 0.1 to the Pit components of the output matrix, as is done in the MC spot position calibration), I could see that there was a small jump in the StripTool trace, but it was much smaller than the ambient fluctuations of the output. 

I just looked back and realized that I must have forgotten to add my screenshot, but it's saved on a desktop on Rossa.  It would be better if I had attached the data, but from that you see that the average of the lockin output signal didn't change very much in the last several minutes of the measurement, but the fluctuations (no misalignment offsets) are pretty big, maybe ~10% or 15% the size of the signal.  Then when I added the misalignment to one mirror (ETMY PIT), there is a very small jump in the lockin signal, but it is much, much smaller than the size of the ambient fluctuations.  Perhaps a long average would result in a "real" value, but by looking by eye, I can't see a discernible difference in the average value of the lockin outputs.

My plan is to do as you say, dithering all 4 optics, and misaligning a single optic's single DoF (Pit or Yaw), and seeing how that misalignment affected each of the sensors (the lockin outputs).  Then put that DoF back to nominal, and misalign a different DoF, rinse and repeat.

Okay, so this is a little stream-of-consciousness-y, and you're going to think I'm really dumb, but I just realized that I haven't set the phase of the lockin demodulators yet.  So I think I need to dither the optics, and go through each of the sensors, and adjust the phase until the peak in TRY in DTT is maximized for the I phase, and minimized for the Q phase (since we use the I-output).  Bah.  Bad Jenne.

Today I spent time locking the YARM in order to calibrate the arm cavity. Here's what I did:

1. Misalign all optics other than the beam splitter, ITMY, ETMY and PZT2

2. Restore BS, ITMY, ETMY, and PZT2

3. Open Dataviewer and run /users/Templates/JenneLockingDataviewer/Yarm.xml from the Restore Settings. This opens the signals C1:LSC-POY11_I_ERR (the Pound-Drever-Hall error signal for this measurement) and C1:LSC-TRY_OUT (the light transmitted through ETMY) in the plot window.

4. Adjust ITMY and ETMY pitch and yaw using the video screens looking at AS and ETMYT as a first, rough guide. It can be helpful at first to increase the gain on the YARM servo filter module in the C1LSC control screen to about 0.3 and decrease it back down to 0.1 as the beam becomes better aligned. You know when to decrease this gain when fuzzy, small oscillations appear on the C1:LSC-TRY_OUT signal. If the mode cleaner is locked you should see a bright spot on the AS camera.

5. Tinker with pitch and yaw while looking at the AS screen until you see a reasonably good circular spot without other fringes extending from a bright center.

6. The overall goal is to maximize C1:LSC-TRY_OUT because the power transmitted through EMTY is proportional to the power within the cavity. A decent target value is 0.85 and today I was able to get it to just over 0.80 at best. At first there will probably be small spikes in C1:LSC-TRY_OUT. You want to adjust pitch and yaw until the deviation in the signal from zero is no longer just a spike, but a sustained, flat signal above zero. By this time there should be light showing up on the ETMYT camera as well.

7. Once that happens, continue to successively adjust ITMY and ETMY doing the pitch adjustments on both first, and then the yaw adjustments, or vice versa. You can also tweak the PZT2 pitch and yaw. Once you've got C1:LSC-TRY_OUT as large as possible, you've locked the cavity.

I saved the pitch and yaw settings I ended up with for ITMY, ETMY, BS and PZT2 in the IFO_ALIGN screen. Before the end of the day I think Jenne restored the rest of the previously misaligned optics because they were restored when I got back from dinner.

I also worked on calibrating the YARM. I opened up DTT using C1:LSC-POY11_I_ERR as the measurement channel and C1:SUS-ITMY_LSC_EXC as the excitation channel. I ran a logarithmic swept sine response measurement with 100 points and an amplitude of 25. The mode cleaner kept losing its lock all day, and if this happened while making this measurement I tried to pause the sweep as quickly as possible. I analyzed the the transfer function and the coherence function that the sweep produced, and thought that some of the odd behavior was due to losing the lock and getting back to a slightly different locked state when resuming the measurement. The measured transfer function and coherence plots are attached below. Both the transfer function and the coherence look good above roughly 30 Hz, but do not look correct at low frequencies. There's also a roll-off in the measured transfer function around 200 Hz, while in the model the magnitude of the transfer function drops only after the corner frequency of the cavity, around several kHz. I have attached a plot of the roughly analogous transfer function from the DARM control loop model (the gains are very large due to the large arm cavity gain and the ADC conversion factor of 2^16/(20 V) ). The measured and the modeled transfer functions are slightly different in that the model does not include the individual mirrors, while the excitation was imposed on ITMY for the measurement.

The next steps are to figure out what's happening in DTT with the transfer function and coherence at low frequencies, and to understand the differences between the model and the measurement.

 We found a PDA255 but it doesn't seem to work. I am not sure if that is one you are mentioning...but I'll ask Steve tomorrow!

  The PDA255 is a good ringdown detector - Steve can find one in the 40m if you ask him nicely.

 To ease the pain of hunting files, optical layout ACAD files have been moved to a new directory 40M_Optical Layout in the repository. Relevant files from directories Upgrade12 and upgrade 08 will be moved to "40M_Optical Layout" very soon and eventually these old directories will be removed. 

From the log, I couldn't understand what has been done.

The procedure we should perform is

1. Dither total 4 dofs of the two mirrors with different frequencies. Some fluctuation of TRY is even visible in dataviewer.
2. The cavity is aligned at the beginning. These 4 peaks in TRY in DTT is small or invisible. Some 2nd harmonics are visible.
3. Misalign one of the dofs. Some peaks get bigger.
4. Correspoding lockin output becomes bigger.

Then you can start measuring the sensing matrix. At which part did the attempt fail?

I turned on the ASS, without closing the loops, to try to measure the sensing matrix. 

The Yarm was locked (Eric did a nice job earlier - he'll ELOG ABOUT IT before he goes home!), and I used an LO CLKGAIN of 300 on all of the TRY Lockins.  Then I put on and took away a 10% offset in pitch, but it's almost impossible to see the difference. 

The attached is a truly awful screenshot, but you can kind of see what's going on.  The big steps are me increasing the LO gain, but around "0" on the x-axis I changed the pitch offset from 10% away to nominal.  Since there are such big oscillations, the change is basically non-existent.  Grrrr. I'll look at it again tomorrow, since I have an exiting bike ride home ahead of me....

Jan and I wanted to measure the ringdown at the IMC. Since the QPD at the MC trans is not fast enough for ringdown measurements, we decided to install a pickoff to include a faster PD while not disturbing much of the current MC trans configuration. The initial configuration had very little space to accommodate the pickoff. So the collimating lens along with the QPD were moved 2 inches closer to the incoming beam. A 50-50 BS was put in front of the QPD and the steering mirror was moved behind to reflect MC trans output to the new PD. The current configuration is shown below with the MC autolocker threshold mentioned in Jenne's elog

The hunt for a faster PD wasn't satisfactory and we found a couple of PDs that were good for measurements actually didn't work after installing them. The one currently installed is also not satisfactorily fast enough for ringdown measurements. We'll hunt for faster PDs at Bridge tomorrow and replace PDA400. Also the IMC unlocked from time to time....may be we were noisy and didn't master the 'interferometer walk' very well.

Jan and Manasa are going to elog about their work later, but it involved putting a BS/window/some kind of pick off in front of the MC Trans QPD, so the total light on the MC Trans QPD is now ~16000 rather than ~26000 counts.  I changed the threshold in the MC autolocker to 5000, so now the MC Trans PD must see at least 5000 counts before the autolocker will engage the boosts, WFS, etc.  Actually, this threshold I believe should have been some several thousand value, but when I went in there, it was set to 500 counts, for low power MC mode during a vent.  It had never gotten put back after the vent to some higher, nominal value.

Hey, the pages got significantly nicer than before. I will continue to give you comments if I find anything.

So far: There are many 10^-100 in logarithmic plots. Once they are removed, we should be able to see the seismic excitation during these recent earth quakes?

Incidentally, where the script is located? "./" isn't the absolute path description.

This week I spent most of my time learning about how the interferometer is calibrated and working on the calibration itself. I also looked more into the Pound-Drever-Hall technique.

Continuing work on the free-swinging Michelson measurements, I changed the signal that I was using to C1:LSC-ASDC_OUT_DQ. This is a proper power signal so that the peak-to-peak amplitude of this error signal can be directly read off the graph. The motivation to measure this amplitude is that it must be known in order to calibrate the actuation of the input and end test masses.

Next I looked into using DTT to make some measurements. I ran the Michelson restore script in the IFO Configure screen to adjust the optics to be near alignment. Then I tweaked the precise settings in the IFO Align screen of pitch and yaw for the ITMX, ITMY, and BS. The goal with this was to minimize the magnitude of the C1:LSC-ASDC_OUT_DQ signal. After it was well-aligned, back in DTT I sent in a sine wave excitation and used a Triggered Time Response measurement to see the output. As a first test I put the excitation signal in the ASDC channel and I was able to plot the resulting OUT signal in DTT. The amplitude was different than I input due to gains between the excitation and the point of measurement, but this can easily be accounted for by adjusting the amplitude in DTT accordingly.

The next step is to work on measurements of a single arm cavity, introducing excitations there and measuring the response.

As Rana pointed out (http://nodus.ligo.caltech.edu:8080/40m/7112), the geophone/accelerometer noise lines from my last eLog (http://nodus.ligo.caltech.edu:8080/40m/7109) were dominated by ADC noise. I checked this today by terminating the ADC channels with 50 Ohm terminators and measuring the noise. The ADC noise line is included on the plot below, and it is clearly dominating the sensor noise data.

I set the accelerometer gain to 100, and will hook up the geophones to the SR560 pre-amp today- this should put both signals above the ADC noise, and I can calculate the sensor noises without the ADC noise being significant.

I have also begun to make some progress in understanding the pre-amp circuitry, and I will post a schematic when I've sketched it all.

Another issue that seems increasingly relevant to me is the power supply to the high voltage amplifier. It appears to go into the high voltage board from the power supply, then into the geophone pre-amp, then back into the high voltage board (see block diagram below). I tested this by inputting a signal after the pre-amp, with the geophones disconnected- the signal only drives the PZT if the pre-amp is plugged in, so the power that returns from the pre-amp must be powering some chips on the high voltage amplifier.

Power flow through the STACIS :

This is somewhat inconvenient, because it means if I want to provide external feedback (with accelerometers, for example) or actuation (such as feedforward), which I want to input after the geophone pre-amp, the pre-amp still needs to be plugged in for the high voltage amplifier to work and drive the PZTs.  I am cataloging all of the pins on the high voltage amplifier and pre-amp so I can figure out how to reroute the power and cut out the geophone pre-amp entirely if necessary. I'll include a pin diagram with the pre-amp circuit sketch.

The main thing that I did this week was write a C block that, given static weights, would classify seismic signals (into one of three categories - Earthquake, Truck, Quiet). I have successfully debugged the C block so that it works without segmentation faults, etc, and have made various versions - one that uses a recurrent neural network, and one that uses a time-delayed input vector (rather than keeping recurrent weights). I've timed my code, and it works very fast (to the point where clock() differences in <time.h> are 0.000000 seconds per iteration). This is good news, because it means that operations can be performed in real-time, whether we are sampling at 2048 Hz, or, as Rana suggested, 256 Hz (currently, my weights are for 256 Hz, and I can decimate the incoming signal which is at 2048 Hz right now).

In order to optimize my code, since at the core it involves a lot of matrix multiplications, I considered how the data is actually stored in the computer, and attempted to minimize pointer movement. Suppose you have an array in C of the form A[num_row][num_col] - the way this array is actually stored on the stack or heap is row_1 / row_2 / row_3 / ... / row_num_row, so it makes sense to move across a matrix from left to right and then down (as though reading on a page). Likewise, there's no efficient algorithm for matrix multiplication which is less that O(N^2) (I think), so it's essentially impossible to avoid double for loops (however, the way I process the matricies, as mentioned before, minimizes this time).

The code is also fast because, rather than using an actual e^-u operation for the sigmoidal activation function, it uses a parametrized hyperbola - this arithmetic operations are the only ones that occur, and this is much faster than exponentiation (which I believe is just computer by Taylor series in the C math library..)

The weight vectors for this block are static (since they're made from training data where the signal type is already known). I am not currently satisfied with the performance of the block on data read from a file, so I am retraining my network. I realized that what is currently happening is that, given a time-dependent desired output vector, the network first trains to output a "quiet" vector, then a "disturbance" vector, and then retrains again to output a "quiet vector" and completely forgets how to classify disturbance. Currently, I am trying to get around this problem by shifting my earthquake data time-series, so that when I train in batch (on all of my data), there is probably an earthquake at all time points, so that the network does not only train on "quiet" at certain iterations. Likewise, I realized that I should perform several epochs (not just one) on the data - I tried this last night, and training performance MSE decreased by a factor of 1 per iteration (when on average, it's about 40, and 20 at best).

After I input the static weight vectors (which shouldn't take long since my code is very generalized), the C block can be added to the c1pem frame, and a channel can be made for the seismic disturbance class. I've made sure to keep with all of the C block rules when writing my code (both in terms of function input/output, and in terms of not using any C libraries).

As for neural networks for control, I talked to Denis about the controller block, and he realized that we should, instead of adding noises, at first attempt to use a reference plant with a lower Q pendulum and a real plant with a higher Q pendulum (since we want pendulum motion to be damped). I've tried training the controller block several times, but each time so far the plant pendulum has started oscillating greatly. My current guess at fixing this is training more.

Also, Jenne and I made a cable for Guralp 1 (I soldered, she explained how to do it), and it seems to work well, as mentioned in my previous E-log. Hopefully it can be used to permanently keep the seismometer all the way down the arm.

This week, I brought my c1lsp model online and fixed up some the medm screens for c1spx. Along the way, I ran into a few interesting problems (and learned a bit about bash scripting and emacs! :D). The screens for the main TM_RESP matrix are not generating automatically, and the medm screens don't want to link to the channels, for some reason. I don't have this problem with the other matrices (i.e. C2DOF and SEN_OUT), so I think it has something to do with TM_RESP being a filter matrix (which the others are not). In addition, the noise overview medm screens for c1spx are practically nonexistent - someone just copied the file for the SUS-ETMX screens into the master directory for c1spx, so they need a complete overhaul. I am willing to do this, but Jamie told me to focus my attentions elsewhere.

So I went back to noise generation. I've been using Matlab to figure out how to recreate the various noise sources (laser amplitude noise, local oscillator phase/amplitude noise, and 60 Hz/ADC. Frequency noise will be added some time this week and seismic noise should be already covered in Jamie's suspension model) in my c1lsp model. I'm doing it the way the RCG does it - by applying a filter to white noise. I'm generating white noise by just using a random number generator and pwelch-ing it to get the power spectral density.

For the filters themselves, I picked z, p, k such that it shaped the white noise PSD to look like the PSD of the noise in question. This was fairly straightforward once I figured out how zeroes and poles affected PSD. Once I'd picked zpk, I applied a bilinear transform to get a discrete zpk out, then converted to a second order section to make computation faster. I then applied that to the white noise (matlab has a convenient "sosfilt" function) and pwelch-ed/graphed it to get the result.

Attached is my attempt at filtering white noise to look like 60 Hz. noise. I can't seem to find a way to pick z and p such that the peak is more narrow (i.e. other than having two complex conjugated poles at 60 Hz.). I took the spectrum of 60 Hz. noise from a terminated ADC channel (Masha kindly let me borrow one of her GURALP channels).

EDIT: I also remembered that I've been looking for how to get a good power spectrum for the rest of the noises. Jenne referred me to Kiwamu's work on this, and I'm mostly going off elog #6133. If you have any other good elogs/data on noises, please feel free to send them my way.

I then measured the PSD of the sensors on the real suspended optics and a PSD of the suspension model. It looks like the OSEMs on the suspension model are only reading white noise, which probably means a lost connection somewhere (Attachment 2 is what the model SHOULD produce, Attachment 3 is what the model ACTUALLY produces). I perused Jamie's model, but couldn't find anything. Not sure where else to check, but I'll continue thinking about it/trying to fix it.

Over the past week, I have been working on my progress report and finalizing the summary pages.  I have a few more things to address in the pages (such as starting at 6 AM, including spectrograms where necessary and generating plots for the days more than ~a week ago) but they are mostly finalized.  I added all of the existing acoustic and seismic channels so the PEM page is up to date.  The microphone plots include information about the transfer factor that I found on their information sheet (http://www.primomic.com/).  If there are any plots that are missing or need editing, please let me know!

I also modified the c1_summary_page.sh script to run either the daily plots or current updating plots by taking in an argument in the command line.  It can be run ./c1_summary_page.sh 2012/07/27
 or ./c1_summary_page.sh now to generate the current day's pages.  (Essentially, I combined the two scripts I had been running separately.)  I have been commenting my code so it is more easily understandable and have been working on writing a file that explains how to run the code and the main alterations I made.  The most exciting thing that has taken place this week is that  the script went from taking ~6 hours to run to taking less than 5 minutes.  This was done by using minute trends for all of the channels and limiting the spectrum plot data.

The summary pages for each day now contain only the most essential plots that give a good overview of the state of the interferometer and its environment instead of every plot that is created for that day.

I am waiting for Duncan to send me some spectrogram updates he has made that downsample the timeseries data before plotting the spectrogram.  This will make it run much more quickly and introduce a more viable spectrogram option.

Today's Summary Pages can be accessed by the link on the wiki page or at:

https://nodus.ligo.caltech.edu:30889/40m-summary/archive_daily/20120808/

There were another couple of earthquakes at about 9:30am and 9:50am local.

All but MC2 were off the watchdogs.  I damped and realigned everything and everything looks ok now.

[Sasha, Masha, Liz, Eric]

A bunch of surfs in the lab just noticed that ETMX is going crazy (laser is shifting everywhere) due to a 4.5 EQ that just hit LA. The optic is already shut down according to the watchdogs.

Looks like you're just measuring the ADC noise. You should add ADC noise to your plot. To compare the geophones with the accelerometers, you have to correct for the preamp gain and plot them both in the same units.

To get above the ADC noise you can use an SR560 preamp. (AC Coupled, G = 100)

 All optics except MC2 and ETMX are crazy

 Just felt an EQ. Impulse moved some vertical blinds by several mm.

Tue Aug 07 23:26:06 2012 

Yesterday I plugged the geophone and accelerometer output into the ADC, rather than the SR785, so I could collect for longer and take more data at once.

As per Rana's suggestion, I am also now taking the geophone output after the first op-amp in the circuitry following the geophone (a low-noise op-amp, OPA227). It acts as a buffer so I'm not just measuring other local noise sources (which explains why the geophone noise curve sort of matched the SR785 noise curve in my old plots).

With these changes, I remeasured the accelerometer and geophone noises as well as collected an ASD of a geophone sitting on the STACIS in open loop operation. I also looked up the noise specs for the various op-amps in the geophone pre-amp and high voltage board; everything I found, I added in quadrature to come up with an approximate op-amp noise value for the STACIS. All of this is plotted below:

I left the y-axis in V/rtHz instead of converting it to m/s/rtHz so that the op-amp noise could be compared to the other noises. All sensor data was taken with the sensors horizontal (noise data taken in granite and foam).

The accelerometer and geophone noise still appear to be similar, and the op-amp noise, at least according to specs, is low compared to the other noises. This implies there's not much to gain from switching the geophones with accelerometers nor with swapping out the op-amps for lower-noise components (unless the ones I couldn't find specs for were high-noise, though it seems like mainly low-noise components were used). 

Please check the summary pages out at the link below and let me know if there are any modifications I should make!  All existing pages are up to date and contain all of the pages I have.

Questions, comments, and suggestions will be appreciated! Contact me at endavison@umail.ucsb.edu

https://nodus.ligo.caltech.edu:30889/40m-summary/

 Good. The power density and max power are not important (especially since you don't define a quantitative way to spec them). We ONLY care about the transmission.

 Sanwiched wall as shown: 1" clear acrylic, 2 layers of 0.004" thick "window tint", 1 layer of  0.007" thermashield and  0.125" yellow acrylic

Visibility: 70 %,    Transmission of 1064 nm  2-3 % at 0-50 degrees incident  power density 0.7 W/mm2

Max power 100 mW

Apparently the last problem was because of root-owned frame directories that daqd was trying to write to.  During debugging Alex had run daqd as root, but it's supposed to run as controls.  All the /frame directories are supposed to be owned by controls.  When daqd was run as root, it created new frame directories owned by root, which controls couldn't write to when I restarted daqd the proper way.  Once we chown'd the directories daqd started running again.

Alex also put in a "fix" for the core dump problem.  He touched an empty core file owned by root:

-rw-r--r-- 1 root root 0 Aug  7 14:38 /opt/rtcds/caltech/c1/target/fb/core

This will prevent any dying daqd process owned by controls from dumping it's core at that location.  Personally I think this is a horribly hacky "solution" that doesn't actually fix any of the issues that were causing the segfaults to begin with, but it might prevent some of the network slow down we see when the core does dump.  It's mostly just masking the problem, though, so I'm tempted to remove it so we all feel the pain when daqd starts shitting all over the network again.

Previously, medmrun didn't accept arguments to pass along to the script it was going to run.  Jamie has graciously taken a moment from fixing the computer disaster to help me update the medmrun script.

Now the ASS scripts are call-able from the screen.

I spoke too soon.  It's still segfaulting, but at a different place. Alex and I are looking into it.

But another mystery solved is the cause of all the network slowness: the daqd core dump.  When daqd segfaults it dumps it's core, which can typically be >4G, to /opt/rtcds/caltech/c1/target/fb/core.  This is of course an NFS mount from linux1, so it's dumping 4G on the network, which not surprisingly clogs the network.

So daqd's problem was apparently the bad/non-running c1sup model.  The c1sup model, which I reported on attempting to get running in 7097, was not running because there were no available CPUs on the c1sus FE machine.  This was due to my stupid undercounting of the number of CPUs.  Anyway, for reasons I don't understand, this was causing daqd to segfault.  Removing c1sup from c1sus "fixed" the problem.

Alex agreed that daqd should definitely not be segfaulting in this circumstance.  It's still unclear exactly what daqd was looking at that was causing it to crash.

I'm going to move c1sup to c1iscex, which has a lot of spare CPUs.

Alex is apparently working on daqd (remotely).  I'll report back when I find out more.

I wrote new setup, on and off scripts for the arm ass.  They take the arm as an argument, so it's the same script for both arms.  Scripts are in ...../scripts/ASS/ , and have been checked in to the 40m svn.

So far the on script doesn't really do anything, since I haven't chosen values for the CLKGAINs of the lockins.  The old values were 30 for lockins 12, 14, 27, 29 and 250 for lockins 7, 9, 22, 24.  Unfortunately, I have no memory of which lockin means what in the old numbered system.  I'll have to look that up somehow.  Or, just dither the optics using some value and look at the spectrum to see the resulting SNR and just pick something that gives me reasonable SNR.

I modified the ASS model slightly:

* Added an overall gain to the ASS_DOF2 library part, between the matrix and the servo inputs so we can do soft startups.  Self - remember that the main ASS screen needs to be modified to reflect this!

* Rearranged the order that the demodulated signals go into the matrix.  I hadn't paid attention, and the old ordering had the transmission (TRX/TRY) demod signals interleaved with the LSC demod signals.  I've changed it to be all the TR signals, then all the LSC signals.  I think this makes more sense, since we will use these inputs separately, so now they're on different halves of the matrix.

When the network / fb went bad this afternoon, I had been working on the ASS model, shortening the names of the filter banks to fix the problem from elog 7092.  I wanted to finish working on that, so the ASS model is now rebuilt with slightly shorter names in the filterbanks (which fixes the problem of the filter banks not working).

------------------------------------------------------------

I mentioned this to Jamie the other day, but here's the error that you get when the GoTo/From tags aren't working:

>>rtcds make c1ass
### building c1ass...
Cleaning c1ass...
Done
Parsing the model c1ass...
IPC 9 8 is C1:LSC-ASS_LSC
IPC 9 8 is ISHME
IPC 10 9 is C1:RFM-LSC_TRX
IPC 10 9 is IPCIE
IPC 11 10 is C1:RFM-LSC_TRY
IPC 11 10 is IPCIE

INPUT XARM_LSC_in is NOT connected

INPUT YARM_LSC_in is NOT connected

***ERROR: Found total of ** 2 ** INPUT parts not connected

make[1]: *** [c1ass] Error 255
make: *** [c1ass] Error 1

I don't know why these tags weren't working, but there was a GoTo tag on the output of the LSC shmem block, and then Froms on each of the XARM_LSC_in and YARM_LSC_in.  The other day I played around with a bunch of different things (grounding inputs, terminating outputs, whatever), but finally replacing the tags with identical ones freshly taken from CDS_PARTS made it happy. 

The PRM was pointing totally the wrong way, so there was no light on the oplev PD.  I restored the PRM, turned the gains back to (0.15, -0.3) as per Yuta's elog 6952, and it seems just fine to me.

I want to check the data from last night / the weekend to see when the mispointing happened, but dataviewer can't connect to the fb, since Jamie is still working his magic.  I'm pretty sure I restored all of the optics after Eric finished playing with MICH Friday night, but it's possible that I forgot one, I suppose.  If it wasn't me, then I'm curious when it happened.

I'm trying to get more of the simplant models running so that we (me and Sasha Surf) can get a full real-time cavity simplant running.  As I reported last week, c1spx is running again on c1iscex.

The two new simplant models are c1sup, which holds the simplant for ITMX, and c1lsp, which holds the IFO simplant, specifically the one we're working on for XARM.

Here's the relevant info:

model  host     dcuid  cpu
c1spx  c1iscex  61     4
c1sup  c1sus    62     6
c1lsp  c1lsc    60     6

c1spx and c1sup will be running the sus_single_plant parts for ETMX and ITMX simplant.  All the simplant suspension channels will be names "SUP" (as opposed to "SUS" for control).

c1lsp is now running, but c1sup won't run for unknown reasons.  The c1sup model is not very complicated, and in fact is more-or-less identical to c1spx.  It compiles and installs and even loads, but it completely unresponsive after loading.  Unfortunately I've had enough CDS bullshit for today, so I'll try to figure out what's going on tomorrow.

I tried running daqd manually, and sure enough it segfaults after about five minutes (see log below).  I've uncommented it from /etc/inittab on fb and I'm leaving it off for now until we can figure out what's going on.

controls@fb /opt/rtcds/caltech/c1/target/fb 0$ /opt/rtcds/caltech/c1/target/fb/daqd -c /opt/rtcds/caltech/c1/target/fb/daqdrc
263596
MX endpoint opened
startup file interpreter thread tid=139790943115536
calling yyparse(5, 6)
[Mon Aug  6 20:15:27 2012] ->5: #set avoid_reconnect
[Mon Aug  6 20:15:27 2012] ->5: set thread_stack_size=102400
[Mon Aug  6 20:15:27 2012] new threads will be created with the stack of size 102400K
[Mon Aug  6 20:15:27 2012] ->5: set allow_tpman_connect_fail
[Mon Aug  6 20:15:27 2012] ->5: #set dcu_status_check=5
[Mon Aug  6 20:15:27 2012] ->5: #set symm_gps_offset=-1
[Mon Aug  6 20:15:27 2012] ->5: #set symm_gps_offset=31535998
[Mon Aug  6 20:15:27 2012] ->5: ##set symm_gps_offset=347155213
[Mon Aug  6 20:15:27 2012] ->5: #set symm_gps_offset=378691215
[Mon Aug  6 20:15:27 2012] ->5: #set symm_gps_offset=378691212
[Mon Aug  6 20:15:27 2012] ->5: #set symm_gps_offset=315964799
[Mon Aug  6 20:15:27 2012] ->5: set symm_gps_offset=315964801
[Mon Aug  6 20:15:27 2012] ->5: set debug=0
[Mon Aug  6 20:15:27 2012] ->5: set log=2
[Mon Aug  6 20:15:27 2012] ->5: set zero_bad_data=0
[Mon Aug  6 20:15:27 2012] ->5: set dcu_status_check=9
[Mon Aug  6 20:15:27 2012] ->5: set controller_dcu=33
[Mon Aug  6 20:15:27 2012] ->5: set master_config="/opt/rtcds/caltech/c1/target/fb/master"
[Mon Aug  6 20:15:30 2012] finished configuring data channels
[Mon Aug  6 20:15:30 2012] ->5: configure channels begin end
GDS server NODE=19 HOST=c1iscex DCUID=19
GDS server NODE=20 HOST=c1sus DCUID=20
GDS server NODE=21 HOST=c1sus DCUID=21
GDS server NODE=22 HOST=c1lsc DCUID=22
GDS server NODE=25 HOST=c1iscex DCUID=61
GDS server NODE=28 HOST=c1ioo DCUID=28
GDS server NODE=33 HOST=c1ioo DCUID=33
GDS server NODE=34 HOST=c1ioo DCUID=34
GDS server NODE=36 HOST=c1sus DCUID=36
GDS server NODE=38 HOST=c1sus DCUID=38
GDS server NODE=39 HOST=c1sus DCUID=39
GDS server NODE=40 HOST=c1lsc DCUID=40
GDS server NODE=42 HOST=c1lsc DCUID=42
GDS server NODE=45 HOST=c1iscex DCUID=45
GDS server NODE=46 HOST=c1iscey DCUID=46
GDS server NODE=47 HOST=c1iscey DCUID=47
GDS server NODE=48 HOST=c1lsc DCUID=48
GDS server NODE=50 HOST=c1lsc DCUID=50
GDS server NODE=51 HOST=c1ioo DCUID=51
GDS server NODE=60 HOST=c1lsc DCUID=60
GDS server NODE=61 HOST=c1iscex DCUID=61
GDS server NODE=62 HOST=c1sus DCUID=62
Unable to find GDS node 90 system c1x00 in INI files
GDS server NODE=91 HOST=c1lsc DCUID=60
Unable to find GDS node 92 system c1tst2 in INI files
Unable to find GDS node 95 system c1x10 in INI files
TP: node = 19, host = c1iscex, dup = 0, prog = 0x31002013, vers = 1
Initialized TP interface node=19, host=c1iscex
TP: node = 20, host = c1sus, dup = 0, prog = 0x31002014, vers = 1
Initialized TP interface node=20, host=c1sus
TP: node = 21, host = c1sus, dup = 0, prog = 0x31002015, vers = 1
Initialized TP interface node=21, host=c1sus
TP: node = 22, host = c1lsc, dup = 0, prog = 0x31002016, vers = 1
Initialized TP interface node=22, host=c1lsc
TP: node = 25, host = c1iscex, dup = 0, prog = 0x31002019, vers = 1
Initialized TP interface node=25, host=c1iscex
TP: node = 28, host = c1ioo, dup = 0, prog = 0x3100201c, vers = 1
Initialized TP interface node=28, host=c1ioo
TP: node = 33, host = c1ioo, dup = 0, prog = 0x31002021, vers = 1
Initialized TP interface node=33, host=c1ioo
TP: node = 34, host = c1ioo, dup = 0, prog = 0x31002022, vers = 1
Initialized TP interface node=34, host=c1ioo
TP: node = 36, host = c1sus, dup = 0, prog = 0x31002024, vers = 1
Initialized TP interface node=36, host=c1sus
TP: node = 38, host = c1sus, dup = 0, prog = 0x31002026, vers = 1
Initialized TP interface node=38, host=c1sus
TP: node = 39, host = c1sus, dup = 0, prog = 0x31002027, vers = 1
Initialized TP interface node=39, host=c1sus
TP: node = 40, host = c1lsc, dup = 0, prog = 0x31002028, vers = 1
Initialized TP interface node=40, host=c1lsc
TP: node = 42, host = c1lsc, dup = 0, prog = 0x3100202a, vers = 1
Initialized TP interface node=42, host=c1lsc
TP: node = 45, host = c1iscex, dup = 0, prog = 0x3100202d, vers = 1
Initialized TP interface node=45, host=c1iscex
TP: node = 46, host = c1iscey, dup = 0, prog = 0x3100202e, vers = 1
Initialized TP interface node=46, host=c1iscey
TP: node = 47, host = c1iscey, dup = 0, prog = 0x3100202f, vers = 1
Initialized TP interface node=47, host=c1iscey
TP: node = 48, host = c1lsc, dup = 0, prog = 0x31002030, vers = 1
Initialized TP interface node=48, host=c1lsc
TP: node = 50, host = c1lsc, dup = 0, prog = 0x31002032, vers = 1
Initialized TP interface node=50, host=c1lsc
TP: node = 51, host = c1ioo, dup = 0, prog = 0x31002033, vers = 1
Initialized TP interface node=51, host=c1ioo
TP: node = 60, host = c1lsc, dup = 0, prog = 0x3100203c, vers = 1
Initialized TP interface node=60, host=c1lsc
TP: node = 61, host = c1iscex, dup = 0, prog = 0x3100203d, vers = 1
Initialized TP interface node=61, host=c1iscex
TP: node = 62, host = c1sus, dup = 0, prog = 0x3100203e, vers = 1
Initialized TP interface node=62, host=c1sus
TP: node = 91, host = c1lsc, dup = 0, prog = 0x3100205b, vers = 1
Initialized TP interface node=91, host=c1lsc
[Mon Aug  6 20:15:30 2012] ->5: tpconfig "/opt/rtcds/caltech/c1/target/gds/param/testpoint.par"
[Mon Aug  6 20:15:30 2012] ->5: set gps_leaps = 820108813
[Mon Aug  6 20:15:30 2012] ->5: set detector_name="CIT"
[Mon Aug  6 20:15:30 2012] ->5: set detector_prefix="C1"
[Mon Aug  6 20:15:30 2012] ->5: set detector_longitude=-90.7742403889
[Mon Aug  6 20:15:30 2012] ->5: set detector_latitude=30.5628943337
[Mon Aug  6 20:15:30 2012] ->5: set detector_elevation=.0
[Mon Aug  6 20:15:30 2012] ->5: set detector_azimuths=1.1,4.7123889804
[Mon Aug  6 20:15:30 2012] ->5: set detector_altitudes=1.0,2.0
[Mon Aug  6 20:15:30 2012] ->5: set detector_midpoints=2000.0, 2000.0
[Mon Aug  6 20:15:30 2012] ->5: set num_dirs = 10
[Mon Aug  6 20:15:30 2012] ->5: set frames_per_dir=225
[Mon Aug  6 20:15:30 2012] ->5: set full_frames_per_file=1
[Mon Aug  6 20:15:30 2012] ->5: set full_frames_blocks_per_frame=16
[Mon Aug  6 20:15:30 2012] ->5: set frame_dir="/frames/full", "C-R-", ".gwf"
[Mon Aug  6 20:15:30 2012] ->5: set trend_num_dirs=10
[Mon Aug  6 20:15:30 2012] ->5: set trend_frames_per_dir=1440
[Mon Aug  6 20:15:30 2012] ->5: set trend_frame_dir= "/frames/trend/second", "C-T-", ".gwf"
[Mon Aug  6 20:15:30 2012] ->5: set raw-minute-trend-dir="/frames/trend/minute_raw"
[Mon Aug  6 20:15:30 2012] ->5: set nds-jobs-dir="/opt/rtcds/caltech/c1/target/fb"
[Mon Aug  6 20:15:30 2012] ->5: set minute-trend-num-dirs=10
[Mon Aug  6 20:15:30 2012] ->5: set minute-trend-frames-per-dir=24
[Mon Aug  6 20:15:30 2012] ->5: set minute-trend-frame-dir="/frames/trend/minute", "C-M-", ".gwf"
[Mon Aug  6 20:15:30 2012] ->5: start main 10
Allocated move buffer size 11616356 bytes
[Mon Aug  6 20:15:32 2012] main started
[Mon Aug  6 20:15:32 2012] ->5: start profiler
[Mon Aug  6 20:15:32 2012] ->5: # comment out this block to stop saving data
[Mon Aug  6 20:15:32 2012] frame saver started
[Mon Aug  6 20:15:32 2012] ->5: start frame-saver
[Mon Aug  6 20:15:33 2012] ->5: sync frame-saver
[Mon Aug  6 20:15:33 2012] ->5: start trender
[Mon Aug  6 20:15:33 2012] trender started
[Mon Aug  6 20:15:33 2012] trend frame saver started
[Mon Aug  6 20:15:33 2012] ->5: start trend-frame-saver
[Mon Aug  6 20:15:34 2012] ->5: sync trend-frame-saver
[Mon Aug  6 20:15:34 2012] minute trend frame saver started
[Mon Aug  6 20:15:34 2012] ->5: start minute-trend-frame-saver
[Mon Aug  6 20:15:34 2012] Done creating ADC structures
[Mon Aug  6 20:15:35 2012] ->5: sync minute-trend-frame-saver
[Mon Aug  6 20:15:35 2012] raw minute trend frame saver started
[Mon Aug  6 20:15:35 2012] ->5: start raw_minute_trend_saver
[Mon Aug  6 20:15:35 2012] ->5: #frame-writer "225.225.225.1" broadcast="131.215.113.0" all
[Mon Aug  6 20:15:35 2012] ->5: #sleep 5
[Mon Aug  6 20:15:35 2012] producer started
[Mon Aug  6 20:15:35 2012] ->5: start producer
[Mon Aug  6 20:15:35 2012] ->5: start epics dcu
[Mon Aug  6 20:15:35 2012] MX receiver thread started
[Mon Aug  6 20:15:35 2012] edcu started
[Mon Aug  6 20:15:35 2012] ->5: start epics server "C0:DAQ-DC0_" "C1:DAQ-DC0_"
[Mon Aug  6 20:15:35 2012] epics server started
[Mon Aug  6 20:15:35 2012] ->5: start listener 8087
[Mon Aug  6 20:15:35 2012] ->5: start listener 8088 1
[Mon Aug  6 20:15:35 2012] ->5: sleep 60
Creating C1:DAQ-DC0_PEM_SLOW_STATUS
Creating C1:DAQ-DC0_PEM_SLOW_CRC_CPS
Creating C1:DAQ-DC0_PEM_SLOW_CRC_SUM
Creating C1:DAQ-DC0_C1X01_STATUS
Creating C1:DAQ-DC0_C1X01_CRC_CPS
Creating C1:DAQ-DC0_C1X01_CRC_SUM
Creating C1:DAQ-DC0_C1X02_STATUS
Creating C1:DAQ-DC0_C1X02_CRC_CPS
Creating C1:DAQ-DC0_C1X02_CRC_SUM
Creating C1:DAQ-DC0_C1SUS_STATUS
Creating C1:DAQ-DC0_C1SUS_CRC_CPS
Creating C1:DAQ-DC0_C1SUS_CRC_SUM
Creating C1:DAQ-DC0_C1OAF_STATUS
Creating C1:DAQ-DC0_C1OAF_CRC_CPS
Creating C1:DAQ-DC0_C1OAF_CRC_SUM
Creating C1:DAQ-DC0_C1ALS_STATUS
Creating C1:DAQ-DC0_C1ALS_CRC_CPS
Creating C1:DAQ-DC0_C1ALS_CRC_SUM
Creating C1:DAQ-DC0_C1X03_STATUS
Creating C1:DAQ-DC0_C1X03_CRC_CPS
Creating C1:DAQ-DC0_C1X03_CRC_SUM
Creating C1:DAQ-DC0_C1IOO_STATUS
Creating C1:DAQ-DC0_C1IOO_CRC_CPS
Creating C1:DAQ-DC0_C1IOO_CRC_SUM
Creating C1:DAQ-DC0_C1MCS_STATUS
Creating C1:DAQ-DC0_C1MCS_CRC_CPS
Creating C1:DAQ-DC0_C1MCS_CRC_SUM
Creating C1:DAQ-DC0_C1RFM_STATUS
Creating C1:DAQ-DC0_C1RFM_CRC_CPS
Creating C1:DAQ-DC0_C1RFM_CRC_SUM
Creating C1:DAQ-DC0_C1PEM_STATUS
Creating C1:DAQ-DC0_C1PEM_CRC_CPS
Creating C1:DAQ-DC0_C1PEM_CRC_SUM
Creating C1:DAQ-DC0_C1X04_STATUS
Creating C1:DAQ-DC0_C1X04_CRC_CPS
Creating C1:DAQ-DC0_C1X04_CRC_SUM
Creating C1:DAQ-DC0_C1LSC_STATUS
Creating C1:DAQ-DC0_C1LSC_CRC_CPS
Creating C1:DAQ-DC0_C1LSC_CRC_SUM
Creating C1:DAQ-DC0_C1SCX_STATUS
Creating C1:DAQ-DC0_C1SCX_CRC_CPS
Creating C1:DAQ-DC0_C1SCX_CRC_SUM
Creating C1:DAQ-DC0_C1X05_STATUS
Creating C1:DAQ-DC0_C1X05_CRC_CPS
Creating C1:DAQ-DC0_C1X05_CRC_SUM
Creating C1:DAQ-DC0_C1SCY_STATUS
Creating C1:DAQ-DC0_C1SCY_CRC_CPS
Creating C1:DAQ-DC0_C1SCY_CRC_SUM
Creating C1:DAQ-DC0_C1ASS_STATUS
Creating C1:DAQ-DC0_C1ASS_CRC_CPS
Creating C1:DAQ-DC0_C1ASS_CRC_SUM
Creating C1:DAQ-DC0_C1CAL_STATUS
Creating C1:DAQ-DC0_C1CAL_CRC_CPS
Creating C1:DAQ-DC0_C1CAL_CRC_SUM
Creating C1:DAQ-DC0_C1MCC_STATUS
Creating C1:DAQ-DC0_C1MCC_CRC_CPS
Creating C1:DAQ-DC0_C1MCC_CRC_SUM
Creating C1:DAQ-DC0_C1MCP_STATUS
Creating C1:DAQ-DC0_C1MCP_CRC_CPS
Creating C1:DAQ-DC0_C1MCP_CRC_SUM
Creating C1:DAQ-DC0_C1LSP_STATUS
Creating C1:DAQ-DC0_C1LSP_CRC_CPS
Creating C1:DAQ-DC0_C1LSP_CRC_SUM
Creating C1:DAQ-DC0_C1SPX_STATUS
Creating C1:DAQ-DC0_C1SPX_CRC_CPS
Creating C1:DAQ-DC0_C1SPX_CRC_SUM
Creating C1:DAQ-DC0_C1SUP_STATUS
Creating C1:DAQ-DC0_C1SUP_CRC_CPS
Creating C1:DAQ-DC0_C1SUP_CRC_SUM
[Mon Aug  6 20:15:35 2012] Epics server started
[Mon Aug  6 20:15:35 2012] EDCU has 2553 channels configured; first=0

Symmetricom status: LOCKED
Starting at gps 1028344552 prev_gps 1028344552 frac 312500000 f 314094022
[Mon Aug  6 20:15:38 2012] Minute trender made GPS time correction; gps=1028344552; gps%60=52
Segmentation fault (core dumped)

When daqd is caught in this state it can not be killed.  It's in "uninterruptable sleep" ('D' state in the top output below).  This usually indicates that it's waiting for the kernel, usually due to some missing or hung IO.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                      
28038 controls  20   0 4430m 2.0g  19m D    0 27.1   0:15.00 daqd                                                                                         

The memory footprint also seems to be getting big.  It's clearly trying to do something stupid that it can't handle.

It looks like daqd is indeed caught in some bad state.  It seems to die at some point after making GPS corrections to minute trender:

...
[Mon Aug  6 19:45:13 2012] Minute trender made GPS time correction; gps=1028342727; gps%60=27
tail: `fb/logs/daqd.log' has been replaced;  following end of new file
263596
MX endpoint opened
startup file interpreter thread tid=140334118615312
calling yyparse(5, 6)
[Mon Aug  6 19:50:08 2012] ->5: #set avoid_reconnect
[Mon Aug  6 19:50:08 2012] ->5: set thread_stack_size=102400
[Mon Aug  6 19:50:08 2012] new threads will be created with the stack of size 102400K
[Mon Aug  6 19:50:08 2012] ->5: set allow_tpman_connect_fail
[Mon Aug  6 19:50:08 2012] ->5: #set dcu_status_check=5
[Mon Aug  6 19:50:08 2012] ->5: #set symm_gps_offset=-1
[Mon Aug  6 19:50:08 2012] ->5: #set symm_gps_offset=31535998
[Mon Aug  6 19:50:08 2012] ->5: ##set symm_gps_offset=347155213
[Mon Aug  6 19:50:08 2012] ->5: #set symm_gps_offset=378691215
[Mon Aug  6 19:50:08 2012] ->5: #set symm_gps_offset=378691212
[Mon Aug  6 19:50:08 2012] ->5: #set symm_gps_offset=315964799
[Mon Aug  6 19:50:08 2012] ->5: set symm_gps_offset=315964801
[Mon Aug  6 19:50:08 2012] ->5: set debug=0
[Mon Aug  6 19:50:08 2012] ->5: set log=2
[Mon Aug  6 19:50:08 2012] ->5: set zero_bad_data=0
[Mon Aug  6 19:50:08 2012] ->5: set dcu_status_check=9
[Mon Aug  6 19:50:08 2012] ->5: set controller_dcu=33
[Mon Aug  6 19:50:08 2012] ->5: set master_config="/opt/rtcds/caltech/c1/target/fb/master"
[Mon Aug  6 19:50:10 2012] finished configuring data channels
[Mon Aug  6 19:50:10 2012] ->5: configure channels begin end
Unable to find GDS node 90 system c1x00 in INI files
Unable to find GDS node 92 system c1tst2 in INI files
Unable to find GDS node 95 system c1x10 in INI files
[Mon Aug  6 19:50:10 2012] ->5: tpconfig "/opt/rtcds/caltech/c1/target/gds/param/testpoint.par"
[Mon Aug  6 19:50:10 2012] ->5: set gps_leaps = 820108813
[Mon Aug  6 19:50:10 2012] ->5: set detector_name="CIT"
[Mon Aug  6 19:50:10 2012] ->5: set detector_prefix="C1"
[Mon Aug  6 19:50:10 2012] ->5: set detector_longitude=-90.7742403889
[Mon Aug  6 19:50:10 2012] ->5: set detector_latitude=30.5628943337
[Mon Aug  6 19:50:10 2012] ->5: set detector_elevation=.0
[Mon Aug  6 19:50:10 2012] ->5: set detector_azimuths=1.1,4.7123889804
[Mon Aug  6 19:50:10 2012] ->5: set detector_altitudes=1.0,2.0
[Mon Aug  6 19:50:10 2012] ->5: set detector_midpoints=2000.0, 2000.0
[Mon Aug  6 19:50:10 2012] ->5: set num_dirs = 10
[Mon Aug  6 19:50:10 2012] ->5: set frames_per_dir=225
[Mon Aug  6 19:50:10 2012] ->5: set full_frames_per_file=1
[Mon Aug  6 19:50:10 2012] ->5: set full_frames_blocks_per_frame=16
[Mon Aug  6 19:50:10 2012] ->5: set frame_dir="/frames/full", "C-R-", ".gwf"
[Mon Aug  6 19:50:10 2012] ->5: set trend_num_dirs=10
[Mon Aug  6 19:50:10 2012] ->5: set trend_frames_per_dir=1440
[Mon Aug  6 19:50:10 2012] ->5: set trend_frame_dir= "/frames/trend/second", "C-T-", ".gwf"
[Mon Aug  6 19:50:10 2012] ->5: set raw-minute-trend-dir="/frames/trend/minute_raw"
[Mon Aug  6 19:50:10 2012] ->5: set nds-jobs-dir="/opt/rtcds/caltech/c1/target/fb"
[Mon Aug  6 19:50:10 2012] ->5: set minute-trend-num-dirs=10
[Mon Aug  6 19:50:10 2012] ->5: set minute-trend-frames-per-dir=24
[Mon Aug  6 19:50:10 2012] ->5: set minute-trend-frame-dir="/frames/trend/minute", "C-M-", ".gwf"
[Mon Aug  6 19:50:10 2012] ->5: start main 10
[Mon Aug  6 19:50:12 2012] main started
[Mon Aug  6 19:50:12 2012] ->5: start profiler
[Mon Aug  6 19:50:12 2012] ->5: # comment out this block to stop saving data
[Mon Aug  6 19:50:12 2012] frame saver started
[Mon Aug  6 19:50:12 2012] ->5: start frame-saver
[Mon Aug  6 19:50:13 2012] ->5: sync frame-saver
[Mon Aug  6 19:50:13 2012] ->5: start trender
[Mon Aug  6 19:50:13 2012] trender started
[Mon Aug  6 19:50:13 2012] trend frame saver started
[Mon Aug  6 19:50:13 2012] ->5: start trend-frame-saver
[Mon Aug  6 19:50:14 2012] ->5: sync trend-frame-saver
[Mon Aug  6 19:50:14 2012] minute trend frame saver started
[Mon Aug  6 19:50:14 2012] ->5: start minute-trend-frame-saver
[Mon Aug  6 19:50:14 2012] Done creating ADC structures
[Mon Aug  6 19:50:15 2012] ->5: sync minute-trend-frame-saver
[Mon Aug  6 19:50:15 2012] raw minute trend frame saver started
[Mon Aug  6 19:50:15 2012] ->5: start raw_minute_trend_saver
[Mon Aug  6 19:50:15 2012] ->5: #frame-writer "225.225.225.1" broadcast="131.215.113.0" all
[Mon Aug  6 19:50:15 2012] ->5: #sleep 5
[Mon Aug  6 19:50:15 2012] producer started
[Mon Aug  6 19:50:15 2012] ->5: start producer
[Mon Aug  6 19:50:15 2012] ->5: start epics dcu
[Mon Aug  6 19:50:15 2012] MX receiver thread started
[Mon Aug  6 19:50:15 2012] edcu started
[Mon Aug  6 19:50:15 2012] ->5: start epics server "C0:DAQ-DC0_" "C1:DAQ-DC0_"
[Mon Aug  6 19:50:15 2012] epics server started
[Mon Aug  6 19:50:15 2012] ->5: start listener 8087
[Mon Aug  6 19:50:15 2012] ->5: start listener 8088 1
[Mon Aug  6 19:50:15 2012] ->5: sleep 60
[Mon Aug  6 19:50:15 2012] Epics server started
[Mon Aug  6 19:50:15 2012] EDCU has 2553 channels configured; first=0

[Mon Aug  6 19:50:18 2012] Minute trender made GPS time correction; gps=1028343032; gps%60=32
...

The "tail:..." line indicates that the log was moved and replaced, which indicates a daqd restart.  As far as I know this was not manually triggered.

After the restart the same thing happens again.  About once every five minutes.

For some reason this afternoon we've been experiencing a lot of problems with the framebuilder, and with the CDS network in general.  The framebuilder has been very unresponsive, although the daqd logs seem to indicate that things are ok.  All models will loose contact with fb for very long stretches.  Attempts to kill/restart daqd don't seem to fix the problem.

These problems seem to be associated with the general CDS network issues as well.  The network seems to become very slow, and the workstations all become very slow.  The later I assume is because of the network and that so much of the work we do is on network mounted filesystems (/opt/rtcds, /ligo, etc.).

My current speculation is that daqd on fb is doing something stupid, like trying to read or write a bunch of stuff from /frames, which is also network mounted, and that clogs up the entire network.  Some serious network debugging is going to be needed to figure out what's going on, though.

I'm afraid daqd is caught in some bad state now, though.  It's not responding to anything, and every attempt to kill it seems to bring it back into the bad state.  Hopefully I can get Alex to help me figure out what's going on tomorrow.   Maybe it will clear up on it's own tonight...

I was trying to load some filters into the ASS so that I can try it out, but for some reason the filter banks aren't working - clicking the on/off buttons doesn't do anything, filters (which exist in the .txt file generated by foton) don't load.

I've emailed cds-announce to see if anyone has any ideas.

The SRM oplev beam is clipping.

ACAD files of the 40m optical layout have been updated as per the vent in Aug 2011.

Files are available at the 40m svn docs-->Upgrade12-->Opt_Layout2011.

 The power outage did not have any effect on the vacuum system.  I had to open VM1 to the RGA because of flaky CC1 cold cathode gauge fluctuations.

We are running on the vertically positioned CC1 again.

Note: rga scans with closed VM1 are back ground scans.

 Earthquake test our suspensions            PRM damping restored.             Oplev servo gains turned to zero.

 After that RFM -> OAF communication through PCIE became bad again. Inside CommData2.c cache flushing is not allowed

// If PCIE comms show errors, may want to add this cache flushing
            #if 0
            if(ipcInfo[ii].netType == IPCIE)
                clflush_cache_range (ipcInfo[ii].pIpcData->dBlock[sendBlock][ipcIndex].data, 16);
            #endif

As a result, a significant part of MC_F and other signals is lost during RFM -> OAF transmission (270 - 330 out of 2048 per second)

Last time when I replaced 0 for 1, it suspended SUS machine because of the code bug. Alex modified a couple of files in the old version and it started to work. Do you know if this bug is fixed in the new version?

The script estimates digital noise produces by online filters. First version of Matlab files and complied c files are in scripts/digital_noise directory.

Algorithm for 1 filter bank (max number of filters = 10):

1.        extract sos - representation from Foton file for each filter (Matlab)
2.        download data from corresponding DQ channel using NDS (Matlab)
3.        find filters that are switched on (Matlab)
4.        filter signal using Df2 and BQF with single and double precision (C)
5.        estimate digital filter noise (Matlab)
6.        calculate power spectral density and plot the result (Matlab)

More details on (2)

Often DQ channels have reduced sampling rate. In this case the script will upsample data adding zeros.

AI filter is not applied. But in the end only the frequency range (0, DQ RATE / 2) is analyzed.

More details on (3):

• input switch ON / OFF => 4 / 0
• filters 1 - 6 ON /OFF
  • 1 => 48 / 0
  • 2 => 192 / 0
  • 3 => 768 / 0
  • 4 => 3072 / 0
  • 5 => 12288 / 0
  • 6 => 49152 / 0
  • If a switch is ON but there is no corresponding filter (one green and one red line under the switch) then the switch value is divided by 3

_SW2R channel value is the sum of the following numbers:

• decimation switch ON / OFF => 512 / 0
• output switch ON / OFF => 1024 / 0
• filters 7 - 10 ON /OFF
  
  • 7 => 3 / 0
  • 8 => 12 / 0
  • 9 => 48 / 0
  • 10 => 192 / 0
  • If a switch is ON but there is no corresponding filter (one green and one red line under the switch) then the switch value is divided by 3

Note: as for now Matlab script assumes that input, output and decimation filters are switched ON and there are no turned ON filter switches that do not correspond to any filters

More details on (5)

Digital noise using double precision is estimated by extrapolation of digital noise with single precision. The last is calculated by subtracting outputs of the filters with single and double precision. Then this noise is multiplied by 3 * 10^-7.

This extrapolation number was achieved by printf tests of the number 0.123456789012345678 with single and double precision on C. Using type 'float' variables 10 significant numbers show up, using type 'double' - 17.

I also did 'calibration tests' to achieve extrapolation number - signal was filters with an aggresive low-pass filter. At high frequencies filter output spectrum is flat => digital noise amplitude must be the same. The plot shows GUR1_X channel filtered with low-pass chebyshev type 1 filter.

However, extrapolation number is not the same for all cases. In the following example of analyzing BS_SUSPOS filter bank using extrapolation 3 * 10^-7 we get noise that is slightly overestimated. In some other examples we need to take a larger number. But in average, I think, this is a good approximation.

To avoid extrapolation problem we can use long double precision (~19 digits). I was able to do this with gcc compiler. However, in mex compiler using long double in filter calculations, I do not get any better precision then using double precision. I'll think more about it.

 Brian Lantz / Dan Clark are looking around their lab to see if they forgot to ship the feet with the T-240.  They had taken the feet off to put it in a pod. 

As we do not have legs for Trillium, I was advised to use shims to adjust the levels. However, they produce extra resonance at ~30 Hz + harmonics. Coherence is lost at these frequencies.

I wanted to try out the ASS tonight, but I wanted some kinds of screens thrown together so I would know what I was doing.  Turns out screens take longer than I thought.  Am I surprised? Not really. 

They're probably at the ~85% mark now, but I should be able to try out the ASS tomorrow I think.

Jamie re-redid the ASS model a few hours ago.

I have just compiled it, and restarted c1ass.  (The model from last night is currently called c1ass3.mdl)  I had to delete and re-put inthe goto and from tags for the LSC signal coming in from the shmem.  For some reason, it kept claiming that the inputs using the from tags were not connected, even when I redid the connections.  Finally deleting and dragging in new goto and from tags made the model happy enough to compile.  Whatever.  I'm going to let Jamie do the svn-ing, since he's the one who made the changes.  Before I had figured out that it was the tags, I was concerned that the shmem was unhappy, so there was no signal connecting to the input of the goto tag, and that was somehow bad....anyhow, I recompiled the LSC model to re-create the shmem sender, but that had no effect, since that wasn't the problem.

The change from last night is that now the library parts are by DoF.  There is only one matrix in each library part, before the servo filters. Now we can DC-actuate on a single mode (ETM or ITM, pitch or yaw), and see how it affects all 4 sensors (the demodulated signals from the lockins). We need to measure the sensing matrix to go from the several sensors to the servo input.

Den and I moved the Streckeisen, Guralp 2, and Trillium seismometers to the isolation box in order to measure the noise of the Streckeisen while we have the Trillium.