I just stumbled upon this while poking around:

Since the great crash of June 2014, the scripts backup script has not been workingon op340m. For some reason, it's only grabbing the PRFPMI folder, and nothing else. 

Megatron seems to be able to run it. I've moved the job to megatron's crontab for now.

Came in, found all front-ends down.

Keyed a bunch of crates, no luck:

Requesting coeff update at 0x40f220 w/size of 0x1e44
No response from EPICS 

Powered off/restarted c1dcuepics.  Still no luck.

Powered off megatron.  Success!  Ok, maybe it wasn't megatron.  I also did c1susvme1 and c1susvme2 at this time.

BURT restored to Nov 26, 8:00am

But everything is still red on the C0_DAQ_RFMNETWORK.adl screen, even though the front-ends are running and synced with the LSC.  I think this means the framebuilder or the DAQ controller is the one in trouble--I keyed the crates with DAQCTRL and DAQAWG a couple of times, with no luck, so it's probably fb40m.    I'm leaving it this way--we can deal with it tomorrow.

Requesting coeff update at 0x40f220 w/size of 0x1e44
No response from EPICS 

 I found the red sea when I came in this morning.

I tried several things.

1. ssh into fb40m: connection refused
2. telnet fb40m 8087: didn't respond
3. shutdown fb40m by physically pushing the power button: it worked and the FB came back to life but still with a red light on the MEDM DAQ_DETAIL screen;
4. powercycled fb40m AND C0DAQCTRL: no improvement
5. shutdown fb40m, C0DAQCTRL, C1DCUEPICS and pushed the reset button on the RF network crate; then I restarted the computers in this order: fb40m, C1DCUEPICS, C0DAQCTRL: it worked: they came back to life and the lights eventually turned green on the MEDM montior screen

I'm now going to restart the single front -ends and burtgooey them if necessary.

Requesting coeff update at 0x40f220 w/size of 0x1e44
No response from EPICS 

Everything is back on.

Restarted all the front ends. As usual c1susvme2 was stubborn  but eventually it came up.

I burt-restored all the front-ends to Nov 26 at 8am.

The mode cleaner is locked.

Taking  a cue from entry 2346, I immediately went for the nuclear option and powered off fb40m.  Someone will probably need to restart the backup script.

 Backup script restarted.

 As I said in the past entry (see this entry), there was unknown loss of about 20dB in the beat detection path.

So I started fully characterizing the beat detection path. 

Today I measured the frequency response of the wideband RFPD using the Jenne Laser.

Since all the data were taken by using a 1064nm laser, the absolute magnitudes [V/W] for 532nm are not calibrated yet.  

I will calibrate the absolute values with a green laser which has a known power.

The data were taken by changing the bias voltage from -150V to 0V.

The shape of the transfer function looks quite similar to that Hartmut measured before (see the entry).

It has 100MHz bandwidth when the bias voltage is -150V, which is our normal operation point.

Theoretically the transfer function must keep flat at lower frequency down to DC.

Therefore for the calibration of this data, we can use the DC signal when a green beam with a known power is illuminating the PD.

I did liso simulation of the circuit in the seis box. I think that AD620 (first amplifier in the circuit) noise might be much less with the signal from guralps from 0.01 Hz. Here is the TF of AD620 output / circuit input.

The noise spectrum is at this node is

The psd of the seismic noise below 1 Hz ~ 1u m/s => circuit input signal is ~1mv.

The TF of the whole circuit is

This result differs from the graph on the circuit sheet, but may be it was done before the resistor parameteres changed. Back of the envelop calculations also show that it is not possible to acheive DC gain 200 while 50-800 Hz gain = 5000. I'll check with the spectrum analyzer.

AD620 might be a weak point in the simulation since this is not a "classical" operational amplifier, it contains a resistor that adjusts the gain. During the liso simulation I assumed that we have an ordinary opamp (with noise, gain and gbw parameters taken from the real ad620 datasheet) with a resistor parallel to the opamp = 50k and a resistor before the inverted input that corrsponds to R2. In this case the gain of the simulated opamp is the same as of the real one given by the formula 1 + 49.9k / R2, though noise parameters may change. This should be also checked with the spectrum analyzer.

I've put the seismometer box back to the 1x1, Guralp is back under MC2. When the seismometer is not plugged in, the noise is

Now, I'm going to collect some data from GUR 1 and MC_F and see if the problem with adaptive filter (increasing errror while decreasing mu) will be gone.

I've replaced R2 resistor that adjusts the gain of the AD620 amplifier. Previous value 5491Ohm, new value 464Ohm, so the gain should increase up to ~200-250. Only at the N/S 1 circuit!

LISO simulation of the circuit transfer function and noise are

LISO predicts gain ~45-46 dB = 200 and noise at the level of 10uV at 1Hz. The transfer function and noise measured are

The noise measured is 5 times higher then predicted by LISO. Though I described AD620 as an ordinary amplifier with 49.9kOhm resistor connecting output and inverted input. I specified the noise spectrum 10 nV and 1/f corner frequency 30 Hz. In the AD620 datasheet noise spectrum is 10 - 100 nV depending on the gain. However, the gain is 200 and noise spectrum should be 10 nV. May be in reality it is not the case. It also possible that the noise model used by LISO is not valid for AD620 as it is not an ordinary operational amplifier.

I've measured the input signal to the seismic box from seismometer Guralp 1. The spectrum of the signal in the "input +" (TP 1) is

The spectrum below 1 Hz is ~250 uV/sqrt(Hz). As the input is differential, then the input voltage is 0.5 mV/sqrt(Hz). The spectrum of the "output +" signal (TP 2) is

So the gain at ~ 1Hz is ~20. I've measured the transfer function between the "input +" and "output +" (TP1 and TP2) for all 9 circuits

The channels 1-6 are of new modification and have gain ~20 at the frequencies 0.2 - 100 Hz. Below 0.2 Hz the gain is reduced. 100 Hz - cut off frequency of the low-pass filters. Meanwhile channels 7-9 (old configuration) have much more gain and 10_50 Hz filters work here.

The coherence between  "input +" and "output +" (TP1 and TP2) for 9 circuits is

We can see that channel VERT 3 is very bad. For others coherence is lost below 0.2 Hz. The spectrum analyzer noise measured is ~1000 times less then the signal at these frequencies. I'll pay more attention to this loss of coherence at low frequencies. Something is noisy.

I've moved GUR1 seismometer from MC2 to the working tables in order not to disturb the MC while working with the seismometer box. The new place for the GUR1 for a few days is near the printer, cables and blue boxes. I've cleaned all mess and wires from the floor, so that seismometer now looks like that

I've connected 2 inputs of the N/S 1 circuit of the seismometer box with a 50 Ohm resistor and measured the noise at the output. The comparison with the seismic signal is

The noise increased at 0.5 Hz and is pretty big. This might explain the loose of coherence at low frequencies.

 This is because spectrum analyzer did not plot the real noise spectrum at the first few points at low frequencies. I've remeasured the noise at 1mHz - 3Hz at "output -" (TP9) and compared it to the seismometer signal

The noise seems to be much less then the signal. I've measured the noise several times and once I got a huge amount of noise

I made another measurement in some time and got the low noise again. A circuit might have a bad contact somewhere.

The plan is to change AD620 adjustable resistor (R2) from 5.49kOhm to 500Ohm to increase the gain from 20 up to 200.

last 20 days - including the pounding from next door

• Expanded the dmf.db file on c1aux to include all accelerometers and the seismometer.
  
• Added the channels to the C0EDCU.ini file and restarted the framebuilder daqd.
  
• Modified seisBLRMS.m to use .conf files for the channels. The .conf files are now residing in the compiled matlab directory that Rob made.
  

The compiled version of seisBLRMS had been running ~2 weeks without crashing as of last night, when I killed it
so it wouldn't interfere with alignment scripts.  I added an EPICS channel C1:DMF-ENABLE, and updated the DMF
executables to check this channel while running.  So far it seems to work.  When you're running alignment acripts,
simply click the DISABLE button on the C1DMF_MASTER.adl screen, and then re-ENABLE when the scripts finish. 
It's not clear why this is necessary though.  Theories include the constant disk access is keeping the
framebuilder busy, reducing its ability to deal with ezcademod commands and DMF programs just flooding the
network with so much traffic that ezcademod-related packets run late and get ignored.

Also, for reasons of aesthetics, I changed the data delay from 6 minutes to 5 minutes.  We'll see if that's enough.

> 
> Also, for reasons of aesthetics, I changed the data delay from 6 minutes to 5 minutes.  We'll see if that's enough.

5 minutes didn't work.  

Can't find hostname 'fb40m'

it only lasted a few hours

 I compiled seisBLRMS.

The tricks were the following:

(1) Don't add path in a deployed command.
It does not make sense to add paths in a compiled command because it may be moved to anywhere. Moreover, it can cause some weird side effects. Therefore, I enclosed the addpath part of mdv_config.m in a "if ~isdeployed ... end" clause to avoid adding paths when deployed. Instead of adding paths in the code, we have to add paths to necessary files with -I options at the compilation time. This way, mcc will add all the necessary files into the CTF archive.

(2) Add mex files to the CTF archive by -a options.
For some reason, mcc does not add necessary mex files into the CTF archive even though those files are called in the m-file which is being compiled. We have to add those files by -a options.

(3) NDS_GetData() is slow for nodus when compiled.
NDS_GetData(), which is called by get_data() stops for a few minutes when using nodus as an NDS server.
This problem does not happen when not compiled. I don't know the reason. To avoid this, I modified seisBLRMS.m so that when an environmental variable $NDS is defined, it will use an NDS server defined in this variable.

I wrote a Makefile to compile seisBLRMS. You can read the file to see the details of the tricks.
I also wrote a script start_seisBLRMS, which can be found in /cvs/cds/caltech/apps/DMF/compiled_matlab/seisblrms/. To start seisBLRMS, you can just call this script.
At this moment, seisBLRMS is running on megatron. Let's see if it continues to run without crashing.

The seisBLRMS has been running on megatron via an open terminal ssh'd into there from allegra with matlab running. This

is because I couldn't get the compiled matlab functionality to work.

Even so, this running script has been dying lately because of some bogus 'NDS' error. So for today I

have set the NDS server for mDV on megatron to be fb40m:8088 instead of nodus.ligo.caltech.edu. If this seems to fix the problem

I will make this permanent by putting in a case statement to check whether or not the mDV'ing machine is a 40m-martian or not.

The green xterm on op540m which is running the seisBLRMS DMF got stuck somehow ~3 days ago and lost its NDS connection. I closed the matlab session and restarted it. Seismic trends are now back online.

I restarted the seisBLRMS DMF monitor by ssh'ing into mafalda and starting up a matlab session. I also have started a StripTool session on rossa by forwarding the process from op440m.

We need to get the modern EPICS installation onto these linux machines by copying what K. Thorne has done at LLO.

I've changed R2 resistor in the seism box for the VERT 1 channel from 464 Ohm to 1051 Ohm to reduce the gain of this channel by a factor of 2. This should help the GUR1Z signal not to be corrupted inside the AA box, so we can use it in the adaptive filtering.

A few weeks ago I found that GUR2_X signal is biased from 0 to 800 counts in average. I decided that the corresponding channel in the readout box is bad - adds DC voltage to the signal. I stopped using GUR2_XYZ channels of the seism readout box. Now the same thing happened with the GUR1_XYZ channels. I checked the signals coming out from the seism box with the oscilloscope and they were fine. So the problem is not in the readout box. Then I applied 1 V sine wave to the input of AA board to the GUR1_X and ACC_MC1_Z channels. GUR1_X channel still shows noise. Something is wrong with these channels inside the AA board or in the ADC.

Edited by Den: GUR1_XYZ_IN1 signals are empty though GUR1_XYZ are fine. So the problem is just that GUR1_XYZ_IN1 are not acquired for now though some of the ACC_IN1 channels contain the signal. I need to correct .ini files.

 So, none of our PEM BLRMS channels are recorded as of right now.  All we have for long-term record is the StripTool on the wall.  The 0.1-0.3Hz and 0.3-1 Hz traces both show these weirdo things, but the 1Hz and up BLRMS don't have any unusual noise.

 Seems like a problem to solve on Monday so that we don't end up without trends like this again.

 Tragically, this is more tricksy than I would have thought. The channels we need are "cdsEpicsOutput"s in the model.  They don't show up in Dataviewer (fast or slow channels) or the regular fast channel .ini file.  Jamie and I don't remember where these channels live and how to get them saved to frames.  I'm on top of it though.

I did notice however, that the striptool for seismic trends is showing the wrong channels for 3-10 and 10-30 Hz.  The other 3 channels are correctly the output after the sqrt is taken, but those two (orange and red on striptool) are before the sqrt, but after the bandpass and low pass.  I'll fix that now...

I reported the procedure to add slow channels to the FB. I guess you already have done Step.1

http://nodus.ligo.caltech.edu:8080/40m/5991

Our only seismometer is at the east end now.

Atm1, Ditch Day morning puzzle. The gardener came after the freshman did leave and cut the grass with the lawn mower.

Atm2, Yesterday afternoon the Aztecs containers moved out.

The earth quake shook ITMX free for a  short while.

Salton See is shaking again.

I tried to figure out where additional (to seismic) noise enters to MC_F, so the coherence below 1 Hz is low (~0.2-0.5). I've examined noise in the path

PD -> DEMOD -> MC BOARD -> FSS -> PZT Controller -> LASER

• I terminated ADC and measured its noise
• connected MC BOARD OUT to ADC, terminated INPUT1 of the MC BOARD and measured noise
• connected DEMOD Q OUT to MC BOARD INPUT1, terminated PD INPUT on the DEMODULATOR and measured noise
• connected PD to DEMOD, blocked the beam incident to the PD and measured noise

 MC BOARD noise shows up only below 0.1 Hz and at 10 Hz where the whitening filter starts to work. SNR is ~2 at 4 Hz, so we might want to slightly improve whitening filter. But other then that path PD->DEMOD->MC BOARD is not responsible for additional noises below 1 Hz.

Next I connected SR785 to the laser and measured the closed loop feedback signal while MC was locked.

Coherence between MC BOARD OUT1 and feedback signal is high enough to assume that FSS and PZT controller are also not responsible for additional noise.

From the other hand MC2 SUSPOS measured by OSEMS shows good coherence with GUR1_X

That means that MC2 is indeed driven by seismic motion. In order to figure out if this is the case for MC1 and MC3, I rotated GUR2 by 45 degrees. When it will calm down, I'll measure coherence between OSEMs and seismic motion.

I locked x,y arms and measured coherence between POS{X,Y}11_I_ERR, MC_F and seismometer signals.

Surprisingly, coherence between POSY and GUR1Y is low, but with GUR1X is relatively high. I wonder if this is due to MC that brings this x-axis noise to the arms.

The shaking has stopped at 9:32am  The AC was turned back on at 11:30am  We still do not have any explanation

 Kiwamu and Steve maybe don't know about how to trend seismic noise. If you just take the mean of the time series, you don't prove that the seismic noise got any higher. The STS has a nominally zero DC output, so the long period level shifts that you see tell you just that there was a DC offset.

This is NOT an increase in seismic noise. To see a seismic trend you should plot the trend of the BLRMS channels that we made especially for this purpose.

[Micro, Den]

Analyzing coherence of seismic noise and mode cleaner length we've figured out that at some days the coherence below 1 Hz is still present. For example, at Nov 13 we can see some coherence compared to most other dates when we are not able to see coherence as shown on the figure. On the top plot - psd of MC_L and GUR1_X at Nov 13 (red and blue) and Nov 15 (black and cyan). On the bottom plot is presented coherence between MC_L and GUR1_X on Nov 13 (red) and Nov 15 (black)

We can divide the psd plot for 2 parts - below 1 Hz and above 1 Hz. Above 1 Hz seismic noise on Nov 15 (cyan) was higher then on Nov 13 (blue) and correspondently MC_L at that region was higher on Nov 15. Below 1 Hz seismic noise was higher on Nov 13 but MC_L is still lower that on Nov 15. That is surprising. From the coherence plot we can say that once we have some more seismic noise than usually, we immediately see coherence.

Because of this we wanted to find out the level of the X noise that makes seismic noise invisible. We injected seismic noise by doing smooth physical exercises near MC_2 (1.5 m and 3 m apart). The MC_2 was in lock during the experiment.

In the coherence plot we can see that coherence between GUR1_X and MC_L increased with noise injection. The highest coherenced we obtained sittind down and standing up smoothly near MC_2 at distance 1.5 m. We did not want to come clother and break the lock. This measurement tells us that the X noise is approximately 3-4 times higher than seismic noise in the range 0.1 - 1 Hz. That means that it is approximately 1e-6 - 1e-8 m/sqrt(Hz) in this region. This noise goes down at frequencies from 2 Hz and not seen because of seismic noise. Actually, seismic noise can be filtered out with the Wiener filter and then we'll see the spectrum of X noise.

We now try to figure out the method to estimate the contribution of OSEM noise to the X noise.

There is still a problem why GUR, STS signals are poorly coherent to MC_L.  But at least we can see coherence at 2-5 Hz. It might be useful to do something with adaptive filtering because it does not work at all for a long time. We start with Wiener filtering. I still doubt that static filtering is useful. Adaptive filter output is linear to its coefficients, so why not to provide adaptive filter with a zero approximation equal to calculated Wiener filter coefficients. Then you automatically have Wiener filter ouput + adaptively control coefficients. But if Wiener filter is already present in the model, I tried to make it work. Then we can compare performance of the OAF with static filter and without it.

I started with GUR1_X and MC_F signals recorded 1 month ago to figure out how stable TF is. Will the same coefficients work now online? In the plot below offline Wiener filtering is presented.

This offline filtering was done with 7500 coefficients. This FIR filter was converted to IIR filter with the following procedure:

1. Calculate frequency responce of the filter. It is presented below.

2. Multiply this frequency response by a window function. This we need because we are interested in frequencies 0.1-20 Hz at this moment. We want this function to be > 1e-3 at ~0Hz, so that the DC component is filtered out from seismometer signal. From the other hand we also do not want huge signal at high frequenies. We know that this signal will be filtered with aggresive low-pass filterd before going to the actuator but still we want to make sure that this signal is not very big to be filtered out by the low-pass filter.

The window function is done in the way to be a differential function to be easier fitted by the vectfit3. Function is equal to 1 for 0.5 - 20 Hz and 1e-5 for other frequencies except neighbouring to the 0.5 and 20 where the function is cosine.

3. I've used vectfit3 software to approximate the product of the frequency response of the filter and window fucntion with the rational function. I've used 10 complex conjugate poles. The function was weighted in the way to make deviation as small as possible for interesting frequencies 0.5 - 10 Hz. The approximation error is big below 20 Hz where the window function is 1e-5 but at least obtained rational function does not increase as real function do at high frequencies.

I tried to make a foton filter out of this approximation but it turns out that this filter is too large for it. Probably there is other problem with this approximation but once I've split the filter into 2 separate filters foton saved it. Wiener21 and Wiener22 filters are in the C1OAF.txt STATIC_STATMTX_8_8 model.

I've tested how the function was approximated. For this purpose I've downloaded GUR and MC_F signals and filtered GUR singla with rational approximation of the Wiener filter frequency response. From power spectral density and coherence plots presented below we can say that approximation is reasonable.

Next, I've approximated the actuator TF and inverted it. If TF measured in p. 5900 is correct then below presented its  rational approximation. We can see deviation at high frequencies - that's because I used small weights there using approximation - anyway this will not pass through 28 Hz low-pass filter before the actuator.

I've inverted this TF p->z , z->p, k->1/k. I've also added "-" sign before 1/k because we subtract the signal, not add it. I placed this filter 0.5Actuator20 to the C1OAF.txt SUS-MC2_OUT filter bank.

The next plot compares online measured MC_L without static filtering and with it. Blue line - with online Wiener filtering, red line - without Wiener filtering.

We can see some subtraction in the MC_L due to the static Wiener filtering in the 2-5 Hz where we see coherence. It is not that good as offline but the effect is still present. Probably, we should measure the actuator TF more precisely. It seems that there some phase problems during the subtraction. Or may be digital noise is corrupting the signal. 

I made a model for our seismometer can using actual data so that we know approximately what the time constant should be when we test it out. I used the appendix in Megan Kelley's report to make a relation for the temperature in terms of time.

 so  and 

In our case, we will heat the can to a certain temerature and wait for it to cool on its own so 

We know that  where k is the k-factor of the insulation we are using, A is the area of the surface through which heat is flowing,  is the change in temperature, d is the thickness of the insulation.

Therefore,

We can take the derivative of this to get

, or  

We can guess the solution to be

 where tau is the time constant, which we would like to find.

The boundary conditions are  and . I assumed we would heat up the can to 40 celcius while the room temp is about 24. Plugging this into our equations,

, so 

We can plug everything back into the derivative T'(t)

Equating the exponential terms on both sides, we can solve for tau

Plugging in the values that we have, m = 12.2 kg, c = 500 J/kg*k (stainless steel), d = 0.1 m, k = 0.26 W/(m^2*K), A = 2 m^2, we get that the time constant is 0.326hr. I have attached the plot that I made using these values. I would expect to see something similar to this when I actually do the test.

To set up the experiment, I removed the can (with Steve's help) and will place a few heating pads on the outside and wrap the whole thing in a few layers of insulation to make the total thickness 0.1m. Then, we will attach the heaters to a DC source and heat the can up to 40 celcius. We will wait for it to cool on its own and monitor the temperature to create a plot and find the experimental time constant. Later, we can use the heatng circuit we used for the PSL lab and modify the parts as needed to drive a few amps through the circuit. I calculated that we'd need about 6A to get the can to 50 celcius using the setup we used previously, but we could drive a smaller current by using a higher heater resistance.

I performed a test with the can last week with one layer of insulation to see how well it worked. First, I soldered two heaters together in series so that the total resistance was 48.6 ohms. I placed the heaters on the sides of the can and secured them. Then I wrapped the sides and top of the can in insulation and sealed the edges with tape, only leavng the handles open. I didn't insulate the bottom. I connected the two ends of the heater directly into the DC source and drove the current as high as possible (around 0.6A). I let the can heat up to a final value of 37.5C, turned off the current and manually measured the temperature, recoding the time every half degree. I then plotted the results, along with a fit. The intersection of the red line with the data marks the time constant and the temperature at which we get the time constant. This came out to be about 1.6 hours, much longer than expected considering that onle one layer instead of four was used. With only one layer, we would expect the time constant to be about 13 min, while for 4 layers it should be 53 min (the area A is 0.74 m^2 and not 2 m^2).

Updated some values, most importantly, the k-factor. I had assumed that it was in the correct units already, but when converting it to 0.046 W/(m^2*K) from 0.26 BTU/(h*ft^2*F), I got the following plot. The time constant is still a bit larger than what we'd expect, but it's much better with these adjustments.

For our next steps, I will measure the time constant of the heater without any insulation and then decide how many layers of it we will need. I'll need to construct and calibrate a temperature sensor like the ones I've made before and use it to record the values more accurately.

For the insulation, I have decided to use this one (Buna-N/PVC Foam Insulation Sheets). We will need 3 of the 1 inch plain backing ones (9349K4) to wrap a few layers around it. I'll try two layers for now, since the insulation seems to be doing quite well according to initial testing.

I've ordered 4 of these from McMaster. Should be delivered to the 40m by noon tomorrow.

Kira and I also discussed the issiue. It would be good if someone can hunt aroun on the web and get some free samples of non-shedding foam with R~4.