found it unresponsive. Restarted fine using procedure documented in wiki
Looks mostly right, but you used the OSEM sensors as readbacks. We are diagonalizing using the cavity sensors. Using the diagonalized input matrix is also good since that will reduce the cross-coupling due to the damping loops.
Its sort of a subtle issue:
Note from Stephen on more sensitive Baslers.
the POS column should be all 1 for the AC balancing. Where did those non-1 numbers come from?
The PSL was too hot, so I turned on the south HEPA on the PSL. The north one was on and the south one was off (or so slow as to be inaudible and no vibration, unlike the north one). Lets watch the trend over the weekend and see if the temperature comes down and if the PMC / WFS variations get less. Fri May 14 17:46:26 2021
Looks like the fan lowered the temperature as expected. Need to get a few more days of data to see if its stabilized, or if that's just a fluke.
The vertical line at 00:00 UTC May 18 is about when I turned the fans up/on.
Fluke. Temp fluctuations are as usual, but the overall temperature is still lower. We ought to put some temperature sensors at the X & Y ends to see what's happening there too.
What are the quantitative root causes for why the statistical uncertainty is so large? Its larger than 1/sqrt(N)
this model doesn't seem to include the analog AA, analog AI, digital AA, digital AI, or data transfer delays in the system. I think if you include those you will get more accuracy at high frequencies. Probably Anchal has those included in his DARM loop model?
I suggest plotting all the traces in the plot so we can see their differences. Also remove the 1/f^2 slope so that we can see small differences. Since the optlev servos all have low pass filters around 15-20 Hz, its not necessary to turn off the optlev servos for this measurement.
I think that based on the coherence and the number of averages, you should also be able to use Bendat and Piersol so estimate the uncertainy as a function of frequency. And we want to see the comparison coil-by-coil, not in the DoF basis.
4 sweeps for BS and 4 sweeps for PRM.
I would expect to see some lower frequency effects. i.e. we should look at the timeseries of the demod with the excitation on and off.
I would guess tat the exc on should show us the variations in the optical gain below 3 Hz, whereas the exc off would not show it.
Maybe you should do some low pass filtering on the time series you have to see the ~DC effects? Also, reconsider your AA filter design: how do you quantitatively choose the cutoff frequency and stopband depth?
I suggest that you
not sure that this is necessary. If you look at teh previous entries Gautam made on this topic, it is clear that the BS/PRM PRMI matrix is snafu, whereas the ITM PRMI matrix is not.
Is it possible that the ~5% coil imbalance of the BS/PRM can explain the observed sensing matrix? If not, then there is no need to balance these coils.
For the oplev, there are DQ channels you can use so that its possible to look back in the past for long measurements. They have names like PERROR
should compare side by side with the ITM PRMI radar plots to see if there is a difference. How do your new plots compare with Gautam's plots of PRMI?
SVG doesn't work in my browser(s). Can we use PDF as our standard for all graphics other than photos (PNG/JPG) ?
rethinking what I said on Wednesday - its not a good idea to put the particle counter on a vac chamber with optics inside. The rumble from the air pump shows up in the acoustic noise of the interferometer. Let's look for a way to mount it near the BS chamber, but attached to something other than vacuum chambers and optical tables.
I have placed a GT321 particle counter on top of the MC1/MC3 chamber next to the BS chamber.
IFOcad model/video of the AEI 10m interferometer:
Please don't put it on c1sus2. Put it on the completely independent test stand as we discussed Wednesday. You must test the controller on the simplant and verify that they thing is stable and works, before putting it in the 40m network.
This looks great. I think what we want to see mainly is just the noise in the 32 bit IIR filtering subtracted from the 64 bit one.
It would be good if Tega can look through your code to make sure there's NO sneaky places where python is doing some funny casting of the numbers. I didn't see anything obvious, but as Chris points out, these things can be really sneaky so you have to be next level paranoid to really be sure. Fox Mulder level paranoia.
And, we want to see a comparison between what you get and what Denis Martynov put in an appendix of his thesis when comparing the Direct Form II, with the low-noise form (also some slides from Matt Evans on thsi from a ~decade agoo). You should be able to reproduce his results. He used matlab + C, so I am curious to see if it can be done all in python, or if we really need to do it in C.
And then...we can make this a part of the IFOtest suite, so that we point it at any filter module anywhere in LIGO, and it downloads the data and gives us an estimate of the digital noise being generated.
We want to maintain the 16 kHz sample rate for the COIL DAQ channels, but nothing wrong with reducing the others.
I would suggest setting the DQ sample rates to 256 Hz for the SUS DAMP channels and 1024 Hz for the OPLEV channels (for noise diagnostics).
Maybe you can put these numbers into a new library part and we can have the best of all worlds?
We notice that all our suspension models need to go through this weird python script modifying auto-generated .ini files to reduce the data rate. Ideally, there is a simpler solution to this by simply adding the datarate 2048 in the '#DAQ Channels' block in the model library part /cvs/cds/rtcds/userapps/trunk/sus/c1/models/lib/sus_single_control.mdl which is the root model in all the suspensions. With this change, the .ini files will automatically be written with correct datarate and there will be no need for using the activateDQ script. But we couldn't find why this simple solution was not implemented in the past, so we want to know if there is more stuff going on here then we know. Changing the library model would obviously change every suspension model and we don't want a broken CDS system on our head at the begining of holidays, so we'll leave this delicate task for the near future.
I updated the mime.local.conf file for the AIC Wiki so as to allow attachments with the .txz format. THis should be persistent over upgrades, since its a local file.
I think our suspension input matrix diagonalization is not so robust usually because we only choose a inverting matrix which gives the best separation for a single suspension alignment.
i.e. we have seen in the past that adjusting the bias for the alignment makes the matrix inversion not work well. Sometime people turn OFF the alignment bias before making the ringdown and that makes the whole measurement invalid.
This is because the sensitivity of the OSEMs to longitudinal and/or transverse motion is significantly different for different alignment.
I wonder if there's a way we can choose a better matrix by putting in random gain errors on the shadow sensor signals and then finding the matrix which gives the best diag under an ensemble of gain errors.
you can hand edit the autoBurt file which the FE uses to set the values after boot up. Just make a python script that amends all of the OFF or ZERO that are needed to make things safe. This would be the autoBurt snap used on boot up only, and not the hourly snaps.
Yeah, this is a known issue actually. We go to ASC screen and manually swich off all the outputs after every reboot. We haven't been able to find a way to set default so that when the model comes online, these outputs remain switched off. We should find a way for this.
nice - please update the particle counter page in the 40m wiki. Its probably years out of date.
I mounted the particle counter over the BS chamber attached to the cable tray as seen in Attachment 1. The signal cable runs through an active 30ft cable to the 1x2 rack. the wire is labeled and runs properly through the cable tray. The particle counter is plugged in at the power strip attached near the cable tray. The power cord is also labeled.
Seems like it should be possible to just remove the transformer (aka as a BALUN ... BALanced, UNbalanced), or replace it with a lower frequency part. Its just a usual mini-circuits part. Maybe you can ask Chris Stoughton about this and ask Tommy to checkout some of the RFSoC user forums for how to go to DC.
After fiddling around with the tone-generators and spectrum analyzer tools in loopback configuration (DAC --> ADC direct connection), we noticed that lower frequency (~ 1 MHz) signals were hardly making it out/back into the board... so we looked at some of the schematics found here and saw that both RF data converters (ADC & DAC) interfaces are AC coupled through a BALUN network in the 10 - 8000 MHz band (see Attachment #1). This is in principle not great news if we want to get this board ready for audio-band DSP.
We decided that while Tommy works on measuring TFs for SHP-200 all the way up to ~ 2 GHz (which is possible with the board as is) I will design and put together an analog modulation/demodulation frontend so we can upconvert all our "slow" signals < 1MHz for fast, wideband DSP. and demodulate them back into the audio band. The BALUN network is pictured in Attachment #2 on the board, I'm afraid it's not very simple to bypass without damaging the PCB or causing some other unwanted effect on the high-speed DSP.
I updated the config file c1pem.ini in /users/public_html/detcharsummary/ConfigFiles, and commited it so I hope it works, but I did not have git push permissions. Does anyone know what is the idea here? Should we do our own personal git clone and modify that way or shoudl we do it with the control account.
Wiki needs to clear out all the outdated information on this workflow.
The changes are to make the y-scales useful. Currently, all of the past seis BLRMS plots are not so useful because the scales have not been set based on the actual signal levels. Let's see if this works, and we can re-evaluate after a few weeks.
what was the result of the inductance measurement? should be ~3.3 mH as measured from the flannge or cable that goes to the flange from sat amp.
Its good that the inductance test passed. This means that the coil is OK. How does the inspection photo look? This is the one you guys took of the ITM OSEM that shows the position of the magnet w.r.t. the coil. Also, how does the free swinging spectra look? Either one of these might indicate a broken magnet, or a sticky EQ stop.
what's the reasoning behind using df/f_beat instead of df/f_laser ?
I took ~ 7 minutes of XALS beatnote data with the XAUX laser locked to the XARM cavity, and the XARM locked to PSL to develop an allan deviation estimator.
this is just the CDS error signal, but is not the electronics noise. You have to go into the lab and measure the noise at several points. It can't be done from the control room. You must measure before and afte the whitening.
I measured electronics noise of WFSs and QPD (of the WFS/QPD, whitening, ADC...) by closing PSL and measuring the error signal. It was needed to put the offset in C1:IOO-MC_TRANS_SUMFILT_OFFSET to 14000 cts (without offset the sum of quadrants would give zero, and 14000 cts is the value when the cavity is locked). For WFS that are RF, if there is intensity noise at low frequencies, it is not affecting the measurement.
In the attachment please find the power spectrum of the error signal when the PSL shutter is on and off.
For the PSL HEPA, we wanted it to remain at full speed during the vent, when anyone is working on the PSL, or when there is a lot of dust due to outside conditions or cleaning in the lab.
For NORMAL conditions, the policy is to turn it to 30% for some flow, but low noise.
I think we ought to lock one of the arms on IR PDH and change the HEPA flow settings and plot the arm error signal, and transmitted power for each flow speed to see what's important. Record the times of each setting so that we can make a specgram later
For the proposed construction in the NW corner of the CES building (near the 40m BS chamber), they did a simulated construction activity on Wednesday from 12-1.
In the attached image, you can see the effect as seen in our seismometers:
this image is calculated by the 40m summary pages codes that Tega has been shepherding back to life, luckily just in time for this test.
Since our local time PDT = UTC - 7 hours, 1900 UTC = noon local. So most of the disturbance happens from 1130-1200, presumably while they are setting up the heavy equipment. If you look in the summary pages for that day, you can also see the IM lost lock. Unclear if this was due to their work or if it was coincidence. Thoughts?
although I know that Yuta knows this, I will just put this here to be clear: the NNN/f^2 calibration is only accurate abouve the pendulum POS eiegenfrequency, so when we estimate the DC part (in diaggui, for example), we have to assume that we have a pendulum with f = 1 Hz and Q ~5, to get the value of DC gain to put into the diaggui Gain field in the calibration tab.
MC WFS Demod board needs some attention.
Tomislav has been measuring a very high noise level in the MC WFS demod output (which he promised to elog today!). I thought this was a bogus measurement, but when he, and Paco and I tried to measure the MC WFS sensing matrix, we noticed that there is no response in any WFS, although there are beams on the WFS heads. There is a large response in MC2 TRANS QPD, so we know that there is real motion.
I suspect that the demod board needs to be reset somehow. Maybe the PLL is unlocked or some cable is wonky. Hopefully not both demod boards are fried.
Please leave the WFS loops off until demod board has been assessed.
I've installed Monit on megatron and nodus just now, and will set it up to monitor some of our common processes. I'm hoping that it can give us a nice web view of what's running where in the Martian network.
as I said to you yesterday, I don't think image 2a shows the output of the demod board. The output of the demod board is actually the output connector ON the demod board. What you are showing in 2a, is the signal that goes from the whitening board to the ADC I believe. I may be msitaken, so please check with Tega for the signal chain.
There was a EQ in Ridgecrest (approximately 200 km north of Caltech). It was around 6:20 PM local time.
All the suspensions tripped. I have recovered them (after some struggle with the weird profusion of multiple conflicting scripts/ directories that have appeared in the recent past...)
ETMY is still giving me some trouble. Maybe because of the HUGE bias on that within the fast CDS system, it had some trouble damping. Also the 'reenable watchdog' script in one of the many scripts directories seems to do a bad job. It re-enables optics, btu doesn't make sure that the beams are on the optical lever QPD, and so the OL servo can smash the optic around. This is not good.
Also what's up with the bashrc.d/ in some workstations and not others? Was there something wrong with the .bashrc files we had for the past 15 years? I will revert them unless someone puts in an elog with some justification for this "upgrade".
This new SUS screen is coming along well, but some of the fields are white. Are they omitted or is there something non-functional in the CDS? Also, the PD variances should not be in the line between the servo outputs and the coil. It may mislead people into thinking that the variances are of the coils. Instead, they should be placed elsewhere as we had it in the old screens.
It looks like Tomislav's measurements of the WFS demod board noise were actually of the cable that goes from the whitening to the ADC. So the huge low frequency excess that he saw is not due to wind, but just the inverse whitening of the digital system?
In any case, today, I looked at the connections from the Whitening to the ADC. It goes through an interface chassis to go from ribbon to SCSI. The D-Sub connectors there have the common problem in many of the LIGO D-sub connectors: namely that the strain relief nuts are too tall and so the D connector doesn't seat firmly - its always about to fall out. JC, can you please take a look at this and order a set of low profile nuts so that we can rework this chassis? Its the one between the WFS whitening and the SCSI cables which go to the ADCs.
After pushing them in, I confirmed that the WFS are working, by moving all 6 DoF of the MC mirrors via bias slider, and looking at the step responses (attached). As you can see, all sensors see all mirrors, even if they are noisy.
Next up: get a breakout for the demod output connector and measure the noise there.
For today, I aligned the IMC by hand, then centred the WFS beams by unlocking the IMC and aligning the bright beam. I noticed that the WFS1 beam was being dumped randomly, so I angled the WFS1 by ~3 deg and dumped the specular reflection on a razor blade dump. To handle the sign change in the MC1 actuation (?), I changed the sign in the MC1 ASC filter banks. MCWFS loops still nto closing, but they respond to mirror alignment.
DARM feedback should go to ETMY - ETMX, not just a single mirror: Differential ARM.
For it to work with 1 mirror the UGF of the CARM loop must be much larger than DARM UGF. But in our case, both have a UGF of ~150 Hz.
In principle, you could run the CARM loop with higher gain by using the CM servo board, but maybe that can wait until the X,Y -> CARM, DARM handoff.
can't we just go back to the old python script that was working for many years, and tested? I imagine that as soon as someone besides you tries to debug the docker setup, this is what will happen.
Added C1:IOO-MC_LOCK to ALConfigMC.yml which solved the isse with FSS Slow. We should tune the FSS Slow Servo PID coefficients for a better performance.
the C1PSL_SLOW.adl screen is not obsolete. It can be used to change the PID coefficients, engage/disengage the PID loop, monitor the PID script blinker, and monitor FAST actuator value C1:PSL-FSS_FAST. the functionality of this screen has not changed from before.
I've also added a wiki page for scripts documentation.
We had several problems with our NDS2 server configuration. It runs on megatron, but I think it may have had issues since perhaps not everyone was aware of it running there.
Since Megatron is currently running the "Shanghai" Quad-core Opteron processor from ~2009, its ~time to replace it with a more up to date thing. I'll check with Neo to see if he has any old LDAS leftovers that are better.
we want to be able to run SimPlant on the teststand, test our new controls algorithms, test watchdogs, and any other software upgrades. Ideally in the steady state it will run some plants with suspensions and cavities and we will develop our measurement scripts on there also (e.g. IFOtest).
I keep getting confused about the purpose of the teststand. The view I am adopting going forward is its use as a platform for testing the compatibility of new hardware upgrade, instead of thinking of it as an independent system that works with old hardware.
for damping and OL loops, we typically don't measure the TF like this because it takes forever and we don't need that detailed info for anything. Just use the step responses in the way we discussed at the meeting 2 weeks ago. There's multiple elog entries from me and others illustrating this. The measurement time is then only ~30 sec per optic, and you also get the cross-coupling for free. No need for test-point channels and overloading, just use the existing DQ channels and read back the response from the frames after the excitations are completed.
The PZT sweeps that we've been making to characterize the ALS-X laser should probably be discarded - the DFD was not setup correctly for this during the past few months.
Since the DFD only had a peak-peak range of ~5 MHz, whenever the beat frequency drifts out of the linear range (~2-3 MHz), the data would have an arbitrary gain. Since the drift was actually more like 50 MHz, it meant that the different parts of a single sweep could have some arbitrary gain and sign !!! This is not a good way to measure things.
I used an ezcaservo to keep the beat frequency fixed. The attacehed screenshot shows the command line. We read back the unwrapped beat frequency from the phase tracker, and feedback on the PSL's NPRO temperature. During this the lasers were not locked to any cavities (shutters closed, but servos not disabled).
For the purposes of this measurement, I reduced the CAL factor in the phase tracker screen so that the reported FINE_PHASE_OUT is actually in kHz, rather than Hz on this plot. So the green plot is moving by 10's of MHz. When the servo is engaged, you can see the SLOWDC doing some action. We think the calibration of that channel is ~1 GHz/V, so 0.1 SLOWDC Volts should be ~100 MHz. I think there's a factor of 2 missing here, but its close.
As you can see in the top plot, even with the frequency stabilized by this slow feedback (-1000 to -600 seconds), the I & Q outputs are going through multiple cycles, and so they are unusable for even a non serious measurement.
The only way forward is to use less of a delay in the DFD: I think Anchal has been busily installing this shorter cable (hopefully, its ~3-5 m long so that the linear range is more. I think a 10 m cable is too long.), and the sweeps taken later today should be more useful.
A design flaw in these initial LIGO RFPDs is that the SMA connector is not strain releieved by mounting to the case. Since it is only mounted to the tin can, when we attach/remove cables, it bends the connector, causing stress on the joint.
To get around this, for this gold box RFPD, connect the SMA connector to the PCB using a S shaped squiggly wire. Don't use multi-strand: this is usually good, since its more flexible, but in this case it affects the TF too much. Really, it would be best to use a coax cable, but a few-turns cork-screw, or pig-tail of single-core wire should be fine to reduce the stress on the solder joint.
Later Anchal noticed that there was no RFPD output (C1:LSC-BH55_I_ERR, C1:LSC-BH55_Q_ERR). I took out the RFPD and opened it up, and the RF OUT SMA to PCB connection wire was broken [Attachment 2]. I re-soldered the wire and closed up the box [Attachment 3]. After placing the RFPD back, we noticed spikes in C1:LSC-BH55_I_ERR and C1:LSC-BH55_Q_ERR channels on ndscope. We suspect there is still a loose connection, so I will revisit the RFPD circuit on Monday.