[Jamie, Xavi Siemens, Chris Pankow]
We built the skeleton of a new calibration model for the LSC degrees of freedom. I named it "c1cal". It will run on the c1lsc FE machine, in CPU slot 4, and has been given DCUID 50.
Right now there's not much in the model, just inputs for DARM_ERR and DARM_CTRL, filters for each input, and the sum of the two channels that is h(t).
Tomorrow we'll extract all the needed signals from c1lsc, and see if we can generate something resembling a calibrated signal for one of the IFO DOFs.
So it turns out that the card that Rolf had given me was not a Dolphin host adapter after all. He did have an actual host adapter board on hand, though, and kindly let us take it. And this one works!
I installed the new board in c1ioo, and it recognized it. Upon boot, the dolphin configuration scripts managed to automatically recognize the card, load the necessary kernel modules, and configure it. I'll describe below how I got everything working.
However, at some point mx_stream stopped working on c1ioo. I have no idea why, and it shouldn't be related to any of this dolphin stuff at all. But given that mx_stream stopped working at the same time the dolphin stuff started working, I didn't take any chances and completely backed out all the dolphin stuff on c1ioo, including removing the dolphin host adapter from the chassis all together. Unfortunately that didn't fix any of the mx_stream issues, so mx_stream continues to not work on c1ioo. I'll follow up in a separate post about that. In the meantime, here's what I did to get dolphin working on c1ioo:
To get the new host recognized on the Dolphin network, I had to make a couple of changes to the dolphin manager setup on fb. I referenced the following page:
Below are the two patches I made to the dolphin ("dis") config files on fb:
--- /etc/dis/dishosts.conf.bak 2014-04-17 09:31:08.000000000 -0700+++ /etc/dis/dishosts.conf 2014-04-17 09:28:27.000000000 -0700@@ -26,6 +26,8 @@ ADAPTER: c1sus_a0 8 0 4 HOSTNAME: c1lsc ADAPTER: c1lsc_a0 12 0 4+HOSTNAME: c1ioo+ADAPTER: c1ioo_a0 16 0 4 # Here we define a socket adapter in single mode. #SOCKETADAPTER: sockad_0 SINGLE 0--- /etc/dis/networkmanager.conf.bak 2014-04-17 09:30:40.000000000 -0700+++ /etc/dis/networkmanager.conf 2014-04-17 09:30:48.000000000 -0700@@ -39,7 +39,7 @@ # Number of nodes in X Dimension. If you are using a single ring, please # specify number of nodes in ring. --dimensionX 2;+-dimensionX 3; # Number of nodes in Y Dimension.
I then had to restart the DIS network manager to see these changes take affect:
$ sudo /etc/init.d/dis_networkmgr restart
I then rebooted c1ioo one more time, after which c1ioo showed up in the dxadmin GUI.
At this point I tried adding a dolphin IPC connection between c1als and c1lsc to see if it worked. Unfortunately everything crashed every time I tried to run the models (including models on other machines!). The problem was that I had forgotten to tell the c1ioo IOP (c1x03) to use PCIe RFM (i.e. Dolphin). This is done by adding the following flag to the cdsParamters block in the IOP:
Once this was added, and the IOP was rebuilt/installed/restarted and came back up fine. The c1als model with the dolphin output also came up fine.
However, at this point I ran into the c1ioo mx_stream problem and started backing everything out.
I added all the _OUT16, _INMON, _EXCMON channels associated with the BS, ITMX, and ITMY channels in SUS_SLOW.ini in the /opt/rtcds/caltech/c1/chans/daq directory. Similarly, I added the channels associated with MC1, MC2, and MC3 to MCS_SLOW.ini and those associated with PRM and the SRM to RMS_SLOW.ini.
Lines pointing to these files were then added to the /opt/rtcds/caltech/c1/target/fb/master file and the frame builder restarted. It took about 4 tries before the frame builder stayed up using ./daqd -c ./daqdrc.
I generated the .ini files with a python script. Its at /opt/rtcds/caltech/c1/scripts/daqscripts/create_inmon_out16_daq_ini.py. It checks the C0EDCU.ini to find what slow channels already exist, then goes through a medm directory given at the command line looking for all _OUT16, _INMON, and _EXCMON channels, adding them if they aren't already accounted for, and then writing out the new file to the location of your choice.
Why EXCMON? I think that we should modify the filter coefficients so that the decimation filter that makes OUT16 is a 2nd order Butterworth at 1 Hz instead of whatever bogus thing it is now. Then we can delete the EXCMON from the DAQ and add either OUTPUT or OUTMON. That way we will have a low frequency signal (OUT16) and a sort of aliased RMS (OUTMON).
ALS looks OK. I tried to lock FPMI using ALS, but I feel like I need 6 hands to do it with current ALS stability. Now I have all computers being so slow.
It was fine for 7 hours after Jamie the Great fixed this, but fb went down couple times and DAQ status for all models now shows 0x4000. I tried restarting mx_stream and restarting fb, but they didn't help.
We rebooted c1psl, c1iscaux and c1aux which were all showing the typical symptom of responding to ping but not to telnet (and also blanked out epics fields on the MEDM screens). Keyed all these crates.
Restored burt snapshots for c1psl, PMC locked fine, and IMC is also locked now.
Johannes forgot to elog this yesterday, but he rebooted c1susaux following the usual procedure to avoid getting ITMX stuck.
Rebooted c1iscaux, c1auxex and c1auxey which were all not reponding to telnet. The watchdogs for the ETMs were turned off and then I keyed all 3 crates. All slow machines are reponding to telnet now. Both green lasers locked to the arms so I didn't do any burt restore.
Had to reboot c1psl, c1susaux, c1auxex, c1auxey and c1iscaux today. PMC has been relocked. ITMX didn't get stuck. According to this thread, there have been two instances in the last 10 days in which c1psl and c1susaux have failed. Since we seem to be doing this often lately, I've made a little script that uses the netcat utility to check which slow machines respond to telnet, it is located at /opt/rtcds/caltech/c1/scripts/cds/testSlowMachines.bash.
The script can be executed by ./testSlowMachines.bash.
After ~3months without any problems on the slow machine front, I had to reboot c1psl, c1susaux and c1iscaux today. The control room StripTool traces were not being displayed for all the PSL channels so I ran testSlowMachines.bash to check the status of the slow machines, which indicated that these three slow machines were dead. After rebooting the slow machines, I had to burt-restore the c1psl snapshot as usual to get the PMC to lock. Now, both PMC and IMC are locked. I also had to restart the StripTool traces (using scripts/general/startStrip.sh) to get the unresponsive traces back online.
Steve tells me that we probably have to do a reboot of the vacuum slow machines sometime soon too, as the MEDM screen for the Vacuum indicator channels are unresponsive.
Steve alerted me that the IMC wouldn't lock. Reboots for c1susaux, c1iool0 today. I tried using the reset button instead of keying the crates. This worked for c1iool0, but not for c1susaux. So I had to key the latter crate. The machine took a good 5-10 minutes before coming back up, but eventually it did. Now IMC locks fine.
Reboots for c1susaux, c1iscaux, c1auxex today. I took this opportunity to squish the Sat. Box. Cabling for MC2 (both on the Sat box end and also the vacuum feedthrough) as some work has been recently ongoing there, maybe something got accidently jiggled during the process and was causing MC2 alignment to jump around.
Relocked PMC to offload some of the DC offset, and re-aligned IMC after c1susaux reboot. PMC and IMC transmission back to nominal levels now. Let's see if MC2 is better behaved after this sat. box. voodoo.
Interestingly, since Feb 6, there were no slow machine reboots for almost 3 months, while there have been three reboots in the last three weeks. Not sure what (if anything) to make of that.
Reboots for c1psl, c1iool0, c1iscaux today. MC autolocker log was complaining that the C1:IOO-MC_AUTOLOCK_BEAT EPICS channel did not exist, and running the usual slow machine check script revealed that these three machines required reboots. PMC was relocked, IMC Autolocker was restarted on Megatron and everything seems fine now.
Reboots for c1susaux, c1iscaux today.
MC autolocker and FSS loops were stuck because c1psl was unresponsive. I rebooted it and did a burtrestore to enable PSL locking. Then the IMC locked fine.
c1susaux and c1iscaux were also unresponsive so I keyed those crates as well, after taking the usual steps to avoid ITMX getting stuck - but it still got stuck when the Sat. Box. connectors were reconnected after the reboot, so I had to shake it loose with bias slider jiggling. This is annoying and also not very robust. I am afraid we are going to knock the ITMX magnets off at some point. Is this problem indicative of the fact that the ITMX magnets were somehow glued on in a skewed way? Or can we make the situation better by just tweaking the OSEM-holding fixtures on the cage?
In any case, I've started listing stuff down here for things we may want to do when we vent next.
MC autolocker was not working - PCdrive was railed at its upper rail for ~2 hours judging by the wall StripTool trace. I tried restarting the init processes on megatron, but that didn't fix the problem. The reason seems to have been related to c1iool0 failing - after keying the crate, autolocker came back fine and MC caught lock almost immediately.
Additionally, c1susaux, c1auxex,c1auxey and c1iscaux are also down. I'm not planning on using the IFO tonight so I am not going to reboot these now.
Eurocrate key turning reboots for c1susaux, c1auxex,c1auxey, c1iscaux, c1iscaux2 and c1aux. Usual precautions were taken for ITMX. Did burtrestore for c1iscaux andc1iscaux2 in order to restore the LSC PD whitening gains.
Un-related to this work: input pointing into PMC was tweaked as the PMC_REFL spot was pretty bright.
MC Autolocker was umnhappy because c1iool0 was unresponsive and hence it couldn't write to the "C1:IOO-MC_AUTOLOCK_BEAT" channel. I keyed the crate and IMC locked almost immediately. I'm moving this channel into the RTCDS model as we did for the IFO_STATE EPICS channel so that the autolocker isn't dependant on c1iool0 (which was the whole point of migrating the IFO-STATE variable anyways). I also commented out all of these channels in /cvs/cds/caltech/target/c1iool0/autolocker.db so that there aren't duplicate channels.
Steve reported problems getting the X arm locked. Alignment sliders were inaccessible. Eurocrate key turning reboots for c1susaux, c1auxex,c1auxey, c1iscaux and c1aux. Usual precautions were taken for ITMX.
This is becoming a once-a-week thing .
[ Gautam , Steve ]
c1susaux & c1iscaux were rebooted manually.
Eurocrate key turning reboots today morning for and c1susaux, c1auxey and c1iscaux. These were responding to ping but not telnet-able. Usual precautions were taken to minimize risk of ITMX getting stuck.
MC autolocker got stuck (judging by wall StripTool traces, it has been this way for ~7 hours) because c1psl was unresponsive so I power cycled it. Now MC is locked.
c1psl, c1susaux, and c1auxey today
It's been a while - but today, all slow machines (with the exception of c1auxex) were un-telnetable. c1psl, c1iool0, c1susaux, c1iscaux1, c1iscaux2, c1aux and c1auxey were rebooted. Usual satellite box unplugging was done to avoid ITMX getting stuck.
All slow machines (except c1auxex) were dead today, so I had to key them all. While I was at it, I also decided to update MC autolocker screen. Kira pointed out that I needed to change the EPCIS input type (in the RTCDS model) to a "binary input", as opposed to an "analog input", which I did. Model recompilation and restart went smooth. I had to go into the epics record manually to change the two choices to "ENABLE" and "DISABLE" as opposed to the default "ON" and "OFF". Anyways, long story short, MC autolocker controls are a bit more intuitive now I think.
Reboot for c1susaux and c1iscaux today. ITMX precautions were followed. Reboots went smoothly.
IMC is shuttered while Jon does PLL characterization...
FSS slow wasn't running so PSL PZT voltage was swinging around a lot. Reason was that was c1psl unresponsive. I keyed the crate, now it's okay. Now ITMX is stuck - Johannes just told be about an un-elogged c1susaux reboot. Seems that ITMX got stuck at ~4:30pm yesterday PT. After some shaking, the optic was loosened. Please follow the procedure in future and if you do a reboot, please elog it and verify that the optic didn't get stuck.
Eurocrate key turning reboots today morning for and c1susaux, c1auxex and c1auxey. Usual precautions were taken to minimize risk of ITMX getting stuck.
The IFO hasn't been aligned in ~1week, so I recovered arm and PRM alignment by locking individual arms and also PRMI on carrier. I will try recovering DRMI locking in the evening.
As far as MC1 glitching is concerned, there hasn't been any major one (i.e. one in which MC1 is kicked by such a large amount that the autolocker can't lock the IMC) for the past 2 months - but the MC WFS offsets are an indication of when smaller glitches have taken place, and there were large DC offsets on the MC WFS servo outputs, which I offloaded to the DC MC suspension sliders using the MC WFS relief script.
I'd like for the save-restore routine that runs when the slow machines reboot to set the watchdog state default to OFF (currently, after a key-turning reboot, the watchdogs are enabled by default), but I'm not really sure how this whole system works. The relevant files seem to be in the directory /cvs/cds/caltech/target/c1susaux. There is a script in there called startup.cmd, which seems to be the initialization script that runs when the slow machine is rebooted. But looking at this file, it is not clear to me where the default values are loaded from? There are a few "saverestore" files in this directory as well:
Are the "default" channel values loaded from one of these?
Eurocrate key turning reboots today morning for c1psl and c1aux.c1auxex and c1auxey are also down but I didn't bother keying them for now. PSL FSS slow loop is now active again (its inactivity was what prompted me to check status of the slow machines).
Note that the EPCIS channels for PSL shutter are hosted on c1aux.But looks like the slow machine became unresponsive at some point during the weekend, so plotting the trend data for the PSL shutter channel would have you believe that the PSL shutter was open all the time. But the MC_REFL DC channel tells a different story - it suggests that the PSL shutter was closed at ~4AM on Sunday, presumably by the vacuum interlock system. I wonder:
The folding crane was fixed and tested this morning by the NNN rigging company. Pictures will be posted by Steve in the morning.
Afterwards, the ITM-east door was installed, jam-nuts checked. No high voltage was on for the in-vac PZTs.
The annulus spaces were roughed down to 350mTorr by Roughing Pump RP1. For this operation, we removed the low flow valve from the RP1 line.
After the spaces came down to ~400 mTorr, we closed their individual valves.
Warning: The VABSSC1 and VABSSC0 valves are incorrect and misleadingly drawn on the Vacuum overview screen.
Our idea is to have a much slower pumpdown this time than the last time when we had a hurricane kick up the dust. Looks like it worked, but next time we should do only 1/2 turn.
The pumpdown started at 4 PM (2300 UTC). At 10 PM, we (Jenne, Jan, and I) opened up the RV1 valve to full open. That's the second inflection point in the plot.
As per Steve's instructions, at 12:43 AM, I used the following steps to stop the pumpdown until the morning:
We have reached 200 Torr at 12 hours of slow pumping speed. Kiwamu stopped the pumping for 11 hrs last night .and I restarted it this morning.
Now RV1 is fully open with butterfly valve in place and the second roughing pump RP3 was just turned on.
How to stop pumping:
1, close RV1 manual valve with torque wheel
2, close V3
3, turn off roughing pumps RP1 & RP3
4, disconnect metal hose connection to butterfly valve
The pump down continued this morning by the removal of the butterfly valve. Two roughing pumps were used to reach 500 mTorr
The Maglev monitoring MEDM screen "Rotating" indicator is not working. It is on all times. Please look at Maglev controller monitor for real information.
Pump down is completed.
Configuration: vacuum normal after 86 days at atm CC1 = 1e-5 Torr
IFO is hungry for light (and maybe some goulash with a little paprikash too)
Atm 2 is showing the butterfly valve that closes down down the orifice at higher pressure to slow down the pumping speed.
See elog entry #2573
Bob and Steve closed BS chamber with the help of the manual Genie lift and the pump down started. The PSL shutter was closed and manual block was placed in the beam path. High voltage power supplies were checked to be off.
Pumping speed ~ 1 Torr/min was achieved at 1/8 of a turn opened roughing valve RV1
I have installed a slow start throttle valve in front of V3 This spring loaded valve will cut down on the flow at high pressures. There will be no more sand storme
and static built-up during pump down.
The PSL shutter was closed. The beam path blocked two places. High voltage power supplies to IOO and OMC PZT were checked to be off. Oplevs are off.
The south arm green cavity was misaligned yesterday
We would like to keep the vent speed at 1 torr / min. I'm venting with N2 now up to 25 PSI. We have 3 cylinder of instrument grade air inside the lab. Additional supply will arrive later. It can be as late as 1pm
Blocked PSL output beam into IFO
Checked: HV at IOO & OMC are off, jam nuts in position,
Closed V1 and VM3, opened VV1 to N2 regulator
We are venting at 1 Torr/min rate
For those of you who want to see plots from slower scan.
Small earth quakes and suspensions. Which one is the most free and most sensitive: ITMX
If I catch anyone putting small booties into the large bootie bin, I will make said person eat small booties.
Gautam and Steve,
The medm monitor & vac control screens were totally blank since ~ May 24, 2017 Experienced vacuum knowledge is required for this job.
IDENTIFY valve configuration:
How to confirm valve configuration when all vac mons are blank? Each valve has a manual-mechanical position indicator. Look at pressure readings and turbo pump controllers. VAC NORMAL configuration was confirmed based on these information.
Preparation: disconnect valves ( disconnect meaning: valve closes and stays paralized ) in this sequence VC2, VC1 power, VA6, V5, V4 & V1 power, at ifo pressure 7.3E-6 Torr-it ( it = InstruTech cold cathode gauge )
This gauge is independent from all other rack mounted instrumentation and it is still not logged.
Switching to this valve configuration with disconnected valves will insure NOT venting of the vacuum envelope by accidental glitching voltage drop or computer malfunction.
RESET v1Vac1 .........in 2-3 minutes........ ( v1Vac1 - 2 ) the vac control screen started reading pressures & position
Connected cables to valves (meaning: valve will open if it was open before it was disconnected and it will be control able from computer ) in the following order: V4, V1 power, V5, VA6, VC2 & VC1 power, at ifo 2E-5 Torr-it.....
....vac configuration is reading VAC NORMAL,
ifo 7.4E-6 Torr-it
We have to hook up the it-cold cathode gauge to be monitored - logged ! this should be the substitute for the out of order CC1 pressure gauge.
We corrected the MICH locking snap file C1configure_MI.req and saved an updated C1configure_MI.snap. Now the 'Restore MICH' script in IFO_CONFIGURE>!MICH>Restore MICH works. The corrections included adding the correct rows of PD_DOF matrices to be at the right settings (use AS55 as error signal). The MICH_A_GAIN and MICH_B_GAIN needed to be saved as well.
We also were able to get to PRMI SB resonance. PRM was misalgined earlier from optimal position and after some manual aligning, we were able to get it to lock just by hitting IFO_CONFIGURE>!PRMI>Restore PRMI SB (3f).
Yesterday, I had the great pleasure to solder a tiny 4 x 4 mm op amp with 16 legs (AD8336).
I figured out that the best and fastest way to do it is
This morning the pencil soldering iron of our Weller WD2000M Soldering Station suddenly stopped working and got cold after I turned the station on. The unit's display is showing a message that says "TIP". i checked out the manual, but it doesn't say anything about that. I don't know what it means. Perhaps burned tip?
Before asking Steve to buy a new one, I emailed Weller about the problem.
There should be a supply of extra tips in the Blue Spinny Cabiney (I can never remember it's French name....) The drawer is something like the top row of one of the bottom sets of drawers. You can pick the shape of tip you want, and stick it in.