git repo - https://git.ligo.org/40m/vac
Finally incorporated the FRGs into the main modbusIOC service and everything seems to be working fine. I have also removed the old sensors (CC1,CC2,CC3,CC4,PTP1,IG1) from the serial client list and their corresponding EPICS channels. Furthermore, the interlock service python script has been updated so that all occurrence of old sensors (turns out to be only CC1) were replaced by their corresponding new FRG sensor (FRG1) and a redundnacy was also enacted for P1a where the interlock condition is replicated with P1a being replaced with FRG1 because they both sense the main volume pressure.
[Tega, Yuta, Paco]
We tried aligning the LO and AS beams on to the BHD beamsplitter. During the alignment process, we noticed that the damping loop for AS1 was not working. Paco drew our attention to the fact that the UR OSEM signal was alway close to zero, so we checked to ensure that the magnet was still within the OSEM recess and it looks OK. Next we checked the electrical connection at the interface between the copper OSEM cables to the blue in-vacuum flat cable and this too looks alright also. Since the AS1 coil driver was recently modified, it is possible we might find the problem there, so I'll ask Koji about this.
So Koji clarified that the coil driver board and SATAMP boards are different so we should connect this issue to the coil driver board.
[JC, Tega, Ian, Paco]
We found that the UR cable was clamped to the table by one of the ITMY OPLEV steering mirror mounts that was recently installed. After freeing the cable, the UR signal is now active again.
This indicates that the AS1 UR OSEM problem is localized in the chamber. Please check if the DSUB pins are touching the table or something else.
We aligned the BS, ITMY, and ETMY PIT and YAW to get the flashing on X-arm whilst also keeping the flashing of Y-arm. From attachment 1, it is clear that POXDC photodiode is not receiveing any light, so our next task is to work on POX alignment.
We needed to sort out the POXDC signal so we could work on X-arm alignment. Given that POXDC channel value was approx 6 compared to POYDC value of approx. 180, we decided to open the ITMX chamber to see if we could improve the situation. We worked on the alignment of POX beam but could not improve the DC level which suggests that this was already optimized for. As an aside, we also noticed some stray IR beam from the BS chamber, just above the POX beam which we cold not identify.
Next we moved on to the POP beam alignment, where we noticed that the beam level on LO1 and POP_SM4 was a bit on the high side. Basically, the beam was completely missing the 1" POP_SM4 mirror and was close to the top edge of LO1. So we changed TT2 pitch value from 0.0143 to -0.2357 in order to move the beam position on POP_SM4 mirror. This changed the input alignment, so we compensated using PR2 (0.0 -> 49.0) and PR3 (-5976.560 -> -5689.800). This did not get back the alignment as anticipated, so we moved ITMY pitch from 0.9297 to 0.9107. All of these alignment changes moved the POP beam down by approx 1/5 of an inch from outside the mirro to the edge of POP_SM4 mirror, where about half of the beam is clipped.
We need to repeat these aligment procedures with say 1.5 time the change in TT2 pitch to center the beam on POP_SM4 mirror.
In order to setup POP camera and RFPD on the ITMX table, we decided to first work on the IMC and X/Y-arm alignment.
We zeroed IMC WFS outputs and aligned IMC manually to get IMC transmission of 1200 and reflection of 0.35.
We used the new video game tool that moves the pairs of mirrors - PR3 & ETMY, ITMY & ETMY - in common and differential modes. This brought the Y-arm flashing to 0.8. Note that we used the _OFFSET bias values for PR3 & ETMY alignment instead of the _COMM bias values.
We repeated the same procedure of moving the pairs of mirrors - BS & ETMX, ITMX & ETMX - in common and differential modes but manually this time. This brought the X-arm flashing to ~1.0.
Started work on the relocating the green transmission optics, cameras and PDs. Before removing the any of the optics, we checked and confirmed that the PDs and Cams are indeed connected to the GRN TRX/Y medm channels. Then added labels to the cables before moving them.
Relocated Optics & PDs & Cameras:
Followed the steps below to complete the ITMX optlev installation. The ITMX optlev return beam now reaches its QPD without being blocked by the input steering mirror.
Although, I centered the ITMX optlev readout, this was not done when the XARM flashing is maximized bcos the IMC chamber was being worked on, so this should be done later when the IR beam is back.
We finally managed to steer the AS beam from ITMY chamber, through BS and IMC chambers, to the in-air AP table.
We moved the AS5 mirror north to its nominal position and we also moved the ASL lens on BS chamber back to its nominal position. Attached photos are taken after today's alignment work.
[JC, Tega, Chub]
Today we installed the 200 lbs doors on the end station chambers.
[Vacuum gauge sensors]
Paco informed me that the FRG sensor EPICS channels are not available on dataviewer, so I added them to slow channels ini file (/opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini). I also commented out the old CC1, CC2, CC3 and CC4 gauges. A service restart is required for them to become available but this cannot be done right now because it would adversely affect the progress of the upgrade work. So this would be done at a later date.
git repo : https://git.ligo.org/40m/scripts/-/tree/main/SUS/OutMatCalc
Here is an update on our recent attempt at diagonalization of the SUS output matrices. There are two different parts to this: the first is coil balancing using existing F2P code which stopped working because of an old-style use of the print function and the second which should now focus on the mixing amongst the various degrees of freedoms (dof) without a DC/AC split I believe. The F2P codes are now working and have been consolidated in the git repo.
I updated the database files for the 7 BHD optics to separate the OSEM variance trigger and the LATCH_OFF trigger operations so that an OSEM variance value exceeding the max of say 200 cnts turns off the damping loop whereas pressing the LATCH_OFF button cuts power to the coil. I restarted the modbusIOC service on c1susaux2 and checked that the new functionality is behaving as expected. So far so good.
Figure out the next layer of watchdogging needed for the BHD optics.
[Anchal, JC, Ian, Paco]
We have now fixed all issues with the PD mons of c1susaux2 chassis. The slow channels are now reading same values as the fast channels and there is no arbitrary offset. The binary channels are all working now except for LO2 UL which keeps showing ENABLE OFF. This was an issue earlier on LO1 UR and it magically disappeared and now is on LO2. I think the optical isolators aren't very robust. But anyways, now our watchdog system is fully functional for all BHD suspended optics.
Implemented ramp down of coil bias voltage when the BHD optics watchdog is tripped. Also added a watchdog reset button to the SUS medm screen that turns on damping and ramps up the coil PIT/YAW bias voltages to their nominal values. I believe this concludes the watchdog work.
Koji noticed that the FRG sensors were not updating due to reference to an obsolete modbusIOC_XGS service, which was used temporarily to test the operation of the serial XGS sensor readout to EPICS. The information in this service was later moved into modbusIOC.service but the dependence on the modbusIOC_XGS.service was not removed from the serial_XGS600.service. This did not present any issue before the shutdown, probably bcos the obsolete service was already loaded but after the restart of c1vac, the obsolete service file modbusIOC_XGS.service was no longer available. This resulted in serial_XGS600.service throwing a failure to load error for the missing obsolete modbusIOC_XGS service. The fix involved replacing two references to 'modbusIOC_XGS' with 'modbusIOC' in /opt/target/services/serial_XGS600.service.
I also noticed that the date logged in the commit message was Oct 2010 and that I could not do a push from c1vac due to an error in resolving git.ligo.org. I was able to push the commit from my laptop git repo but was unable to do a pull on c1vac to keep it synced with the remote repo.
IPC issue still unresolved.
Updated shared memory tag so that 'SUS' -> 'SU2' in c1hpc, c1bac and c1su2. Removed obsolete 'HPC/BAC-SUS' references from IPC file, C1.ipc. Restarted the FE models but the c1sus2 machine froze, so I did a manual reboot. This brought down the vertex machines---which I restarted using /opt/rtcds/caltech/c1/scripts/cds/rebootC1LSC.sh---and the end machines which I restarted manually. Everything but the BHD optics now have their previous values. So need to burtrestore these.
# IPC file:
# Model file locations:
# Log files:
SUS overview medm screen :
We were able to fix the shared memory issue by updating the receiver model name from ''SUS' to 'SU2' and the ADC zero issue by including both ADC0 and ADC1 in the c1hpc and c1bac models as well as removing the grounding of the unused ADC channels (including chn#16 and chn#17 which are actually used in c1hpc) in c1su2. We also used shared memory to move the DCPD_A/B error signals (after signal conditioning and mixing A/B; now named A_ERR and B_ERR) from c1hpc to c1bac.
C1:HPC-DCPD_A_IN1 and C1:HPC-DCPD_B_IN1 are now available (they are essentially the same as C1:LSC-DCPD_A_IN1 and C1:LSC-DCPD_B_IN1, except for they are ADC-ed with different ADC; see elog 40m/16954 and Attachment #1).
Dolphin IPC error in seding signal from c1hpc to c1lsc still remains.
From the 40m wiki, I was able to use the instructions here to map out what to do to get the IPC issue resolved. Here is a summary of my findings.
I updated the /etc/dis/dishost.conf file on the frame builder machine to include the c1sus2 machine which runs the sender model, c1hpc, see below. After this, the file becomes available on c1sus2 machine, see attachment 1, and the c1sus2 node shows up in the dxadmin GUI, see attachment 2. However, the c1sus2 machine was not active. I noticed that the log file for the dis_nodemgr service, see attachment 3, which is responsible for setting things up, indicated that the dis_irm service may not be up, so I checked and confirmed that this was indeed the case, see attachment 4. I tried restarting this service but was unsuccessful. I restarted the machine but this did not help either. I have reached out to Jonathan Hanks for assistance.
[Paco, Tega, Yuta]
Today, we made a custom MEDM screen for the BHD Homodyne Phase Control, which is basically an overview of the c1hpc model. See Attachments 1 & 2 for details.
[Yuta, Tega, Yehonathan]
To investigate the BHD power imbalance and clipping issues, we did some shaking test of the mirrors in the LO path and AS paths. The results suggests the following:
We performed both PIT and YAW shaking of all mirrors and looked at the output at DCPD_A and DCPD_B, see table below for details. Since we only see the dithering signal in DCPD_A, it suggests that the clipping is ocurring after the BHD BS and is also confined to the path between BHD BS and DCPD_A. We also swapped the camera location from DCPD_A to DCPD_B on ITMY table and confirmed that the beam was clipped for DCPD_A but not for DCPD_B.
This result discounts the possiblity of clipping being responsible for the power imbalance and therefore suggests that the power imbalance may actually be due to BHD BS not being 50:50. From the measurement in elog 17040, the transmission of BHD BS is 440.3% and the reflectivity is 560.3%. Note that DCPD_A is the transmission of BHD BS for AS beam, whereas DCPD_B is the reflection of BHD BS for AS beam, elog 16932.
We expect the shaking of PR2 to give no signal in either DCPD_A or DCPD_B when the LO beam is purely in trasmission, however, we see a signal in DCPD_A sugesting that the LO beam transit path through PR2 may not be as expected, i.e. the beam might be exiting the side of PR2 instead of the AR coated surface.
Finally, we measured the coherence between the dithering dof and DCPD_A/B & POP, see attachment 2, where we noticed that both DCPD_A/B have high coherence in the 1Hz-10Hz frequency band whereas ther was no coherence in POP as expected. This suggests that there may also be some small clipping in DCPD_B path.
LO Beam Shaking (LO1, LO2, PR2):
Signal in DCPD_A & No signal in DCPD_B
AS Beam Shaking (AS1 and AS4)
From our previous work (elog 17044) of shaking PR2 and seeing a signal in DCPD_A and the fact that LO beam power is far smaller than the expected nominal value, we decided to use TT1 and TT2 to realign the LO beam. This resulted in LO beam power going up by a factor of 6 and an improvement in the LO beam shape. We are still unable to find LO and AS alignment which realize BHD fringe with no clipping everywhere.
Deformed LO beam issue: Following the TT1 and TT2 alignments, used PR2 and PR3 to recover the transmission of the X and Y arms to 1. We also used LO1 and LO2 offsets to further reduce the beam deformation by eliminating the HOM concentric fringes that surounded the LO beam and to maximize the DCPD outputs. BHD optics in ITMY table was tweaked a lot to keep the LO beam centered on the BHD DCPDs and camera. The improved LO beam is still astigmatic in the yaw direction but at least now looks like a TEM00 mode. We also repositioned the DCPD_A path camera lens to remove the circular diffused fringes due to lens clipping. After the alignment, power was measured to be the following. It also reduced the coherence between DCPD outputs and suspension motions (see attached).
C1:HPC-DCPD_A_OUT16 +127.50 +96.24 (ITMX single bounce consistent to 40m/17040)
C1:HPC-DCPD_B_OUT16 +120.51 +141.52
Power at viewport A 504 uW (almost as expected 40m/17040)
Power at viewport B 385 uW
AP table AS beam clipping: We also noticed clipping in the AS beam in AP table which we removed by moving SR2 and AS1 in YAW and then used AS4 to keep the BHD AS beam centered in the BHD DCPDs.
BHD fringe: After overlaping the LO and AS beams, we saw diagonal fringes indicating beam tilt of LO wrt AS, so we tried to remove the AS beam tilt using AS1 and AS4 but failed to do so because the AS4 mirror seemed to completely distort the beam, so intead we decided to use SR2 and AS1 to remove the tilt between LO and AS beams, which realized BHD fringe. But the motion of SR2 and AS1 then moved the AS beam that it is no longer seen in AP table. The alignment to realize LO and AP AS beam without clipping, and that to realize BHD fringe are attached.
We decided to give the dolphin debugging another go. Firstly, we noticed that c1sus2 was no longer recogonising the dolphin card, which can be checked using
lspci | grep Stargen
or looking at the status light on the dolphin card of c1sus2, which was orange for both ports A and B.
We decided to do a hard reboot of c1sus2 and turned off the DAQ chassis for a few minutes, then restared c1sus2. This solved the card recognition problem as well as the 'dis_irm' driver loading issue (I think the driver does not get loaded if the system does not recognise a valid card, as I also saw the missing dis_irm driver module on c1testand).
Next, we confirmed the status of all dolphin cards on fb1, using
It looks like the dolphin card on c1sus2 has now been configured and is availabe to all other nodes. We then restated the all FE machines and models to see if we are in the clear. Unfortunately, we are not so lucky since the problem persisted.
Looking at the output of 'dmesg', we could only identity two notable difference between the operational dolphin cards on c1sus/c1ioo/c1lsc and c1sus2, namely: the card number being equal to zero and the memory addresses which are also zero, see image below.
Anyways, at least we can now eliminate driver issues and would move on to debugging the models next.
[Yuta, Tega, Chris]
We did it!
Following Chris's suggestion, we added "pciRfm=1" to the CDS parameter block in c1x07.mdl - the IOP model for c1sus2. Then restarted the FE machines and this solved the dolphin IPC problem on c1sus2. We no longer see the RT Netstat error for 'C1:HPC-LSC_DCPD_A' and 'C1:HPC-LSC_DCPD_B' on the LSC IPC status page, see attachement 1.
Attachment 2 shows the module dependencies before and after the change was made, which confirms that the IOP model was not using the dolphin driver before the change.
We encountered a burt restore problem with missing snapfiles from yesterday when we tried restoring the EPICS values after restarting the FE machines. Koji helped us debug the problem, but the summary is that restarting the FE models somehow fixed the issue.
Koji and JC looked around the lab today and found some supermicro boxes which I was told to look into to see if they have any useful computers.
Boxes next to Y-arm cabinets (3 boxes: one empty)
We were expecting to see a smaller machine in the first box - like top machine in attachement 1 - but it turns out to actually contain the front-end we need, see bottom machine in attachment 1. This is the same machine as c1bhd currently on the teststand. Attachment 2 is an image of the machine in the second box (maybe a new machine for frambuilder?). The third box is empty.
Boxes next to X-arm cabinets (3 boxes)
Attachement 3 shows the 3 boxes each of which contains the same FE machine we saw earlier at the bottom of attachement 1. The middle box contains the note shown in attacment 4.
Box opposite Y-arm cabinets (1 empty box)
In summary, it looks like we have 3 new front-ends, 1 new front-end with networking issue and 1 new tower machine (possibly a frame builder replacement).
Added 3 FE machines - c1ioo, c1lsc, c1sus - to the teststand following the instructions in elog15947. Note that we also updated /etc/hosts on chiara by adding the names and ip of the new FE since we wish to ssh from there given that chiara is where we land when we connect to c1teststand.
Two of the FE machines - c1lsc & c1ioo - have the 6-core X5680 @ 3.3GHz processor and the BIOS were already mostly configured because they came from LLO I believe. The third machine - c1sus - has the 6-core X5650 @ 2.67GHz processor and required a complete BIOS config according to the doc.
Next Step: I think the next step is to get the latest RTS working on the new fb1 (tower machine), then boot the frontends from there.
KVM switch note:
All current front-ends have the ps/2 keyboard and mouse connectors except for fb1, which only has usb ports. So we may not be able to connect to fb1 using a ps/2 KVM switch that works for all the current front-ends. The new tower machine does have a ps/2 connector so if we decide to use that as the bootserver and framebuilder, then we should be fine.
We diagnosed the suspension damping of the IMC/BHD/recycling optics by kicking the various degree of freedom (dof) and then tuning the gain so that we get a residual Q of approx. 5 in the cases where this can be achieved.
MC2: SIDE-YAW coupling, but OK
MC3: Too much coupling between dofs, NEEDS ATTENTION
AS1: POS-PIT coupling, close to oscillation, cnt2um off, NEEDS ATTENTION
AS4: PIT-YAW coupling, cannot increase YAW gain because of coupling, No cnt2um, No Cheby, NEEDS ATTENTION
PR2: No cnt2um, No Cheby
PR3: POS-PIT coupling, cannot increase POS/PIT/YAW gain because of coupling, No cnt2um, No Cheby, NEEDS ATTENTION
SR2: No cnt2um
Here is a summary of what needs doing following the chat with Jamie today.
Jamie brought over the KVM switch shown in the attachment and I tested all 16 ports and 7 cables and can confirm that they all work as expected.
1. Do a rack space budget to get a clear picture of how many front-ends we can fit into the new rack
2. Look into what needs doing and how much effort would be needed to clear rack 1X7 and use that instead of the new rack. The power down on Friday would present a good opportunity to do this work on Monday, so get the info ready before then.
3. Start mounting front-ends, KVM and dolphin network switch
4. Add the BOX rack layout to the CDS upgrade page.
I keep getting confused about the purpose of the teststand. The view I am adopting going forward is its use as a platform for testing the compatibility of new hardware upgrade, instead of thinking of it as an independent system that works with old hardware.
The initial idea of clearing 1X7 cannot be done for now, because I missed the deadline for providing a detailed enough plan before Monday power up of the lab, so we are just going to go ahead and use the new rack as was initially intended and get the latest hardware and software tested here.
We mounted the DAQ, subnet and dolphin IX switches, see attachement 1. The mounting ears that came with the dolphin switch did not fit and so could not be used for mounting. We looked around the lab and decided to used one of the NavePoint mounting brackets which we found next to the teststand, see attachment 2.
We plan to move the new rack to the current location of the teststand and use the power connection from there. It is also closer to 1X7 so that moving the front-ends and switches to 1X7 should be straight forward after we complete all CDS upgrade testing.
- Disk full
I updated the configuration file '/etc/logrotate.d/rsyslog' to set a file sise limit of 50M on 'syslog' and 'daemon.log' since these are the two log files that capture caget & caput terminal outputs. I also reduce the number of backup files to 2.
controls@c1vac:~$ cat /etc/logrotate.d/rsyslog
invoke-rc.d rsyslog rotate > /dev/null
invoke-rc.d rsyslog rotate > /dev/null
invoke-rc.d rsyslog rotate > /dev/null
- Vacuum gauge
The XGS-600 can handle 6 FRGs and we currently have 5 of them connected. Yes, having a spare would be good. I'll see about placing an order for these then.
- Disk Full: Just use the usual /etc/logrotate thing
I rather feel not replacing P1a. We used to have Ps and CCs as they didn't cover the entire pressure range. However, this new FRG (=Full Range Gauge) does cover from 1atm to 4nTorr.
Why don't we have a couple of FRG spares, instead?
Questions to Tega: How many FRGs can our XGS-600 controller handle?
Moved the rack to the location of the test stand just behind 1X7 and plan to remove the other two small test stand racks to create some space there. We then mounted the c1bhd I/O chassis and 4 front-end machines on the test stand (see attachment 1).
Installed the dolphin IX cards on all 4 front-end machines: c1bhd, c1ioo, c1sus, c1lsc. I also removed the dolphin DX card that was previously installed on c1bhd.
Found a single OneStop host card with a mini PCI slot mounting plate in a storage box (see attachment 2). Since this only fits into the dual PCI riser card slot on c1bhd, I swapped out the full-length PCI slot OneStop host card on c1bhd and installed it on c1lsc, (see attachments 3 & 4).
[JC, Tega, Chris]
After moving the test stand front-ends, chiara (name server) and fb1 (boot server) to the new rack behind 1X7, we powered everything up and checked that we can reach c1teststand via pianosa and that the front-ends are still able to boot from fb1. After confirming these tests, we decided to start the software upgrade to debian 10. We installed buster on fb1 and are now in the process of setting up diskless boot. I have been looking around for cds instructions on how to do this and I found the CdsFrontEndDebian10page which contains most of the info we require. The page suggests that it may be cleaner to start the debian10 installation on a front-end that is connected to an I/O chassis with at least 1 ADC and 1 DAC card, then move the installation disk to the boot server and continue from there, so I moved the disk from fb1 to one of the front-ends but I had trouble getting it to boot. I decided to do a clean install on another disk on the c1lsc front-end which has a host adapter card that can be connected to the c1bhd I/O chassis. We can then mount this disk on fb1 and use it to setup the diskless boot OS.
Here is a proposal for what we would like to do in terms of reshuffling a few rack-mounted equipments for the CDS upgrade.
I believe all of this can be done in one go followed by CDS validation. Please comment so we can improve the plan. Should we move FB1 to 1X7 and remove old FB & JetStor during this work?
Attachment 1: Reshuffling proposal
Attachment 2: Front of 1X7 Rack
Attachment 3: Rear of 1X7 Rack
Attachment 4: Front of 1X6 Rack
Attachment 5: Rear of 1X6 Rack
Attachment 6: Martian switch connections
We got the 3 front-ends from LLO today. The contents of each box are:
We received the remaining 3 front-ends from LHO today. They each have a timing card and an OSS host adapter card installed. We also receive 3 dolphin DX cards. As with the previous packages from LLO, each box contains a rack mounting kit for the supermicro machine.
[Tega, Paco, JC]
We moved the GPS network time server and the Frequency distribution amplifier from 1X7 to 1X6 and the PEM AA, ADC adapter and Martian network switch from 1X6 to 1X7. Also mounted the dolphin IX switch at the rear of 1X7 together with the DAQ and martian switches. This cleared up enough space to mount all the front-ends, however, we found that the mounting brackets for the frontends do not fit in the 1X7 rack, so I have decided to mount them on the upper part of the test stand for now while we come up with a fix for this problem. Attachments 1 to 3 show the current state of racks 1X6, 1X7 and the teststand.
Attachment 1: Front of racks 1X6 and 1X7
Attachment 2: Rear of rack 1X7
Attachment 3: Front of teststand rack
Thursday [CAN I GET THE IFO ON THIS DAY PLEASE?]
[Tega, Radhika, JC]
We wired the front-ends for power, DAQ and martian network connections. Then moved the I/O chassis from the buttom of the rack to the middle just above the KVM switch so we can leave the top og the I/O chassis open for access to the ports of OSS target adapter card for testing the extension fiber cables.
Attachment 1 (top -> bottom)
When I turned on the test stand with the new front-ends, after a few minutes, the power to 1x7 was cut off due to overloading I assume. This brought down nodus, chiara and FB1. After Paco reset the tripped switch, everything came back without us actually doing anything, which is an interesting observation.
After this event, I moved the test stand power plug to the side wall rail socket. This seems fine so far. I then brought chiara (clone) and FB1 (clone) online. Here are some changes I made to get things going:
sudo service isc-dhcp-server restart
Getting the new front-ends booting off FB1 clone:
1. I found that the KVM screen was flooded with setup info about the dolphin cards on the LLO machines. This actually prevented login using the KVM switch for two of these machines. Strangely, one of them 'c1sus' seemed to be fine, see attachment 2, so I guessed this was bcos the dolphin network was already configured earlier when we were testing the dolphin communications. So I decided to configure the remaining dolphin cards. To do so, we do the following
1a. Ideally running
sudo /opt/DIS/sbin/dis_mkconf -fabrics 1 -sclw 8 -stt 1 -nodes c1lsc c1sus c1ioo c1iscex c1iscey c1sus2 -nosessions
should set up all the nodes, but this did not happen. In fact, I could no longer use the '/opt/DIS/sbin/dis_admin' GUI after running this operation and restarting the 'dis_networkmgr.service' via
sudo systemctl restart dis_networkmgr.service
so I logged into each front-end and configured the dolphin adapter there using
After which I shut down FB1 (clone) bcos restarting it earlier didn't work, I waited a few minutes and then started it. Everything was fine afterward, although I am not quite sure what solved the issue as I tried a few things and I was glad to see the problem go!
1b. I later found after configuring all the dolphin nodes that 2 of them failed the '/opt/DIS/sbin/dis_diag' test with an error message suggesting three possible issues of which one was 'faulty cable'. I looked at the units in question and found that swapping both cables with the remaining spares solved the problem. So it seems like these cables are faulty (need to double-check this). Attachment 3 shows the current state of the dolphin nodes on the front-ends and the dolphin switch.
2. I noticed that the NFS mount service for the mount points '/opt/rtcds' and '/opt/rtapps' in /etc/fstab exited with an error, so I ran
sudo mount -a
3. edit '/etc/hosts' to include c1iscex and c1iscey as these were missing
To test the PCIe extension fiber cables that connect the front-ends to their respective I/O chassis, we run the following command (after booting the machine with the cable connected):
controls@c1lsc:~$ lspci -vn | grep 10b5:3
If we see the output above, then both the cable and OSS card are fine (We know from previous tests that the OSS card on the I/O chassis is good). Since we only have one I/O chassis, we repeat the step above 8 times, also cycling through the six new front-end as we go so that we are also testing the installed OSS host adapter cards. I was able to test 4 cables and 4 OSS host cards (c1lsc, c1sus, c1ioo, c1sus2), but the remaining results were inconclusive (i.e. it seems to suggest that 3 out of the remaining 5 fiber cables are faulty, which in itself would be considered unfortunate but I found the reliability if the test to be in question when I went back to test the functionality to the 2 remaining OSS host cards using a cable that passed the test earlier and it didn't pass. After a few retries, I decided to call it a day b4 I lose my mind) and need to be redone again tomorrow.
Note: We were unable to lay the cables today bcos these tests were not complete, so we are a bit behind the plan. Would see if we can catch up tomorrow.
We laid 4 out of 6 fiber cables today. The remaining 2 cables are for the I/O chassis on the vertex so we would test the cables the lay it tomorrow. We were also able to identify the problems with the 2 supposedly faulty cable, which are not faulty. One of them had a small bend in the connector that I was able to straighten out with a small plier and the other was a loose connection in the switch part. So there was no faulty cable, which is great! Chris wrote a matlab script that does the migration of all the model files. I am going through them, i.e. looking at the CDS parameter block to check that all is well. Next task is to build and install the updated models. Also need to update the '/opt/rtcds' and '/opt/rtapps' directory to the latest in the 40m chiara system.
We built, installed and started all the 40m models on the teststand today. The configuration we employ is to connect c1sus to the teststand I/O chassis and use dolphin to send the timing to other frontends. To get this far, we encounterd a few problems that was solved by doing the following:
0. Fixed frontend timing sync to FB1 via ntp
1. Set the rtcds enviroment variable `CDS_SRC=/opt/rtcds/userapps/trunk/cds/common/src` in the file '/etc/advligorts/env'
2. Resolved chgrp error during models installation using sticky bits on chiara, i.e. `sudo chmod g+s -R /opt/rtcds/caltech/c1/target`
3. Replaced `sqrt` with `lsqrt` in `RMSeval.c` to eliminate compilation error for c1ioo
4. Created a symlink for 'activateDQ.py' and 'generate_KisselButton.py' in '/opt/rtcds/caltech/c1/post_build'
5. Installed and configured dolphin for new frontend 'c1shimmer'
6. Replaced 'RFM0' with 'PCIE' in the ipc file, '/opt/rtcds/caltech/c1/chans/ipc/C1.ipc'
We still have a few issues namely:
1. The user models are not running smoothly. the cpu usage jumps to its maximum value every second or so.
2. c1omc seems to be unable to get its timing from its IOP model (Issue resolved by changing the CDS block parameter 'specific_cpu' from 6 to 4 bcos the new FEs only have 6 cores, 0-5)
3. The need to load the `dolphin-proxy-km` library and start the `rts-dolphin_daemon` service whenever we reboot the front-end
Something strange happened to the ITMX osem reading around 4.pm. PDT as shown below.
Also the there was no response of the reading as we adjusted the PITCH and YAW. :(
Note that we restarted the slow machine: c1susaux,c1ausex this afternoon because of the unresponced interface.
In order to figure out the difference betweent simulated result and measurement, I tried to measuren the electronic noise by following ways as show in attachment 1
1.measure from the satellite box by SR785 at ETMY ,calibrate to counts by divide by 3267.8. while at that conditin, the set up is in suspension.
2. measure after ADC by diagnostics test tools, with set up on table in history and on uspension currently.
3. use the caculated butterfly channel.
the results are shown in attachmemt 2. The overall nosie level are still much higher than simulation.
If we have some data with one of the optics clamped and the open light hitting the PD, or with the OSEMs removed and sitting on the table, that would be useful for evaluating the end-to-end noise of the OSEM circuit. It seems like we probably have that due to the vent work, so please post the times here if you have them.
The ETMX OSEMs have been attached to its Satellite box and plugged in for the last 10 days or so, with the PD exposed to the unobstructed LED. I pulled the spectrum of one of the sensors (mean detrended, I assume this takes care of removing the DC value?). The DQed channels claim to record um (the raw ADC counts are multiplied by a conversion factor of 0.36). For comparison, re-converted the y-axis for the measured curve to counts, and multiplied the total noise curve from the LISO simulation by a factor of 3267.8cts/V (2^16cts/20V) so the Y axis is noise in units of counts/rtHz. At 1Hz, there is more than an order of magnitude difference between the simulation and the measurement which makes me suspect my y-axis conversion, but I think I've done this correctly. Can such a large discrepancy be solely due to thick film resistors?
In the attachment please find IMC ASC simulation plots. Let me know what you think, if you want some other plots, and if you need any clarification.
In the attachment please find a comparison of error signals of simulation and reality. For C1:IOO-WFS1/2_PIT_IN1 excess signal ('belly') between a few Hz up to 70-80 Hz might be caused by air turbulence (which is not included in the simulation).
In the attachment please find a comparison of error signals of simulation and reality after including air turbulence as input noise.
In the attachment please find plots comparing controller output, local damping output, and error signals.
Input noises of the simulation are seismic noise, osem noise, input power fluctuations, sensing noises of WFSs and QPD, and air turbulence noise for WFSs. There is also optical torque noise (radiation pressure effect).
The procedure to get optical gains and sensing noises:
Having the actuator response A rad/cnts @ 3 Hz. I was shaking MC1/2/3 in pitch with B cnts @ 3 Hz and getting WFS1/2 QPD signals of C cnts @ 3 Hz, which means WFS1/2 QPD optical gain is D cnts/rad = C / (A * B) cnts/rad. So, if WFS1/2 QPD IN1 has a noise spectrum (at higher freqs) of E cnts/rtHz, that corresponds to E/D rad/rtHz of sensing noise for WFS1/2 QPD.
Actuator response [rad/cts] I was getting shaking mirrors at 3 Hz and measuring amplitudes of OSEM output (knowing the geometry of the mirror). I scaled it to DC. From here I was getting ct2tau_mc (knowing the mirror's moment of inertia, Q, and natural pitch frequencies). OSEM calibration factors [cts/rad] I was getting from the input matrix and geometry of the mirror.
The flat noise at higher frequencies from the local damping and controller output channels is presumably quantization/loss of digits/numerical precision noise which I don't include in simulations for now?!
Regarding air turbulence, in KAGRA it has been reported that air turbulence introduces phase fluctuations in laser fields that propagate in air. According to Kolmogorov’s theory, the PSD of phase fluctuations caused by air turbulence scales as ∝ L*V^(5/3)*f^(−8/3). Here, L is the optical path length and V is a constant wind speed. Since it is not obvious how can one estimate typical V in the beam paths I was taking this excess noise from the error signals data between 10 Hz and 50 Hz, extrapolated it taking into account ∝ f^(−8/3) (not for frequencies below 2 Hz, where I just put constant, since it would go too high). I expect that I won't be able to get a parameterized model that also predicts the absolute value. The slope is all I can hope to match, and this I already know. QPD chamber is much smaller (and better isolated?) and there is no this excess noise.
Regarding other things in simulations (very briefly): beam-spots are calculated from angular motions, length change is calculated from beam-spots and angular motion, cavity power depends on length change and input power, and torque on the mirrors depends on beam-spots and cavity power. From other things, local-sensor basis conversion (and vice versa) is worth noting.
I measured electronics noise of WFSs and QPD (of the WFS/QPD, whitening, ADC...) by closing PSL and measuring the error signal. It was needed to put the offset in C1:IOO-MC_TRANS_SUMFILT_OFFSET to 14000 cts (without offset the sum of quadrants would give zero, and 14000 cts is the value when the cavity is locked). For WFS that are RF, if there is intensity noise at low frequencies, it is not affecting the measurement.
In the attachment please find the power spectrum of the error signal when the PSL shutter is on and off.
For whitening electronics noise for WFS1, I get (attachment). This doesn't seem right, right?
We measured the electronics noise of the demodulation board, whitening board, and ADC for WFSs, and OPLEV board and ADC for DC QPD in MC2 transmission. We were using SR785.
Regarding the demodulation board, we did 2 series of measurements. For the first series of measurements, we were blocking WFS (attachment 1) and measuring noise at the output of the demod board (attachment 2a). This measurement includes dark noise of the WFS, electronics noise of demod board, and phase noise from LO. For the second series of the measurements, we were unplugging input to the demod board (attachment 2b & 2c is how they looked like before unplugging) (the mistake we made here is not putting 50-ohm terminator) and again measuring at the output of the demod board. This measurement doesn't include the dark noise of the WFS. We were measuring it for all 8 segments (I1, I2, I3, I4, Q1, Q2, Q3, Q4). The dark noise contribution is negligible with respect to demod board noise. In attachments 3 & 4 please find plots that include detection and demodulation contributions for both WFSs.
For whitening board electronics noise measurement, we were terminating the inputs (attachment 5) and measuring the outputs (attachment 6). Electronics noise of the whitening board is in the attachments 7 & 8.
For ADC electronics noise we terminated ADC input and measured noise using diaggui (attachments 9 & 10). Please find these spectra for WFS1, WFS2, and MC TRANS in attachments 11, 12 & 13.
For MC2 TRANS we measured OPLEV board noise. We did two sets of measurements, as for demod board of WFSs (with and without QPD dark noise) (attachments 14, 15 & 16). In the case of OPLEV board noise without dark noise, we were terminating the OPLEV input. Please find the electronics noise of OPLEV's segment 1 (including dark noise which is again much smaller with respect to the OPLEV's electronics noise) in attachment 17.
For the transfer functions, demod board has flat tf, whitening board tf please find in attachment 18, ADC tf is flat and it is (2**16 - 1)/20 [cts/V], and dewhitening tf please find in attachment 19. Also please find the ASD of the spectral analyzer noise (attachment_20).
Measurements for WFS1 demod and whitening were done on 5th of July between 15h and 18h local time. Measurements for WFS2 demod and whitening were done on 6th of July between 15h and 17h local time. All the rest were done on July 7th between 14h and 19h. In attachment 21 also find the comparison between electronics noise for WFSs and cds error signal (taken on the 28th of June between 17h and 18h). Sorry for bad quality of some pictures.
To access the board remotely through the 40m lab ethernet port, use
ssh -N -L localhost:1137:localhost:9090 xilinx@<ip_address>
Then in the browser go to
Other SSH commands using different ports or without the -N -L seemed to fail to open Jupyter. This way has been successful thereafter.
Since last week I've worked with tommy on getting the RFSoC 2x2 board to get some TFs from simple minicircuits type filters. The first thing I did was set up the board (which is in the office area) for remote access. I hooked up the TCP/IP port to a wall ethernet socket (LIGO-04) and the caltech network assiggned some IP address to our box. I guess eventually we can put this behind the lab network for internal use only.
After fiddling around with the tone-generators and spectrum analyzer tools in loopback configuration (DAC --> ADC direct connection), we noticed that lower frequency (~ 1 MHz) signals were hardly making it out/back into the board... so we looked at some of the schematics found here and saw that both RF data converters (ADC & DAC) interfaces are AC coupled through a BALUN network in the 10 - 8000 MHz band (see Attachment #1). This is in principle not great news if we want to get this board ready for audio-band DSP.
We decided that while Tommy works on measuring TFs for SHP-200 all the way up to ~ 2 GHz (which is possible with the board as is) I will design and put together an analog modulation/demodulation frontend so we can upconvert all our "slow" signals < 1MHz for fast, wideband DSP. and demodulate them back into the audio band. The BALUN network is pictured in Attachment #2 on the board, I'm afraid it's not very simple to bypass without damaging the PCB or causing some other unwanted effect on the high-speed DSP.