40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 81 of 339  Not logged in ELOG logo
ID Date Authorup Type Category Subject
  14505   Mon Apr 1 12:01:52 2019 JonUpdateCDS 

I brought c1susaux back online this morning for suspension-channel test scripting. It had been dead for some time. I followed the procedure outlined in #12542. ITMY became stuck during this process, which Gautam tells me always happens since the last vacuum access, but ITMX is not stuck.

  14508   Tue Apr 2 15:02:53 2019 JonUpdateCDSITMY freed

blushI renamed all channels on c1susaux2 from "C1:SUS-..." to "C1:SUS2-..." to avoid contention. When the new system is ready to install, those channel names can be reverted with a quick search-and-replace edit.

Quote:

While doing this work, I noticed several errors corresponding to EPICS channel conflicts. Turns out the c1susaux2 EPICS server was left running, and the MEDM screens (and possibly several scripts) were confused. There has to be some other way of testing the new crate, on an isolated network or something - please do not leave the modbus service running as it potentially interferes with normal IFO operation. For good measure, I stopped the process and shut down the machine since I saw nothing in the elog about any running tests.

  14514   Wed Apr 3 16:17:17 2019 JonUpdateVACTP2 forepump replaced

I can't explain the mechanical switching sound Gautam reported. The relay controlling power to the TP2 forepump is housed in the main AC relay box under the arm tube, not in the Acromag chassis, so it can't be from that. I've cycled through the pumpdown sequence several times and can't reproduce the effect. The Acromag switches for TP2 still work fine.

In any case, I've made modifications to the vacuum interlocks that will help with two of the issues:

  1. For the "AC power loss" over-triggering: New logic added requiring the UPS to be out of the "on line power, battery OK" state for ~5 seconds before tripping the interlock. This will prevent electrical transients from triggering an emergency shutdown, as seems to be the case here (the UPS briefly isolates the load to battery during such events).
  2. PSL interlocking: New logic added which directly sets C1:AUX-PSL_ShutterRqst --> 0 (closes the PSL shutter) when the main volume pressure is 3 mtorr-500 torr. Previously there was a channel exposed for this interlock (C1:Vac-interlock_high_voltage), but c1aux was not actually monitoring it. Following the convention of every vac interlock, after the PSL shutter has been closed, it has to be manually reopened. Once the pressure is out of this range, the vac system will stop blocking the shutter from reopening, but it will not perform the reopen action itself. gautam: a separate interlock logic needs to be implemented on c1aux (the shutter machine) that only permits the shutter to be opened if the Vac pressure range is okay. The SUS watchdog style AND logic in the EPICS database file should work just fine.

After finishing this vac work, I began a new pumpdown at ~4:30pm. The pressure fell quickly and has already reached ~1e-5 torr. TP2 current and temp look fine.

Quote:

However, when opening V4 (the foreline of TP1 pumped by TP2), I heard a loud repeated click track (~5Hz) from the electronics rack. Shortly after, the interlocks shut down all the TPs again, citing "AC power loss". Something is not right, I leave it to Jon and Chub to investigate.

Attachment 1: IMG_3180.jpg
IMG_3180.jpg
  14536   Thu Apr 11 12:04:43 2019 JonUpdateSUSStarting some scripted SUS tests on ITMY

Will advise when I'm finished, will be by 1 pm for ALS work to begin.

  14538   Thu Apr 11 12:57:48 2019 JonUpdateSUSStarting some scripted SUS tests on ITMY

Testing is finished.

Quote:

Will advise when I'm finished, will be by 1 pm for ALS work to begin.

  14539   Thu Apr 11 17:30:45 2019 JonUpdateSUSAutomated suspension testing with susPython

Summary

In anticipation of needing to test hundreds of suspension signals after the c1susaux upgrade, I've started developing a Python package to automate these tests: susPython

https://git.ligo.org/40m/suspython

The core of this package is not any particular test, but a general framework within which any scripted test can be "nested." Built into this framework is extensive signal trapping and exception handling, allowing actuation tests to be performed safely. Namely it protects against crashes of the test scripts that would otherwise leave the suspensions in an arbitrary state (e.g., might destroy alignment).

Usage

The package is designed to be used as a standalone from the command line. From within the root directory, it is executed with a single positional argument specifying the suspension to test:

$ python -m suspython ITMY

Currently the package requires Python 2 due to its dependence on the cdsutils package, which does not yet exist for Python 3.

Scripted Tests

So far I've implemented a cross-consistency test between the DC-bias outputs to the coils and the shadow sensor readbacks. The suspension is actuated in pitch, then in yaw, and the changes in PDMon signals are measured. The expected sign of the change in each coil's PDMon is inferred from the output filter matrix coefficients. I believe this test is sensitive to two types of signal-routing errors: no change in PDMon response (actuator is not connected), incorrect sign in either pitch or yaw response, or in both (two actuators are cross-wired).

The next test I plan to implement is a test of the slow system using the fast system. My idea is to inject a 3-8 Hz excitation into the coil output filter modules (either bandlimited noise or a sine wave), with all coil outputs initially disabled. One coil at a time will be enabled and the change in all VMon signals monitored, to verify the correct coil readback senses the excitation. In this way, a signal injected from the independent and unchanged fast system provides an absolute reference for the slow system.

I'm also aware of ideas for more advanced tests, which go beyond testing the basic signal routing. These too can be added over time within the susPython framework.

  14561   Mon Apr 22 21:33:17 2019 JonUpdateSUSBench testing of c1susaux replacement

Today I bench-tested most of the Acromag channels in the replacement c1susaux. I connected a DB37 breakout board to each chassis feedthrough connector in turn and tested channels using a multimeter and calibrated voltage source. Today I got through all the digital output channels and analog input channels. Still remaining are the analog output channels, which I will finish tomorrow.

There have been a few wiring issues found so far, which are noted below.

Channel Type Issue
C1:SUS2-PRM_URVMon Analog input No response
C1:SUS2-PRM_LRVMon Analog input No response
C1:SUS2-BS_UL_ENABLE Digital output Crossed with LR
C1:SUS2-BS_LL_ENABLE Digital output Crossed with UR
C1:SUS2-BS_UR_ENABLE Digital output Crossed with LL
C1:SUS2-BS_LR_ENABLE Digital output Crossed with UL
C1:SUS2-ITMY_SideVMon Analog input Polarity reversed
C1:SUS2-MC2_UR_ENABLE Digital output Crossed with LR
C1:SUS2-MC2_LR_ENABLE Digital output Crossed with UR
     
     

 

  14563   Tue Apr 23 18:48:25 2019 JonUpdateSUSc1susaux bench testing completed

Today I tested the remaining Acromag channels and retested the non-functioning channels found yesterday, which Chub repaired this morning. We're still not quite ready for an in situ test. Here are the issues that remain.

Analog Input Channels

Channel Issue
C1:SUS-MC2_URPDMon No response
C1:SUS-MC2_LRPDMon No response

I further diagnosed these channels by connecting a calibrated DC voltage source directly to the ADC terminals. The EPICS channels do sense this voltage, so the problem is isolated to the wiring between the ADC and DB37 feedthrough.

Analog Output Channels

Channel Issue
C1:SUS-ITMX_ULBiasAdj No output signal
C1:SUS-ITMX_LLBiasAdj No output signal
C1:SUS-ITMX_URBiasAdj No output signal
C1:SUS-ITMX_LRBiasAdj No output signal
C1:SUS-ITMY_ULBiasAdj No output signal
C1:SUS-ITMY_LLBiasAdj No output signal
C1:SUS-ITMY_URBiasAdj No output signal
C1:SUS-ITMY_LRBiasAdj No output signal
C1:SUS-MC1_ULBiasAdj No output signal
C1:SUS-MC1_LLBiasAdj No output signal
C1:SUS-MC1_URBiasAdj No output signal
C1:SUS-MC1_LRBiasAdj No output signal

To further diagnose these channels, I connected a voltmeter directly to the DAC terminals and toggled each channel output. The DACs are outputting the correct voltage, so these problems are also isolated to the wiring between DAC and feedthrough.

In testing the DC bias channels, I did not check the sign of the output signal, but only that the output had the correct magnitude. As a result my bench test is insensitive to situations where either two degrees of freedom are crossed or there is a polarity reversal. However, my susPython scripting tests for exactly this, fetching and applying all the relevant signal gains between pitch/yaw input and coil bias output. It would be very time consuming to propagate all these gains by hand, so I've elected to wait for the automated in situ test.

Digital Output Channels

yes Everything works.

  14564   Tue Apr 23 19:31:45 2019 JonUpdateSUSWatchdog channels separated from autoBurt.req

For the new c1susaux, Gautam and I moved the watchdog channels from autoBurt.req to a new file named autoBurt_watchdogs.req. When the new modbus service starts, it loads the state contained in autoBurt.snap. We thought it best for the watchdogs to not be automatically enabled at this stage, but for an operator to manually have to do this. By moving the watchdog channels to a separate snap file, the entire SUS state can be loaded while leaving just the watchdogs disabled.

This same modification should be made to the ETMX and ETMY machines.

  14574   Thu Apr 25 10:32:39 2019 JonUpdateVACVac interlocks updated

I slightly cleaned up Gautam's disabling of the UPS-predicated vac interlock and restarted the interlock service. This interlock is intended to protect the turbo pumps after a power outage, but it has proven disruptive to normal operations with too many false triggers. It will be reenabled once a new UPS has been installed. For now, as it has been since 2001, the vac pumps are unprotected against an extended power outage.

  14580   Fri Apr 26 12:32:35 2019 JonUpdatePSLmodbusPSL service shut down

Gautam and I are removing the prototype Acromag chassis from the 1x4 rack to make room for the new c1susuax hardware. I shut down and disabled the modbusPSL service running on c1auxex, which serves the PSL diagnostic channels hosted by this chassis. The service will need to be restarted and reenabled once the chassis has been reinstalled elsewhere.

  14581   Fri Apr 26 19:35:16 2019 JonUpdateSUSNew c1susaux installed, passed first round of scripted testing

[Jon, Gautam]

Today we installed the c1susaux Acromag chassis and controller computer in the 1X4 rack. As noted in 14580 the prototype Acromag chassis had to first be removed to make room in the rack. The signal feedthroughs were connected to the eurocrates by 10' DB-37 cables via adapters to 96-pin DIN.

Once installed, we ran a scripted set of suspension actuation tests using PyIFOTest. BS, PRM, SRM, MC1, MC2, and MC3 all passed these tests. We were unable to test ITMX and ITMY because both appear to be stuck. Gautam will shake them loose on Monday.

Although the new c1susaux is now mounted in the rack, there is more that needs to be done to make the installation permanent:

  • New 15V and 24V power cables with standard LIGO connectors need to be run from the Sorensenn supplies in 1X5. The chassis is currently powered by bench supplies sitting on a cart behind the rack.
  • All 24 new DB-37 signal cables need to be labeled.
  • New 96-pin DIN connectors need to be put on two ribbon cables (1Y5_80 B, 1Y5_81) in the 1X4 rack. We had to break these connectors to remove them from the back of the eurcrates.
  • General cleanup of any cables, etc. left around the rack. We cleaned up most things this evening.
  • Rename the host computer c1susaux2 --> c1susaux, and update the DNS lookup tables on chiara.

On Monday we plan to continue with additional scripted tests of the suspensions.


gautam - some more notes:

  • Backplane connectors for the SUS PD whitening boards, which now only serve the purpose of carrying the fast BIO signals used for switching the whitening, were moved from the P1 connector to P2 connector for MC1, MC2, MC3, ITMX, ITMY, BS and PRM.
  • In the process, the connectors for BS and PRM were detatched from the ribbon cable (there wasn't any good way to unseat the connector from the shell that I know of). These will have to be repaired by Chub, and the signal integrity will have to be checked (as they have to be for the connectors that are allegedly intact).
  • While we were doing the wiring, I disconnected the outputs of the coil driver board going to the satellite box (front panel DB15 connector on D010001). These were restored after our work for the testing phase.
  • The backplane cables to the eurocrate housing the coil driver boards were also disconnected. They are currently just dangling, but we will have to clean it up if the new crate is performing alright.
  • In general the cable routing cleanliness has to be checked and approved by Chub or someone else qualified. In particular, the power leads to the eurocrate are in the way of the DIN96-DB37 adaptor board of Johannes' design, particularly on the SUS PD eurocrate.
  • Tapping new power rails for the Acromag chassis will have to be done carefully. Ideally we shouldn't have to turn off the Sorensens.
  • There are some software issues we encountered today connected with the networking that have to be understood and addressed in a permanent way.
  • Sooner rather than later, we want to reconnect the Acromag crate that was monitoring the PSL channels, particularly given the NPRO's recent flakiness.
  • The NPRO was turned back on (following the same procedure of slowly dialing up the injection current). Primary motivation to see if the mode cleaner cavity could be locked with the new SUS electronics. Looks like it could. I'm leaving it on over the weekend...
Attachment 1: IMG_3254.jpg
IMG_3254.jpg
Attachment 2: IMG_3256.jpg
IMG_3256.jpg
  14585   Mon Apr 29 19:23:49 2019 JonUpdateComputer Scripts / ProgramsScripted tests of suspension VMons using fast system

I've added a scripted VMon/coil-enable test to PyIFOTest following the suggestion in #15542. Basically, a DC offset is added to one fast coil output at a time, and all the VMon responses are checked.

After resolving the swapped ITMX/ITMY eurocrate slots described in #14584, I ran the new scripted VMon test on all eight optics managed by c1susaux. All of them passed: SRM, BS, MC1, MC2, MC3, PRM, ITMX, ITMY. This is not the final suspension test we plan to do, but it gives me reasonably good confidence that all channels are connected correctly.

  14588   Thu May 2 10:59:58 2019 JonUpdateSUSc1susux in situ wiring testing completed

Summary

Yesterday Gautam and I ran final tests of the eight suspensions controlled by c1susaux, using PyIFOTest. All of the optics pass a set of basic signal-routing tests, which are described in more detail below. The only issue found was with ITMX having an apparent DC bias polarity reversal (all four front coils) relative to the other seven susaux optics. However, further investigation found that ETMX and ETMY have the same reversal, and there is documentation pointing to the magnets being oppositely-oriented on these two optics. It seems likely that this is the case for ITMX as well. 

I conclude that all the new c1susaux wiring/EPICS interfacing works correctly. There are of course other tests that can still be scripted, but at this point I'm satisfied that the new Acromag machine itself is correctly installed. PyIFOTest has been morphed into a powerful general framework for automating IFO tests. Anything involving fast/slow IO can now be easily scripted. I highly encourage others to think of more applications this may have at the 40m.

Usage and Design

The code is currently located in /users/jon/pyifotest although we should find a permanent location for it. From the root level it is executed as

$ ./IFOTest <PARAMETER_FILE>

where PARAMETER_FILE is the filepath to a YAML config file containing the test parameters. I've created a config file for each of the suspended optics. They are located in the root-level directory and follow the naming convention SUS-<OPTIC>.yaml.

The code climbs a hierarchical "ladder" of actuation/readback-paired tests, with the test at each level depending on signals validated in the preceding level. At the base is the fast data system, which provides an independent reference against which the slow channels are tested. There are currently three scripted tests for the slow SUS channels, listed in order of execution:

  1. VMon test:  Validates the low-frequency sensing of SUS actuation (VMon channels). A DC offset is applied in the final filter module of the fast coil outputs, one coil at a time. The test confirms that the VMon of the actuated coil, and only this VMon, senses the displacement, and that the response has the correct polarity. The screen output is a matrix showing the change in VMon responses with actuation of each coil. A passing test, roughly, is diagonal values >> 0 and off-diagonal values << diagonal.

  2. Coil Enable test:  Validates the slow watchdog control of the fast coil outputs (Coil-Enable channels). Analogously to (1), this test also applies a DC offset via the fast system to one coil at a time and analyzes the VMon responses. However, in this case, the offset is enabled to all five coils simulataneously and only one coil output is enabled at a time. The screen output is again a \Delta VMon matrix interpreted in the same way as above.

     

  3. PDMon/DC Bias test:  Validates slow alignment control and readback (BiasAdj and PDMon channels). A DC misalignment is introduced first in pitch, then in yaw, with the OSEM PDMon responses measured in both cases. Using the gains from the PIT/YAW---> COIL output coupling matrix, the script verifies that each coil moves in the correct direction and by a sufficiently large magnitude for the applied DC bias. The screen output shows the change in PDMon responses with a pure pitch actuation, and with a pure yaw actuation. The output filter matrix coefficients have already been divided out, so a passing test is a sufficiently large, positive change under both pitch and yaw actuations.

     

  14589   Thu May 2 15:15:15 2019 JonOmnistructureComputerssusaux machine renamed

Now that the replacement susaux machine is installed and fully tested, I renamed it from c1susaux2 to c1susaux and updated the DNS lookup tables on chiara accordingly.

  14590   Thu May 2 15:35:54 2019 JonOmnistructureUpgradec1susaux upgrade documentation

For future reference:

  • The updated list of c1susaux channel wiring (includes the "coil enable" --> "coil test" digital outputs change)
  • Step-by-step instructions on how to set up an Acromag system from scratch
  14596   Mon May 6 11:05:23 2019 JonUpdateSUSAll vertex SUS watchdogs were tripped

Yes, this was a consequence of the systemd scripting I was setting up. Unlike the old susaux system, we decided for safety NOT to allow the modbus IOC to automatically enable the coil outputs. Thus when the modbus service starts/restarts, it automatically restores all state except the watchdog channels, which are left in their default disabled state. They then have to be manully enabled by an operator, as I should have done after finishing testing.

Quote:

I found the 8 vertex watchdogs tripped today morning. The ETMs were fine, suggesting this was not an actual earthquake. I suspect it was connected to this remote work? Was there a reason why they were left tripped?

  14600   Thu May 9 22:26:39 2019 JonOmnistructurePSLSecond ADC added to PSL Acromag crate

This evening I added a second ADC module to the prototype Acromag chassis. This chassis can now read out all the PSL diagnostic channels.

I configured the second ADC identically to the first ("ADC0"), and assigned it IP address 192.168.113.122. I confirmed it is visible on the martian network.

There was an existing but unused DB-15 feedthrough which I used for ADC1 channels 1-7. The eighth channel I left unwired, but there are slots available in the neighboring DB-25 feedthough, if that channel is needed in the future. The channel wiring assignments are as follows.

ADC1 Channel DB-15 Feedthrough Pin
0+ 1
0- 9
1+ 2
1- 10
2+ 3
2- 11
3+ 4
3- 12
4+ 5
4- 13
5+ 6
5- 14
6+ 7
6- 15
7+ not connected
7- not connected

I tested all seven of these channels by applying a calibrated voltage source and measuring the response via the Windows Acromag software. All work and are correctly calibrated to better than 0.1%.

Attachment 1: IMG_3291.jpg
IMG_3291.jpg
  14604   Sat May 11 11:48:54 2019 JonUpdatePSLSome work on/around PSL table

I took a look at the error being encountered by the modbusPSL service. The problem is that the /run/modbusPSL.pid file is not being generated by procServ, even though the -p flag controlling this is correctly set. I don't know the reason for this, but it was also a problem on c1vac and c1susaux. The solution is to remove the custom kill command (ExecStop=...) and just allow systemd to stop it via its default internal kill method.

modbusPSL.service - ModbusIOC Service via procServ
   Loaded: loaded (/etc/systemd/system/modbusPSL.service; disabled)
   Active:
active (running) since Fri 2019-05-10 13:17:54 PDT; 2h 13min ago
  Process: 8824 ExecStop=/bin/kill -9 ` cat /run/modbusPSL.pid`
(code=exited, status=1/FAILURE)
 Main PID: 8841 (procServ)
   CGroup: /system.slice/modbusPSL.service
           ├─8841 /usr/bin/procServ -f -L /home/controls/modbusPSL.log -p /run/modbusPSL.pid 8009 /cvs/cds/rtapps/epics-3.14.12.2_long/module...
           ├─8842 /cvs/cds/rtapps/epics-3.14.12.2_long/modules/modbus/bin/linux-x86_64/modbusApp /cvs/cds/caltech/target/c1psl2/npro_config.c...
           └─8870 caRepeater

May 10 13:17:54 c1auxex systemd[1]: Started ModbusIOC Service via procServ.

  14801   Tue Jul 23 21:59:08 2019 JonUpdateCamerasPlan for GigE cameras

This afternoon Gautam and I assessed what to do about restoring the GigE camera software. Here's what I propose:

  • Set up one of the new rackmount Supermicros as a dedicated camera feed server
  • All GigE cameras on a local subnet connected to the second network interface (these Supermicros have two)
  • Put the SnapPy, pypylon, and pylon5 binaries on the shared network drive. These all have to be built from source.
  • All other dependencies can be gotten through the package managers, so create requirements files for yum and pip to automatically install these locally.

I've started resolving the many dependencies of this code on rossa. The idea is to get a working environment on one workstation, then generate requirements files that can be used to set up the rest of the machines. I believe the dependencies have all been installed. However, many of the packages are newer versions than before, and this seems to have broken SnapPy. I'll continue debugging this tomorrow.

  14806   Wed Jul 24 16:45:32 2019 JonUpdateCamerasUpgraded Pylon from 5.0.12 to 5.2.0

I upgraded Pylon, the C/C++ API for the GigE cameras, to the latest release, 5.2.0. It is installed in the same location as before, /opt/rtcds/caltech/c1/scripts/GigE/pylon5, so environment variables do not change. The old version, 5.0.12, still exists at opt/rtcds/caltech/c1/scripts/GigE/backup_pylon5.

The package contains a GUI application (/bin/PylonViewerApp) for streaming video. The old version supports saving still images, but Milind discovered that the new version supports saving video as well. This required installing a supplementary package supporting MPEG-4 output.

Basler's GUI application is launched from the terminal using the alias pylon. I've tested it and confirm it can save both videos and still-image formats. I recommend to also try grabbing images using this program and check the bit resolution. It would be a useful diagnostic to know whether it's a bug in Joe B.'s code or something deeper in the camera settings.

Attachment 1: IMG_3525.jpg
IMG_3525.jpg
  14810   Thu Jul 25 09:19:32 2019 JonUpdateCamerasUpgraded Pylon from 5.0.12 to 5.2.0

I'll keep developing the camera server on a parallel track using the "new_..." directory naming convention. One thing I forgot to note is that the new pylon/pypylon packages require Python 3, so will not work with any of the 2.7 scripts. All of the environment I need to set up is exclusively Python 3. I won't change anything in the Python 2.7 environment in current use.

Also, I found the source of the bit resolution issue: Joe B's code loads a set of initialization parameters from a config file. One of them is "Frame Type = Mono8" which sets the dynamic range of the stream. I'll look into how this should be changed. 

Quote:

Since there are multiple SURF projects that rely on the cameras:

  1. I moved the new installs Jon made to "new_pylon5" and "new_pypylon". The old installs were moved back to be the default directories.
  2. The bashrc alias for pylon was updated to allow the recording of videos (i.e. it calls the PylonViewerApp from new_pypylon).
  3. There is a script that can grab images at multiple exposures and save 12-bit data as uint16 numpy arrays to an HDF5 file. Right now, it is located at /users/kruthi/scripts/grabHDR.py. We can move this to a better place later, and also improve the script for auto adjusting the exposure time to avoid saturations.
  14814   Fri Jul 26 19:53:53 2019 JonOmnistructureCamerasGigE Camera Server

I've started setting up the last new rackmount SuperMicro as a dedicated server for the GigE cameras. The new machine is currently sitting on the end of the electronics test bench. It is assigned the hostname c1cam at IP 192.168.113.116 on the martian network. I've installed Debian 10, which will be officially supported until July 2024.

I've added the /cvs/cds NFS mount and plan to house all the client/server code on this network disk. Any dependencies that must be built from source will be put on the network disk as well. Any dependencies that can be gotten through the package manager, however, will be installed locally but in an automated way using a reqs file.

We should ask Chub to reorder several more SuperMicro rackmount machines, SSD drives, and DRAM cards. Gautam has the list of parts from Johannes' last order.

  14839   Fri Aug 9 20:58:33 2019 JonUpdateElectronicsBorrowed Variac transformer

I borrowed an old-looking Variac variable transformer from the power supplies cabinet along the y-arm. It is currently in the TCS lab.

  14855   Fri Aug 23 18:46:17 2019 JonUpdateCDSc1iscaux remaining work

I added the list of new c1iscaux channels to /opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini and restarted the framebuilder. Koji had thought some of these channels might have previously existed under slightly different names. However, after looking through C0EDCU.ini and the other _SLOW.ini files, I did not find any candidates for removal. As far as I can tell, all of these channels are being recorded for the first time.

Quote:Koji 
  1. Modify C0EDCU.ini to trend the new slow channels we may want long-term monitoring of (e.g. LO power levels to the Demod boards). Anyone want to volunteer to take care of this?
  14856   Fri Aug 23 19:10:02 2019 JonUpdateCamerasGigE camera server is online

Following the death of rossa, which was hosting the only working environment for the GigE camera software, I've set up a new dedicated rackmount camera server: c1cam (details here). The Python server script is now configured as a persistent systemd service, which automatically starts on boot and respawns after a crash. The server depends on a set of EPICS channels being available to control the camera settings, so c1cam is also running a softIOC service hosting these channels. At the moment only the ETMX camera is set up, but we can now easily add more cameras.

Usage

Instructions for connecting to a live video feed are posted here. Any machine on the martian network can stream the feed(s). The only requirement is that the client machine have GStreamer 0.10 installed (all the control room workstations satisfy this).

Code Locations

As much as possible, the code and dependencies are hosted on the /cvs/cds network drive instead of installed locally. The client/server code and the Pylon5, PyPylon, and PyEpics dependencies are all installed at /cvs/cds/rtcds/caltech/c1/scripts/GigE. The configuration files for the soft IOC are located at /cvs/cds/caltech/target/c1cam.

Upgrade Goals

The 40m GigE camera code is a slightly-updated version of the 10+ year-old camera code in use at the sites. Consequently every one of its dependencies is now deprecated. Ultimately, we'd like to upgrade to the following:

  • Python 2.7 --> 3.7
  • Basler Pylon 5.0.12 --> 5.2.0
  • PyPylon 1.1.1 --> 1.4.0
  • GStreamer 0.10 --> 1.2

This is a long-term project, however, as many of these APIs are very different between Python 2 and 3.

  15093   Wed Dec 11 15:01:57 2019 JonSummaryPSLPMC cavity ringdown measurement

[Jon, Yehonathan]

We carried out a set of cavity ringdown measurements of the PMC. The 1/e decay time scale is found to be 35.2 +/- 2.4 (systematic) μs. The statistical error is negligible compared to the systematic error, which is taken as the maximum absolute deviation of any measurement from the average value.

To make the measurement, we injected the first order deflection beam of an 80 MHz AOM, then extinguished it quickly by cutting the voltage offset to the AOM driver provided by an RF function generator. A 100 MHz oscilloscope configured to trigger on the falling voltage offset was used to sample the cavity in transmission as sensed by a PDA55. We found the detector noise of the DC-coupled output of the 35.5 MHz REFL PD to be too high for a reflection-side measurement.

Further loss analysis is forthcoming.

Attachment 1: IMG_0101.jpg
IMG_0101.jpg
  15111   Mon Jan 6 15:36:55 2020 JonUpdatePSLAssembly underway for c1psl upgrade

[Jon, Yehonathan]

We've begun assembling the new c1psl Acromag chassis based on Yehonathan's final pin assignments. So far, parts have been gathered and the chassis itself has been assembled.

Yehonathan is currently wiring up the chassis power and Ethernet feedthroughs, following my wiring diagram from previous assemblies. Once the Acromag units are powered, I will help configure them, assign IPs, etc. We will then turn the wiring over to Chub to complete the Acromag to breakout board wiring.

I began setting up the host server, but immediately hit a problem: We seem to have no more memory cards or solid-state drives, despite having two more SuperMicro servers. I ordered enough RAM cards and drives to finish both machines. They will hopefully arrive tomorrow.

  15114   Tue Jan 7 18:51:51 2020 JonUpdatePSLNew c1psl server assembled

I've assembled a new SuperMicro rackmount machine to replace c1psl. It is currently set up on the electronics bench.

  • OS: Debian 10.2
  • Hostname: c1psl1 (will become c1psl after installation)
  • IP: 192.168.113.54 (registered in the martian DNS)
  • Network drive mount point set up (/cvs/cds), which provides all the EPICS executables.
  15125   Wed Jan 15 14:10:28 2020 JonConfigurationPSLNew EPICS database for C1PSL + C1IOO

Summary

I have completed the new EPICS channel database for the c1psl and c1ioo channels (now combined into the new c1psl Acromag machine). I've tested a small subset of channels on the electronics bench to confirm that the addressing and analog channel calibrations are correct in a general sense. At this point, we are handing the chassis off to Chub to complete the wiring of the Acromag terminals to Dsub feedthroughs. At the 40m meeting today, we identified Feb. 17-22 as a potential window for installation in the interferometer (Gautam is out of town then). Below are some implementaton details for future reference.

Analog channel calibration for Acromag

For analog input (ai) channels, the Acromag outputs raw values ranging from +/-30,000 counts, but the EPICS IOC interprets the data type as ranging from +/-2^15 = 32,768. Similarly, for analog output (ao) channels, the Acromag expects a drive signal in the range +/-30,000 counts. To achieve proper scaling, Johannes had previously changed the EGUF and EGUL fields from +/-10 V to +/-10.923 V. However, changing the engineering fields makes it much harder for a human to read off the real physical I/O range of the channel.

A better way to achieve the correct scaling is to simply set the field  ASLO=1.09225 (65,536 / 60,001) in addition to the normal EGUF and EGUL field values (+/-10 V). Setting this field forces a rescaling of the number of raw counts that works as so (assuming a 16-bit bipolar ADC or DAC, as are the Acromags):

OVAL = (RVAL * ASLO + AOFF + 2^15) * (EGUF - EGUL) / 2^16 + EGUL

In the above mapping, OVAL is the value of the channel in engineering units (e.g., V) and RVAL is its raw value in counts. It is not the case that either the ASLO/AOFF or EGUF/EGUL fields are used, but not both. The ASLO/AOFF parameters are always applied (but their default values are ASLO=1 and AOFF=0, so have no effect unless changed). The EGUF and EGUL parameters are then additionally applied if the field LINR="LINEAR" is set.

This conversion allows the engineering fields to remain unchanged from the real physical range. The ASLO value is the same for both analog input and output channels. I have implemented this on all the new c1psl and c1ioo channels and confirmed it to work using a calibrated input voltage source.

  15140   Wed Jan 22 16:19:47 2020 JonUpdateVACTP3 controller errors

Gautam and I debugged a communications problem with TP3 that was causing its python service to fail. We traced the problem back to the querying of the pump controller for its operational parameters (speed, voltage, temp). Some small percentage of the time (~5%, indeterministically), the pump controller is returning an invalid response which causes the service to shut itself down and signal a NO COMM error.

As a temporary fix, I wrapped the failing query in an exception handler to continue past this particular error. However, we suspect the microprocessor in the TP3 controller may be beginning to fail. There is a spare controller sitting right next to it in the vacuum rack. We will ask Chub to install the replacement in the near future.

gautam: this pump is responsible for pumping the annular volume under normal operations. while this problem is being resolved, the annular volume is valved off (as it has been since July 2019 anyway which is when this problem first manifested).

  15150   Thu Jan 23 23:07:04 2020 JonConfigurationPSLc1psl breakout board wiring

To facilitate wiring the c1psl chassis and scripting loopback tests, I've compiled a distilled spreadsheet with the Acromag-to-breakout board wiring, broken down by connector. This information is extractable from the master spreadsheet, but not easily. There were also a few apparent typos which are fixed here.

The wiring assignments at the time of writing are attached below. Here is the link to the latest spreadsheet.

Attachment 1: c1psl_feedthrough_wiring.pdf
c1psl_feedthrough_wiring.pdf c1psl_feedthrough_wiring.pdf c1psl_feedthrough_wiring.pdf c1psl_feedthrough_wiring.pdf
  15151   Fri Jan 24 13:56:21 2020 JonUpdateBHDBHD optics specifications

I've started a spreadsheet for the BHD optics specifications and populated it with my best initial guesses. There are a few open questions we still need to resolve, mostly related to mode-matching:

  • PR2 replacement: What transmission do we need for a ~100 mW pickoff? Also, do we want to keep the current curvature of -700 m?
  • LO mode-matching telescope: What are the curvatures of the two mirrors?
  • Lenses: We have six of them in the current layout. What FLs do we need?

The spreadsheet is editable by anyone. If you can contribute any information, please do!

  15168   Tue Jan 28 19:12:30 2020 JonConfigurationPSLSpare channels added to c1psl chassis

After some discussion with Gautam, I decided to build more spare channels into the new c1psl machine. This is anticipation of adding new laser and ISS channels in the near future, to avoid having to disconnect the installed chassis and pull it out of the rack. The spare channels will be wired to DB37M feedthroughs on the front side of the chassis, with enough wire length to be able to pull the breakout boards out of the front to reconfigure their wiring as needed (e.g., split off channels onto a separate connector).

To have enough overhead, this will require installing 1 additional ADC unit (XT1221) and 1 additional DAC (XT1541). We have enough spare BIO channels among the existing units (both sinking and sourcing). This will give us:

  • 13 spare ADC channels
  • 14 spare DAC channels
  • 16 spare sinking BIO channels
  • 12 spare sourcing BIO channels

The updated c1psl chassis wiring assignments are attached. It adds 4 new DB37M connectors for the spare channels (highlighted in yellow) and fixes one typo Jordan found while wiring today. The most current spreadsheet is available here.

Attachment 1: c1psl_feedthrough_wiring_v2.pdf
c1psl_feedthrough_wiring_v2.pdf c1psl_feedthrough_wiring_v2.pdf c1psl_feedthrough_wiring_v2.pdf c1psl_feedthrough_wiring_v2.pdf c1psl_feedthrough_wiring_v2.pdf
  15176   Thu Jan 30 12:52:10 2020 JonUpdateBHDMetal OMCs procured

Last night Yehonathan and I located the two steel PMCs in the QIL, with help from Anchal. They are currently sitting on my desk in Bridge, inside a box that also contains optics and other OMC parts. I will bring them over to the 40m the next time I come.

  15178   Thu Jan 30 17:31:28 2020 JonUpdatePSLErrant FSS_INOFFSET change

A script I was testing errantly set C1:PSL-FSS_INOFFSET => 10 V at about 5:30 pm. I manually reverted the channel value to 0, but I don't know what the value was initially. Someone please check this value if there are problems locking the FSS.

  15184   Mon Feb 3 15:22:39 2020 JonUpdatePSLc1psl progress/Acromag ADC grounding

I tested the c1psl AO channels on the electronics bench on Friday. While I found all the wiring to be correct, some of the channels exhibited excess noise with all appearances of a grounding problem.

Today Jordan, Gautam, and I investigated this further. It is indeed a grounding problem, but actually with the Acromag ADCs. The Acromag DAC outputs are single-ended (return is grounded), so (for the purpose of a loopback test) I would expect to leave the ADC inputs ungrounded. This is the configuration I tested Friday. Today we also tested driving the ADC with a floating source. The ADC noise behavior is exactly the same, whether the source end is grounded or not.

However, grounding the minus pin of the ADC channel eliminates the noise. We don't understand why this seems to be required irrespective of the driving source, so there something we're missing about the ADC design. As it turns out, this same fix was made to the AI channels of the previously-upgraded Acromag machines. I know Chub and I had to do this for the AI channels of c1vac, but at the time we thought the source grounding was causing the issue. However, today Jordan and I looked inside c1iscaux, which Chub wired, and confirmed that its AI channels are wired in the same way.

So in any case, Jordan is grounding the c1psl AI channels in the same way as c1iscaux. Once this is done, we'll continue with the bench testing tomorrow.

gautam: here are my notes about this issue when i was doing the c1iscaux testing. As I note there, "previously-upgraded Acromag machines" in the plural may be a bit of a stretch - I have no idea what the grounding situation is in c1susaux / c1auxex for example.

  15194   Thu Feb 6 21:54:13 2020 JonUpdatePSLc1psl bench testing complete

Today I engineered the last piece of the new c1psl system: the multi-bit binary output (mbbo) channels that control the MC servo board gains. These 6-bit channels have to be split across two 4-bit Acromag registers. To enforce synchronous switching, I adapted the latch.py script developed by Gautam to address this problem in c1iscaux. Analogously to the c1iscaux implementation, I scripted the code to automatically run as a systemd service which is launched by the main modbusIOC service. I tested this all using the DB37 LED test board and confirmed it to work.

This now completes the electronics bench testing.

There are still several DB37 connectors to be wired, which carry only spare channels for future use and are not interfaced with the EPICS IOC. Jordan and I discussed this today and he or Chub will complete it shortly. To allow time for the spare channel wiring to be completed (as well as for more locking progress before interruption), Gautam and I think Monday/Tuesday next week would be the earliest possible window to install the new system.

  15226   Wed Feb 26 21:43:48 2020 JonSummaryBHDProjected IFO noise budget, post-BHD upgrade

To supplement the material presented during the BHD review, I've put together a projected noise budget for the 40m. Note these are the expected interferometer noises (originating in the IFO itself), not BHD readout noises. The key parameters for each case are listed in the figure title. Also attached is a tarball (attachment 4) containing the ipython notebook, data files, and rolled-back version of pygwinc which were used to generate these figures.

Attachment 1: Phase quadrature readout.

Attachment 2: Comparison to aLIGO design sensitivity (phase quadrature).

Attachment 3: Amplitude quadrature readout.

Attachment 1: 40m_phase_quad.pdf
40m_phase_quad.pdf
Attachment 2: 40m_aligo_comp.pdf
40m_aligo_comp.pdf
Attachment 3: 40m_ampl_quad.pdf
40m_ampl_quad.pdf
Attachment 4: noise_budget.tar
  15241   Mon Mar 2 23:49:03 2020 JonSummaryBHDProjected IFO noise budget, post-BHD upgrade

Updated noise budget curves, now computed using the latest version of pygwinc. This resolves the inconsistency between the gwinc quantum noise curves and Gautam's analytic calculations. As before, the key configuration parameters are listed in the figure titles.

Attachment 1: Phase quadrature

Attachment 2: Amplitude quadrature

Attachment 3: Comparison to aLIGO design (phase quadrature)

Quote:

The quantum noise curves here are not correct. c.f. amplitude quadrature noise budget.

Attachment 1: 40m_phase_quad.pdf
40m_phase_quad.pdf
Attachment 2: 40m_ampl_quad.pdf
40m_ampl_quad.pdf
Attachment 3: 40m_aligo_comp.pdf
40m_aligo_comp.pdf
  15244   Tue Mar 3 18:11:05 2020 JonSummaryBHDProjected IFO noise budget, post-BHD upgrade

Revised noise estimates, correcting a couple of factor of 2 and factor of pi errors found in the coil driver noise calculation. Also resolves a strain vs. displacement units confusion using the new pygwinc. Gautam and I have checked these noises against the analytical predictions and believe they are now accurate. Attachments are again:

Attachment 1: Phase quadrature

Attachment 2: Amplitude quadrature

Attachment 3: Comparison to aLIGO design (phase quadrature)

Attachment 1: 40m_phase_quad.pdf
40m_phase_quad.pdf
Attachment 2: 40m_ampl_quad.pdf
40m_ampl_quad.pdf
Attachment 3: 40m_aligo_comp.pdf
40m_aligo_comp.pdf
  15253   Wed Mar 4 22:38:31 2020 JonUpdatePSLc1psl communications problem resolved

I investigated the problem reported earlier today with the BIO1 channels. By logging the systemd messages generated when the IOC starts, I was immediately able to determine that the problem was not limited to BIO1. The modbus communications were failing for several other units as well.

Because some in-situ rewiring of a handful of channels had recently been done (more on this soon), I initially suspected that one of the Acromags had been damaged in the process. However, removing BIO1 (or other non-communicating modules) did not restore communications with the rest of the modules. To test whether the chassis was the source of the problem at all, we set up a fresh ADC (new out of the package) and directly connected it to the secondary Ethernet interface of c1psl. With only the one new ADC connected, the modbus IOC failed in exactly the same way.

To confirm that the new ADC did in fact work, we connected it to c1auxex in the same configuration. The unit worked fine connected to c1auxex. This established that the source of the problem was the c1psl host. After some extensive debugging, I traced the problem to a pre-execution script (part of the modbus IOC systemd service) which resets the secondary network interface (the one connected to the Acromag chassis) prior to launching the IOC. This was to ensure the secondary interface always had the correct IP address. It appears this reset was somehow creating a race condition that allowed the modbus initializations (first communications with the Acromags) to sometimes start before the network interface had actually come back up.

I still don't understand how this was happening, or why the pre script worked just fine up until yesterday, but eliminating the network interface reset fixes the problem in 100% of the trials we ran. Unfortunately we lost the entire day to debugging this problem, so the final round of testing is still to be completed. We plan to pick it back up tomorrow afternoon.

  15256   Thu Mar 5 19:45:23 2020 JonSummaryPSLC1PSL in-situ test results

We've completed almost all of the in-situ testing of the c1psl channels. During this process, we identified several channels which needed to be rewired to different Acromags (BIO sinking v. sourcing). We also elected to change the connector type of a few channels for practical advantages. Those modifications and other issues found during testing are detailed below. Also attached are the updated channel assignments, with a column indicating the in-situ testing status of each channel.

Post-installation modifications

  • All four channels connected to the sourcing BIO module were found to in fact require sinking I/O. They were reassigned to sinking BIO modules. Affected channels:
    • C1:PSL-FSS_FASTSWEEP
    • C1:PSL-FSS_SW1
    • C1:PSL-FSS_SW2
    • C1:PSL-PSL_Shutter
  • Added a new AI channel:
    • C1:PSL-FSS_MIXERM
  • Removed an unneccessary AI channel:
    • C1:PSL-FSS_LODET
  • Moved two AI channels from BNC connectors to a new Dsub connector (labelled DB25M-2 in the spreadsheet).
    • C1:PSL-FSS_RCTEMP
    • C1:PSL-FSS_RMTEMP_VOLTS

Issues identified during testing

  • Digital calibration. The following channels work, but we need to verify their EPICS calibration parameters (EGUF/EGUL):
    • C1:PSL-FSS_FASTGAIN
    • C1:PSL-FSS_FAST
    • C1:PSL-PMC_RFADJ
    • C1:PSL-PMC_MODET
  • IMC servo board. The Acromag channels themselves were found to work, but the linearity of the mbbo gain stages are in question (i.e., a potential problem with the board). GV is currently testing the servo board.
  • PSL QPD board apears to be dead. We connected a scope directly to the test points on the board and measured a high level of noise and no signal (for all four of the QPD channels). I understand this QPD has not been used in some time, so it may not have been noticed before.
  • WFS DC channels are saturating when the IMC is unlocked. The acceptance range of the Acromag ADC is only +/-10 V, but we measured sensor voltages as high as ~14 V. It appears that the old ADCs were somehow accepting a range of 0 to +20 V instead of -10 to +10 V. However, the Acromags do not support the input range 0-20 V. Since SNR is not critical for these channels (they're used only for initial alignment), I propose we simply install a voltage divider inside the chassis, just before the Acromag, for each of these signals.
Attachment 1: c1psl_feedthrough_wiring_-_By_Connector_(3).pdf
c1psl_feedthrough_wiring_-_By_Connector_(3).pdf c1psl_feedthrough_wiring_-_By_Connector_(3).pdf c1psl_feedthrough_wiring_-_By_Connector_(3).pdf c1psl_feedthrough_wiring_-_By_Connector_(3).pdf
  15276   Fri Mar 13 20:00:50 2020 JonUpdateComputersLoopback monitoring for slow machines

Summary

Today I finished implementing loopback monitors of the up/down state of the slow controls machines. They are visible on a new MEDM screen accessible from Sitemap > CDS > Slow Machine Status (pictured in attachment 1). Each monitor is a single EPICS binary channel hosted by the slow machine, which toggles its state at 1 Hz (an alive "blinker"). For each machine, the monitor is defined by a separate database file named c1[machine]_state.db located in the target directory.

This is implemented for all upgraded machines, which includes every slow machine except for c1auxey. This is the next and final one slated for replacement.

Implementation

The blinkers are currently implemented as soft channels, but I'd like to ultimately convert them to hard channels using two sinking/sourcing BIO units. This will require new wiring inside each Acromag chassis, however. For now, as soft channels, the monitors are sensitive to a failure of the host machine or a failure of the EPICS IOC. As hard channels, they will additionally be sensitive to a failure of the secondary network interface, as has been known to happen.

Each slow machine's IOC had to be restarted this afternoon to pick up the new channels. The IOCs were restarted according to the following procedure.

c1auxex

  • Disabled OPLEV servos on ETMX
  • Zeroed slow biases
  • Disabled watchdog
  • Restarted IOC
  • Reverted 1-3

​c1vac

  • Closed V1, VM1
  • Restarted IOC
  • Returned valves to original state

c1psl

  • Disabled IMC autolocker
  • Closed PSL shutter
  • Restarted IOC
  • Reverted 1-2

c1iscaux

  • ​Restarted IOC

c1susaux

  • Disabled IMC autolocker
  • Closed shutter
  • Disabled OPLEV servos on: MC1, MC2, MC3, BS, ITMX, ITMX, PRM, SRM
  • Zeroed slow biases
  • Disabled watchdogs
  • Restarted IOC
  • Reverted 1-5

The intial recovery of c1susaux did not succeed. Most visibly, the alignment state of the IFO was not restored. After some debugging, we found that the restart of the modbus service was partially failing at the final burt-restore stage. The latest snapshot file /opt/rtcds/caltech/c1/burt/autoburt/latest/c1susaux.snap was not found. I manually restored a known good snapshot from earlier in the day (15:19) and we were able to relock the IMC and XARM. GV and I were just talking earlier today about eliminating these burt-restores from the systemd files. I think we should.

Attachment 1: Screen_Shot_2020-03-13_at_7.59.55_PM.png
Screen_Shot_2020-03-13_at_7.59.55_PM.png
  15284   Thu Mar 26 17:41:18 2020 JonOmnistructureBHDBHD docs compilation

Since there has been a proliferation of BHD Google docs recently, I've linked them all from the BHD wiki page. Let's continue adding any new docs to this central list.

  15292   Thu Apr 2 16:31:33 2020 JonUpdateCDSC1AUXEY wiring + channel list
Quote:

I have made a wiring + channel list that need to be included in the new C!AUXEY Acromag.

I used Yehonathan's wiring assignments to lay the rest of groundwork for the final slow controls machine upgrade, c1auxey. Actions completed:

  • Created an internal wiring diagram for assembling the Acromag chassis (log in with LIGO.ORG credentials to view/edit)
  • Created a new target directory on the network drive:
/cvs/cds/caltech/target/c1auxey1

The "1" will be dropped after the new system is permanently installed.

  • Populated the target directory with files:
    • modbusIOC.service - wraps the EPICS IOC as a systemd service
    • ETMYaux.env - defines the EPICS environment variables
    • ETMYaux.cmd - command file to set up the EPICS IOC
    • ETMYaux.sh - enables DAC outputs to the suspension (executed lastly)
  • Created the EPICS channel databases:
    • ETMYaux.db - migration of the existing database
    • c1auxey_state.db - contains logic for loopback monitoring of the IOC "alive" state (visible from Sitemap > CDS > Slow Controls Status)

Hardware-wise, this system will require:

  • 2 Acromag XT-1221 units (ADC)
  • 1 Acromag XT-1541 unit (DAC)
  • 1 Acromag XT-1111 unit (sinking BIO)

I know that we do have these quantities left on hand. The next steps are to set up the Supermicro host and begin assembling the Acromag chassis. Both of these activities require an in-person presence, so I think this is as far as we can advance this project for now.

  15294   Fri Apr 3 12:09:53 2020 JonUpdateCDSC1AUXEY wiring + channel list
Quote:

We want to migrate the end shutter controls from c1aux to the end acromags. Could you include them to the list if not yet?

This will let us remove c1aux from the rack, I believe.

Yehonathan's list does include C1:AUX-GREEN_Y_Shutter and I copied its definition from /cvs/cds/caltech/target/c1aux/ShutterInterlock.db into the new ETMYaux.db file.

I noticed ShutterInterlock.db still contains about a dozen channels. Some of them appear to be ghosts (like the C1:AUX-PSL_Shutter[...] set, which has since become C1:PSL-PSL_Shutter[...] hosted on c1psl) but others like C1:AUX-GREEN_X_Shutter appear to still be in active use.

  15295   Fri Apr 3 13:40:07 2020 JonUpdateBHDBHD front-end complication

I wanted to pass along a complication pointed out by K. Thorne re: our plan to use Gen1 (old) Dolphin IPC cards in the new real-time machines: c1bhd, c1sus2. The implication is that we may be forced to install a very old OS (e.g., Debian 8) for compatibility with the IPC card driver, which could lead to other complications like an incompatibility with the modern network interface.

Hardware is easy - you will also need a DX switch and the cables

As for the driver - the last update (version 4.4.5) was in 2016.  The notes on it say valid for Linux kernel 2.6 to 3.x.  This implies that it will not work with Linux kernel 4.x and greater

So - Gentoo with 3.0 kernel OK, SL7 (kernel 3.10)  - OK,   Debian 8 (kernel 3.16) - OK   

But Debian 9 (kernel 4.9),Debian 10 (kernel 4.19) - NOT OK

We have Gentoo with kernel 3.0  boot server, etc. [used in L1,H1 production right now, but not much longer] The hard part here will be making sure we have network drivers for the SuperMicro 5018-MR.

CDS was never able to get real-time builds to work well on Linux kernels from 3.2 on up until we got to Debian 9. This is not to say that the tricks and stripped-down RCG we found worked for real-time on Debian 9 and 10 won’t work on, say, Debian 8.  But we have not tried.

I have a query out to Dolphin asking:

  1. Have they done any testing of these old drivers on Linux kernel 4.x (e.g., Debian 9/10)?
  2. Is there any way to buy modern IPC cards for the two new machines and interface them with our existing Gen1 network?

I'll add more info if I hear back from them.

  15299   Tue Apr 7 10:56:39 2020 JonUpdateBHDBHD front-end complication
Quote:

I have a query out to Dolphin asking:

  1. Have they done any testing of these old drivers on Linux kernel 4.x (e.g., Debian 9/10)?
  2. Is there any way to buy modern IPC cards for the two new machines and interface them with our existing Gen1 network?

Answers from Dolphin:

  1. No, and kernel 4.x (modern Linux) definitely will not work with the Gen1 cards.
  2. No, cards using different PCIe chipsets cannot be mixed.

Since upgrading every front end is out of the question, our only option is to install an old OS (Linux kernel < 3.x) on the two new machines. Based on Keith's advice, I think we should go with Debian 8. (Link to Keith's Debian 8 instructions.)

  15300   Tue Apr 7 15:30:40 2020 JonSummaryNoiseBudget40m noise budget migrated to pygwinc

In the past year, pygwinc has expanded to support not just fundamental noise calculations (e.g., quantum, thermal) but also any number of user-defined noises. These custom noise definitions can do anything, from evaluating an empirical model (e.g., electronics, suspension) to loading real noise measurements (e.g., laser AM/PM noise). Here is an example of the framework applied to H1.

Starting with the BHD review-era noises, I have set up the 40m pygwinc fork with a working noise budget which we can easily expand. Specific actions:

  • Updated the 40m fork to the latest pygwinc version (while preserving the commit history).
  • Added a directory ./CIT40m containing the 40m-specific noise budget files (created by GV).
  • Added an ipython notebook CIT40m.ipynb at the root level showing how to generate a noise budget.
  • Integrated our DAC and seismic noise estimators into pygwinc.
  • Marked the old 40m NB repo as obsolete (last commit > 2 yrs ago). Many of these noise estimates are probably stale, but I will work with GV to identify which ones can be migrated.

I set up our fork in this way to keep the 40m separate from the main pygwinc code (i.e., not added to as a built-in IFO type). With the 40m code all contained within one root-level directory (with a 40m-specific name), we should now always be able to upgrade to the latest pygwinc without creating intractable merge conflicts.

ELOG V3.1.3-