I propose we set up a temperature sensor network as described in attachment 1.

Here there are two types of units:

• BASE-GATEWAY
  • Holds the processor to talk to the network through Modbus TCP/IP protocol.
  • This unit itself has a temperature sensor in it. It is powered by a power adaptor or PoE from the switch.
  • Each base unit can have at max 2 extended temperature probes ENV-TEMP.
• ENV-TEMP
  • This is just a temperature probe with no other capabilities.
  • It is powered via PoE from the BASE-GATEWAY unit.

Proposal is

• to put 2 ENV-TEMP sensors (#1 and #2) along the Y-arm at the end and midway. These are powered and read through a BASE-GATEWAY (#A) at the vertex. The BASE-GATEWAY (#A) will serve as temperature sensor for the vertex.
• We put one ENV-TEMP(#3) at the X-end powered and read through by another BASE-GATEWAY (#B) at the midpoint of X-arm.
• Both BASE-GATEWAY are connected by ethernet cables to the network switch. That's it.

These sensors can be configured over network by going to their assigned IP addresses. I'm not sure at the moment how to configure the dB files to get them to write on slow EPICS channels.

We will have an unused port on the BASE-GATEWAY (#B) which can be used to run another temperature sensor, maybe at an important rack, PSL table or somewhere else.

In future, if more sensors are required, there are expansion (network switch like) options for BASE-GATEWAY or daisy-chain options for the probes.

Edit Fri Jun 18 16:28:13 2021 :

See this [wiki page](https://wiki-40m.ligo.caltech.edu/Physical_Environment_Monitoring/Thermometers) for updated plan and final quote.

Anchal mentioned it would be good to put more details about how I arrived at the values needed to configure the modbus drive for the temperature sensor, since this information is not in the manual and is hard to find on the internet, so here is a breakdown.

So the generic format is:

drvAsynIPPortConfigure("<TCP_PORT_NAME>","<UNIT_IP_ADDRESS>:502",0,0,1)
modbusInterposeConfig("<TCP_PORT_NAME>",0,5000,0)
drvModbusAsynConfigure("<PORT_NAME>","<TCP_PORT_NAME>",<slaveAddress>,<modbusFunction>,<modbusStartAddress>,<modbusLength>,<dataType>,<pollMsec>,<plcType>)

which in our case become:

drvAsynIPPortConfigure("c1pemxendtemp","192.168.113.240:502",0,0,1)
modbusInterposeConfig("c1pemxendtemp",0,5000,0)
drvModbusAsynConfigure("C1PEMXENDTEMP","c1pemxendtemp",0,4,199,2,0,1000,"ServerCheck")

As can be seen, the parameters of the first two functions "drvAsynIPPortConfigure" and "modbusInterposeConfig" are straight forward, so we restrict our discussion to the case of third function "drvModbusAsynConfigure". Well, after hours of trolling the internet, I was able to piece together a coherent picture of what needs doing and I have summarised them in the table below.

drvModbusAsynConfigure

Once the asyn IP or serial port driver has been created, and the modbusInterpose driver has been configured, a modbus port driver is created with the following command:

drvModbusAsynConfigure(portName,                # used by channel definitions in .db file to reference this unit)
                       tcpPortName,             # reference to portName created with drvAsynIPPortConfigure command
                       slaveAddress,            # 
                       modbusFunction,          # 
                       modbusStartAddress,      # 
                       modbusLength,            # length in dataType units
                       dataType,                # 
                       pollMsec,                # how frequently to request a value in [ms]
                       plcType);                #

Useful links

https://nodus.ligo.caltech.edu:8081/40m/16214

https://nodus.ligo.caltech.edu:8081/40m/16269

https://nodus.ligo.caltech.edu:8081/40m/16270

https://nodus.ligo.caltech.edu:8081/40m/16274

http://manuals.serverscheck.com/InfraSensing_Sensors_Platform.pdf

http://manuals.serverscheck.com/InfraSensing_Modbus_manualv5.pdf

https://community.serverscheck.com/discussion/comment/7419#Comment_7419

https://wiki-40m.ligo.caltech.edu/CDS/SlowControls

https://www.slac.stanford.edu/grp/ssrl/spear/epics/site/modbus/modbusDoc.html#Creating_a_modbus_port_driver

https://github.com/riptideio/pymodbus

https://en.wikipedia.org/wiki/Modbus

https://deltamotion.com/support/webhelp/rmctools/Communications/Ethernet/Supported_Protocols/Ethernet_Modbus_TCP.htm

For the past couple of days the 0.1 to 0.3 Hz RMS seismic noise along BS-X has increased. Attachment 1 shows the hour trend in the last ~ 10 days. We'll keep monitoring it, but one thing to note is how uncorrelated it seems to be from other frequency bands. The vertical axis in the plot is in um / s

Looks like this increase is correlated for BS/EX/EY. So it is likely to be real.

Comparison between 9/15 (UTC) (Attachment 1) and 9/10 (UTC) (Attachment 2)

I have placed a GT321 particle counter on top of the MC1/MC3 chamber next to the BS chamber. The serial cable is connected to c1psl computer on 1X2 using 2 usb extenders (blue in color) over the PSL enclosure and over the 1X1 rack.

The main serial communication script for this counter by Radhika is present in 40m/labutils/serial_com/gt321.py.

A 40m specific application script is present in the new git repo for 40m scripts, in 40m/scripts/PEM/particleCounter.py. Our plan is to slowly migrate the legacy scripts directory to this repo overtime. I've cloned this repo in the nfs shared directory at /opt/rtcds/caltech/c1/Git/40m/scripts which makes the scripts available at all computers and keep them upto date in all computers.

The particle counter script is running on c1psl through a systemd service, using service file 40m/scripts/PEM/particleCounter.service. Locally in c1psl, /etc/systemd/system/particleCounter.service is symbollically linked to the file in the file.

Following channels for particle counter needed to be created as I could not find any existing particle counter channels.

[C1:PEM-BS_PAR_CTS_0p3_UM]
[C1:PEM-BS_PAR_CTS_0p5_UM]
[C1:PEM-BS_PAR_CTS_1_UM]
[C1:PEM-BS_PAR_CTS_2_UM]
[C1:PEM-BS_PAR_CTS_5_UM]

These are created from 40m/softChansModbus/particleCountChans.db database file. Computer optimus is running a docker container to serve as EPICS server for such soft channels. To add or edit channels, one just need to add new database file or edit database files in thsi repo and on optimus do:

controls@optimus|~> sudo docker container restart softchansmodbus_SoftChans_1
softchansmodbus_SoftChans_1

that's it.

I've added the above channels to /opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini to record them in framebuilder. Starting from 11:20 am Oct 20, 2021 PDT, the data on these channels is from BS chamber area. Currently the script is running continuosly, which means 0.3u particles are sampled every minute, 0.5u twice in 5 minutes and 1u, 2u, and 5u particles are sampled once in 5 minutes. We can reduce the sampling rate if this seems unncessary to us.

The particle count channel names were changes yesterday to follow naming conventions used at the sites. Following are the new names:

C1:PEM-BS_DUST_300NM
C1:PEM-BS_DUST_500NM
C1:PEM-BS_DUST_1000NM
C1:PEM-BS_DUST_2000NM
C1:PEM-BS_DUST_5000NM
 

The legacy count channels are kept alive with C1:PEM-count_full copying C1:PEM-BS_DUST_1000NM channel and C1:PEM-count_half copying C1:PEM-BS_DUST_500NM channel.

Attachment one is the particle counter trend since 8:30 am morning today when the HVAC wokr started. Seems like there was some peak particle presence around 11 am. The particle counter even counted 8 counts of particles size above 5um!

SVG doesn't work in my browser(s). Can we use PDF as our standard for all graphics other than photos (PNG/JPG) ?

rethinking what I said on Wednesday - its not a good idea to put the particle counter on a vac chamber with optics inside. The rumble from the air pump shows up in the acoustic noise of the interferometer. Let's look for a way to mount it near the BS chamber, but attached to something other than vacuum chambers and optical tables.

I have done some reading about where would be the best place to put the particle counter. The ISO standard (14644-1:2015) for cleanrooms is one every 1000 m^2 so one for every 30m x 30m space. We should have the particle counter reasonably close to the open chamber and all the manufactures that I read about suggest a little more than 1 every 30x30m. We will have it much closer than this so it is nice to know that it should still get a good reading. They also suggest keeping it in the open and not tucked away which is a little obvious. I think the best spot is attached to the cable tray that is right above the door to the control room. This should put it out of the way and within about 5m of where we are working. I ordered some cables to route it over there last night so when they come in I can put it up there. 

[temperature sensor]

git repo: https://git.ligo.org/40m/tempsensor.git

todo

Update the temp sensor channels to fit with cds format, ie. "C1:PEM-TEMP_EX", "C1:PEM-TEMP_EY", "C1:PEM-TEMP_BS"

- Use FLOAT32_LE data format for the database file (/cvs/cds/caltech/target/c1pem1/tempsensor/C1PEMaux.db) to create the new channels.

- Keep the old datadase code and channels so we can compare with new temp channels afterwards. Also we need a 1-month overlap b4 deleting the old channels.

[sus medm screen]

git repo: https://git.ligo.org/40m/susmedmscreen.git

todo (from talk with Koji)

- Link stateword display to open "C1CDS_FE_STATUS.adl"

- Damp filter and Lock filter buttons should open a 3x1 filter screen so that the 6 filters are opened by 2 buttons compared to the old screen that has 3 buttons connected to 2X1 filter screen

- Make the LOCKIN signla modulation flow diagramlook more like the old 40m screen since that is a better layout

- Move load coefficient button to top of sus medm screen (beside stateword)

- The rectangular red outline around the oplev display is confusing and needs to be modified for clarity

- COMM tag block should not be 3D as this suggests it is a button. Make it flat and change tag name to indicate individual watchdog control as this better reflect its functionality. Rename current watchdog switch to watchdog master is it does what the 5 COMM switches do at once.

- Macro pass need to be better documented so that when we call the sus screens from locations other than sitemap, we should know what macro variables to pass in, like DCU_ID etc.

- Edit sitemap.adl to point only to the new screens. Then create a button on the new screen that points to the old screen. This way, we can still access the old screen without clogging sitemap.

- Move the new screen location to a subfolder of where the current sus screens reside, /opt/rtcds/userapps/trunk/sus/c1/medm/templates

- Rename the overview screen (SUS_CUST_HSSS_OVERVIEW.adl) to use the SUS_SINGLE nomenclature, i.e. SUS_SINGLE_OVERVIEW.adl

- Keep an eye of the cpu usage of c1pem as we add BLRMS block for other optics. 

[Temperature sensor]

Added new temp EPICs channels to database file (/cvs/cds/caltech/target/c1pem1/tempsensor/C1PEMaux.db)

Added new temp EPICs channels to slow channels ini file (/opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini)

[SUS medm screen]

Moved new SUS screen to location : /opt/rtcds/userapps/trunk/sus/c1/medm/templates/NEW_SUS_SCREENS

Place button on the new screen to link to the old screen and replace old screens link on sitemap.

Fixed Load Coefficient button location issue

Fixed LOCKIN flow diagram issue

Fixed watchdog labelling issue

Linked STATE WORD block to FrontEnd STATUS screen

Replaced the 2x1 pit/yaw filter screens for LOCK and DAMP fliters with 3x1 LPY filter screen

*Need some more time to figure out the OPTLEV red indicator

I mounted the particle counter over the BS chamber attached to the cable tray as seen in Attachment 1. The signal cable runs through an active 30ft cable to the 1x2 rack. the wire is labeled and runs properly through the cable tray. The particle counter is plugged in at the power strip attached near the cable tray. The power cord is also labeled. 

I restarted the particle counter service in the c1psl computer in the /etc/systemd/system/ folder using the commands

I cannged the usb hub assigned in the service file to ttyUSB0 which is what we saw the computer had named it.

Checking the channels from this elog show the same particle count as when testing with the buttons and checking the screen. It seems that the channels had been down but are now restarted.

nice - please update the particle counter page in the 40m wiki. Its probably years out of date.

For the proposed construction in the NW corner of the CES building (near the 40m BS chamber), they did a simulated construction activity on Wednesday from 12-1.

In the attached image, you can see the effect as seen in our seismometers:

this image is calculated by the 40m summary pages codes that Tega has been shepherding back to life, luckily just in time for this test.

Since our local time PDT = UTC - 7 hours, 1900 UTC = noon local. So most of the disturbance happens from 1130-1200, presumably while they are setting up the heavy equipment. If you look in the summary pages for that day, you can also see the IM lost lock. Unclear if this was due to their work or if it was coincidence. Thoughts?

I pressed the Auto-Z(ero) button for ~ 3 seconds at ~9:55 local (pacific) time on the trillium interface on 1X5.

This nicely brought the sensing signal back to ~zero. See attachment

Some basic info:

• BS Seismometer is T240 (Trillium)
• The interface unit is at 1X5 Slot 26. D1002694
• aLIGO Trillium 240 Interface Quick Start Guide T1000742

thanks, this seems to have recentered well.

It looks like it started to act funny at 0400 UTC on 10/24, so thats 9 PM on Sunday in the 40m. What was happening then?

I implemented the BLRMS on WFS1/2_I_PIT/YAW outputs and using there value, a HEPA state can be defined. I've currently set it up to use average of 10-30 Hz noise on WFS signals low passed at 0.3 Hz. For ON threshold of 100 and OFF threshold of 50, it is working for my limited testing time. To read HEPA state, one can do caget C1:IOO_HEPA_STATE. I've turned OFF HEPA for tonight's shimmer test. This is bare bones quick attempt, people can make the screen more beautiful and add more complexity if required in future.

this is very exciting! Its the beginning of the task of reducing vast amounts of PEM data into human-useful (bite sized) info. Imagine if we had 60 Hz BLRMS on various channels - we would know exactly when ground loops were happening or disappearing.

Over the past few days I have been trying to understand the existing sensor and heater related hardware by manual inspection and combing elogs. From what I understand, I believe a lot of the hardware was built from scratch by Kira in 2017-18. She put in place the insulation, built the sensor and heater circuits and installed and interfaced everything with EPICS. This let her successfully do step response tests and also execute PID control of the temperature of the can. As suggested by Paco, I am trying to summarise my findings in this elog. This is the block diagram of the overall system.

In the lab I was able to locate two identical temperature sensor boards, one for monitoring ambient temps and another for the can temp, which has a sensor attached to the inner seismometer can. Attachment 1 shows the actual board.

The schematic for the temperature sensor can be found here. It was later clarified that she is using AD590 and not the AD592. I tested the ambient sensor board by hooking it up to an oscilloscope and heating the sensor up. It seems to work like she described, and the output voltage goes down with heat. I will later figure out interfacing it with an ADC and calibrating it if I cannot find Kira's old work related to this. The setup there also has +/-15V Sorensen's for power (I probed these and they work), a BNC for temp readout and a bunch of other wiring for the heater. This wiring comes from the nearby rack which should have the heater circuit, but I haven't yet been able to locate it because there is too much stuff.

Coming to the heater circuit - I found the schematic for it here. I am still unsure about what the power range it can operate over, but this plot seems to imply it saturates at around 55W. This circuit underwent many different iterations so I am unsure what load resistance was actually used finally and it is not clear from the elogs. It might be a pair of heaters combined in series or parallel. Additionally, this elog by Shruti from June 2018 implies that the original circuit that Kira made ended up breaking at several different points. She said there were attempts to fix it, but I cannot find any updates after that and I think her SURF ended. I don't know the current status of the heater circuit. I am not sure where it is stored.

I was hoping to utilise the old EPICS channels set up by Kira for heater control as well as sensor readout to verify if everything was still functional and initially just read out the temperature of the can and find out how much it fluctuates. I planned to later also try to replicate the PID results. But Paco explained how the Acromag system worked and told me that utilising the old channels will not be possible right now, as all the channels have been used up for more critical tasks and none of the old interfaces that Kira had set up can be used now, even if the hardware was fine. 

Also, the can temperature sensor board has a wobbly solder joint at one of the power connectors, and we cannot move it to the lab because the sensor is anchored to the inner can. There is no power indicator LED to signal if the connection is secure so its kind of unusable right now. Paco and JC told me that soldering near the X end is not possible because the fumes would harm the optics. We will look for solutions to this. One option would be to make an entirely new board and keep a mechanical connector for the sensor. 

I wanted to mention here that I have a printed circuit board design (LIGO-D1800304) for using AD590 as temperature sensors. I believe printed boards and all required components are stored on the wire shelf in the WB EE shop on a box labeled with this DCC number. This circuit is also meant to be read by Acromag, maybe it can come of some use to you. You can check out the use in CTN lab in WB near the south-east end of the table.

I measured the dimensions of the puck we shall be using as the mass for the toy model. It has a diameter of 75 mm and a thickness of 25 mm. Assuming a density of 2.7 g/cm^3, this brings the mass to 298.2 g.

How about measuring the actual weight with a scale? There are a couple of scales on Yuta's desk.

Thanks, I was actually looking for a scale earlier but could not find it after asking a couple of people. One of the scales reads 298 g, while the other has a limit of 200g. Looks like it is indeed aluminium.

Paco suggested I start putting together the beginnings of a heating system so that I can get a better idea of what parts and equipment we need and can order them ASAP. This is also to try and match the real data with simulation models for the basic system and then add complexities like insulation foam (which is also currently not in hand) later. This should give insights into loopholes in the models and also into the estimation of parameters like the convective transfer factor (h) which aren't trivial to accurately constrain in simulation according to what I've read.

I will edit in a circuit schematic by tomorrow (done), but the circuit is quite similar to the new one made by Kevin and Kira which I linked in a previous elog. For now, I have used a makeshift and messy combination of resistors (there are very few high wattage ones that we could find in both 40m and the EE shop) which manages to achieve a max current of around 0.25A through a resistive element of 50 ohms, leading to a peak power output of 3.22 W which is good enough for the puck. This resistor comes in a nice brass housing (visible below the puck in image) which makes it convenient to attach to the puck and pump heat into it relatively efficiently. I tested this circuit by turning it on for 30 minutes and the puck did warm up a bit. I will work on calibrating and attaching the old temperature sensor to this tomorrow so I can start doing step responses and quantifying things.

One critical difference from the old circuit is that this one is meant to drive the MOSFET only in fully-on or fully-off mode to minimize heat dissipation through it, so we basically use PWM to regulate the heat from zero to the peak value. JC and I also set up the ethernet wiring for the Raspberry Pi 2 at the experiment as well as my desk so that I can control it from my laptop over LAN. I tested PWM using Python with an LED that can be seen in the image. Now I basically just have to plug the Pi into the circuit input and measure the RMS current to confirm that it is also controllable in a similar way and linear as we would expect, and also check out the output waveform.

[andrei, advait]

I have analyzed the temperature data that we have collected for the past 10 days or so from the X-end temperature sensor that is accessible via Acromag.

The average temperature was 20.74 Celsius, noticeably cooler than the average 24 Celsius in the control room. The day-night temperature variations are clearly visible.

And here is the power spectrum density of these variations:

The red line indicates the frequency of the day-night cycle (1/24hrs), while the green line indicates twice the frequency of the day-night cycle. We can see that the two lines align fairly well with the peaks, which is to be expected. Here I used n_avg=3 in Paco's spec_dens() function, but I am not sure this is the right value to use. Please let me know if you have suggestions in this regard.

As for the noise estimates of the detector used to collected these data, I am not sure how to get those. The above temperature data was collected using a detector that no one knows where exactly it is in the lab, and I don't even know what type of sensor it is. Perhaps there is a way to get a noise estimate from the data that I collected? Again, please let me know.

However, we have a rough estimate for the noise of the AD590 sensor made by Kira that we calibrated and used for the step response test. We estimated this by calculating the standard deviation of the signal measured from the 0 C ice bath. This translates to 0.0134 C. We will probably repeat this measurement in a better way (once we build our own version of the sensor) because the duration of the actual 0 C measurement was just around 2 minutes.

As this circuit is inefficient and can waste a lot of power through the R resistance, Rana asked me to come up with a better design that does not have this issue and mainly dissipates heat only through the heater.

I realised that the kind of circuit I originally made with the op-amp feeding into the FET is unnecessarily complicated, and came up with a more simple one that does the job and does not dissipate extra heat - 

The input to this is a RPi PWM signal that goes through an op-amp based non-inverting amplifier. I used this to comfortably clear the GS threshold of the MOSFET, which has a quoted maximum value of 4V. I implemented this on a breadboard and it seems to work fine with a 1 kHz PWM.

Please let me know if anybody has any feedback or suggestions for improvement.

I found the sensor that we used to measure the tmeperature in the lab. It is a SensorGateway (BASE-WIRED) and according to its datasheet it has a temperature resolution of 0.1 Celsius and an accuracy of 1 Celsius. Regardless, I also let it collect some data overnight to get a better grasp of the noise.

I put the sensor in a box, wrapped around two towels. I tried closing the box as much as possible, but the cables connecting to the sensor prevented me from closing the box all the way. I put the box in the same spot where the sensor sat when the environment data was collected. Here is the picture of the device:

I started the data collection yesterday at around 5pm. Here is the raw temperature data collected by the sensor:

The temperature first increased due to the heat put out by the sensor unit itself (this is a server temp sensor, which has its network server for easy access among other features). We can see that the temperature peaked around 4 hours after the experiment started, which is to be expected, around that time (9pm) the temperature in the lab starts to decrease dramatically. The sensor reached a fairly stable equilibrium around hour 10 which would translate to around 3am, which is again to be expected considering our environment temperature measurements. I used the stable region between hours 10 and 13 for noise estimates:

We can see here that the sensor kept "jumping" between two values: 30.9592 Celsius and 30.8967 Celsius. The difference between these readings is 0.0625 Celsius, and it can be seen that there are no intermediary values between these 2 values. I also performed a spectrum analysis just to see what happens:

The dip at around 0.9 Hz is also visbile in the environment temperature spectrum.

I connected

On Wednesday, Rana showed me that there are two AD590 sensors in the PSL/FSS refcav. It seems that one of the sensors is inside a box which is insulated with foam, while the other sensor is just outside the foam. It is likely that even the sensor outside the foam is not reading the environment temperature, but the difference between the two sensors gives us a good idea of the effects of the foam. Here are the plots:

As you can see, RCTEMP seems to have higher variations than RMTEMP. There is also an offset, probably due to the heat disippated by the hardware inside the box. In the freuquency domain:

where we can clearly see that there is a big difference in intensity at frequencies higher than 0.01 Hz. Moreover, RMTEMP (which is indside the box) has the lower intensity, thus showing that the foam is a low-pass filter. The same story can be seen in the coherence plot:

I have since wanted to hopefully use RCTEMP's readings to make a frequency spectrum like the one I made before using the server temperature sensor. However, I am still struggling to download the data from the channel for periods larger than a few hours. I would need data for at least 5 days in order to capture the very low end like 1/24hr.

[andrei, advait]

We wrapped the puck in foam. We also performed a calibration on the AD590 sensors @ 25.3 Celsius and then proceeded with an additional 6-point calibration using a digital thermometer (DT310LAB).

This is the current setup There is also a lid similar in size with the bottom lid, and the entire foam is currently held together with a heavy wire role that was put on top of the lid.

We couldn't replace the red tape with the yellow-transparent tape because the yellow tape is not strong enough and an air gap forms between the resistor and the puck. As a result, we will avoid getting above ~45 Celsius in to prevent the red tape from becoming too hot. We are now proceeding with a step response measurement and will follow up on that soon.

Update: we have a working "I" controller for the puck. Here is the data:

In achieving this, we had to make a few changes;

 1. In the heater circuit, the resistor is now connected to its own power supply set for 10V. This enables us to have around 50% duty cycle at equilibrium for T_ref=45C:

*This is the heater profile for the above "I" controller run
 2.  We changed the frequency of the Pulse Width Modulation (PWM) of the RaspberryPi that is used to switch the MOSFET. It is now 100Hz compared to the previous 1kHz. I observed using the oscilloscope that the PWM was delivering ~870Hz when set for 1kHz and that it also struggled to deliver accurate duty cycles, especially for low duty cycles (below 5%). The duty cycles should always be kept between 5%-95% for accurate modulation.
 3. In case we want to change the power supply voltage, the controller code can be easily adjusted by just changing the maximum voltage constant.

We decided to verify the PWM accuracy by adjusting the oscilloscope to fit a hundred cycles of the signal and measuring the mean voltage - the variation is linear as expected, although it does diverge at the extremeties as noted earlier.

Additionally, I compared the experimental data in the previous plots with a simulation using the model that we constructed, and after accounting for time delays in heating the system, we see a very good agreement between the two - 

On Friday afternoon, I added a couple of fuses to complete the circuit for a power supply for the heater that is inside the seismometer can. A cable had been set up previously and connected to the X end rack, but the connection was broken. I am adding before and after images to show the change.

As you can see, the fuses were already there on the rack but not connected to anything. The red and black wires to the left lead to the bottom most power supply, which is currently set at 25V. The black and white wires to the right lead to the seismometer. I was accompanied by Yehonathan while I did this. 

We will also set up some Acromag channels for the heater control and temperature readout soon.

Also, if someone could share resources for adding channels it would be great. There were two old slow Acromag channels, C1:PEM-SEIS_EX_TEMP_MON and C1:PEM-SEIS_EX_TEMP_CTRL_WATTS before, but these were disconnected and are now in use for something else. Yehonathan said there are no Acromag channels available now, but I might be able to set up fast channels to replace these.

JC shared this wiki with me, and while the "To add or edit a channel to an existing subsystem" section seems relevant (to add new channels to the SEIS subsystem), the folder structure seems very different from what is described there and these instructions seem outdated. It also does not explain where the physical connections of the BNC cables would need to be made.

Any help with this would be appreciated.

[Koji, Advait]

Summary

• Spare ADC/DAC channels were prepared for Advait's experiment.
  C1:PEM-SEIS_EX_TEMP2_MON gives us the raw count number for the temp sensor signal.
  C1:PEM-SEIS_EX_TEMP2_CTRL can provide the output for the heater control (calibration ~0.5/2^15 V/count)
• The remote switching of the X End shutter was restored.

End X configuration

• Advait already had the temp sensor connected to the power supply. And the heater cable was laid out between the seismometer and the 1X9 rack.
• We went to the X End:
  • Connected the temp sensor signal to the ADC adapter chassis at CH5 (fifth connector ... the numbering starts from CH1) - This corresponds to the ADC channel of CH28.
  • DAC connection
    • We were unsure if the DAC adapter was configured to be single-ended (SE) or differential. The unit is S2200003 . Fortunately, this DCC entry had a photo of the PCB board. We confirmed that the unit was configured to have SE outputs.
    • The DAC adapter unit has four general BNC output ports (CH13-CH16). CH13 was reserved for the ALS Laser Slow control.
    • (Unrelated topic) Connected green shutter TTL switching input of Unibritz shutter controller to CH14
    • The heater input was connected to CH15 (the channel last but one)

Software configuration

• There is no Acromag at the X End right now. Therefore, for now, we want to use the fast channels to implement the channels.
• Opened c1scx model:
  • Added ADC0 CH28 to the adc ch selector
  • Added a PEM box with "top_names" option. ADC0 CH28 signal goes into this box.
  • In the PEM box, the input goes into EPICS OUT supposed to be "C1:PEM-SEIS_EX_TEMP2_MON".
  • The EPICS IN, supposed to be "C1:PEM-SEIS_EX_TEMP2_CTRL", goes to the output of the box. This output is connected to DAC CH14 (i.e. 15th channel)
     
  • Added an AUX box. In the box, the EPICS channel "C1:AUX-GREEN_Shutter" is read. This goes to a filter module (C1:AUX-GREEN_X_SHUTTER_CAL).
  • The output of the AUX box goes into DAC CH13.
• Opened c1asx model:
  • Checked the DAC output. Some spare filter modules were connected to the DAC channels we wanted to use. They were cleaned up.
• Edited c1auxex EPICS database
  • Now the shutter channels are double booked. So, commented out C1:AUX-GREEN_Shutter entry in /cvs/cds/caltech/target/c1auxex/ETMXaux.db .
  • Restarted modbusIOC by "sudo systemctl restart modbusIOC.service" on c1auxex
• The model was built and installed.
  $ rtcds build-install c1scx
$ rtcds build-install c1asx
$ rtcds build-install c1x01
• Upon restarting the models, all the machines crashed with DK. As per Yehonathan's suggestion, /opt/rtcds/caltech/c1/Git/40m/scripts/cds/restartAllModels.sh was used to bring everything back online. It was painful, but it worked after a few times of repetitions.
  • By the way, I checked what's wrong with burtrestore at this opportunity.
  • First of all, when the restart script is used, most of the time something is already wrong. So in situ recording of the epics values does not provide a healthy snapshot of the system.
  • Secondly, applying auto snapshot at once may cause the incomplete restoration whatever reason (like the command line is too long etc).
  • It'd be good to write/use a script to apply a bunch of snapshots one by one.
  • Maybe cleaning of the request file is also necessary.

Hardware check again

• We noticed that the ADC channel for the temp sensor was saturated. It was confirmed that the temp sensor sig has -14V. This needs to be fixed before the cable is reconnected to the ADC adapter chassis.
• Checked the DAC adapter ports' calibration:  +30000, 0, -30000 count to C1:PEM-SEIS_EX_TEMP2_CTRL gave +4.563V, +0.0047V, -4.573V respectively.
  This corresponds to 1.523e-4 V/count, This number is close to 1/2^16 V/count.
• The output saturates around +/-5V because... the diff to se converter in the DAC adapter has the gain of 1/2.
• The X end green shutter:
  • The shutter output was amplified by 30000 at C1:AUX-GREEN_X_SHUTTER_CAL filter module.
  • The shutter mode was reverted to "REMOTE".
  • It was confirmed that the shutter can be switched by giving 0 (close) or 1 (open) to C1:AUX-GREEN_X_Shutter.
    I quickly made buttons to switch the shutter from the ALS and IFO_ALIGN screens. Also now the shutter toggle scripts controlls the shutter as before.

Yesterday during the recovery work from the CDS restart, I found ASS was not working for both arms. Both arms were flashing and locking, but their alignment seemed suboptimal.

Also, something seemed wrong: For example, the ITMX was too excited when the OPLEV servos were on. The arm locking was achieved when the ITMX oplev was turned off.

Yes, it seems like ITMX UL sensor issue has reared its ugly head again (see attachment).

[advait, andrei]

We needed two additional ADC channels for the seismometer setup, to read out the temperature of the seismometer can/enclosure, and another for the ambient temperature near the can. Koji let me know that there are two EPICS ADC channels, C1:X01-MADC0_EPICS_CH29 and C1:X01-MADC0_EPICS_CH30 that are currently unused. These are also conveniently CH 6 and 7 on the ADC board referred to in the previous post. 

Andrei and I laid out a couple of new BNC cables from the 1x9 rack to the seismometer, along the older cables. The image below shows the current configuration of the ADC - 

The cable labelled EX-SEIS ... is for the temperature of the seismometer connected to CH 5, SEIS_AMB_TEMP is for the ambient temp connected to CH 6, and SEIS_CAN_TEMP is for the can temp connected to CH 7.

All three channels yield a count number on caget-ing, so I had to calibrate these too. The calibration is given by voltage = [ -3.0593-4 * (counts) - 0.00367 ] V and I obtained this by reading values for three positive voltages. 

As we move towards the goal of controlling the temperature of the seismometer near the X end, I am setting up hardware and interfaces to provide sensor readout and heater control there.

Seismometer temperature sensor configuration:

• Yesterday night I removed the foam-insulated can off the seismometer to attach AD590 temperature sensors (we are using the TO-52 package).
• I attached one to curved surface of the can and another to the seismometer inside :

• The wires go through a hole in the concrete base and are long enough to reach the top of the whole enclosure. In the image of the can, the thick red wires to the right lead to the heater (a parallel combination of two resistances, which I measured to be 12.9 ohms), while the sensor is on the left. 
• I've used duct tape to secure the sensors, and ensured there is good contact and it doesn't wobble around. The wires are also taped down so that the sensor doesn't move a lot if someone tugs them.

There was already an older sensor attached to the seismometer, presumably from 5 years ago when Kira was working on this. This one used the 8-pin SOIC package, and in our experience it was hard to achieve and maintain good contact with this variety of the sensor because of the awkward shape, so this sensor was replaced. The new sensor circuit also has pin headers for connection, unlike the old one where the sensor was permanently soldered to a board.

Here is a rundown of my work so far on RL controllers for the puck:

Environment: I implemented an environment that simulates the puck's temperature, according to our crude fits of the step response and pid controller of the puck in foam. Currently, this environment assumes constant ambient temperature and doesn't simualte the delays between the applied heat and the measured temperature change. However, one can easily adjust this environment by just changing the underlying Tdot equation. The environment works in steps of 10s and executes 3600*5 steps (for a total simulated time of 50h). The reward function is L2: at every timestep, it adds -(T_puck - T_ref)^2 to the total reward. The reward is only disclosed to the agent on the last timstep, so far this seemed to have work better, perhaps because the reward you get at any timestep is heavily dependent on the previous state (in 10s you can't raise the temperature enough if you are 5 degrees away from T_ref, so the agent gets penalized again).

Agents: I have made the first implementations for two agents: DQN and PPO, that could be used to controller the puck's temperature. Currently, these implementations are rough and require hyperparameter tuning and bigger networks.

• PPO: this agent works on continuous environemnts, meaning that it outputs any real number between min and max as its action. In the case of the puck, the action is the applied heat (in Watts). However, this agent has proven very slow to train on the LIGO-Caltech cluster (despite the networks being very small: 3 dense layers, 2->10->5->output for both critic and policy networks), so I only have a very rough training done, of 20 episodes Here is a plot:
  
  Here T_ref is represented by the red line and the blue line is the output bahvior of the policy. Keep in mind that this is only 20 episodes, and at the time of writing the training was killed (don't know why) but the measured total reward at 25 episodes was around 10% better than the above policy (sadly, no plot for that).
• DQN: this agent works on discrete environments, meaning that I had to modify the training environment to be dsicrete (which can be done in 1 line of code using a wrapper, see env.ipynb). There is an argument to be made that the actuator (in our case the PWM of the Raspberry Pi) has only a limited number of possible outputs: for example, the RPi PWM doesn't produce any measureable difference in duty cycles that differ by less than 0.05%. As Rana suggested, it would be best to discretize according to the number of bits that the plant can work with (I am not sure how many that is for the case of the RPi). Anyways, I am currently training the DQN on the env discretized to 100 actions, but it currently seems very unstable, even after I lowered the learning rate. The great thing about DQN is that it has been very fast to train, even on a cpu. Hence, I think the next step would be to just increase the size of the q-network, as the currrent 3 dense layers seems inadequate (2->10->5->100). I will come back with a better plot of this agent very soon.

Moving on, I want to:

1. Implement a reverb replay server for the PPO in hopes that it will speed up the training
2. Add T_env noise and sensor noise simulations to the environment.
3. Start hyperparameter tuning
4. Check results and compare with lthe inear controller

[Koji, Hiroki]

We resotred the noisy ITMX UL sensor reported in elog #17713

Based on Koji's experience, we checked the cables from ITMX sensors and pressed their connectors (Attachment 1).

We confirmed that the noise spectrum of the ITMX UL sensor was improved (Attachment 2: measuered w/o damping) but we do not know the exact connector that had been affecting the spectrum.

Even though the noise issue of an OSEM was resolved, the issue with ITMX still continues.
OPLEV pitch servo can't be turned on as it induces the suspension running. So there is something systematically wrong with the suspension.

This requires some checking of the whitening / dewhitening situation, or even other unknown cause.

I added a heater circuit to the seismometer, and moved the 4 channel temperature readout circuit that I built in there as well. The heater uses the lowermost power supply available on the 1x9 rack, visible below reading 24V -

• It was turned off prior to my using it, and there is no current drawn when I'm not using it - so I think it is not connected to any other device. I mentioned how I added DIN fuses to complete the circuit in another elog.
• The heater circuit is simply a MOSFET being operated as a switch, with a 2x scaled version of the DAC signal being fed into the gate, letting current through when the DAC is at 4.6V (30000 counts).
• The resistance of the heating pads is 13 ohms. This means we can achieve a max power of roughly 44W, which is delivered to the can that weighs 15 kg.

I also interfaced the temperature sensors to the BNCs we laid earlier, enabling readout through EPICS. This let us do a step response test where I heated the can for 4 hours and let it cool after that, letting us observe the temperature profiles - 

This plot is a lot noisier than the ones we got through the RPi ADC for the puck, and we think this is due to the EPICS system being worse. The noise is initially absolutely terrible, and then suddenly improves after around 1.5 hours. We are not sure what would cause this behaviour and will have to investigate.

The code for this is available at /users/Mayank/step.py

[Koji, Hiroki]

We investigated the cause of the unstable OPLEV damping in ITMX and found some clues.
 

Transfer functions from coils to OPLEVs:

We measured the transfer functions from the OSEM coils (UL, UR, LR, LL) to the OPLEV sensors (PIT, YAW) for ITMX, ITMY, ETMX and ETMY under the condition of OSEM damping.
Attachment 1 shows the transfer function from the ITMX UR coil to the ITMX OPLEV as an example.
We compared the obtained transfer functions at 10Hz: 

COIL    OL     Response @10 Hz [dB]
                      ITMX    ITMY    ETMX    ETMY
LL       PIT      -78      -93*      -87       -52
LL       YAW    -72      -97*      -87       -50
LR      PIT      -93*     -94*      -88       -53
LR      YAW    -70      -94*      -84       -46
UL      PIT      -73      -65       -91        -46
UL      YAW    -73      -65       -87        -50
UR     PIT       -91*    -63       -87        -53
UR     YAW     -73     -63       -87        -52

Noteworthy poinsts:

• For ITMY, responses from LL and LR are small compared to UL and UR. Something wrong with LL and LR?
• For ITMX, responses from LR and UR to PIT are small. I wonder why the responses to PIT are different from that to YAW.
   

OLTF of OPLEV servo for ITMX PIT:

We measured the OLTF of the OPLEV servo for ITMX PIT with the condition of only OSEM damping (OPLEV servo loop was disconnected during the measurement).
Attachment 2 shows the modelled OLTF and Attachment 3 shows the measured OLTF.
We found some structure and its huge phase delay around 1 Hz. This structure seems to be affecting the stability of ITMX OPLEV servo.

The shape of this structure changed with diffrent OLTF gain of ITMX POS for some reason.

We tried to avoid this structure by increasing the gain of OPLEV servo (0.2 -> ~40).
As a result, the movement of the OPLEV beam spot was stabilized but ITMX POS got noisy (large fringing at AS).

I had a suspicion that some of the ITMX coils were not properly working. So individual coils of the 4 test masses were excited while the arm-locking servo output was observed.

The servo output of the arm locking (denoted as ) can be expressed as:

,

where G is the open loop TF, A is the actuator TF, and dx is the displacement noise of the FP arm.
When the arm is shaken with a coil, dx is A' dv where A' is the actuator TF of the coil in question, and dv is the applied excitation.

i.e.

If the length coupling of the coil is all identical, A' is expected to be A/4 and this TF should be ~1/4 well below the control bandwidth.

For ETMs the TFs were close to 1/4 and all the phases were the same, although I could not understand why the gain goes down at low frequency.
This needs more careful thinking. But at least the TF were close to identical between the coils.

However, the situation is different for the ITMs:
- ITMX LL and LR seemed OK. However, the UL response was too low (by ~1/25) and the UR response had an opposite phase.
- ITMY LL and LR seemed OK. However, the UL and UR responses were too low (by ~1/25)

We want to check if the CDS is behaving properly, or this is the electronics issue.

[Hiroki, Yehonathan, Yuta, Koji]

Summary

• The c1sus DAC0 issue seemed to be the driving capability of the DAC card.
• The DAC issue was solved by replacing 2x Anti-Imaging modules (D000186 RevB) with 2x Rev C equipped with input amps.
• This may mean that the previous DAC card failure was also the same issue. (i.e. The card could still be OK)
   
• This change may have increased the coil driver responses of the vertex suspensions by a factor of 2
  as the conventional RevB had the single-end inputs, but Rev C has the differential ones.
   
• Enabling the full actuation range caused significant misalignment of the suspensions. But the IFO was realigned and now looks much more normal than before.
  • The ITMX OPLEV servos could be engaged with the conventional gain. No instability is observed anymore.
  • The BS OPLEV gains were too high. |G|=0.4 -> 0.2 to maintain stability.
  • Confirmed Yarm ASS functioning well as before
  • Xarm ASS is partially functioning as before. Some gains were reduced to keep the stability, but still, careful inspection is necessary.

Details

• The monitor outputs of the Anti Anti-Imaging modules (D000186 RevB) were checked while the corresponding coil EXC channels were excited via awggui.
• When the DAC was fully driven, we observed the output was only limited to the negative side of the full scale. The negative side was clipped at -5V. (See Attachment 1). This was the case for all 16 channels for DAC0. We found no issue for DAC1 and DAC2.
• Disconnect the DAC0 SCSI cable from the SCSI-IDC40 adapter unit to isolate the DAC from the analog chain. Could we observe the bipolar signal? Indeed yes, we saw the bipolar signal. (Attachment 2). (We use a SCSI-DSUB9 adapter board.)
• We wondered if this was the failure of the AI modules. Replacing the left D000186 with the same Rev B spare. No change.
• Checked if power cycling the AI module reset it. As this was done with the DAC connected, we lost positive and negative outputs of the DAC (OMG!)
• Rebooting c1sus solved the issue, and indeed the DAC exhibited a bipolar signal even with the Rev B cards.
• We thought we solved the issue! Came back to the control room and worked on a boot fest. After about an hour of struggle, all the RTS are back online. (GJ!)
• But we found that we are back to the original state. (same as Attachment 1)

Some break ....

• Went back to the rack. My thought was that the DAC could work when it was not connected to the AI modules. Once it is connected to the AI modules, the positive side disappears. Probably can't DAC drive all the load?
• Disconnected the 2nd IDC40 cable from the adapter (i.e., 2nd AI isolated). This brings the bipolar signals back! (Attachment 3)
• If the 1st IDC40 was this connected while the 2nd cable was connected, this brings the bipolar signals back to the later half 8chs (Attachment 4)
   
• We have a couple of D000186 Rev C spares. I checked the circuit diagram of Rev C (it is not on DCC but in Jay's circuit diagram archive on 40m BOX). I found that each channel of Rev C has an INA134. This is not a true instrumentation amplifier but a differential amplifier with R=25kOhm. So it was promising.
• Indeed, replacing a RevB with a RevC already solved the issue. Just in case, I've replaced both RevBs for DAC0.

• I wondered what the input impedance of RevB was. The Frequency Devices Filter module has the min input impedance of 10kOhm. However, the Rev B input is single-ended. I believe the negative side is shorted to GND! This probably mean that the gain with Rev C is x2 of the conventional one
   
• IFO alignment / locking / oplev test: as described at the summary.

Update

1. I have finished setting up Neptune for easy access to the train/eval metrics of the RL models. Here is a link where you can track my progress. Neptune is an extremely useful tool that allows you to see in real time how the models are bahving: you can see nice plots of losses and returns, as well as my source code, the stdout and stderr, and even the resources plots for the node that my jobs got allocated to. Feel free to play around with it.
2. Currently, there are are only 2 runs (I just started them): one with DQN (RLCON-8), the other with PPO (RLCON-7).
3. As outlined before, I have tried improving the PPO agent by replacing the replay buffer with a reverb server (which is supposed to be the fast way of doing a replay buffer). However, no big speed ups have been achieved. The problem is that PPO is using a special type of layer: a distributional layer. I am unsure as to how this layer works, but certainly it is not a normal "dense" layer. After changing all other components, it seems that the PPO algorithm is very slow not because of the replay as I initially thought, but because of the actual training of the agent at line agent.train(experience).  However, I still have a lead: there is an error signal about tensorflow retracing (see run RLCON-8). I am now looking into that.
4. In the mean time, I will start increasing the size of the Q-net for the DQN and increasing the num_actions to 201 (so a resolution of 0.5% in duty cycle). That is because RLCON-7 seems to not learn too much: the loss is really unstable and the evaluation avg_return is terrible.

Aside on avg_return if you wonder what this number means: basically, at each step of the environment (the code that simulates the temperature of the puck) I am takig the error to be . Then, for an entire episode, the reward is given by the average of these errors (so it gives you average error for all steps in one episode). Then, the reported avg_return is calculated by taking the mean of average errors for 5 (as of now) episodes. Basically, sqrt(|avg_return|) gives you an average deviation away from T_ref: so if avg_return=-25, then you should expect the model to be consistently around 5 degrees away from T_ref.

As the preparation of the flow sensor installation, I added two slow channels via c1psl (Acromag IOC).
C1:PEM-HEPA_FLOW_PSL1
C1:PEM-HEPA_FLOW_PSL2

This entry explains how this was done because there is a tricky hidden process that is scarcely explained elsewhere upon adding the slow channels
Tag: How to add slow channels / C0ECDU.ini / Acromag / Advanced LIGO RTS stand-alone EPICS data concentrator

1. Edited the db file

I learned that the spare ADC channels ADC4 CH5 and CH6 are available on c1psl
(See http://nodus.ligo.caltech.edu:8080/40m/[ELOG 17727] )

Created /cvs/cds/caltech/target/c1psl/psl_pem.db

c1psl>cat psl_pem.db
# Greated for Acromag PEM - Koji Arai 2023/8/3
#

grecord(ai,"C1:PEM-HEPA_FLOW_PSL1")
{
        field(DESC,"HEPA Air Flow Speed Monitor Unit 1")
        field(SCAN,".1 second")
    field(DTYP,"asynInt32")
    field(INP,"@asynMask(C1PSL_XT1221_ADC04 5 -16)MODBUS_DATA")
        field(PREC,"3")
        field(LINR,"LINEAR")
        field(EGUF,"10")
        field(EGUL,"-10")
        field(EGU,"Volts")
        field(HOPR,"10")
        field(LOPR,"-10")
        field(ASLO,"1.09225")
}

grecord(ai,"C1:PEM-HEPA_FLOW_PSL2")
{
        field(DESC,"HEPA Air Flow Speed Monitor Unit 2")
        field(SCAN,".1 second")
    field(DTYP,"asynInt32")
    field(INP,"@asynMask(C1PSL_XT1221_ADC04 6 -16)MODBUS_DATA")
        field(PREC,"3")
        field(LINR,"LINEAR")
        field(EGUF,"10")
        field(EGUL,"-10")
        field(EGU,"Volts")
        field(HOPR,"10")
        field(LOPR,"-10")
        field(ASLO,"1.09225")
}

2. Added the db file to cmd file.
Added the following line to /cvs/cds/caltech/target/c1psl/C1_PSL.cmd as

dbLoadDatabase("/cvs/cds/caltech/target/c1psl/psl_pem.db")

3. Restart modbusIOC at c1psl

c1psl>ssh c1psl
controls@c1psl's password:
...
controls@c1psl:~$ sudo systemctl restart modbusIOC.service

4. Added the channels to C0EDCU.ini

Add the following lines to /opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini

Leave the first ~20 channels untouched, as they appears on dataviewer as the default channels.

#HEPA Air Flow Mon channels @c1psl

[C1:PEM-HEPA_FLOW_PSL1]
[C1:PEM-HEPA_FLOW_PSL2]

5. Restart fb

telnet fb 8087
Trying 192.168.113...
Connected to fb.martian.
Escape character is '^]'.
daqd> shutdown

6. Ssh into c1sus and run the following command:

This is a new process. Until this is done, the channels are not recorded.

controls@nodus|daq> ssh c1sus

controls@c1sus:~$ sudo systemctl restart rts-edc_c1sus

As reported already, we resolved the c1sus DAC0 issue by introducing the differential receivers for the DAC output CHs.

The issue remained in DAC1 (SRM/MC2/MC1/MC3) and DAC2 (All 8 vertex side OSEMs). Their outputs were still received by the single-ended AI boards.
This meant that DAC1 and DAC2 negative side outputs were shorted to ground. This was not great.
(See Attachment 1 for the configuration)

Mitigation

The DAC SCSI-IDC40 adapter units were modified to resolve the issue. This modification was done for the two adapter units among the three.
The modified units were indicated as "DAC Neg out isolated"

These adapter boxes will eventually be removed when we make the transition to the aLIGO electronics. So it's convenient us to touch them.
In the box, the + and - outputs were lined up with a narrow gap. So the modification was a bit cumbersome. The below is the procedure

• First, the solder resist was removed at the corner of the wiring, and copper lines were cut one by one.
• The isolation of the DAC +/-, the isolation of the output sides each other were checked for all the channels.
• Note1: I used a spare PCB found on the workbench for the first modification. I found that one of the left channel (CH9-16) has the output +/- pin connected internally. This would cause + output shorted to the gnd when RevB Ai board is connected. This problem was resolved by cutting one of the IDC40 pin. So don't be surprised if you find one of the pin is missing for the left IDC40 connector for DAC1.
• Note2: One of the + signal line was damaged why the PCB for DAC2 was prepared.

Result

The replacement was done with the watchdogs shutdown. After the replacement, the watchdogs were reloaded and the MC locking were resumed.
I found the apparent behavior of the suspension looked OK but we should keep checking if everything is absolutely fine.

It was surprising that the BS/PRM/SRM OPLEVs lost their spots, but it turned out that it is not related to the electronics work. (See next elog)