[Kira, Gautam]

We closed the loop today and implemented the PID script. I have attached the StripTool graph for an integral value of 0.5 and proportional value of 20. We had some issues getting it to work properly and it would oscillate between some low values of the control voltage. The set point here was -3.20, which corresponds to about a 20 degree increase in temperature. The next step would be to find which values of Kp, Ki, and Kd would work in this case and low pass filter the signal from the temperature sensor, and also create an MEDM screen for easier PID control.

Can't really figure out what this plot means. We need to see the sensor (in units of deg C) and the control signal (in heating power (W)). The plot should show a few step responses with the PID loop on, so that we can see the loop response time. Please zoom in on the axes so that we can see what's happening.

I created two new channels today, C1:PEM-SEIS_EX_TEMP_MON_CELCIUS, which turns the output voltage signal into degrees C, and C1:PEM-SEIS_EX_TEMP_CTRL_WATTS, which takes the input voltage from the DAC and turns it into a value of watts. I'm trying to stabilize the temperature at 35 degrees, but it's taking a lot longer than expected. Perhaps we'll need to use different values for P and I and decrease the noise in the sensor, since right now there's about a 10 degree variation between the highest and lowest values.

I did a step response for the loop from 35 degrees to 40 degrees. The PID is not properly tuned, so the signal oscillates. In the graph, the blue curve is the temperature of the can in celcius and the green curve is the heating power in watts. The x-axis is in minutes. Before, the signal was too noisy to do a proper step response, so I placed a 3.3 microF capacitor in parallel with the resistor in my temperature sensor circuit (I'll draw and attach this updated version). This created a 5 Hz low pass filter and the signal is now pretty clean.

-----

I also added in new Epics channels so that we could log the data using Data Viewer. The channels I added were C1:PEM-SEIS_EX_TEMP_MON_CELCIUS and C1:PEM-SEIS_EX_TEMP_CTRL_WATTS. I used 13023 as a guide on how to do this.

Update: the channels work and show data in Data Viewer

-----

Edit: I've attached a photo of the circuit with the capacitor indicated. It is in parallel with the resistor below it. I've attached an updated circuit diagram as well.

I have been trying to tune the PID and have managed to descrease the oscillations without saturating the actuator. I'm going to model the system to calculate the exact values of P, I and D in order to get rid of the oscillations altogether. I was going to record the data using Data Viewer, but there seems to be some issue with that, so I'm using StripTool for now.

Made some changes:

1. Set P and D gains to zero. We only need slow drift control.
2. Changed names of the python script and .ini file to distinguish it from the FSS stuff. Lives in scripts/PEM/
3. removed debug flag from argParse. To run in non-debug mode you use the "-O" option of python as usual.
4. Fixed the upper/lower limit convention for the heater. Was backwards.
5. Removed the "rail" function that was defined. We can just use numpy.clip since that's already built in.

There is also now a StripTool file in the scripts/PEM directory which has appropriate channel names and scales for PID loop tuning. Use this file!

I'm leaving it running over the weekend with K_I = -0.003. There is a StripTool on rossa which you can watch. The code itself is running on a tmux session on megatron. Let's ONLY run this code there until we're satisfied that things are good.

Update Sun Apr 8 00:40:11 2018: Lowered gain by factors of 3 down to -0.0001 Saturday afternoon. Seems like still oscillating a bit, now with a ~4 hour period. Setting it to -3e-5 now. Usually we have a linear feedback loop, but our actuation voltage actually gets squared (P = I^2 R) before being integrated to produce temperature. Wonder if we should think of linearizing the feedback control signal to make the loop act nicer.

Update Sun Apr 8 21:09:48 2018: Set K_I = -1e-5 earlier today. Seems to have stabilized nearly, but temperature swings are still +/- 1 K. Will need to add some proportional feedback (K_P) to increase the loop bandwidth, but system is at least sort of stable now. Probably should start construction of EY,BS systems now.

Earth quake M5.3    2018-04-05 19:29:16UTC          Santa Cruz Island, CA

I created an MEDM screen for the PID control. In addition, I added a new EPICS channel for the setpoint so that it could be adjusted using the MEDM screen.

Edit: forgot to mention the channel name is C1:PEM-SEIS_EX_TEMP_SETPOINT

Edit #2: the path for the MEDM is /opt/rtcds/caltech/c1/medm/c1pem/C1PEM_SEIS_EX_TCTRL.adl

An update to the screen. I changed the min/max values for some of the parameters, as well as changing the script so that I could specify the integral gain in terms of 1e-5. I've also added this screen to the PEM tab in the sitemap.

Another update. I've changed the on/off button so that it's visible which state it's in. I did that by changing the type of C1:PEM-SEIS-EX_TEMP_SLOWLOOP from ai to bi (I checked the FSS script and copied the entry for the slowloop). Previously, MEDM was giving me an error that it wasn't an ENUM value when I wanted to use a choice button to indicate the value of slowloop, and this solved the issue. I've also added a StripTool button.

changed the setpoint of the EX Seismomter T ctrl servo from 35 to 39 C to see if this helps the stability by decreasing the cooldown time constant.

I've updated the sketches and added in front panels for the seismometer block and the 1U panel (attachments 3 and 4). There was an issue when it came to the panel on the block because the hole is only big enough for the cable that already exists there and there is no space to add in the D-9 connector. Not quite sure how to resolve this issue. Attachment 7 is the current panel on the seismometer block. Attachments 5 and 6 are the updated temperature circuit and the heater circuit.

The boxes will be located in the short racks at EX and EY to minimize cable length.

Can you please add dimensions to the drawing, so we can see if things fit and what the cable lenghts need to be?

For the panel on the granite slab, we should use a thinner piece of metal and mount it with an offset so that the D-sub cable can be fished through the hole in the slab. The hole is wide enough for 2 cables, but not 2 connectors.

Attached is a 8-day minute trend of the heater control signals, as well as the in-loop temperature sensor (which underestimates the true fluctuations; we really need an out-of-loop sensor attached to the can or seismometer).

You can see that since the last tuning (on the 13th), its been stable at the set point of 39 C with 8.5 - 10 W of heating power. Need to add the PID loop settings (all the sliders on the MEDM screen) to the frames so that we can help in diagnosing. Also, fix the spelling of "Celcisususs".

Yesterday, I changed the P gain of the PID loop from zero to  +0.1. Seems good so far; will monitor for a couple days to see if we're in the right ballpark. Main issue in the stability may now be that the quantization noise is too big for the temperature sensor. If so, we should consider subtracting off the DC value (with a V ref) and then amplifying before ADC.

I've added in the dimensions to my sketch.

It seems like placing the two connectors right next to each other would allow both cables to just barely go through the hole in the block.

since we're just going from the short rack (not the tall rack) to the seismometer, can't we use a cable shorter than 45' ?

the panel should be completely replaced like I described. We don't want to try to squeeze it in artificially and torque the wires. It just needs to be separated from the slab by a few more cm.

If we lay the cable along the floor then it should be around 6' to the current setup and about 20' to the actual seismometer.

Edit: 16 gauge wire should be good.

Steve secured the GPS time server in the rack above the AA board and removed the wooden block that it was resting on. The new rack is shown in attachment 1.

I then opened the AA board to see why the channels aren't working. Even though the board was powered and outputting 4.6 V, none of the chips were getting power. I must have shorted something while trying to diagnose this and the board is no longer powered either.

The schematic is given in D990147. The D68L8EX filter is bypassed on all the channels, as can be seen in attachment 3, so the board isn't really doing anything. Rana suggested that we could just bypass the whole circuit by wiring the IN channels directly to the OUT channels going to the ADC. I'll try that next for a single channel.

This shows a step response of the EX seis temp control with K_I = -1 and K_P = -0.1. The time constants for both heatup and cooldown are ~2 hours.

I'm not so sure if the PID code itself makes sense though:

  # The basic finite-difference PID approximation
  e[0] = (p-s)
  print("Error signal = {}" .format(e[0])) 

  # These are the main equations of the PID Process
  u[0] = u[1]
  u[0] = u[0] + Kp * (e[0] - e[1])
  u[0] = u[0] + Ki * (e[0])
  u[0] = u[0] + Kd * (e[0] - 2*e[1] + e[2])
 

Seems like the Proportional term uses the difference (or derivative) of the error signal. This makes it more likely to pick up some high frequency noise; maybe we should low pass this signal somewhat, or at least implement a running average.

Since we still don't have an out of loop sensor or a PSL room temperature monitor or a particle counter in the frames, I've disabled the PID loop to see how much the can temperature varies with no feedback. Please leave it this way for a few days.

I've attached the final sketch for the panel on the granite block.

In the ongoing attempt to recover the seismometer BLRMS, I removed the AA board from the rack and modified the BS seismometer Z channel. The BS_Z BLRMs seem to be recovered after this modification.

I removed the three resistors from the output of the circuit and wired the input and from the seismometer directly to the input to the ADC. The modified schematic is shown in attachment 1. Attachments 2 and 3 show the top and bottom of the modified board. The board is doing nothing now other than serving as a connector for this channel.

I put the board back in the rack and injected a 2 Vpp signal into the BS_Z channel and saw +/- 1600 cts in C1PEM-SEIS_BS_Z. I then plugged the seismometer back into the board and took the spectrum shown in attachment 4. This shows the working Z channel giving a reasonable seismic spectrum. Note that X and Y are not modified yet.

If there are no objections, I will modify all the other channels on the board in the same way tomorrow.

I wired all 32 channels going to the AA board directly to the ADC as described in the previous log. However, instead of using the old AA board and bypassing the whole circuit, I just used a breakout board as is shown in the first attachment. I put the board back in the rack and reconnected all of the cables.

The seismic BLRMs appear to be working again. A PSD of the BS seismometers is shown in attachment 2. Tomorrow I'll look at how much the ADC alone is suppressing the common mode 60 Hz noise on each of the channels.

Steve: 5 of ADC DAC In Line Test Boards [ D060124 ] ordered. They should be here within 10 days.

Increased the Integral gain (from -1 to -4) on the EX temperature controller. This didn't work a few weeks ago, but now with the added P gain, it seems stable. Daily temperature swings are now ~3x smaller.

Notes for Kira on what we need to do tomorrow (Friday):

1. add the MEDM screen EPICS values to the DAQ so that we can plot those trends DONE
2. add the out-of-loop sensor to the EX can
3. reboot the AUX-EX so we can pick up the new channels and the fixed spelling of the old channels DONE
4. Re-install EX seismometer and hook up seismometer channels to PEM DAQ so we can start testing its performance.

For those who are flabbergasted by the way I calibrated the TEMP_MON channel from volts to deg C, here's how:

XMgrace->Data->Transformations->Geometric Transforms...

use the 'scale' and 'translate' fields to change the slope and offset for calibration in the obvious ways

Yesterday I wired the outputs from the seismometers directly to the ADC input bypassing the old AA board circuit as is described in this elog. The old circuit converted the single-ended output from the seismometers to a differential signal. Today I looked at whether 60 Hz noise is worse going directly into the ADC due to the loss of the common mode rejection previously provided by the conversion to differential signals.

I split the output from the BS Z seismometer to the new board and to an SR785. On the SR785 I measured the difference between the inner and outer conductors of the seismometer output, i.e. A-B with A the center conductor and B the outer conductor, with grounded input. At the same time I took a DTT spectrum of C1:PEM-SEIS_BS_Z_IN1. Both spectra were taken with 1 Hz bandwidth and 25 averages. The setup is shown in attachment 1.

The spectra are shown in attachment 2. The DTT spectrum was converted from counts to volts by multiplying by 2 * 10 V/32768 cts where the extra factor of 2 is from converting from single-ended to differential input. If there was common 60 Hz noise that the ADC was picking up we would expect to see less noise at 60 Hz in the SR785 spectrum measured directly at the output from the seismometer since that was a differential measurement. Since both spectra have the same 60 Hz noise, this noise is differential.

After fixing the spelling of the EX temperature readback, I also added all of the MEDM sliders to the C0EDCU.ini file (making sure to add an even number of channels). Restarted FB (after installing telnet on rossa):

telnet fb 8083

> shutdown

preferred method of posting DataViewer images: print as a SVG image (since its vectorized). Then from the command line do:

inkscape steven.svg --export-pdf=vass.pdf

for whatever reason, I am unable to get minute or second trends from nodus for any channels (IMC, PEM, etc) since the reboot. has there been some more recent FB failure or is this still a bug since last years FB catastrophe?

I've attached a sketch of how the panel will be mounted. We should make a small rectangular box that would raise the panel from the block by 1 cm or so to allow the cables to fit into the hole in the block without getting bent. It also has to be airtight so maybe having a thin layer of rubber between the mount and block would be good.

I added an out of loop sensor to the can by placing the lab temperature sensor inside the can. I'm not sure which channel is logging this temperature though. I also noticed that the StripTool still had the old misspelled name for the temperature readout so I fixed that as well.

I've attached a picture of the setup.

[Kira, Johannes]

I connected up the channels for the ADC Acromag a while back and we were planning to install it today so that we could set up a new channel for the out of loop sensor. Unfortunately, the Acromag seems to be broken. We connected up a precision 10V voltage to one of the channels, but the Acromag read out ~7V and it kept fluctuating. Even after calibration, we still got the same result. When enabling the legacy support, we got ~11V. But when we measured the voltage at the screw terminals with a multimeter and it showed 10V, so the issue is not with the wiring. All of the channels have this same issue. We will be ordering more Acromags soon, so hopefully we'll be able to set up the channel soon. I've attached a picture of the Acromag along with the front panel with the channels labeled

As part of investigation into this issue, Jonathan Hanks pointed out that the "minute trends" being recorded by our system were actually only being recorded every 120 seconds (a.k.a. 2 minutes). He had fixed the appropriate line in the parameter file, but had not restarted the FW processes. I had restarted it on Friday. (but failed to elog it !) 

To check if this made any difference, I pulled 1 hour of "minute trend" data for the PSL table temperature channel from ~1 hour ago, and compared the number of datapoints against a 1 hour minute trend time series from 2 May. I've put the display with # of datapoints (for an identical length of time) from before [left] and after [right] the restart next to the plots in Attachment #1. Seems like we are getting minute trends written every 60 seconds now, as it should be .

The unavailability of trends from nodus is a separate issue for which JH has suggested another fix, to be elogged separately.

Here are a few things I will be working on:

• Design PCB boards for the heater circuit and temperature sensor circuits [by wednesday]
• Order the front panel I've designed for the seismometer block [today]
• [next week?] install the new Acromag when it comes

The replacement Acromag we scooped from the West Bridge E-Shop does actually seem to work, although we thought it was broken - at first it was just outputting zeros, but after I did the calibration procedure, applying +10 V and -10 V, respectively, it was reporting voltage correctly, over the full range. I don't know why the factory settings would be messed up, but it had been out of the box before. I did this only with channel 7, so you need to calibrate channels 0-6 and confirm that they indeed also work properly.

I tried calibrating the other channels today, but they still fluctuate. Sometimes they do stabilize at +/- 10V, but then suddenly drop to 5 or 6 V before climbing back up to 10. Turning the legacy off made it go only up to 6.67V. This happens for all the channels, even after doing a factory reset and recalibrating. Not sure what's happening here.

As described in this elog, the ADC for the seismometers now has the signals wired directly to the ADC instead of going through an AA board or other circuit to remove any common mode noise. This elog describes one test of the common mode rejection of this setup. Guantanamo suggested comparing directly with a recent spectrum taken a few months before the new setup described in this elog.

Today I took a spectrum (attachment 1) of C1:PEM-MIC_2 (Ch17) and C1:PEM-MIC_3 (Ch18) with input to the ADC terminated with 50 Ohms. These are two of the channels plotted in the previous spectrum, though I don't know how that plot was normalized. It's clear that there are now strong 60 Hz harmonic peaks that were not there before, so this new setup does have worse common mode rejection.

Earlier today Kira and I reconnected the EX seismometer. I just took some spectra of all three seismometers, shown in the attachments, to compare with past data and to do a rough check of the calibration.

This elog has a spectra from 2010 (GUR1 is now EY) and this elog has one for BS at lower frequencies from 2017. Note that the EX seismometers now have strong peaks that are not at 60 Hz harmonics. Other than these peaks, these old spectra roughly match up with the ones taken today, so the callibration is still roughly the same. I couldn't find any old data for EX (GUR2) though so I don't know for sure that these peaks weren't there before.

gautam 20180517 0930: In 2017, Gur2 (now EX) looked like this. Still peaky, but the peaks seem shifted in frequency. Steve also informed me that the Gur1 and Gur2 cables were swapped n times, so perhaps we shouldn't read too much into that.

I've moved my setup to the actual seismometer. I attached the temperature sensor to the seismometer (attachment 1) with duct tape, though this is temporary. I will be monitoring the temperature fluctuations of the seismometer for a whole day then take the can off and repeat the test. The can isn't clamped down so the insulation isn't perfect, so I'd expect to see some noticeable fluctuations even with the can on. I've also labeled the long cable for the temperatuse sensor readout (attachments 2 and 3). There will also be an out of loop sensor added in later, but for this test since I am not running the loop it doesn't matter which sensor I monitor. Attachment 4 is a picture of the current setup.

Here is the result of my test. I think I'll leave the can on over the weekend because there's a long period of time where the seismometer heated up by 0.8 degrees so I can't fully see the fluctuations over a full 24 hour period.

I guess it's fine for now while we are still finalizing the setup at EX, but we should eventually line up the seismometer axes with the IFO axes. Is there a photo of the orientation of the seismometer pre heater can tests? If not, probably good to make some sort of markings on the granite slab / seismometer to allow easy lining up of these axes...

I have attached the graph for the seismometer temperature fluctuations over 3 days. As we can see, there is a noticeable fluctuation in daily temperature as well as a difference between days in the maximum and minimum temperatures. I will repeat this test but take the can off to see if there's any difference between having the can on or off.

It appears that one of the wires was disconnected overnight or this morning so I wasn't able to gather data over a full 24 hour period. Perhaps someone accidentally kicked it. I placed some cones in that area so hopefully the wires won't be in the way as much and I can get the data tomorrow. From the data I do have it seems that the seismometer is at a colder temperature when the can is not on, though it is difficult to see by how many degrees the temperature fluctuates. I've included the data from 5 days back to see the comparison.

This time the test went without issue. The first attachment is the data for the past 24 hours and the second attachment is the full data over 6 days. The average temperature fluctuations (from highest point to lowest point) for the can on was 0.43 C and for the can off it came out to 0.55 C. In addition the seismometer with the can off is about 1 C cooler than with the can on. I'd like to leave the can off until the end of the week so we can get a comparable data set for both the can on and off. Eventually I'll need to figure out a way to clamp the can down to the block in order to get better insulation and hopefully get even smaller temperature fluctuations.

Rana said that it wasn't necessary to gather more data on the temperature fluctuations so I have reconnected the heater circuit and restarted the PID loop with the can on the seismometer.

We will need to order a few things for our final setup.

1. 1U box to place the heater circuit and temperature circuits in. The minimum depth that will fit all the electronics is 10 inches according to my sketch.
   • I found two possibilities online for this. I don't know exactly what our budget is, but this one is $144. According to the datasheet, the front panel is less than 19 inches wide, so if we are to order this one, I've adjusted the width of the panel I designed to match the width of the panel that comes with it I've labeled it in the attached file as 1U-panel-1.
   • The other possibility is this one. It comes with handles already which is quite nice. I wasn't able to find a price for it on the website.
2. Front panel for the 1U box and block panel. I've attached them as .fpd files below in one .zip file. Not sure if this is the correct way to attach them, though.
3. We'll also need a 16 gauge cable that has 6 wires bunched together. This is to connect up the heater circuit and AD590s. The other cables that we will need can be found in the lab.

I've updated the parts list to be an excel document and included every single part we will need. This is ony a first draft so it will probably be updated in the future. I also made a mistake in hole sizing for the front panel so I've updated it and attached it as well (second attachment).

Edit: re-attached the EX can panel fpd file so that everything is in one place

Chris replaced some air condition filters and ordered some replacement filter today.

[koji, gautam]

We noticed quite a strong burning smell in the office area and control room ~20mins ago. We did a round of the bake lab, 40m VEA and the perimeter of the CES building, and saw nothing burning. But the smell persists inside the office area/control room (although it may be getting less noticeable). There is a whining noise coming from the fan belt on top of the office area. Anyways, since nothing seems to be burning down, we are not investigating further.

Steve [ 10am 5-31 ] we should always check partical count in IFO room

Service requested 

I have attached the result of running the PID script on the seismometer with the can on. The daily fluctuations are no more than 0.07 degrees off from the setpoint of 39 degrees. Not really sure what happened in the past day to cause the strange behavior. It seems to have returned back to normal today.

Got this 1U box from the Y arm that we could potentially use (attachment 1). It doesn't have handles on the front but I guess we could attach them if necessary. Attachment 2 is a switch that could be used instead of a light up switch, but now we need to add LEDs on the front panel that indicate that the switch is functional. Attachment 3 is a terminal block that we can use to attach the 16 gage wire to since it is thick and attaching it directly to the board would be difficult. If this is alright to use then I'll change up my designs for the front panel and PCB to accomodate these parts.

We have 6 of these boards now in cabinet E7