Need a working IO chassis connected to c1ioo in order to bring the MC_L into the digital realm, and then via RFM transmit to the c1sus machine.
Move the c1iscey IO chassis to c1ioo while the c1ioo chassis is at downs.
The c1iscey chassis however doesn't seem to be talking to the c1ioo computer. I tried changing the host interface card on the c1ioo chassis. I took out One Stop Systems HIB2-x4-H interface card with serial number 26638 from the c1ioo computer and put in the One Stop Systems HIB2-x4-H with serial number 35242 in from c1iscey into c1ioo. Still didn't work.
All the lights are red on the interface card on the actual chassis and its cooling fan isn't spinning.
Using dmesg on c1ioo shows that it does not see any of the ADC/DAC/BO cards.
I'm going to wait until tomorrow morning when Rolf gets a chance to look at the c1ioo chassis over at Downs to determine the next step. If we fix the c1ioo chassis, I'm move the c1iscey chassis and its host interface board back to the end.
[Tega, Anchal, Paco]
After talking to Anchal, it was made clear that chiara is not the place to host the modbus service for the temperature sensors. The obvious machine is c1pem, but the startup cmd script loads c object files and it is not clear how easy it would integrate the modbus functionality since we can only login via telnet, so we decided to instead host the service on c1susaux. We also modified the /etc/motd file on c1susaucx which displays the welcome message during login to inform the user that this machine hosts the modbus service for the temperature sensor. Anchal plans to also document this information on the temperature sensor wiki at some point in the future when the page is updated to include what has been learnt so far.
We might also consider updating the database file to a more modern way of reading the temperature sensor data using FLOAT32_LE which is available on EPICs version 3.14 and above, instead of the current method which works but leaves the reader bemused by the bitwise operations that convert the two 16 bits words (A and B) to IEEE-754 32-bit float, via
field(INPC, "0x8000") # Hi word, sign bit
field(INPD, "0x7F80") # Hi word, exponent mask
field(INPE, "0x00FF") # Hi word, mantissa mask (incl hidden bit)
field(INPF, "150") # Exponent offset plus 23-bit mantissa shift
field(INPG, "0x0080") # Mantissa hidden bit
field(INPJ, "65536") # Hi/Lo mantissa ratio
as opposed to the more modern form
When I put away the lenses we had used for measuring the RF transfer functions of the QPD heads, I saw that I'd removed them from the cabinet containing green endtable optics, but hadn't noticed the sign forbidding their removal. I'll talk with Koji/Gautam about what happened and what should be done.
We moved our one STS1 from the x-arm end to the vortex. We record the data as STS1 in c1pem @ 256Hz. x is still north-south.
JD: This is actually an STS-2. We just call it C1:PEM-SEIS_STS1.... to differentiate the 3 STSs that we have from one another (assuming we plug in the other two).
[Paco, Ian, Tega]
We moved the white rack (formerly unused along the YARM) to a position between 1X3, and 1X4. For this task we temporarily removed the hepas near the enclosures, but have since restored them.
I looked at the CAD layout and it seems like we will clearly be clipping POY if we move SRM by 7.5cm. Since POY is not visible at low power, we cannot be sure about the clipping.
We should have a plan B before we move everything. I suggest we move a combination of SRM and SR2 to get the desired SRC length.
Moving SR2 will require extra effort to walk the beam unclipped through all the 6 output steering mirrors that follow; but there will be little room for error if we use irides to propagate the beam through the first 4 mirrors that are in the BS and ITMY chamber.
We turned off the power of the seismometers and moved the Guralp1 close to the STS. Both are now situated below the center of the mode cleaner vacuum tube.
We oriented the X axis of the STS & Guralp1 along the X axis of the interferometer. Then we turned on the power again, but the STS channels don't give any signal. We think this is, because we didn't push the auto zero button.
After pressing the auto-zero button (a lot of times) of the STS breakout box & aligning the bubble in the STS, we could finally get data from STS (Bacardi). So, now STS2 (Bacardi - Serial NR. 100151) is working!
The teststand has some non-trivial issue with Myrinet card (either software or hardware) which even teh experts are saying they don't remember how to fix it. CDS with mx was iin use more than a decade ago, so it is hard to find support for issues with it now and will be the same in future. We need to wrap up this test procedure one way or another now, so I have following two options moving forward:
So these are the two options we have. We should discuss which one to take in the mattermost chat or in upcoming meeting.
I've finished the MEDM portion of the RCG FiltMuxMatrix part. Now it generates an appropriate medm screen for the matrix, with links to all the filter banks. The filter bank .adl files are also generated, and placed in a sub directory with the name of the filter matrix as the name of the sub directory.
The input is the first number and the output is the second number. This particular matrix has 5 inputs (0 through 4) and 15 outputs (0 through 14). Unfortunately, the filter names can't be longer than 24 characters, which forced me to use numbers instead of actual part names for the input and output.
The key to the numbers is:
To get this working required modifications to the feCodeGen.pl and the creation of mkfiltmatrix.pl (which was based off of mkmatrix.pl). These are located in /cvs/cds/caltech/cds/advLigoRTS/src/epics/util/
In related news, I asked Valera if he could load the simulated plant filters he had generated, and after several tries, his answer was no. He says it has the same format as those filter they pass to the feed forward banks down in Livingston, so he's not sure why they won't work.
I tested my script, FillFotonFilterMatrix.py, on some simple second order section filters (like gain of 1, with b1 = 1.01, b2 = 0.02, a1 = 1.03, a2 = 1.04), and it populated the foton filter file correctly, and was parsed fine by Foton itself. So I'm going to claim the script is done and its the fault of the filters we're trying to load. This script is now living in /cvs/cds/caltech/chans/ , along with a name file called lsp_dof2pd_mtrx.txt which tells the script that DARM is 0, CARM is 1, etc. To run it, you also need a SOS.txt file with the filters to load, similar to the one Valera posted here, but preferably loadable.
I also updated my progess on the wiki, here.
Now I am studying about the behavior of the Q-factor in the resonant circuit because the Q-factor of the circuit directly determine the performance as the EOM driver.
Here I summarize the fundamental which explains why Q-factor is important.
The EOM driver circuit can be approximately described as shown in figure below
Z represents the impedance of a resonant circuit.
In an ideal case, the transformer just raise the voltage level n-times larger. Rin is the output impedance of the signal source and usually has 50[Ohm].
The transformer also makes the impedance Z 1/n^2 smaller. Therefore this configuration gives a following relation between Vin and Vout.
Where G is the gain for the voltage. And G goes to a maximum value when Rin=Z/n2. This relation is shown clearly in the following plot.
Note that I put Rin=50 [Ohm] for calculating the plot.
Under the condition Rin=Z/n2( generally referred as impedance matching ), the maximum gain can be expressed as;
It means that larger Z makes more efficient gain. In our case, interested Z is considered as the impedance at a resonance.
The key point of the story is:
"The recipe to exploit maximum benefit from a resonant EOM"
- Make a resonant EOM circuit. Measure the impedance Z at the resonance.
- This Z determines the optimum turn ratio n of the step-up transformer.
(n2 = Z/Rin where Rin is 50Ohm in our case.)
- This n gives the maximum gain Gmax (= n/2) that can be obtained with the step up transformer.
And, the impedance matching is also satisfied in this condition.
OK: The larger Z, the better. The higher Q, the Z larger, thus the better.
(Although the relationship between Z and Q were not described in the original post.)
So, how can we make the Q higher? What is the recipe for the resonant circuit?
=> Choose the components with smaller loss (resistance). The details will be provided by Kiwamu soon???
When I was young (3 months ago), I thought...
I was just too thoughtless. In reality, they are closely related each other.
A high Q resonant circuit has a high residual resistance at the resonant frequency. As far as the impedance is higher than the equivalent output impedance of the driving circuit (i.e. Z>Rin n2), we get the benefit of increasing the turn ratio of the transformer. In other words, "the performance of the resonant EOM is limited by the turn ratio of the transformer." (give us more turns!)
OK. So can we increase the turn ratio infinitely? No. Once Rin n2 gets larger than Z, you no longer get the benefit of the impedance transforming. The output impedance of the signal source yields too much voltage drop.
There is an optimum point for n. That is the above recipe.
So, a low Q resonant EOM has a destiny to be useless. But high Q EOM still needs to be optimized. As far as we use a transformer with a low turn ratio, it only shows ordinary performance.
I have been working about multi-resonant EOM since last week.
In order to characterize the behavior of the each components, I have made a simple LC tank circuit.
You can see the picture of the circuit below.
Before constructing the circuit, I made an "ideal" calculation of the transfer function without any assumptions by my hand and pen.
Most difficult part in the calculation is the dealing with a transformer analytically. Eventually I found how to deal with it in the analytical calculation.
The comparison of the calculated and measured transfer function is attached.
It shows the resonant frequency of ~50MHz as I expected. Those are nicely matched below 50MHz !!
For the next step, I will make the model of the circuit with stray capacitors, lead inductors, ... by changing the inductance or something.
There were several small/medium earthquakes in Ridgecrest and one medium one in Blackhawk CA at about 2000 UTC (i.e. ~ 2 hours ago), one of which caused BS, ITMY, and ETM watchdogs to trip. I restored the damping just now.
I have created the attached EOM circuit with resonances at 11 MHz, 29.5 MHz, and 55 MHz (the magnitude and phase of the voltage across the EOM are shown in the attached plot). The gain is roughly the same for each resonant peak. Although I have managed to get the impedances at all of the resonant frequencies to equal each other, I am having more trouble getting the impedances to be 50 Ohms (they are currently all around 0.66 Ohms).
For the current circuit, initial calculations show that we will need around 4.7 - 14.2 A of current to drive the EOM at the desired voltage (8 - 24 V); this is much higher than the current rating of most of the available transformers (250 mA), but the necessary current will change as the impedance of the circuit is corrected, so this is probably not a cause for concern. For example, the necessary driving voltages for the current circuit are (2.8 - 8.5 V); if we assume that the 50-Ohm impedance will be purely resistive, then we get a current range of 56 - 170 mA.
Since last week, I have come up with a new circuit, which is shown in the attached figure. The magnitude (solid) and phase (dashed) of the voltage across the EOM (red), the ratio between the voltage across the EOM and the voltage across the primary nodes of the transformer (blue), and the impedance through the primary port of the transformer (red) are also shown in an attached figure. As can be seen on the plot, resonance occurs at 11 MHz, 29.5 MHz, and 55 MHz, as specified. Also, at these resonant frequencies, the impedance is about 50 Ohms (34 dB). The gain between the voltage across the EOM and the voltage across the primary nodes of the transformer (or output of the crystal oscillator) is about 12 dB; we'd like a higher gain than this, but this gain is primarily governed by the ratio between the secondary and primary inductances in the transformer, and we are using the largest available ratio (on the Coilcraft website) that has the necessary bandwidth. Because of this, we will likely have to add another component between the crystal oscillator and the EOM circuit, to get the voltage to the desired 8.5 Vp across the EOM (for an optical modulation depth of 0.1 rad).
The current and power through the primary port of the tranformer are 43-85 mA and 25-92 mW, respectively. Since the transformer ratings are 250 mA and 1/4 W for current and power; these values should be safe to use with the intended transformer. Also, the highest power dissipated by a resistor in the circuit (not including the 50 Ohm resistor that is part of the crystal oscillator setup) is around 74 mW.
This week, I've been working on adapting the last week's circuit to make it buildable. Mostly this has involved picking components that are already in the lab, adding tunable components when necessary, and planning roughly how the components should be laid out on a board. I then built the circuit and put it in a box with BNC connectors for easy connection during testing. A picture of the built circuit is attached.
For initial testing, the transformer was removed from the design; since this changed the response of the circuit, I added two resistors to correct the response. A figure showing a schematic of the built circuit is attached. The expected responce of the circuit is also shown; the magnitude (solid) and phase (dashed) of the voltage across the EOM are shown in green, and the impedance of the circuit is shown in blue. While this response has sharp peaks and 50 Ohms (34 dB) of impedance at resonances, the gain is low compared to the circuit with the transformer. This means that, as is, this circuit cannot be used to drive the EOM; it is simply for testing purposes.
This week I've been working on testing the first version of the prototype circuit. Initially, I tested the circuit that I built last week, which had resistors in the place of the transformer. The magnitude and phase of the transfer function, as measured by the Agilent 4395A, are shown in the attached plot (first plot, MeasuredTransferFunction_R.jpg). The transfer function doesn't look like the simulated transfer function (second plot, BuiltCkt_ExpectedResponse.png), but I think I see the three peaks at least (although they're at the wrong frequencies). I spent some time trying to recreate the actual transfer function using LTSpice, and I think it's reasonable that the unexpected response could be created by extra inductance, resistance, capacitance and interaction between components.
When the transformer arrived yesterday, I replaced the resistors in the circuit with the transformer, and I have measured the following response (last plot, MeasuredTransferFunction.jpg). The gain is much lower than for the circuit with the resistors; however, I am still trying to track down loose connections, since the measured transfer function seems very sensitive to jiggled wires and connections.
Meanwhile, the parts for a flying-component prototype circuit have been ordered, and when they arrive, I'll build that to see if it works a little better.
Using FET probes, I was able to measure a transfer function that looks a little more like what I expected. There are only two peaks, but I think this can be explained by a short between the two capacitors (and two tunable capacitors) in the LC pairs, as shown (in red) in the circuit diagram attached. The measured transfer function (black), along with the simulated transfer functions with (red) and without (blue) the short are shown in the attached plot. The measured transfer function doesn't look exactly like the simulated transfer function with the short, but I think the difference can be explained by stray impedances.
I have built a version of the circuit with flying components; the completed circuit is shown in the attached picture. I built the circuit in segments and measured the transfer function after each segment to see whether it matched the LTSpice simulation after each step. The segments are shown in the circuit diagram.
After building the first segment, the measured transfer function looked pretty much the same as the simulated transfer function; it appears shifted in the attached plot, but this is because I didn't do a careful job of tuning at this point, and I'm relatively sure that I could have tuned it to match the simulation. After adding the second segment of the circuit, the measured and simulated transfer functions were similar in shape, but I was unable to increase the frequency of the peaks (through tuning) any more than what is shown in the plot (I could move the peaks so that their frequency was lower, but they are shown as high as they will go). When I added the final segment to complete the circuit, the measured and simulated transfer functions no longer had the same shape; two of the peaks were very close together and I was barely able to differentiate one from the other.
In order to understand what was happening, I tried making modifications to the LTSpice model to recreate the transfer function that was measured. I was able to create a transfer function that closely resembles the measured transfer function in both the circuit as of the 2nd segment and the completed circuit by adding extra inductance and capacitance as shown in red in the circuit diagram. The transfer functions simulated with these parasitic components are shown in red in both plots. While I was able to recreate the response of the circuit, the inductance and capacitance needed to do this were much larger than I would expect to occur naturally within the circuit (2.2uH, 12 pF). I'm not sure what's going on with this.
After speaking with Rana and realizing that it would be better to use smaller inductances in the flying-component circuit (and after a lot of tinkering with the original), I rebuilt the circuit, removing all of the resistors (to simplify it) and making the necessary inductance and capacitance changes. A picture of the circuit is attached, as is a circuit diagram.
A plot of the measured and simulated transfer functions is also attached; the general shape matches between the two, and the resonant frequencies are roughly correct (they should be 11, 29.5, and 55 MHz). The gain at the 55 MHz peak is lower than the other two peaks (I'd like them all to be roughly the same). I currently have no idea what the impedance is doing, but I'm certain it is not 50 Ohms at the resonant peaks, because there are no resistors in the circuit to correct the impedance. Next, I'll have to add the resistors and see what happens.
This is a quite nice measurement. Much better than the previous one.
1) For further steps, I think now you need to connect the real EOM at the end in order to incorporate
the capacitance and the loss (=resistance) of the EOM. Then you have to measure the input impedance
of the circuit. You can measure the impedance of the device at Wilson house.
(I can come with you in order to consult with Rich, if you like)
Before that you may be able to do a preparatory measurement which can be less precise than the Wilson one,
but still useful. You can measure the transfer function of the voltage across the input of this circuit,
and can convert it to the impedance. The calibration will be needed by connecting a 50Ohm resister
on the network analyzer.
2) I wonder why the model transfer function (TF) has slow phase changes at the resonance.
Is there any implicit resistances took into account in the model?
If the circuit model is formed only by reactive devices (Cs and Ls), the whole circuit has no place to dissipate (= no loss).
This means that the impedance goes infinity and zero, at the resonance and the anti-resonance, respectively.
This leads a sharp flip of the phase at these resonances and anti-resonances.
The real circuit has small losses everywhere. So, the slow phase change is reasonable.
For the past couple of days I have been trying to understand and perform Koji's method for impedance measurement using the Agilent 4395A Network Analyzer (without the impedance testing kit). I have made some headway, but I don't completely understand what's going on; here's what I've done so far.
I have made several transfer function measurements using the attached physical setup (ImpedanceTestingPhysicalSetup.png), after calibrating the setup by placing a 50 Ohm resistor in the place of the Z in the diagram. The responses of the various impedances I've measured are shown in the attached plot (ImpResponses.png). However, I'm having trouble figuring out how to convert these responses to impedances, so I tried to derive the relationship between the measured transfer function and the impedance by simplifying the diagram Koji drew on the board for me (attached, ImpedanceTestingSetup.png) to the attached circuit diagram (ImpedanceTestingCktDiagram.png), and finding the expected value of VA/VR. For the circuit diagram shown, the equation should be VA/VR = 2Z/(50+Z). 50 Ohms is good to use for calibration because the expected value of the transfer function for this impedance is 1 (0 dB).
So I used this relationship to find the expected response for the various impedances, and I also calculated the impedance from the actual measured responses. I've attached a plot of the measured (red) and expected (black) response (top) and impedance (bottom) for a 1 nF capacitor (1nF.png). The expected and measured plots don't really match up very well; if I add extra inductance (7.6 nH, plots shown in blue), the two plots match up a little better, but still don't match very well. I suspect that the difference may come from the fact that for my analysis, I treated the power splitter as if it were a simple node, and I think that's probably not very accurate.
Anyway, the point of all this is to eventually measure the impedance of the circuit I created on Friday, but I don't think I can really do that until I understand what is going on a little better.
I checked the setup and found RF reflection at the load was the cause of the unreasonable response in the impedance measurement.
So, I have put a total 22dB attenuation (10+6+6 dB) between the power splitter and the load to be measured. See the picture.
This kind of attenuators, called as PADs, is generally used in order to improve the impedance matching, sacrificing the signal amplitude at the load.
Then, It looks the measurements got reasonable up to 100MHz (at least) and |Z|<1kOhm.
For the measurements, I just followed the procedure that Stephanie described.
Stephanie has measured the impedance of her resonant circuit.
As a test of the method, I measured impedances of various discrete devices. i.e. 50Ohm, 10-1000pF Cap, Inductances, circuit opened.
a) 50Ohm (marine-blue) was calibrated to be recognized as 50Ohm.
b) The mica capacitances (orange 10pF, yellow 100pF, green 1000pF) appeared as the impedances f^-1 in the low freq region. It's nice.
At high frequency, the impedances deviate from f^-1, which could be caused by the lead inductance. (Self Resonance)
So 1000pF mica is not capacitance at 50MHz already.
c) Open BNC connector (Red) looks have something like 5pF. (i.e. 300Ohm at 100MHz)
d) I could not get good measurements with the inductors as I had 200nH (and some C of ~5pF) for a Pomona clip (blue).
This is just because of my laziness such that I avoid soldering the Ls to an RF connector!
I was able to make an impedance measurement of the flying-component circuit using Koji's method for impedance measurement. I first measured the impedance of the circuit with a 10 pF capacitor in the place of the EOM (as shown in the circuit diagram). This impedance plot is attached. I then added resistance to adjust the impedance slightly, attached the circuit to a New Focus KTP 4064 EOM, and took another impedance measurement (circuit diagram and impedance plot attached). The peaks are relatively close to 50 Ohms; they are at least the same order of magnitude.
I put the flying-component circuit in a box; a photo is attached. I also measured the impedance; it looks exactly the same as it looked before I put the circuit in the box.
I have spent the past couple of days gathering optics and mounts so that I can observe the modulation of the EOM attached to the circuit I built using the optical spectrum analyzer (OSA). A rough diagram of the planned layout is attached.
I also built a short SMA cable so that the EOM did not have to be connected directly to the circuit box. The cable is shown attached to the EOM and circuit box in the attached photo. After checking to make sure that all of the connections in the cable were sound, I remeasured the input impedance of the circuit; the impedance measurement (black) is shown in the attached plot with the impedance before the SMA cable was added with and without the box (green and blue, respectively--these two are almost identical). The new impedance has a strange shape compared to the original measurements; I'd like to understand this a little better, since adding extra inductance in LTSpice doesn't seem to have that effect. Since I had already taken apart the setup used for the previous impedance measurements, I had to rebuild and recalibrate the setup; I guess the difference could be something about the new calibration, but I don't really think that that's the case.
I have spent the past couple of days gathering optics and mounts so that I can observe the modulation of the EOM attached to the circuit I built using the optical spectrum analyzer (OSA). A rough diagram of the planned layout is attached.
After investigating this a bit further, I discovered that some of the components in the circuit were pressed firmly up against the inside of the box, and when they were moved, the impedance plot changed shape dramatically. I think that originally, the components were not pressed against the box, but the box's SMA joint was rather loose; when I connected this to the SMA cable, I tightened it, and this seems to have twisted the circuit around inside the box, pushing the components up against the side. I have fixed the twisting, and since the SMA joint is now tight, the circuit should no longer have any twisting problems.
A new plot is attached, showing the impedance of the circuit with nothing attached (blue), with the SMA cable and EOM attached (green), and with the EOM attached directly to it taken last friday with the old calibration of the setup (red). All three curves look roughly the same; the center peak is shifted slightly between the three curves, but the circuit with SMA and EOM is the version we'll be using, and it's central peak is close to the correct value.
I was able to observe the three sets of modulation sidebands created by the EOM + triply resonant circuit yesterday. Quantitative results will be posted later.
I measured the magnitude of modulation as a function of frequency using the optical spectrum analyzer and an oscilloscope while generating signals using a Marconi signal generator; the results are shown in the attached plot and are compared to the expected modulation given the measured transfer function of the circuit and the nominal modulation index of the EOM used (13 mrad/V). Using the oscilloscope, I found the resonant peaks to be at 11.11 MHz, 29.57 MHz, and 54.70 MHz. There are several different colors on the plot; this is because I had to take the data in several different segments and had to switch to measuring a different sideband partway through the measurment. I also separately found the modulation at each resonant peak for each sideband. The magnitude of modulation was measured by finding the ratio between the magnitude of the carrier and sideband powers using an oscilloscope, and calculating the magnitude of modulation from this. This method was also used to quantify the dependence of modulation magnitude on input power at each resonant peak; these results are also attached. These same results can also be plotted as modulation magnitude as a function of voltage into the resonant circuit; this is also attached (I'm not sure which is more useful).
In order to produce these results (get the measurements in mrad/V) it was necessary to measure the gain of the amplifier. I used the signal generator to input signals of varying power and measured the output signal voltage using the oscilloscope; I then repeated this process at each resonant frequency. From this I was able to calculate the gain of the amplifier to be 28.1 dB at 11.11 MHz, 27.4 dB at 29.57 MHz, and 25.7 dB at 54.70 MHz. These values are in the same ballpark as the values in the Mini Circuits data sheet (all values are ~25-28 MHz).
Calendars: The calendar issue discussed previously (http://nodus.ligo.caltech.edu:8080/40m/8098) where the numbers are squished together is very difficult for me to find. I am not going to worry about it for the time being.
Multiprocessing: Reviewed the implementation of Multiprocessing in python (using Multiprocessing package). Wrote a simple test function and ran it on megatron, to verify that multiprocessing could successfully take advantage of megatron’s multiple cores – it could. Now, I will work on implementing multiprocessing in the program. I began testing at a section in the program where a for loop calls process_data() (which has a long runtime) multiple times. The megatron terminals I had open began to run very slowly. Why? I believe that the process_data() function loads data into global variables to accomplish its task. The global variables in the original implementation were cleared before the subsequent calls to process_data(). But in the multiprocessing version, the data is not cleared, meaning the memory fills quickly, which drastically reduces performance. In the short term, I could try generating fewer processes at a time, wait for them to finish, then clearing the data, then generating more processes, etc. This will probably generate a nominal performance boost. In the long-term, restructuring of the way the program handles data may help (but not for sure). In the coming week I will experiment with these techniques and try to decrease the run time of the program.
Overview: In order to make the code more maintainable, I need to factor it into different well-documented classes. To do this carefully and rigorously, I need to run tests every time I make changes to the code. The runtime of the code is currently quite high, so I will work on improving the runtime of the program before factoring it into classes. This will be more efficient (minimize testing time) and allow me to factor more quickly. So, my current goal is to improve runtime as much as possible.
I invented a simple way to implement multiprocessing in the summary_pages.py file. Here is an example: in the code, there is a process_data() function, which is run 75 times and takes rather long to run. I created multiple processes to run these calls concurrently, as follows:
Original Code: (around line 7840)
for sec in datasections:
for run in run_opts:
run_opt = 'run_%s_time' % run
if hasattr(tabs[sec], run_opt) and getattr(tabs[sec], run_opt):
process_data(cp, ifo, start, end, tabs[sec],\
cache=datacache, segcache=segcache, run=run,\
# free data memory
keys = globvar.data.keys()
for ch in keys:
The weakness in this code is that process_data() is called many times, and doesn't take advantage of megatron's multiple threads. I changed the code to:
Modified Code: (around line 7840)
... etc... (same as before)
if hasattr(tabs[sec], run_opt) and getattr(tabs[sec], run_opt):
# Create the process
p = multiprocessing.Process(target=process_data, args=(cp, ifo, start, end, tabs[sec], datacache, segcache, run, veto_table[run], do['plots'], do['subplots'], do['html_only']))
# Add the process to the list of processes
plist += [p]
Then, I run the process in groups of size "numconcur", as follows:
numconcur = 8
curlist = 
for i in range(len(plist)):
curlist += [plist[i]]
if (i % numconcur == (numconcur - 1)):
for item in curlist:
for item in curlist:
keys = globvar.data.keys()
for ch in keys:
curlist = 
The value of numconcur (which defines how many threads megatron will use concurrently to run the program) greatly effects the speed of the program! With numconcur = 8, the program runs in ~45% of the time of the original code! This is the optimal value -- megatron has 8 threads. Several other values were tested - numconcur = 4 and numconcur = 6 had almost the same performance as numconcur = 8, but numconcur = 1 (which is essentially the same as the unmodified code) has a much worse performance.
Why does numcores = 4 have almost the same performance as numcores = 8? I monitored the available memory of megatron, and it is quickly consumed during these runs. I believe that once 4 or more cores are being used, the fact that the data can't all fit in megatron's memory (which was entirely filled during these trials) counteracts the usefulness of additional threads.
Summary of Improvements:
Original Runtime of all process_data() statements: (approximate): 8400 sec
Runtime with 8 processes (approximate): 3842 sec
This is about a 55% improvement for speed, in this particular sector (not in the overall performance of the entire program). It saves about 4600 seconds (~1.3 hours) per run of the program. Note that these values are approximate (since other processes are running on megatron during my tests. They might be inflated or deflated by some margin of error).
This same optimization method will be applied to all repetitive processes with reasonably large runtimes.
At first I thought that this was goofy, but then I logged in and saw that Megatron only has 8GB of RAM. I guess that used to be cool in the old days, but now is tiny (my laptop has 8 GB of RAM). I'll see if someone around has some free RAM for a 4600; in the meantime, I've killed a MEDM that was running on there and using up a few hundred MB.
Run your ssh-MEDMs elsewhere or else I'll make a cronjob to kill them periodically.
Okay, more memory would definitely be good. I don't think I have been using MEDM (which Jamie tells me is the controls interface) so making a cronjob would probably be a good idea.
Upon investigation, the other methods besides process_data() take almost no time at all to run, by comparison. The process_data() method takes roughly 2521 seconds to run using Multiprocessing with eight threads. After its execution, the rest of the program only takes 120 seconds to run. So, since I still need to restructure the code, I won't bother adding multiprocessing to these other methods yet, since it won't significantly improve speed (and any improvements might not be compatible with how I restructure the code). For now, the code is just about as efficient as it can be (in terms of multiprocessing). Further improvements may or may not be gained when the code is restructured.
Multiprocessing: In its current form, the code uses multiprocessing to the maximal extent possible. It takes roughly 2600 seconds to run (times may vary depending on what else megatron is running, etc.). Multiprocessing is only used on the process_data() function calls, because this by far takes the longest. The other function calls after the process_data() calls take a combined ~120 seconds. See http://nodus.ligo.caltech.edu:8080/40m/8218 for details on the use of Multiprocessing to call process_data().
Crontab: I also updated the crontab in an attempt to fix the problem where data is only displayed until 5PM. Recall that previously (http://nodus.ligo.caltech.edu:8080/40m/8098) I found that the crontab wasn't even calling the summary_pages.py script after 5PM. I changed it then to be called at 11:59PM, which also didn't work because of the day change after midnight.
I decided it would be easiest to just call the function on the previous day's data at 12:01AM the next day. So, I changed the crontab.
59 5,11,17,23 * * * /users/public_html/40m-summary/bin/c1_summary_page.sh 2>&1
0 6,12,18 * * * /users/public_html/40m-summary/bin/c1_summary_page.sh 2>&1
1 0 * * * /users/public_html/40m-summary/bin/c1_summary_page.sh $(date "+%Y/%m/%d" --date="1 days ago") 2>&1
For some reason, as of 9:00PM today (March 4, 2013) I still don't see any data up, even though the change to the crontab was made on February 28. Even more bizarre is the fact that data is present for March 1-3. Perhaps some error was introduced into the code somehow, or I don't understand how crontab does its job. I will look into this now.
Once I fix the above problem, I will begin refactoring the code into different well-documented classes.
[Rana, Jenne, Manasa]
We looked at the I vs. Q separation in several of the Refl PDs, while driving the PRM, while the PRMI was locked on sidebands.
For REFL 55, we adjusted the demod phase to try to minimize the peak in the Q signal, and were only able to get it to be about 1/10th the size of the I peak. This is not good, since it should be more like 1/100, at least.
For both REFL 11 and REFL 165, we were able to get the Q peaks to less than 1/100 of the I peak height.
We changed the REFL55 phase from 17 to 16, and the REFL165 phase from -160.5 to -162.5.
Since we believed that we had done a good job of setting the demod phase for REFL165, we used it to also check the balance of BS/PRM for MICH locking. I drove the BS with an arbitrary number (0.5), which creates a peak in the I phase of REFL165, and then I put in a drive on the PRM and tweaked it around until that peak was minimized. I came up with the same ratio as Koji had last Friday: BS=0.5, PRM=-0.2625. (The old ratio we were using, up until ~December when we started locking MICH with the ITMs, was BS=0.5, PRM=-0.267).
Also, while we were locked using REFL55 I&Q, we noticed that the other REFL PDs had lots of broadband noise in their I signals, as if some noise in the REFL55 diode is being injected into the PRM, that we are then seeing in the other PDs.
Some checks that we need to do:
* Inject a calibration line, set all the peak heights equal, and look at the noise floors of each PD.
* Use the calibration line to calibrate the PDs (especially REFL165) into meters, so that we know that it's noise is low enough to hold the PRC through the CARM offset reduction.
* Check out the state of the transmission QPDs - what is their noise, and is it good enough to use for holding the arms after we transition from green beatnote locking? Does the whitening switching do anything? What is the state of the whitening?
The following will be a stream-of-consciousness, approximately chronological story of my last hour or so of looking at screens....
In the old OAF days, we used to bypass the analog dewhitening in the coil driver path, using the XYCOMS. See, ex. elog 2548.
I began to wonder if we needed to do the same thing now. I checked several optics, to see how the switching works.
For the whitening on the OSEM sensor input, FM1 is linked to the Contec binary I/O. FM1 is the inverse whitening filter. Turn it on, and the analog whitening is on (bit in the binary I/O screen turns red). Turn it off, and the analog whitening is bypassed (bit in the binary I/O screen turns gray). Good. Makes sense. Either way, the net transfer function is flat.
The dewhitening is not so simple. In FM9 of the Coil Output filter bank, we have "SimDW", and in FM10, we have "InvDW". Clicking SimDW on makes the bit in the binary I/O screen gray (off?), while clicking it off makes it red (on?). Clicking InvDW does nothing to the I/O bits. So. I think that for dewhitening, the InvDW is always supposed to be on, and you either have Simulated DW, or analog DW enabled, so that either way your transfer function is flat. Fine. I don't know why we don't just tie the analog to the InvDW filter module, and delete the SimDW, but I'm sure there's a reason.
All optics have this setup, except MC1 and MC3. They don't have the SimDW or InvDW filter modules. Instead, in FM9 (which on all the other suspensions is SimDW, and controls the binary I/O) there is a 28Hz Elliptic Low Pass filter. The only thing I can find about these is elog 1405 where Rana talks about implementing ELP28's in MC2. But right now there is no ELP in the MC2 coil output filters. So, if Rana's old elog is to be believed, we need to fix up the ELP28 situation. But that elog was from a long time ago, so maybe things are different now? If MC1 and MC3 need the SimDW and InvDW (why wouldn't they?) then the ELP28 needs to move to another filter module. Because right now, when I click the ELP28's on and off, it changes bits in the binary I/O. Which I don't think it should. Maybe. I don't really know.
Okay. So. Now we know where everything is, and which buttons do what. Maybe not why, but at least what.
In the old world, Rob had lots and lots of trouble (elog 2027) with locking when the analog dewhitening was bypassed. But right now, I think that all of the analog dewhitening filters are bypassed, for every single optic we have. So. Which way do we want things? What's the new game plan. What's going on??
The input/whitening filters used to be in a similarly confused state as the output filters, but they have been corrected. There might have been a reason for this setup in the past, but it's not what we should be doing now. The output filters all need to be fixed. We just haven't gotten to it yet.
As with the inputs, all output filters should be set up so that the full output transfer function is always flat, no matter what state it's in. The digital anti-dewhitening ("InvDW") and analog dewhitening should always be engaged and disengaged simultaneously. The "SimDW" should just be removed.
This is on my list of things to do.
Although the morning MC tuning looked stable, Koji pointed out that the MC_REFL_OFFSET was changed from its nominal value.
The offset was reset and this caused drift in the MC_TRANS_SUM.
To fix this:
- disabled the WFS servo
- aligned MC using MC1 and MC3
- centered beam on the MC_REFL
- reset WFS offsets
- locked MC
MC looks happy now.
ALS is in a very different state from a couple of days ago when we could successfully lock the arms and scan.
The green alignment to the arms had drifted.
PSL green alignment on the PSL table was off. The PSL green was not even on the steering mirror. Did anyone work around the PSL table in the last couple of days?
After aligning and finding the beat note, I found the ALS servo very noisy. The error signal had 10 times more rms noise than what was achieved earlier this week and there were some new 60Hz peaks as well.
Overall, we could not do any PRMI+ALS arms today
Today, I worked with Kakeru on ISS.
The problem is sort of elusive. Some time, the laser power looks fine, but after a while you may see many sharp drops in the power. Some times, the power drops happen so often that they look almost like an oscillation.
We made several measurements today and Kakeru is now putting the data together. Meanwhile, I will put my speculations on the ISS problem here.
The other day, Kakeru took the transfer function of the ISS feedback filter (he is supposed to post it soon). The filter shape itself has a large phase margin ( more than 50deg ?) at the lower UGF (~3Hz) if we assume the response of the current shunt to be flat. However, when we took the whole open loop transfer function of the ISS loop, the phase margin was only 20deg. This leads to the amplification of the intensity noise around the UGF. The attached plot is the spectrum of the ISS monitor PD. You can see a broad peak around 2.7Hz. In time series, this amplified intensity noise looks like semi-oscillation around this frequency.
Since it is very unlikely that the PD has a large phase advance at low frequencies, the additional phase advance has to be in the current shunt. We measured the response of the current shunt (see Kakeru's coming post). It had a slight high-pass shape below 100Hz (a few dB/dec). This high-pass response produces additional phase advance in the loop.
There seems to be no element to produce such a high-pass response in the current shunt circuit ( http://www.ligo.caltech.edu/docs/D/D040542-A1.pdf )
This Jamie's document shows a similar high-pass response of the current ( http://www.ligo.caltech.edu/docs/G/G030476-00.pdf page 7 )
Now the question is what causes this high-pass response. Here is my very fishy hypothesis :-)
The PA output depends not only on the pump diode current but also on the mode matching with the NPRO beam, which can be changed by the thermal lensing. If the thermal lensing is in such a condition that an increase in the temperature would reduce the mode matching, then the temperature increase associated with a pump current increase could cancel the power increase. This thermal effect would be bigger at lower frequencies. Therefore, the intensity modulation efficiency decreases at lower frequencies (high-pass behavior). If this model is true, this could explain the elusiveness of the problem, as the cancellation amount depends on the operation point of the PA.
To test this hypothesis, we can change the pump current level to see if the current shunt response changes. However, the PA current slider on the MEDM screen does not work (Rob told me it's been like this for a while). Also the front panel of the MOPA power supply does not work (Steve told me it's been like this for a while). We tried to connect to the MOPA power supply from a PC through RS-232C port, which did not work neither. We will try to fix the MEDM slider tomorrow.
For purpose of the automation and my record, I summarized my locking procedure as a chart.
Sorry to spam the e-log, but did someone come knock loudly on the emergency exit door a few moments ago? It gave Sasha and I quite a fright, and we are rather worried.
Probably, security. You can call 5555 and ask them. Otherwise you can ask them to come and check everything.
You mean 5000?
Probably, security. You can call 5555 and ask them. Otherwise you can ask them to come and check everything.
Looks like there was some mysterious MC alignment shift around 5:30 PM today, but no elog.....?? Now things are drifting much more than this morning or yesterday. Who did what and why???
Looks like there was a mysterious loss of data overnight; since there's nothing in the elog I assume that its some kind of terrorism. I'm going to call Rolf to see if he can come in and work all night to help diagnose the issue.
In addition to the other fixes, Alex rebuilt the daqd process. I failed to elog this. When he rebuilt it, he needed change the symmerticom gps offset in the daqdrc file (located in /opt/rtcds/caltech/c1/target/fb).
On Friday night, Kiwamu contacted me and let me know the frame builder had core dumped after a seg fault. I had him temporarily disable the c1ass process (the only thing we changed that day), and then replaced Alex's rebuilt daqd code with the original daqd code and restarted it. However, I did not change the symmetricom offset at this point. Finally, I restarted the NDS process. At that point testpoints and trends seemed to be working.
The daqd process was restarted sometime on Sunday night (by Valera i believe). Apparently this restart finally had the symmetricom gps offset kick in (perhaps because it was the first restart after the NDS was restarted?). So data was being written to a future gps time.
Kiwamu had problems with testpoints and trends and contacted me. I tracked down the gps offset and fixed it, but the original daqd process only started once successfully, after that is was segfault, core dump non-stop. I tried Alex's rebuilt daqd (along with putting the gps offset to the correct value for it), and it worked. Test points, trends, excitations were checked at the point and found working.
I still do not understand the underlying causes of all these segmentation faults with both the old and new daqd codes. Alex has suggested some new open mx drivers be installed today.