koji.arai@cr9:/opt/rtcds/userapps/trunk/asc/l1/scripts/initial_alignment$ cat autostart_strips.sh
koji.arai@cr9:/opt/rtcds/userapps/trunk/asc/l1/scripts/initial_alignment$ cat autostart_strips.sh
# Baffle window setup 1500x480
StripTool -xrm 'StripTool.StripGraph.geometry:1500x470+0+24' /opt/rtcds/userapps/trunk/asc/l1/scripts/initial_alignment/baffle_pd.stp &
# DC signals setup
StripTool -xrm 'StripTool.StripGraph.geometry:1500x470+0+470' /opt/rtcds/userapps/trunk/asc/l1/scripts/initial_alignment/dc_signals.stp &
# WFS prx mich sry setup
StripTool -xrm 'StripTool.StripGraph.geometry:1500x470+0-24' /opt/rtcds/userapps/trunk/asc/l1/scripts/initial_alignment/wfs_prx_mich_sry.stp &
In order to do online static IIR Wiener filtering one needs to do the following,
1) Get data for FIR Wiener filter witnesses and target.
2) Measure the transfer function (needs to be highly coherent, ~ 0.95 for all points) from the actuactor to the control signal of interest (ie. from MC2_OUT to MC_L).
3) Invert the actuator transfer function.
4) Use Vectfit or (LISO) to find a ZPK model for the actuator transfer and inverse transfer functions.
5) Prefilter your witness data with the actuator transfer function, to take into account the actuator to control transfer function when designing the offline Wiener FIR filter.
6) Calculate the frequency response for each witness from the FIR coefficients.
7) Vectfit the frequency reponses to a ZPK model, this is the FIR to IIR Wiener conversion step.
8) Now, either, divide the IIR transfer function by the actuator transfer function or more preferably, multiply by the inverse transfer function.
9) Use Quack to make SOS model of the IIR Wiener filter and inverse transfer function product that goes into foton for online implementation.
10) Load it into the system.
The block diagram below summarizes the steps above.
Frustrated by the single pixel width of the windows and how hard that makes it to drag things around, I explored StackExchange:
which showed how there is a .xml file which can be edited to increase this. I've changed the border size to 4 pixels on Rossa - its nice.
For CARM and DARM, the A channels are used for the ALS signals, whereas the B channels are used for blending the RF signals.
For the DRMI, the A channels are used for the 1F signals, whereas the B channels are used for the 3F signals. The settings for transitioning to 1F after locking the DRFPMI have not yet been determined.
These settings are currently saved in the DRMI configurator, but the demod angles are set for DRFPMI lock, so the settings don't reliably work for misaligned arms.
The REFL33 element in SRCL_B is to reduce the PRCL coupling, was found empirically by tuning the relative gains with the arms misaligned and looking at excitation line heights. The offsets were found by locking the DRMI on 1F signals with arms misaligned, and taking the average value of these 3F error signals.
The CARM and DARM ALS settings are largely scripted by scripts/ALS/Transition_IR_ALS.py, which takes you from arms POX/POY locked to CARM and DARM ALS locked. The DRMI settings are usually restored from the IFO_CONFIGURE screen.
When arms are POX/POY locked, and the green beatnotes are appropriately configured, calling scripts/DRFPMI/carm_cm_up.sh initiates the following sequence of events:
When CARM and DARM are buzzing around true zero, powers maximized:
This is as far as we've taken the DRFPMI so far, but the CARM bandwidth is still only at a few kHz. Based on PRFPMI locking, the next steps will be:
Today image capture did not work again, though it had worked 3 days before. I also found that red indicator light on the front pannel of SENSORAY was not turned on, which had been turned on 3 days before (you can find SENSORAY on the floor near Pianosa). Possible reason that it did not work again was I restarted Pianosa last night. Anyway, it works now. Here I report what I did to make it work.
I ran thes commands in shell, following the instruction of the manual of SENSORAY 2253 (Page 5; link or you can find the manual in /users/sensoray; I put it there).
> cd /users/sensoray/sdk_2253_1.2.2_linux
> make all
> sudo make install
> modprobe s2253
Then the red light got turned on, and image capture worked.
If you recieve an error like "No such file or directory: /dev/video0" at the beginning of the error message when you run image capture scripts from the medm screen, or if you notice that the red indicator light of SENSORAY is not on, this procedure could help you.
Do we need "make" everytime? Do you mean just running "modprobe" didn't work?
I don't know if just running "modprobe" will work or not, because I didn't try it... When the same problem happens again, we can try just running "modprobe" first.
The door was not locked this morning.
Please do not use this door if you can not close it!
Last person leaving the lab should check that the latch is cut by the strike plate.
This is one of those unsolved door lock acquisition problems. Its been happening for years.
Please ask facilities to increase the strength of the door tensioner so that it closes with more force.
It was requested this morning.
The attached file is a python notebook that you can use to get data. Minimal syntax.
"## Get some 40m data using NDS"
Minute trend data seems not available using the NDS2 server. Its super slow using dataviewer from the control room.
Did some digging into the NDS2 config on megatron. It hasn't been updated in 2 years.
All of the stuff is run by the user 'nds2mgr'. The CronTab for this user was running all the channel name updates and server restarts at 3 AM each day; I've moved it to 5:05 AM. I don't know the password for this user, so I just did 'sudo su nds2mgr' to become him.
On megatron, in /home/nds2mgr/nds2-megatron/ there is a list of channels and configs. The file for the minute trend (C-M-ChanList.txt), hasn't been updated since Nov-2015. ???
I have written a code(a basic one which needs a lot of improvements, but still does the job) for taking multiple measurements from the AG4395A. I have also written a separate code for plotting the data taken from the previoius code along with the error bars upto 1 standard deviation.
Details on How To Operate AG4395A:
Brief Details on How the 'AGmeasure' command works:
AGmeasure is a python script developed by some of the people who work at 40m. It is set as a global command and can be used from within any directory. The source code is in the scripts folder on the network, or else it can also be found in Eric Quintero's git repository. This command accepts at the very least a parameter file. This is supposed to be a .yml file. A template (TFAG4395Atemplate.yml) can be found in the scripts folder or in Eric's repo. There are some other options that can be passed to this command, see the help for more details.
The Multi_Measurement Script:
This script calls the 'AGmeasure' command repetitively and keeps storing the data files in a folder. Right now, the script needs to be fed in th template file manually at prompt.
The Test_Plotting Script:
This script plots the a set of data files obtained from the above mentioned script and produces a plot along with the errors bands upto 1 standard deviation of the data. The format (names) and total number of text files need to be explicitly known, for now at least.
Update: Increased the font size in the plot. Added a few comments to the two scripts
To Do: Need to consider the transfer function as a single physical quantity (both the magnitude and phase) and then take the averages and calculate the standard deviation and then plot these results.
The attachment with the test files and the code now also contains a pdf with all the relations/equations I have used to calculate the averages and errors.
The following elog describes the procedure followed for generating a sample simulation for a cavity scan, fitting an actual cavity scan and calculating the relevant paramaters using the cavity scan and fit data.
1. Cavity Scan Simulation:
2. Fitting a Cavity Scan:
3. Calculating Physically Relevant Parameters:
Most of the above info and some smaller details can be found in the markdown readme file in this git repo.
I've modified the __init.py__ file located at /ligo/apps/linux-x86_64/cdsutils-480/lib/python2.7/site-packages/cdsutils/__init__.py so that you can now simply import pyawg from cdsutils. On the control room workstations, iPython is set up such that cdsutils is automatically imported as "cds". Now this import also includes the pyawg stuff. So to use some pyawg function, you would just do (for example):
One could also explicitly do the import if cdsutils isn't automatically imported:
from cdsutils import awg
Linking this useful instructional elog from Chris here: https://nodus.ligo.caltech.edu:8081/Cryo_Lab/1748
? Why aren't you able to just import 'awg' directly? You shouldn't have to import it through cdsutils. Something must be funny with the config.
Going through some astronomy CCD calibration resources (-), I gather that there are in general 3 distinct types of correction that are applied:
The flat-field calibration seems to be the most complicated - the idea is to use a source of known radiance, and capture an image of this known radiance with the CCD. Then assuming we know the source radiance well enough, we can use some math to back out what the actual response function of individual pixels are. Then, for an actual image, we would divide by this response-map to get the actual image. There are a number of assumptions that go into this, such as:
I am not sure what error is incurred by ignoring 2 and 3 in the list at the beginning of this elog, perhaps this won't affect our ability to estimate the scattered power from the test-masses to within a factor of 2. But it may be worth it to do these additional calibration steps.
I also wonder what the uncertainty in the 1.5V/A number for the photodiode is (i.e. how much do we trust the Ophir power meter at low power levels?). The datasheet for the PDA100A says the transimpedance gain at 60dB gain is 1.5 MV/A (into high impedance load), and the Si responsivity at 1064nm is ~0.25A/W, so naively I would expect 0.375 V/uW which is ~factor of 4 lower. Is there a reason to trust one method over the other?
Also, are the calibration factor units correct? Jigyasa reported something like 0.5nW s / ct in her report.
The incident power can be calculated as Pin =CF*Total(Counts-DarkCounts)/ExposureTime.
Disclaimer: Wrong calibration factors! See https://nodus.ligo.caltech.edu:8081/40m/13391
The factors were indeed enormously off. The correct table reads:
I did subtract a 'dark' frame from the images, though not in the sense of your point 1, just an exposure of identical duration with the laser turned off. This was mostly to reduce the effect of residual light, but given similar initial conditions would somewhat compensate for the offset that pre-existing charge and electronics noise put on the pixel values. The white field is of course a difference story.
I wonder how close we can get to a white field by putting a thin piece of paper in front of the camera without lenses and illuminate it from the other side. A problem is of course the coherence if we use a laser source... Or we scrap any sort of screen/paper and illuminate directly with a strongly divergent beam? Then there wouldn't be a specular pattern.
I'm not sure I understand your point about the 1.5V/A. Just to make sure we're talking about the same thing I made a crude drawing:
The PD sees plenty of light at all times, and the 1.5V/uW came from a comparative measurement PD<-->Ophir (which took the place of the CCD) while adjusting the power deflected with the AOM, so it doesn't have immediate connection to the conversion gain of silicon in this case. I can't remember the gain setting of the PD, but I believe it was 0dB, 20dB at most.
I calculated a better lens solution for the ETMX side view with the simple python script that's attached. The camera is still not as close to the viewport as we would like, and now the front lens is almost all the up to the end of the tube. With a little more playing around there maybe a better way, especially if we expand the repertoire of focal lengths. Using Steve's wonderful camera fixture I put the beam spot in focus. I turned the camera sideways for better use of the field of view, and now the beam spot actually fills the center area of the beam, to the point where we probably don't want more magnification or else we start losing the tails of the Gaussian.
We'll take a serious of images tomorrow, and will have an estimate of the scatter loss by the end of tomorrow.
Telescope front lens to wall distance 25 cm, GigE camera lenght 6 cm and cat6 cable 2cm
Atm3, Existing short camera can has 16cm lenght to lexan guard on viewport. Available 2" od periscope tube lenght is 8cm. The one in use 16 cm long.
Note: we can fabricate a lite cover with tube that would accomodate longer telescope.
Can we calibrate the AR coated M5018-SW and compare it's performance agains the 2" periscope
Look at the Edmond Optics 3" od camera lens with AR
This lower priced 1" apeture Navitar lens can be an option too.
Atm1, Now I can see dust. This is much better. The focus is not right yet.
Atm2, Chamber viewport wiped and image refocused. Actually I was focusing on the dust.
Atm1, Now I can see dust. This is much better. The focus is not right yet.
We don't really have to calibrate the lens, just the CCD, which we've done. It's more about knowing the true aperture size to know how much solid angle you're capturing to infer the total amount of scatter. For our custom lens tubes this is the ID of the retaining ring.
The Edmund Optics lens tube looks tempting, but itcomes at a price. Thorlabs sells lens tubes that offer a more flexibility than what we have right now, so I bought a few different ones, and also more 150mm 2" lenses. This will allow for more compact solutions and offer some in-situ focusing ability that doesn't require detaching the lens tube like now. Should be here in a couple of days, then we'll be able to enclose the GigE camera in the viewport can with a similar field of view we have now.
I also bought a collimation package for the AS port fiber stuff so we can move ahead with the ringdown measurements and also mode spectroscopy.
The units were still off in my previous post. Here's the corrected, sanity-checked version:
I estimated the uncertainties based on a linear fit to the data I recorded with 75nW incident on the CCD and assumed a 5% uncertainty in that number. This is just an upper limit, to be safe. I had calibrated the power reading placing the Ophir power meter where the CCD would otherwise be and comparing it to the PD voltage of a picked off beam. In my previous figures the axes were mislabeled, so I reproduce them here:
Using the current camera position I recorded 50 exposures both with and without beam (XARM locked vs PSL shutter closed) and averaged the images to see how much the reading fluctuates. The exposure time was 10 ms, which left the maximum reported pixel value in all exposures below 3800 out of 4096. The gain setting was 100, which is what I used to calibrate the CCDs.
(*) I calculated the lens positions to focus at a plane 65cm from the front lens. We're pretty close to that, but I can't confirm the actual distance easily, so I assumed a 5cm error on the distance, which is where most of the error is coming from. This is also assuming uniform scatter.
(**) This is assuming 10W of circulating power
Since we're getting ready to put the replacement slow DAQ for c1auxex in I wanted to bring the IFO back to operating condition after the PMC hasn't been locked for days. Something seems wrong with the CDS system though, many of the frontent models have red background and don't seem to be responsive. I followed the instructions laid out in https://wiki-40m.ligo.caltech.edu/Computer_Restart_Procedures.
In the attached screenshot, initially all c1ioo models were red, and on c1iscex only c1x01 was blue, the other ones red. I was able to ssh into both machines and tried to restart indivitual models, which didn't work and instead turned their background white. Still following the wiki page, I restarted both machines but they don't respond to pinging anymore and thus I cannot use ssh to reach them. Not sure what to do, I also rebooted fb over telnet.
So far I couldn't find any records of how to fix this situation.
Once a realtime machine was rebooted, it did not come back. I suspect that the diskless hosts have a difficulty to boot up.
The issue was partially fixed and the interferometer is in workable condition now.
What -probably- fixed it was restarting the dhcp server on chiara
sudo service isc-dhcp-server restart
Afterwards the frontends were restarted one by one. SSH access was possible and the essential models for IFO operation were started.
c1iscex reported initially that no DAQ card was found, and inside the IO chassis the LED indicator strip was red. Turning off the machine, checking the cables and rebooting fixed this.
I checked the calibration of the Oplevs for both ITMs, both ETMs and the BS. The table below summarizes the old and new cts->urad conversion factors, as well as the factor describing the scaling applied. Attachment #1 is a zip file of the fits performed to calculate these calibration factors (GPS times of the sweeps are in the titles of these plots). Attachment #2 is the spectra of the various Oplev error signals (open loop, so a measure of seismic induced angular motion for a given optic, and DoF) after the correction. Loop TF measurements post calibration factor update and loop gain adjustment to be uploaded tomorrow.
Now that the ETMX calibration has been updated, let's keep an eye out for a wandering ETMX.
Gautam and Steve,
We have calibrated the load cells. The support beams height monitoring is almost ready.
The danger of this measurment that the beams height changes can put shear and torsional forces on this formed (thin walled) bellow
They are designed for mainly axial motion.
The plan is to limit height change to 0.020" max
0, center oplev at X arm locked
1, check that jack screws are carrying full loads and set height indicator dials to zero ( meaning: Stacis is bypassed )
2, raise beam height with aux leveling wedge by 0.010" on all 3 support point and than raise it an other 0.005"
3, replace levelling wedge with load cell that is centered and shimmed. Dennis Coyne pointed out that the Stacis foot has to be loaded at the center of the foot and formed bellow can shear at their limits.
4, lower the support beam by 0.005" ......now full load on the cells
Note: jack screw heights will not be adjusted or touched.......so the present condition will be recovered
We could use similar load cells to make the actual weight measurement on the Stacis legs. This seems practical in our case.
I have had bad experience with pneumatic Barry isolators.
Our approximate max compression loads are 1500 lbs on 2 feet and 2500 lbs on the 3rd one.
[ Dennis Coyne' precision answer ]
Differential Height between Isolators
According to a note on the bellows drawing (D990577-x0/A), the design life of the bellows at ± 20 minutes rotational stroke is 10,000 cycles. A 20 minute angular (torsional) rotation of the bellows corresponds to 0.186" differential height change across the 32" span between the chamber support beams (see isolator bracket, D000187-x0/B).
Another consideration regarding the bellows is the lateral shear stress introduced by the vertical translation. The notes on the bellows drawing do not give lateral shear limits. According to MDC's web page for formed bellows in this size range the lateral deflection limit is approximately 10% of the "live length" (aka "active length", or length of the convoluted section). According to the bellows drawing the active length is 3.5", so the maximum allowable lateral deflection should be ~0.35".
Of course when imposing a differential height change both torsional and lateral shear is introduced at the same time. Considering both limits together, the maximum differential height change should be < 0.12".
One final consideration is the initial stress to which the bellows are currently subjected due to a non-centered support beam from tolerances in the assembly and initial installation. Although we do not know this de-centering, we can guess that it may be of the order of ~ 0.04". So the final allowable differential height adjustment from the perspective of bellows stress is < 0.08". Steve: accumulated initial stress is unknown. We used to adjust the original jack screws for IFO aligment in the early days of ~1999. This kind of adjustment was stopped when we realized how dangereous it can be. The fact is that there must be unknown amount of accumulated initial stress. This is my main worry but I'm confident that 0.020" change is safe.
So, with regard to bellows stress alone, your procedure to limit the differential height change to <0.020" is safe and prudent.
However, a more stringent consideration is the coplanarity requirement (TMC Stacis 2000 User's Manual, Doc. No. SERV 04-98-1, May 6, 1991, Rev. 1), section 2, "Installation",which stipulates < 0.010"/ft, or < 0.027" differential height across the 32" span between the chamber support beams. Again, your procedure to limit the differential height change to < 0.02" is safe.
Centered Load on the STACIS Isolators
According to the TMC Stacis 2000 User's Manual (Document No. SERV 04-98-1, May 6, 1991, Rev. 1), section 2, "Installation", typical installations (Figure 2-3) are with one payload interface plate which spans the entire set of 3 or 4 STACIS actuators. Our payload interface is unique.
Section 2.3.1, "Installation Steps": "5. Verify that the top of each isolator is fully under the payload/interface plate; this is essential to ensure proper support and leveling. The payload or interface plate should cover the entire top surface of the Isolator or the entire contact area of the optional jack."
section 2.3.2, "Payload/STACIS Interface": "... or if the supporting points do not completely cover the top surface of each Isolator, an interface plate will be needed."
The sketch in Figure 2-2 indicates an optional leveling jack which appears to have a larger contact surface area than the jacks currently installed in the 40m Lab. Of course this is just a non-dimensioned sketch. Are the jacks used by the 40m Lab provided by TMC, or did we (LIGO) choose them? I beleive Larry Jones purchased them.
A load centering requirement is not explicitly stated, but I think the stipulation to cover the entire top surface of each actuator is not so much to reduce the contact stress but to entire a centered load so that the PZT stack does not have a reaction moment.
From one of the photos in the 40m elog entry (specifically jack_screw.jpg), it appears that at least some isolators have the load off center. You should use this measurement of the load as an opportunity to re-center the loads on the Isolators.
In section 2.3.3, "Earthquake Restraints" restraints are suggested to prevent damage from earth tremors. Does the 40m Lab have EQ restraints? Yes, it has
Screw Jack Location
I could not tell where all of the screw jacks will be placed from the sketch included in the 40m elog entry which outlines the proposed procedure.
Load Cell Locations
The sketch indicates that the load cells will be placed on the center of the tops of the Isolators. This is good. However while discussing the procedure with Gautam he said that he was under the impression that the load cell woudl be placed next to the leveling jack, off-center. This condition may damage the PZT stack. I suggest that the leveling jack be removed and replaced (temporarily) with the load cell, plus any spacer required to make up the height difference. Yes
If you have any further question, just let me know.
Chief Engineer, LIGO Laboratory
California Institute of Technology
MC 100-36, 1200 E. California Blvd.
follow up email from Dennis 5-13-2018. The last line agrees with the numbers in elog13821.
Hi Steve & Gautam,
I've made some measurements of the spare (damaged) 40m bellows. Unfortunately neither of our coordinate measurement arms are currently set up (and I couldn't find an appropriate micrometer or caliper), so I could not (yet) directly measure the thickness. However from the other dimensional measurements, and a measurement of the axial stiffness (100 lb/in), and calculations (from the Standards of the Expansion Joint Manufacturers Association (EJMA), 6th ed., 1993) I infer a thickness of 0.010 inch in . This is close to a value of 0.012 in used by MDC Vacuum for bellows of about this size.
I calculate that the maximum allowable torsional rotation is 1.3 mrad. This corresponds to a differential height, across the 32 in span between support points, of 0.041 in.
In addition using the EJMA formulas I find that one can laterally displace the bellows by 0.50 inch (assuming a simultaneous axial displacement of 0.25 inch, but no torsion), but no more than ~200 times. I might be good to stay well below this limit, say no more than ~0.25 inch (6 mm).
If interested I've uploaded my calculations as a file associated with the bellows drawing at D990577-A/v1.
BTW in some notes that I was given (by either Larry Jones or Alan Weinstein) related to the 40m Stacis units, I see a sketch from Steve dated 3/2000 faxed to TMC which indicates 1200 lbs on each of two Stacis units and 2400 on the third Stacis.
I copied the netgpibdata folder onto rossa (under the directory ~/Agilent/), which contains all the necessary scripts and templates you'll need to remotely set up, run, and download the results of measurements taken on the AG4395A network analyzer. The computer will communicate with the network analyzer through the GPIB device (plugged into the back of the Agilent, and whose communication protocol is found in the AG4395A.py file in the directory ~/Agilent/netgpibdata/).
The parameter template file you'll be concerned with is TFAG4395Atemplate.yml (again, under ~/Agilent/netgpibdata/), which you can edit to fit your measurement needs. (The parameters you can change are all helpfully commented, so it's pretty straightforward to use! Note: this template file should remain in the same directory as AGmeasure, which is the executable python script you'll be using). Then, to actually set up, run, and download your measurement, you'll want to navigate to the ~/Agilent/netgpibdata/ directory, where you can run on the command line the following: python AGmeasure TFAG4395Atemplate.yml
The above command will run the measurement defined in your template file and then save a .txt file of your measured data points to the directory specified in your parameters. If you set up the template file such that the data is also plotted and saved after the measurement, a .pdf of the plot will be saved along with your .txt file.
Now if you want to just download the data currently on the instrument display, you can run: python AGmeasure -i 192.168.113.105 -a 10 --getdata
Those are the big points, but you can also run python AGmeasure --help to learn about all the other functions of AGmeasure (alternatively, you can read through the actual python script).
Happy remote measuring! :)
Today, Rana had me key the PSL crate.
Locking the PMC:
Yutaro talked about the BIO bug in KAGRA elog. http://klog.icrr.u-tokyo.ac.jp/osl/?r=9536
I think I made the similar change for the 40m model somewhere (don't remember), but be aware of the presense of this bug.
While going to take some transfer functions of the MC WFS loop, LSC was down. When we tried to restart the FE using 'rtcds restart --all', c1lsc crashed and froze. We manually reset c1lsc, then laboriously determined the correct order of machines to reboot. Here's what works best:
rtcds start c1x04 c1lsc c1ass c1oaf c1cal c1daf
Starting c1dnn crashes the other FE
rtcds restart --all
rtcds restart c1rfm c1sus c1mcs
restarting c1pem crashes the other FE on c1sus
We're seeing a lot of red IPC indicators--perhaps it's an issue with the order we're restarting?
via Polish chat, GV tells us to RTFE
As suggested, I ran the script cds/rebootC1LSC.sh
I got a timeout error when the script tried closing the PSL shutter ('C1:AUX-PSL_ShutterRqst' not found), but Rana and I closed the shutter before leaving last night. c1sus is down, so the script found no route to host c1sus; I'm thinking I need to reset c1sus for the script to run completely. Nonetheless, c1lsc was rebooted, which crashed c1ioo and left the c1lsc FE all red (probably because c1sus wasn't restarted).
I reset c1lsc, c1sus, and c1ioo.
I noticed that the script gives the command 'ssh c1XXX', but we have been getting no route to host using this command. Instead, the machines are currently only reachable as c1XXX.martian. I'm not sure why this is, so I just appended .martian in rebootC1LSC.sh
This time, the script does run. I did get 'no route to host' on c1ioo, so I think I need to reset that machine again. After reset, the script failed to login to c1ioo and c1lsc.
Fri Sep 6 13:09:05 2019
After lunch, I reset the computers again, and try the script again. There is again no route to host for c1ioo. I'm going inside to shutoff the power to c1ioo, since the reset buttom seems to not be working. I still can't login from nodus, so I'm bringing a keyboard and monitor over to plug in directly.
On reset, c1ioo repeatedly reaches the screen in attachment 1, before going black. Holding down shift or ctrl+alt+f1 doesn't get me a command prompt. After waiting/searching the elog for >>3 min, we decided to follow these instructions to cycle the power of c1ioo. The same problem recurred following power up. I found online some instructions that the SunSystems 4600 can hang during reboot if it has become too hot ("reboot during a thermal shutdown"); I did notice that the temperature light was on earlier in this procedure, so perhaps that is the problem. I followed the wiki instructions to shut down the computer again (pressed power button, unplugged 4 power supplies from back of machine), and left it unplugged for 10-30 min (Fri Sep 6 14:46:18 2019 ).
Fri Sep 6 15:03:31 2019
Rana plugged in the power supplies and reset the machine again.
Fri Sep 6 16:30:37 2019
c1ioo is still unreachable! I pressed reset once, and the reset button flashes white. The yellow warning light is still on.
Fri Sep 6 16:54:21 2019
The reset light has stopped flashing, but I still can't access c1ioo. I reset once more, this time watching c1ioo on a monitor directly. I'm still seeing the same boot screen repeatedly. I do see that CPU0 is not clocking, which seems weird.
Following gautam's elog here, I found the Sun Fire X4600 manual for locating faulty CPUs. After the white reset light stopped flashing, I held down the power button to turn off the system. Before shutdown, all of the CPU displayed amber lights; after shutdown, only the leftmost CPU (as viewed from the back, presumably CPU0) displays an amber light. The manual says this is evidence that the CPU or DIMM is faulty. Following the manual, I remove the standby power, then checked out these Instructions for replacing the CPU to remove the CPU; Gautam also has done this before.
Fri Sep 6 20:09:01 2019 Fri Sep 6 20:09:02 2019
I pulled the leftmost CPU module out, following the instructions above. The CPU module matches the physical layout and part number of the Sun Fire X4600 M2 8-DIMM CPU module; pressing the fault reminder light gives amber indicators at the DIMM ejectors, indicating faulty DIMMs (see). The other indicator LEDs did not illuminate.
I located several spare DIMMs in the digital cabinet along Y arm (and a couple with misc computer components in the control room), but didn't find the correct one for this CPU module. The DIMM is Sun PN 371-1764-01; I found it online and ordered eight. Please let me know if this is incorrect.
To protect the CPU module, I've put it in an ESD safe bag with some bubble wrap and a note. It's on the E shop bench.
Assuming you are at pianosa, /etc/resolv.conf is like
# Generated by NetworkManager
But this should be like
as indicated in https://nodus.ligo.caltech.edu:8081/40m/14767
I did this change for now. But this might get overridden by Network Manager.
Q1 Can we run the machine with the reduced # of cores?
Q2 We might be able to order them quickly. What's the spec and configuration of the DIMMs (like DDR2-667MHz ECC 4GBx4, and even more specs (like Samsung 2GB DDR2 RAM PC2-6400 240-Pin DIMM M378T5663EH3) so that we are to identify the exact spec).
Q3 Can we scavenge the old OMC RT machine or even megatron to extract the memories?
Saw these slightly delayed.
Q1: Not sure--is it a safe operation for me to remove the DIMM on CPU0, replace CPU0 (with no DIMM), and boot up to try this?
Q2: Specifically, it's this DIMM. The CPU core is compatible with DDR2, clock rate up to 333 MHz (DDR2-667) and 1, 2, or 4 GB of memory.
One pair of DIMM cards from the Sunstone box had the same Sun part number as those in c1ioo, so I swapped them in and reinstalled c1ioo's CPU0. c1ioo now boots up an seems ready to go, I'm able to log on from nodus. I also reinstalled optimus' CPU0, and optimus boots up with no problems.
While opening up optimus, I noticed a box labelled 'SUNSTONE' sitting below the rack--it contains two CPU modules a similar type as in c1ioo! I'm going to try swapping in the DIMM cards from this SUNSTONE box; I didn't find any elogs about sunstone--where are these modules from?
I reset c1lsc and c1sus, then ran rebootC1LSC.sh as before. All models started by the script are running with minimal red lights; c1oaf, c1cal, c1dnn, c1daf, and c1omc are not started by the script. I manually started these in the order c1cal->c1oaf->c1daf->c1dnn. Starting c1dnn crashed the other FE on c1ioo, so I reset all three FE again, and ran the script again (this time, including the startup for c1cal, c1oaf, and c1daf, but excluding c1dnn).
Except for c1dnn and c1omc, all models are started. The status lights are attached.
When I put away the lenses we had used for measuring the RF transfer functions of the QPD heads, I saw that I'd removed them from the cabinet containing green endtable optics, but hadn't noticed the sign forbidding their removal. I'll talk with Koji/Gautam about what happened and what should be done.
Came across this while looking up the BIO situation at 1Y2. For reference, the fix Koji mentions can be seen in the attached screenshot (one example, the other BIO cards also have a similar fix). The 16th bit of the BIO is grounded, and some bit-shifting magic is used to implement the desired output.
New DIMM cards have arrived. I stored them in the digital cabinet along y arm.
Here is the procedure for setting up the three new BHD front-ends (c1bhd, c1sus2, c1ioo - replacement). This plan is based on technical advice from Rolf Bork and Keith Thorne.
The overall topology for each machine is shown here. As all our existing front-ends use (obsolete) Dolphin PCIe Gen1 cards for IPC, we have elected to re-use Dolphin Gen1 cards removed from the sites. Different PCIe generations of Dolphin cards cannot be mixed, so the only alternative would be to upgrade every 40m machine. However the drivers for these Gen1 Dolphin cards were last updated in 2016. Consequently, they do not support the latest Linux kernel (4.x) which forces us to install a near-obsolete OS for compatibility (Debian 8).
I used an Acromag XT1221 in CTN to play around with different wiring and see what works. Following are my findings:
Floating Single Ended Source (Attachment 2):
Differential Source (Attachment 3):
Comments and suggestions are welcome.
Related elog posts:
Edit Tue Jan 26 12:44:19 2021 :
Note that the third wiring diagram mentioned actually does not work. It is an error in judgement. See 40m/15762 for seeing what happens during this.
Thanks for the systematic effort.
I'm working on a better wiring diagram that takes into account multiple power supplies, how their GND is passed forward to the circuits or sensors using those power supplies and what possible wiring configurations on Acromag would give low noise. I think I have two configurations in mind which I will test and update here with data and better diagrams.
I took some striptool images earlier yesterday. So I'm dumping them here for further comments or inferences.
Taking inspiration from SR785 on how it reads differential signal, I figured that acromag too always need a way to return current through RTN ports always. That must be the reason why everything goes haywire when RTN is not connected to IN-. Now for single ended signals, we can always short RTN to IN- and keep same GND but then we need to be careful in avoiding ground loops. I'm gonna post a wiring diagram in next post to show how if two signal sources connect to each other separately, a GND loop can be formed if we tie each IN- port to RTN on an acromag.
Coming to the issue of reading a differential signal, what SR785 does is that it connects 50 Ohm resistance between Earth GND and differential signal shields (which are supposed to signal GND). In a floating GND setting, SR785 connects a 1 MOhm resistor between input shield and Earth GND. This can be used to read a differential signal through a single BNC cable since the shiled can take arbitrary voltages thanks ti the 1 MOhm resistor.
We can do the same in acromag. Instead of shorting RTN to IN- ports, we can connect them through a large resistor which would let IN- float but will give a path for current to return through RTN ports. Attached here are few scenarios where I connected IN- to RTN throguh wire, 820 Ohms, 10kOhms and 1MOhms in two sub cases where RTN was left open or was shorted to Earth GND. In all cases, the signal was produced by a 9V battery outputing roughly 8.16V. It seems that 10kOhm resistor between RTN and IN- with RTN connected to Earth GND is the best scenario noise wise. I'll post more results and a wiring diagram soon.
Here I present few wiring diagrams when using Acromag to avoid noisy behavior and ground loops.
Edit Wed Jan 27 13:38:19 2021 :
This solution is not acceptable as well. Even if it is successfull in reading the value, connecting resistor between IN- and RTN will not break the ground loops and the issue of ground loops will persist. Further, IN- connection to RTN breaks the symmetry between IN- and IN+, and hence reduces the common mode rejection which is the intended purpose of differential signal anyways. I'll work more on this to find a way to read differential signals without connecitng IN- and RTN. My first guess is that it would need the GND on the source end to be connected to EarthGND and RTN on acromag end to be connected to EarthGND as well.
I found a white paper from Acromag which discusses how to read differential signal using Acromag units. The document categorically says that differential signals are always supposed to be transmitted in three wires. I provides the two options of either using the RTN to connect to the signal ground (as done in Attachment 3) or locally place 10k-100k resistors between return and IN+ and IN- both (Attachment 2).
I have provided possible scenarios for these.
Using an acromag card without making any connection with RTN is basically not allowed as per this document.