This afternoon Gautam and I assessed what to do about restoring the GigE camera software. Here's what I propose:
I've started resolving the many dependencies of this code on rossa. The idea is to get a working environment on one workstation, then generate requirements files that can be used to set up the rest of the machines. I believe the dependencies have all been installed. However, many of the packages are newer versions than before, and this seems to have broken SnapPy. I'll continue debugging this tomorrow.
I have been trying a couple of HDR algorithms, all of them seem to give very different results. I don't know how suitable these algorithms are for our purpose, because they are more concerned with final display. I'm attaching the HDR image I got by modifying Jigyasa's code a bit (this image has been be modified further to make it suitable for displaying). Here, I'm trying compare the plots of images that look similar. The HDR image has a dynamic ratio of 700:1
PS: 300us_image.png file actually looks very similar to HDR image on my laptop (might be an issue with elog editor?). So I'm attaching its .tiff version also to avoid any confusion.
I upgraded Pylon, the C/C++ API for the GigE cameras, to the latest release, 5.2.0. It is installed in the same location as before, /opt/rtcds/caltech/c1/scripts/GigE/pylon5, so environment variables do not change. The old version, 5.0.12, still exists at opt/rtcds/caltech/c1/scripts/GigE/backup_pylon5.
The package contains a GUI application (/bin/PylonViewerApp) for streaming video. The old version supports saving still images, but Milind discovered that the new version supports saving video as well. This required installing a supplementary package supporting MPEG-4 output.
Basler's GUI application is launched from the terminal using the alias pylon. I've tested it and confirm it can save both videos and still-image formats. I recommend to also try grabbing images using this program and check the bit resolution. It would be a useful diagnostic to know whether it's a bug in Joe B.'s code or something deeper in the camera settings.
At the lab meeting today, Rana suggested that I use the Pylon app to collect more data if that's what I need. Following this, Jon helped me out by updating the pylon version and installing additional software to record video. Now I am collecting data at
Consequently I have dithered the MC2 optic from around 9:00 PM.
Since there are multiple SURF projects that rely on the cameras:
My changes were necessary because the grabHDR.py script was throwing python exceptions, whereas it was running just fine before Jon's changes. We can move the "new_*" dirs to the default once the SURFs are gone.
Let's freeze the camera software config in this state until next week.
Somehow I never got around to doing the pixel sum thing for the new real data from the GigE. Since I have to do it for the presentation, I'm putting up the results here anyway. I've normalized this and computed the SNR with the true readings.
SNR = (power in true readings)/ (power in error signal between true and predicted values)
Attachment #2 is SNR of best performing CNN for comparison.
I'll keep developing the camera server on a parallel track using the "new_..." directory naming convention. One thing I forgot to note is that the new pylon/pypylon packages require Python 3, so will not work with any of the 2.7 scripts. All of the environment I need to set up is exclusively Python 3. I won't change anything in the Python 2.7 environment in current use.
Also, I found the source of the bit resolution issue: Joe B's code loads a set of initialization parameters from a config file. One of them is "Frame Type = Mono8" which sets the dynamic range of the stream. I'll look into how this should be changed.
I've started setting up the last new rackmount SuperMicro as a dedicated server for the GigE cameras. The new machine is currently sitting on the end of the electronics test bench. It is assigned the hostname c1cam at IP 192.168.113.116 on the martian network. I've installed Debian 10, which will be officially supported until July 2024.
I've added the /cvs/cds NFS mount and plan to house all the client/server code on this network disk. Any dependencies that must be built from source will be put on the network disk as well. Any dependencies that can be gotten through the package manager, however, will be installed locally but in an automated way using a reqs file.
We should ask Chub to reorder several more SuperMicro rackmount machines, SSD drives, and DRAM cards. Gautam has the list of parts from Johannes' last order.
I've put the analog camera back and disconnected the 151 unit GigE. But I ran out of time and wasn't able to replace the beamsplitter. I've put all the equipments back to the place where I took them from. The chopper and beam dump mount, that Koji had got me for the scatterometer, are kept outside, on the table I was working on earlier, in the control room. The camera lenses, additional GigEs, wedge beamsplitter, 1050nm LED and all related equipments are kept in the GigE box. This box was put back into CCD cameras' cabinet near the X arm.
Note: To clean stuff up, I had entered the lab around 9.30pm on Monday. This might have affected Yehonathan's loss measurement readings (until then around 57 readings had been recorded).
Sorry for the late update.
Following the death of rossa, which was hosting the only working environment for the GigE camera software, I've set up a new dedicated rackmount camera server: c1cam (details here). The Python server script is now configured as a persistent systemd service, which automatically starts on boot and respawns after a crash. The server depends on a set of EPICS channels being available to control the camera settings, so c1cam is also running a softIOC service hosting these channels. At the moment only the ETMX camera is set up, but we can now easily add more cameras.
Instructions for connecting to a live video feed are posted here. Any machine on the martian network can stream the feed(s). The only requirement is that the client machine have GStreamer 0.10 installed (all the control room workstations satisfy this).
As much as possible, the code and dependencies are hosted on the /cvs/cds network drive instead of installed locally. The client/server code and the Pylon5, PyPylon, and PyEpics dependencies are all installed at /cvs/cds/rtcds/caltech/c1/scripts/GigE. The configuration files for the soft IOC are located at /cvs/cds/caltech/target/c1cam.
The 40m GigE camera code is a slightly-updated version of the 10+ year-old camera code in use at the sites. Consequently every one of its dependencies is now deprecated. Ultimately, we'd like to upgrade to the following:
This is a long-term project, however, as many of these APIs are very different between Python 2 and 3.
We noticed last week that the MC2 trans camera has pitch and yaw swapped; I rotated what I thought is the correct camera by 90 degrees clockwise (as viewed from above, like in the attachment), but I now have doubts. It's the camera on the right in the attachment.
The left one is analog and 90deg rotated.
See also: This issue tracker
MC2 analog camera was rotated by 90 degrees. Orientation correctness was verified by exciting the MC2 Yaw degree of freedom.
Attached before and after photos of the camera setup.
There's this elog from Stephen about better 1064 sensitivity from Basler. We should consider getting one if he finds that its actual SNR is as good as we would expect from the QE improvement.
Might allow for better scatter measurements - not that we need more signal, but it could allow us to use shorter exposure times and reduce blurring due to the wobbly beams.
Nice, and we should also permanently install the camera server (c1cam) which is still sitting on the electronics bench. It is running an adapted version of the Python 2/Debian 8 site code. Maybe if COVID continues long enough I'll get around to making the Python 3 version we've long discussed.
Note from Stephen on more sensitive Baslers.
We found Mon7 in control room dead today afternoon. It's front power on green light is not lighting up. All other monitors are working as normal.
This monitor was used for looking at IMC camera analog feed. It is one of the most important monitors for us, so we should replace it with a different monitor.
Yehonathan and Paco disconnected the monitor and brought it down. We put it under the back table if anyone wants to fix it. Paco has ordered a BNC to VGA/HDMI converter to put in any normal monitor up there. It will happen this Wednesday. Meanwhile, I have changed the MON4 assignment from POP to Quad2 to be used for IMC.
We replaced the Mon 7 with an LCD monitor from back bench. It is fed the analog signal from BNC converted into VGS with a converter box that Paco bought. We can replace this monitor with another monitor if it is required on the back bench. For now, we definitely need a monitor to show IMC camera's up there.
Tested the Nikon batteries for the camera. they are supposed to be 7V batteries but they don't hold a charge. I confirmed this with multi-meter after charging for days. Ordered new ones Nikon EN-EL9
I believe that the Nikon has an exposure problem and that's why we bought the Canon.
Coil Drivers LO2, SR2, AS4, and AS1 have been updated a reinstalled into the system.
LO2 Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 Unit: S2100008
LO2 Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 Unit: S2100530
SR2 Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 Unit: S2100614
SR2 Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 Unit: S2100615
AS1 Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 Unit: S2100610
AS1 Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 Unit: S2100611
AS4 Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 Unit: S2100612
AS4 Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 Unit: S2100613
Nice. Please put this information on the DCC pages of the coil driver units. You'll find links to all the units in this document tree LIGO-E2100447. For each page, click on "Change Metadata" from the left panel and add the change made to the resistor (including the resistor name on PCB, previous and new value), and add a link to your previous elog post which has more details like photos, to "Notes and Changes", and upload an updated version of the circuit schematic by creating an annotation in the previous circuit schematic pdf. Every unit that has a serial number in the lab has a DCC page (if not, we should create one) where we should track all such hard changes.
18 (9 pairs) Coil Drivers have been modified. Namely ETMX/ITMX/ITMY/BS/PRM/SRM/MC1/MC2/MC3.
ETMX Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100624 ETMX Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100631
ITMX Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100620 IMTX Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100633
ITMY Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100623 ITMY Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100632
BS Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100625 BS Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100649
PRM Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100627 PRM Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100650
SRM Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100626 SRM Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100648
MC1 Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100628 MC1 Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100651
MC2 Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100629 MC2 Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100652
MC3 Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 S2100630 MC3 Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 S2100653
Will be updating this linking each coil driver to the DCC
The DCC has been updated, along with the modified schematic. Links have been attached.
>> which decimate
if (`pgrep lockMZ | wc -l` > 1) then
echo lockMZ is already running!
pgrep `basename $0` | grep -v $$ > /dev/null
echo Another copy of this program is already running. Exiting!