[Gautam Koji]

After this work, we recovered the nominal RTCDS state. The main points were:

1. We needed to restart the bind9 service on chiara such that the FEs knew their IP addresses upon reboot and hence, could get their root filesystems over NFS.
2. We recovered suspension local damping, IMC locking and POX/POY locking with nominal arm transmission.

Some stuff that is not working as usual:

1. The EX QPD is reporting strange transmission values - even with the PRM completely misaligned, it reports transmission of ~30. But we were able to lock the Xarm with the Thorlabs PD and revover transmission of ~1.15.
2. The X arm green does not stay locked to the cavity - the alignment looks fine, and the green flashes are strong, but the lock does not hold. This shouldn't be directly connected to anything we did today since the Green PDH servo is entirely analog.

I made a model change in c1x03 (the IOP model on c1ioo) to add a DAC part. The model compiled, installed and started correctly, and looking at dmesg on c1ioo, it recognises the DAC card as what it is. Next step is to use a core on c1ioo for a c1omc model, and actually try driving some signals.

Note that the only change made to the c1ioo expansion chassis was that a DAC card was installed into the PCIe bus. The adaptor card which allows interfacing the DAC card to an AI board was already in the expansion chassis, presumably from whenever the DAC was removed from this machine.

*I think I forgot to restart optimus after this work...

The main motivation behind adding a DAC card in c1ioo was to setup an RTCDS model for the OMC. Attachment #1 shows the new look CDS overview screen. Here is what I did.

Mostly, I followed instructions from when I setup the model for the EX green PZTs.

Simulink model:

The model is just a toy for now (CDS parameters, ADC block and 2 CDS filter modules). I leave it to Aaron to actually populate it, check functionality etc. The path to the model is /opt/rtcds/caltech/c1/userapps/release/isc/c1/models/c1omc.mdl. I am listing the parameters set on the CDS_PARAMETERS block:

• host = c1ioo
• site = c1
• rate = 16k
• dcuid = 27 (which I chose after making sure that this dcuid was not used on this list which I also updated by adding c1omc and moving c1imc to "old")
• specific_cpu = 6 (again chosen after checking the available CPUs in the above list and confirming using the cset utility).
• adc_Slave = 1
• shmem_daq = 1
• no_rfm_dma = 1
• biquad = 1

Building and installing model:

Once the model was installed, I logged into c1ioo, and built and installed the models using the usual rtcds make and rtcds install instructions. Before starting the model, I edited /diskless/root.jessie/etc/rtsystab to allow c1omc to be run on c1ioo. Using sudo cset set, I verified that CPU #6 is no longer listed (if I understand correctly, the RTCDS system takes over the core).

MEDM:

To reflect all this on the MEDM CDS OVERVIEW screen, I just edited the screen.

• Moved the orange explanation of bits over to the c1iscey panel to make space in the c1ioo panel.
• Edited the macros to reflect the c1omc parameters.

DAQD:

Finally, I followed the instructions here to get the channels into frames and make all the indicators green. Went into fb and restarted the daqd processes. All looks good . I'm going to leave the model running overnight to investigate stability. I forgot to svn commit the model tonight, will do it tomorrow.

The testing plan (at least initially) is to install the AA and AI boards from the OMC rack in 1X1/1X2. Then we will have short SCSI cables running from the ADC/DAC to these. The actual HV driving stages will remain in the OMC rack (NE corner of AS table).

@Steve, can we get 10 Male-Female D9 cables so that we can run them from 1X1/1X2 to the OMC rack?

Unrelated to this work: There were 2 crashes of the models on c1lsc, one ~6pm and one right now ~1030pm. The restart script brought everything back gracefully  ...

Actually, c1lsc had crashed again sometime last night so I had to reboot everything this morning. I used the reboot script again, but I increased the sleep time between trying to start up the models again so that I could walk into the VEA and power cycle the c1lsc expansion chassis, as this kind of frequent model crash has been fixed by doing so in the past. Sure enough, there have been no issues since I rebooted everything at ~1030 in the morning. 

The c1omc model itself has been stable as well, though of course, there is nothing in there at the moment. I may do a check of the newly installed DAC tomorrow just to see that we can put out a sine wave.

Steve has ordered the D-sub cabling that will allow us to route signals between AA/AI boards in 1X1/1X2 to the HV PZT electronics in the OMC rack. Things look setup for a measurement next week. Aaron will post a block diagram + photoz of what box goes where in the electronics racks.

I'm going to guess that this was me: I was disconnecting some octopus power strip nonsense down there (in particular illuminators and cameras), so I might have turned off the AUX rack by mistake.

Follow up:

- At least it was confirmed that the local backup (4TB->2TB) is regularly running every morning.

- The 2TB disk was used up to 95%. To ease the size of the remaining space, I have further compressed the burt snapshot folders. (~2016). This released another 150GB. The 2TB is currently used up to  87%.

Prev

Filesystem      1K-blocks       Used  Available Use% Mounted on
/dev/sdc1      3845709644 1731391748 1918967020  48% /home/cds
/dev/sdd1      2113786796 1886162780  120249888  95% /media/40mBackup

Now

Filesystem      1K-blocks       Used  Available Use% Mounted on
/dev/sdc1      3845709644 1731706744 1918652024  48% /home/cds
/dev/sdd1      2113786796 1728124828  278287840  87% /media/40mBackup

LIGO GC notified us that nodus had SSL2.0 and SSL3.0 enabled. This has been disabled now.
The details are described on 40m wiki.

The VEA vertex laptop, paola, has a flashing orange indicator which I take to mean some kind of battery issue. When the laptop is disconnected from its AC power adaptor, it immediately shuts down. So this machine is kind of useless for its intended purpose of being a portable computer we can work at optical tables with. The actual battery diagnostics (using upower) don't report any errors. 

donatella was one of our last workstations running ubuntu12. we installed SL7 on there today

1. had to use a DVD; wouldn't boot from USB stick
2. made sure to use userID=1001 and groupID=1001 at the initial install part
3. went to the Keith Thorne LLO wiki on SL7
4. The 'yum update' command failed due to a gstreamer conflict. I did "yum remove gstreamer1-plugins-ugly-free-1.10.4-3.el7.x86_64" and then it continued a bit more.

5. Then there are ~20 errors related to gds-crtools that look like this:Error: Package: gds-crtools-2.18.12-1.el7.x86_64 (lscsoft-production) Requires: libMatrix.so.6.14()(64bit)

6. I re-ran the yum install .... command using the --skip-broken command and that seemed to complete, although I guess the GDS stuff will not work.
7. Installed: terminator, inconsolata-fonts, 
8. Installed XFCE desktop as per K Thorne:  yum groupinstall "Xfce" -y
9.  

The old IBM laptop (Asia) has died from a fan error after 7 years. WE have a new Lenovo 330 IdeaPad to replace it:

1. to enter bios, the usual FN keys don't work. Power off laptop. Insert paperclip into small hole on laptop side with upside-down U symbol. Laptop powers up into BIOS setup.
2. Insert SL 7.6 DVD into drive
3. Change all settings from modern UEFI into Legacy support. Change Boot order to put CDROM first.
4. Boot.
5. Touchpad is not detected. Hookup mouse for setup.
6. Delete windows partition.
7. Setup wireless network according to (https://wiki-40m.ligo.caltech.edu/Network). Computer name = asia.martian. 
8. Set root password. Do not create user (we want to make the controls acct later using the command line so that we can set userID and groupID both to 1001).
9. Begin install...lots of disk access noises for awhile...

Install done. Touchpad not recognized by linux - lots of forum posts about kernel patching...Arrgh!

I have swapped our martian router's WiFi security over to WPA2 (AES) from the previous, less-secure, system. Creds are in the secrets-40-red.

had trouble using YUM to update. This turned out to be a config problem with our Martian router, not the new laptop. Since I've changed the WiFi pwd awhile ago for the martian access for the CDS laptops, you'll have to enter that in order to use the laptops.

turned out to be some Access Control nonsense inside of the router. Even loggin in as admin with a cable gave some of the fields the greyed out color (had to hover over the link and then type the URL directly in the browser window). ASIA is now able to connect and use YUM + usual connections. Gautam and I have also moved the router a little to get easier view of its LED lights and not blockk its WiFi signal with the cable tray. We'll get a little shelf so that we can mount it ~1 foot off of the wall.

still, this seems like a bad laptop choice: the Lenovo Ideapad 330 will not have its touchpad supported by SL7 without compiling a new version of the kernel 

Now that the replacement susaux machine is installed and fully tested, I renamed it from c1susaux2 to c1susaux and updated the DNS lookup tables on chiara accordingly.

• setup controls user using K Thorne LLO CDS offsite workstation instructions
• modified /etc/fstab ala pianosa to NFS mount disks
• set up symlinks as other workstations
• troubles with libsasl2 and libmetaio libraries as usual for SL7 - doing symlink tricks
• setup shared .bashrc
• now running 'yum install gds-all' to see if we need more local libraries to run GDS from the shared disks...

Following instructions here, I installed ndscope on this machine. DTT still could not be be run from this machine, and I want to use this today - so I ran the following commands from the K. Thorne setup instructions.

yum clean metadata
yum update
yum install cds-workstation pcaspy subversion redhat-lsb  gnuradio google-chrome-stable xorg-x11-drv-nvidia epel-release redhat-lsb

Now diaggui can be opened, and spectra can be made. I'm moving this laptop to its new home at EY.

Milind pointed out that all boxes on the medm screens were white. I didn't have diagnostics from the medm screens, so I started following the troubleshooting steps on the restart procedures page.

It seemed like maybe a frontend problem. I tried telnet-ing into several of the fe, and wasn't able to access c1psl. The section on c1psl mentions that if this machine crashes, the screens will go white and the crate needs to be turned off and on. Millind did this.

Now, most of the status lights are restored (screenshot).

Milind: I did a burtrestore following this and locked the PMC following the steps described in this elog.

Kruthi noticed that she could not login to rossa from giada.

I checked /etc/resolv.conf and it was

nameserver 127.0.0.1

so obviously it is useless to refer localhost (i.e. giada) as a nameserver.

I copied our usual resolv.conf to giada as following:

nameserver 192.168.113.104
nameserver 131.215.125.1
nameserver 8.8.8.8

search martian

Giada's ssh known_host had unupdated entry for rossa, so I had to clean it up, but after that we can connect to rossa from giada just by "ssh rossa".

Case closed.

• copied over /etc/fstab lines from pianosa sothat the NFS mounts work correctly
• added symlinks so that the NFS dirs mount in the right dirs
• installed Opera browser
• symlink libsasl2.so.3 -> libsasl2.so.2 and now DTT runs and can get data now and in the past
• DTT can natively produce PDF so you don't have to take screen caps of your camera phone and make a chalk drawing of that anymore
• sitemap/MEDM is working
• after editing fonts.alias as detailed in Thorne wiki, I ran 'sudo xset fp rehash' to get MEDM to notice new fonts. MEDM Scalable fonts are back!!
• installed Grace and now dataviewer works
• ndscope not running: yum install ndscope breaks because it can't find a couple of python34 packages
• tested that AWGGUI also runs and puts in real sine waves

I think rana did some more changes to this workstation to make it useful for commissioning activities - but the MEDM screens were still white blanks. The problem was that the firewalld wasn't disabled (last two steps of the KThorne setup wiki). I disabled it. Now donatella can run MEDM, ndscope and StripTool. DTT doesn't work to get online data because of a "Synchronization Error", I'm not bothering with this for now. I think Kruthi successfully demonstrated the fetching of offline data with DTT.

I brought one CPU (Dell T3500) and one 28" monitor from Mike Pedraza's office in Downs to the 40m. It is on Steve's desk right now, pending setup. The machine already has Solidworks and Altium installed on it, so we can set it up at our leisure. The login credentials are pasted on the CPU with a post-it should anyone wish to set it up.

Chub brought the replacement Supermicro we ordered to the 40m today. I stored it at the SW entrance to the VEA, along with the other Supermicro. At the time of writing, we have, in hand, two (unused) Supermicro machines. One is meant for EY and the other is meant for c1psl/c1iool0. DDR3 RAM and 120 GB SSD drives have also been ordered, but have not yet arrived (I think, Chub, please correct me if I'm wrong).

Update 20190802: The DDR3 RAM and 120 GB SSD drives arrived, and are stored in the FE hardware cabinet along the east arm. So at the time of writing, we have 2 sets of (Supermicro + 120GB HD + 4GB RAM).

The VEA laptop asia was configured to be able to connect to too many WiFi networks - it was getting conflicted in its default position at the vertex and trying to hop between networks, for some reason trying to connect to networks that had poor signal strength. I deleted all options from the known networks except 40MARS. Now the network connection seems much more stable and reliable.

cdsutils is not working on rossa.

Import cdsutils produces this error:

In [2]: import cdsutils
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-2-949babce8459> in <module>()
----> 1 import cdsutils

/ligo/apps/linux-x86_64/cdsutils-480/lib/python2.7/site-packages/cdsutils/__init__.py in <module>()
     53 
     54 try:
---> 55     import awg
     56 except ImportError:
     57     pass

/ligo/apps/linux-x86_64/cdsutils-480/lib/python2.7/site-packages/cdsutils/awg.py in <module>()
     30 """
     31 
---> 32 import sys, numpy, awgbase
     33 from time import sleep
     34 from threading import Thread, Event, Lock

/ligo/apps/linux-x86_64/cdsutils-480/lib/python2.7/site-packages/cdsutils/awgbase.py in <module>()
     17 libawg = CDLL('libawg.so')
     18 libtestpoint = CDLL('libtestpoint.so')
---> 19 libSIStr = CDLL('libSIStr.so')
     20 
     21 ####

/ligo/apps/anaconda/lib/python2.7/ctypes/__init__.pyc in __init__(self, name, mode, handle, use_errno, use_last_error)
    364 
    365         if handle is None:
--> 366             self._handle = _dlopen(self._name, mode)
    367         else:
    368             self._handle = handle

OSError: libSIStr.so: cannot open shared object file: No such file or directory

please no one touch the UPS: last time it destroyed ROSSA. Please ask Chub to order the replacement batteries so we can do this in a controlled way (fully shutting down ALL workstations first). Last time we wasted 8 hours on ROSSA rebuilding.

Chub wanted to get the correct part number for the replacement UPS batteries which necessitated opening up the UPS. To be cautious, all the workstations were shutdown at ~9:30am while the unit is pulled out and inspected. While looking at the UPS, we found that the insulation on the main power cord is damaged at both ends. Chub will post photos.

However, despite these precautions, rossa reports some error on boot up (not the same xdisp junk that happened before). pianosa and donatella came back up just fine. It is remotely accessible (ssh-able) though so maybe we can recover it...

I booted Rossa in rescue mode; though I see no errors on bootup, I still see the same error ("a problem has occurred") after boot, and a prompt to logout. I powered rossa off/on (single short press of power button), no change.

Booting in debug mode, I see that the error occurs when mounting /cvs/cds, with the error

Which is odd, because when I boot in recovery mode, is mounts /cvs/cds successfully. 

I booted in emergency mode by adding to the boot command

systemd.unit=emergency.target

but didn't have the appropriate root password to troubleshoot further (the usual two didn't work).

Formatted and re-installing OS on rossa for the 3rd or 4th time this year. I suggest that whoever is installing software and adjusting video settings please stop.

If you feel you need to tinker deeply, use ottavia or zita and then be prepared to show up and fix it.

While I was moving the UPS around, the network lights went out for Rossa, so I may have damaged the network interface or cable. Debugging continues.

Got the network to work again just by unplugging the power cord and letting it sit for awhile. But corrupted OS by trying to install Nvidia drivers.

https://www.advancedclustering.com/act_kb/installing-nvidia-drivers-rhel-centos-7/

back on new Rossa from Xi computing

1. switched to using Display Port for video; this works. The DVi, HDMI, VGA ports are connected to the motherboard rather than the video card, so they are not active.
2. runs super slow w/ SL 7.6; maybe some service is running after startup?
3. install repos and update according to LLO CDS wiki
4. add controls user and group according to LLO wiki
5. remove gstreamer ugly because it breaks yum update
6. run 'yum update --skip-broken' because GDS doesn't work
7. turn off old selinux stuff
8. modify fstab to get NFS

Next:

1. finish mounting
2. xfce
3. figure out why the LLO install instructions can't install any CDS software (e.g. root, DTT, etc)

Update: Sun Nov 3 18:08:48 2019

1. moved the SL7 fresh install repos back into etc/yum.repos.d/. The LLO instructions has me remove them, but the LLO supplied repos are no good for standard apps. After putting these back was able to install standard apps (terminator, root, diaggui)
2. copied over /etc/fstab lines from pianosa sothat the NFS mounts work correctly
3. added symlinks so that the NFS dirs mount in the right dirs
4. symlink libsasl2.so.3 -> libsasl2.so.2 and now DTT runs and can get data now and in the past
5. install XFCE
6. sitemap / MEDM works
7.  Did "sudo ln -s /usr/lib64/libXm.so.4 /usr/lib64/libXm.so.3" to enable StripTool.

Update: Fri Nov 15 00:00:26 2019:

1. random hanging of machine while doing various window moving or workspace switching
2. turned off power management in XFCE
3. turned off power management on monitor
4. disabled SELINUX
5. firewalld was already off
6. installed most, pdftk, htop, glances, qtgrace, lesstif
7. dataviewer now works and QTgrace is much nicer than XMGrace

and so it begins...until this is finished I have turned off the projector and moved the striptools to the big TV (time to look for Black Friday deals to replace the projector with a 120 inch LED TV)

the upgrade seems to have been successfully executed - the machine was restarted at ~430pm local time. Projector remains off and diagnostic striptools are on the samsung.

The IBM laptop at EX was running Ubuntu 14, so I allowed it to start upgrading itself to Ubuntu16 as it desired. After it is done, I will upgrade it to 18.04 LTS. We should have them all run LTS.

I noticed recently that Megatron was running Ubuntu 12, so I've started its OS upgrade.

1. Unlocked the IMC + disabled the autolocker from the LockMC screen + closed the PSL shutter (IMC REFL shutter doesn't seem to do anythin)
2. Disabled the "FSS" slow servo on the FSS screen
3. did sudo apt-get update, sudo apt-get upgrade, and then sudo apt-get do-release-upgrade which starts the actual thing
4. According to the internet, the LTS upgrades will go in series rather than up to 18 in one shot, so its now doing 12 -> 14 (Trusty Tapir)

Megatron and IMC autolocking will be down for awhile, so we should use a different 'script' computer this week.

Mon Dec 9 14:52:58 2019

upgrade to Ubuntu 14 complete; now upgrading to 16

Megatron is now running Ubuntu 18.04 LTS.

We should probably be able to load all the LSC software on there by adding the appropriate Debian repos.

I have re-enabled the cron jobs in the crontab.

The MC Autolocker and the PSL NPRO Slow/Temperature control are run using 'initctl', so I'll leave that up to Shruti to run/test.

wiped and install Debian 10 on rossa today

still to be done: config it as CDS workstation

please don't try to "fix" it in the meantime

upgrade was done

cronjob testing wasn't one by one 😢 

burt snapshots were gone

i brought them back home 🏠 

The burt snapshotting is still not so reliable - for whatever reason, the number of snapshot files that actually get written looks random. For example, the 14:19 backup today got all the snaps, but 15:19 did not. There are no obvious red flags in either the cron job logs or the autoburt log files. I also don't see any clues when I run the script in a shell. It'll be good if someone can take a look at this.

I've also been noticing that the IMC Autolocker scripts are running rather sluggishly on Megatron recently. Some evidence - on Feb 11 2019, the time between the mcup script starting and finishing is ~10 seconds (I don't post the raw log output here to keep the elog short). However, post upgrade, the mean time is more like ~45-50 seconds. Rana mentioned he didn't install any of the modern LIGO software tools post upgrade, so maybe we are using some ancient EPICS binaries. I suspect the cron job for the burt snapshot is also just timing out due to the high latency in channel access. Rana is doing the software install on the new rossa, and once he verifies things are working, we will try implementing the same solution on megatron. The machine is an old Sun Microsystems one, but the system diagnostics don't signal any CPU timeouts or memory overflows, so I'm thinking the problem is software related...

There were a bunch of medm processes stalled on megatron (connected with screenshot taking). To see if they were interfering with the other scripts, I killed all of the medm processes, and commented out the line in the crontab that runs the screenshots every 10 mins. Let's see if this improves stability.

[Jon, gautam]

We found that the caput commands were taking much longer to execute on megatron than on pianosa (for example). Suspecting that this had something to do with the fact that megatron was using EPICS binaries from the shared NFS drive which were compiled for a much older OS, I installed the latest stable release of EPICS on megatron. The new caput commands execute much faster. I also added the local EPICS directory to the head of the $PATH variable used by the MC autolocker and FSS Slow scripts, so that they use the new caput command. But mcup is still slow - maybe my new path definition isn't picked up and it is still using the NFS binaries? To be looked into...

Allegra had no network cable and no mouse. We found Allegra'snetwork cable (black) and connected it.

I found a dirty old school mouse and connected it.

I wiped Allegra and now I'm currently installing debian 10 on allegra following Jon's elog.

04/01 update: I forgot to mention that I tried installing cds software by following Jamie's instruction: I added the line in /etc/apt/sources.list.d/lscsoft.list: "deb http://software.ligo.org/lscsoft/debian/ stretch contrib". But this the only thing I managed to do. The next command in the instructions failed.

Summary

Today I finished implementing loopback monitors of the up/down state of the slow controls machines. They are visible on a new MEDM screen accessible from Sitemap > CDS > Slow Machine Status (pictured in attachment 1). Each monitor is a single EPICS binary channel hosted by the slow machine, which toggles its state at 1 Hz (an alive "blinker"). For each machine, the monitor is defined by a separate database file named c1[machine]_state.db located in the target directory.

This is implemented for all upgraded machines, which includes every slow machine except for c1auxey. This is the next and final one slated for replacement.

Implementation

The blinkers are currently implemented as soft channels, but I'd like to ultimately convert them to hard channels using two sinking/sourcing BIO units. This will require new wiring inside each Acromag chassis, however. For now, as soft channels, the monitors are sensitive to a failure of the host machine or a failure of the EPICS IOC. As hard channels, they will additionally be sensitive to a failure of the secondary network interface, as has been known to happen.

Each slow machine's IOC had to be restarted this afternoon to pick up the new channels. The IOCs were restarted according to the following procedure.

c1auxex

• Disabled OPLEV servos on ETMX
• Zeroed slow biases
• Disabled watchdog
• Restarted IOC
• Reverted 1-3

​c1vac

• Closed V1, VM1
• Restarted IOC
• Returned valves to original state

c1psl

• Disabled IMC autolocker
• Closed PSL shutter
• Restarted IOC
• Reverted 1-2

c1iscaux

• ​Restarted IOC

c1susaux

• Disabled IMC autolocker
• Closed shutter
• Disabled OPLEV servos on: MC1, MC2, MC3, BS, ITMX, ITMX, PRM, SRM
• Zeroed slow biases
• Disabled watchdogs
• Restarted IOC
• Reverted 1-5

The intial recovery of c1susaux did not succeed. Most visibly, the alignment state of the IFO was not restored. After some debugging, we found that the restart of the modbus service was partially failing at the final burt-restore stage. The latest snapshot file /opt/rtcds/caltech/c1/burt/autoburt/latest/c1susaux.snap was not found. I manually restored a known good snapshot from earlier in the day (15:19) and we were able to relock the IMC and XARM. GV and I were just talking earlier today about eliminating these burt-restores from the systemd files. I think we should.

In an effort to make a second usable workstation, I did the following (remotely) on rossa today (not necessarily in this order, I wasn't maintaining a live log so I forgot):

1. Fixed /etc/resolv.conf, so that the other martian machines can be found.
2. Copied over .bashrc file, and the appropriate lines from /etc/fstab from pianosa to rossa.
3. Ran sudo apt install nfs-common. Then ran sudo mount -a to get /cvs/cds mounted.
4. Made symlinks for /users and /opt/rtcds , and /ligo. All of these are used by various environment-setting scripts and I chose to preserve the structure, though why we need so many symlinks, I don't know...
5. Set up the shell variable $NDSSERVER using export NDSSERVER=fb:8088. I'm not sure how, but I believe DTT, awggui etc use this on startup to get the channel list (any
6. Followed instructions from Erik von Reis at LHO to install the cds workstation packages and dependencies. Worked like a charm 🎃! 
7. As a test, I plotted the accelerometer spectra in DTT, see Attachment #1. I also launched foton from inside awggui, and confirmed that the sample rate is inherited and I could designate a filter. But I haven't yet run the noise injection to test it, I'll do that the next time I'm in the lab.
8. Also checked that medm, StripTool and ndscope, and anaconda python all seem to work 👍🏾.

So, in summary, rossa is now all set up for use during lock acquisition. However, until this machine has undergone a few months of testing, we should freeze the pianosa config and not mess with it.

Note that this version of the "crtools" is rather new. Please, use them and if there is an issue, report the errors! I am going to occassionally try lock acquisition using rossa. 

maybe we should make a "dd" copy of pianosa in case rossa has issues and someone destroys pianosa by accidentally spilling coffee on it.

I did

sudo usermod -a -G lpadmin controls

and then was able to add Grazia to the list of printers for Rossa by following the instructions on the 40m Wiki.

I installed color syntax highlighting on Rossa using the internet (https://superuser.com/questions/71588/how-to-syntax-highlight-via-less). Now if you do 'less genius_code.py', it will be highlighting the python syntax.

when I try 'sitemap' on rossa I get:

medm: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such file or directory
 

This is strange - I was definitely able to launch medm when I was working on this machine remotely on Friday. But now, there does seem to be a problem with this shared library being missing.

First of all, I installed mlocate to find where the shared library files are installed. Then I made the symlink, and now sitemap seems to work again.

Weirdly, my changes to /etc/resolv.conf got overwritten somehow. Was this machine rebooted? Uptime suggests it's only been running for ~6 hours at the time of writing of this elog.

yes, I rebooted yesterday to fix the 'steaking white lines' problem in the video/display

maybe we're supposed to edit something besides resolv.conf since that gets over-written on boot for some linux OS

Indeed, this is now fixed by following instructions from here. I rebooted rossa at ~1250 PDT and confirmed that resolv.conf didn't get overwritten. The resolv.conf file also now has the following useful lines at the head:

I wanted to try using rossa as my locking workstation today. However, a few problems became quickly evident. Basically, any of our scripts that rely on the cdsutils package (there are MANY) will not work on rossa, because of some library error. This machine is running Debian 10, while the cdsutils package is being loaded from a pre-compiled install on the shared drive, so perhaps this isn't surprising?

Digging a little more, I found that actually, a version of cdsutils that actually works with python3 is actually shipped with the standard cds-workstation meta-package. This is great news, and we should try and use this where possible I guess. Deferring further debugging for daytime work.

Anyway, I added a symlink: sudo ln -s /usr/lib/x86_64-linux-gnu/libncurses.so.6 /usr/lib/x86_64-linux-gnu/libncurses.so.5, and installed wmctrl using sudo apt install wmctrl. 

I noticed these streaky lines again today (but they were not a problem last night). It is annoying if we have to reboot this machine all the time. I wonder if this has something to do with missing drivers. When I ran sudo apt update && sudo apt upgrade, I got several lines like (this isn't the whole stack trace)

Is this indicative of the graphics drivers being installed incorrectly? I am hesitant to mess with this because I think in the past, it was always trying to update some graphics driver that crashed the whole machine into some weird state where we have to wipe the drive and do a fresh re-install of the OS.

Should we just follow these instructions? The graphics card is apparently Quadro P400, which is one of the supported ones according to the list of supported devices.

Or just swap donatella and rossa monitors and defer the problem for later?

After some consultation with Erik von Reis at LHO, this workstation is progressing towards being usable for most commissioning tasks. DTT, awggui, foton, and MEDM are all now working well. The main limitation now comes from the fact that many of our python scripts are written for python2, and rossa doesn't have many dependencies installed for python2. I see no reason to build these dependencies on rossa for python2, we should not have to work with an unsupported language. But at the same time, I don't want to completely wipe all our python2 scripts, and make them python3, because this would involve a lot of tedious testing that I'm not prepared to undertake at the moment (the problem is compounded by the fact that pianosa does not have many dependencies installed for python3).

So what I have done in the interim is make python3 versions of the most important scripts I need to get the PRFPMI locking working - they are in the scripts directory and have the same names as their python2 counterparts, but have a 3 appended to their names. So when working on rossa, these are the scripts that are called. Eventually, after a lot more testing, we can depracate the old scripts. Currently, where applicable, the MEDM screens allow for either the python2 or python3 version of the script to be called.

Please, for the time being, do not try and install any new packages on rossa unless you are prepared to debug any problems caused and return the machine to a workable state. If you find some issue with a missing package on rossa, (i) make a note of it on the elog, and (ii) if possible, set up your own conda environment for testing and install dependencies to that environment only.