[anchal, ian]

We went and collected some information for the overlords to fix the c1teststand DAQ network issue.

• from c1teststand, c1bhd and c1sus2 computers were not accessible through ssh. (No route to host). So we restarted both the computers (the I/O chassis were ON).
• After the computers restarted, we were able to ssh into c1bhd and c1sus, ad we ran rtcds start c1x06 and rtcds start c1x07.
• The first page in attachment shows the screenshot of GDS_TP screens of the IOP models after this step.
• Then we started teh user models by running rtcds start c1bhd and rtcds start c1su2.
• The second page shows the screenshot of GDS_TP screens. You can notice that DAQ status is red in all the screens and the DC statuses are blank.
• So we checked if daqd_ services are running in the fb computer. They were not. So we started them all by sudo systemctl start daqd_*.
• Third page shows the status of all services after this step. the daqd_dc.service remained at failed state.
• open-mx_stream.service was not even loaded in fb. We started it by running sudo systemctl start open-mx_stream.service.
• The fourth page shows the status of this service. It started without any errors.
• However, when we went to check the status of mx_stream.service in c1bhd and c1sus2, they were not loaded and we we tried to start them, they showed failed state and kept trying to start every 3 seconds without success. (See page 5 and 6).
• Finally, we also took a screenshot of timedatectl command output on the three computers fb, c1bhd, and c1sus2 to show that their times were not synced at all.
• The ntp service is running on fb but it probably does not have access to any of the servers it is following.
• The timesyncd on c1bhd and c1sus2 (FE machines) is also running but showing status 'Idle' which suggested they are unable to find the ntp signal from fb.
• I believe this issue is similar to what jamie ficed in the fb1 on martian network in 40m/16302. Since the fb on c1teststand network was cloned before this fix, it might have this dysfunctional ntp as well.

We would try to get internet access to c1teststand soon. Meanwhile, someone with more experience and knowledge should look into this situation and try to fix it. We need to test the c1teststand within few weeks now.