40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Fri Sep 8 12:08:32 2017, Gabriele, Summary, LSC, Good reconstruction of PRMI degrees of freedom with deep learning full_real_data_2017_09_06b_100x4_4000_300_20_free_swinging_time_domain.png.pngfull_real_data_2017_09_06b_100x4_4000_300_20_free_swinging_free_swinging_histograms2d.png.pnggru_2017_08_13a_100x4_4000_300_20_free_swinging_simulation_reference.png.png
    Reply  Fri Oct 6 12:56:40 2017, gautam, Summary, LSC, RTCDS NN post_NN_test.png
       Reply  Tue Oct 17 17:53:25 2017, jamie, Summary, LSC, prep for tests of Gabriele's neural network cavity length reconstruction c1dnn.png
          Reply  Wed Oct 18 12:14:08 2017, jamie, Summary, LSC, prep for tests of Gabriele's neural network cavity length reconstruction 
             Reply  Thu Oct 19 15:42:03 2017, jamie, Summary, LSC, MICH/PRCL reconstruction neural network running on c1lsc NN.pngC1DNN_GDS.pngC1DNN_CPU_METER.png
                Reply  Tue Oct 24 20:14:21 2017, jamie, Summary, LSC, further testing of c1dnn integration; plugged in to DAQ NN.pngc1dnn_out.png
                   Reply  Wed Oct 25 09:32:14 2017, Gabriele, Summary, LSC, further testing of c1dnn integration; plugged in to DAQ 
                   Reply  Mon Nov 6 18:22:48 2017, jamie, Summary, LSC, current procedure for running c1dnn code 
                      Reply  Thu Nov 9 10:51:37 2017, gautam, Summary, LSC, current procedure for compiling and installing c1dnn code 
Message ID: 13395     Entry time: Thu Oct 19 15:42:03 2017     In reply to: 13390     Reply to this: 13400
Author: jamie 
Type: Summary 
Category: LSC 
Subject: MICH/PRCL reconstruction neural network running on c1lsc 

Gabriele's PRCL/MICH reconstruction neural network is now running on c1lsc.  Summary:

  • front-end model is called c1dnn, and is running as an experimental user-space process
  • c1dnn is getting most of it's needed inputs from existing SHMEM IPC outputs from c1lsc
  • none of the output signals from the network are being sent anywhere yet (grounded)
  • c1dnn has not been integrated in any way, into the DAQ etc.  it is being run manually by hand, and will be completely shut down after this test

Simple MEDM screen I made to monitor the input/output signals:

The RTS process seems to run fine, but there is quite a bit of jitter in the CPU_METER, at the 50% level:

It's not running over the limit, but it is jumping around more than I think it should be.  Will look into that...

cpuset for cpu isolation for user-space model

The c1dnn model is running on CPU6 on c1lsc.  CPU6 was isolated from the rest of the system using cpuset.  The "cset" utility was used to create a "system" CPU set that was assigned to CPU0, and the kernel was instructed to move all running processes to that set:

controls@c1lsc:~ 2$ sudo cset set
cset:
         Name       CPUs-X    MEMs-X Tasks Subs Path
 ------------ ---------- - ------- - ----- ---- ----------
         root        0,6 y       0 y   343    0 /
controls@c1lsc:~ 0$ sudo cset set -c 0 -s system --cpu_exclusive
cset: --> created cpuset "system"
controls@c1lsc:~ 0$ sudo cset set
cset:
         Name       CPUs-X    MEMs-X Tasks Subs Path
 ------------ ---------- - ------- - ----- ---- ----------
         root        0,6 y       0 y   342    1 /
       system          0 y       0 n     0    0 /system
controls@c1lsc:~ 0$ sudo cset proc --move -f root -t system -k
cset: moving all tasks from root to /system
cset: moving 292 userspace tasks to /system
cset: moving 0 kernel threads to: /system
cset: --> not moving 50 threads (not unbound, use --force)
[==================================================]%
cset: done
controls@c1lsc:~ 0$ sudo cset set
cset:
         Name       CPUs-X    MEMs-X Tasks Subs Path
 ------------ ---------- - ------- - ----- ---- ----------
         root        0,6 y       0 y    50    1 /
       system          0 y       0 n   292    0 /system
controls@c1lsc:~ 0$ sudo cset proc --move -f root -t system -k --force
cset: moving all tasks from root to /system
cset: moving 50 kernel threads to: /system
[==================================================]%
cset: **> 29 tasks are not movable, impossible to move
cset: done
controls@c1lsc:~ 0$ sudo cset set
cset:
         Name       CPUs-X    MEMs-X Tasks Subs Path
 ------------ ---------- - ------- - ----- ---- ----------
         root        0,6 y       0 y    29    1 /
       system          0 y       0 n   313    0 /system
controls@c1lsc:~ 0$

I then created a set for the RTS process ("rts-c1dnn") on CPU6, and executed the c1dnn model in that set:

controls@c1lsc:~ 0$ sudo cset set -c 6 -s rts-c1dnn --cpu_exclusive
cset: --> created cpuset "rts-c1dnn"
controls@c1lsc:~ 0$ sudo cset set
cset:
         Name       CPUs-X    MEMs-X Tasks Subs Path
 ------------ ---------- - ------- - ----- ---- ----------
         root        0,6 y       0 y    24    2 /
    rts-c1dnn          6 y       0 n     0    0 /rts-c1dnn
       system          0 y       0 n   340    0 /system
controls@c1lsc:~ 0$ sudo cset proc -s rts-c1dnn --exec /opt/rtcds/caltech/c1/target/c1dnn/bin/c1dnn -- -m c1dnn
cset: --> last message, executed args into cpuset "/rts-c1dnn", new pid is: 27572
sysname = c1dnn
....

When done I just hit Ctrl-C.

I left the cpusets as they are, with all system processes in the "system" set.  This should not pose any problems since it's the identical configuration as would be if a normal kernel-level model was running in CPU6.

The c1dnn process and it's EPICS sequencer were shutdown after this test.

ELOG V3.1.3-