How to remove endling pending jobs
==================================
Sometimes it happens that the web page of the slow control 
show jobs which are running since many days,
for example (Oct 28):

Run	Filename	Started	Farm
32526 	032526N_ALL_f06_1_1_1_20.000_0 	Oct 25 21:13 	fibm39
32526 	032526N_ALL_f07_1_1_1_20.001_2 	Oct 25 21:13 	fibm39
32526 	032526N_ALL_f07_1_1_1_20.000_1 	Oct 25 21:13 	fibm39

Ths is a warning situation, since usually it takes 2 hours to complete a
job.

Often this is a "fake" message, in the
sense that the run has been completely reconstructed, 
even if it seems to be in reconstruction.
This is due to the fact that a file  
which tells the script of job submission that the cpu for that maichine 
is availble, is not removed.
In fact each time a job start on fibmXX, a file
run_nrN_ALL_f0x_1_1_1_dbv.00X_X
is created in the area
/runcond/datarec/jobs/status/cpu/fibmXX/


For example, in our case the files:

0 -rw-r--r--   1 kloerec  kloe           0 Oct 25 21:13 032526N_ALL_f06_1_1_1_20.000_0
0 -rw-r--r--   1 kloerec  kloe           0 Oct 25 21:13 032526N_ALL_f07_1_1_1_20.000_1
0 -rw-r--r--   1 kloerec  kloe           0 Oct 25 21:13 032526N_ALL_f07_1_1_1_20.001_2

are created in the area:
/runcond/datarec/jobs/status/cpu/fibm39/

ACTIONS to take:
===============

1) Look if the jobs pending have been completely reconstructed.
This can be done, for example by typing the command:
"list_runs run_nr" and look if the dbv number is displayed under the "ana"
column.
Another way is type the command "list_datarecs run_nr" and look if the
files that appear as pending
have been  really recontructed. For example, in our case,  
do:
 list_datarecs 32526 | grep -i f06 | grep 000
 32526      20 f43 160  kpm032526N_ALL_f06_1_1_1_20.000 -  293363712      16394 17.9kb    Deleted       Archived
 32526      20 f43 160  ksl032526N_ALL_f06_1_1_1_20.000 -  347865088      20440 17.0kb    Deleted       Archived
 32526      20 f43 160  rpi032526N_ALL_f06_1_1_1_20.000 -   79857664       5496 14.5kb    Deleted       Archived
 32526      20 f43 160  rad032526N_ALL_f06_1_1_1_20.000 -  205041664      17925 11.4kb    Deleted       Archived
 32526      20 f43 160  clb032526N_ALL_f06_1_1_1_20.000 -   22284288       1734 12.9kb    Deleted       Archived
 32526      20 f43 160  bha032526N_ALL_f06_1_1_1_20.000 -  384864256      31170 12.3kb    Deleted       Archived
 32526      20 f43 160  lsb032526N_ALL_f06_1_1_1_20.000 -   12042240       3781  3.2kb    Deleted       Archived

This show that all the stream have been done for the file
032526N_ALL_f06_1_1_1_20.000 .

Do the same for the other two files:
 list_datarecs 32526 | grep -i f07 | grep 000
 list_datarecs 32526 | grep -i f07 | grep 001


2)If the file has been completely reconstructed login on fibm35 as kloerec
  and remove the file run_nrN_ALL_f0x_1_1_1_dbv.00X_X in the area
  /runcond/datarec/jobs/status/cpu/fibmXX/

In our case go to /runcond/datarec/jobs/status/cpu/fibm39

do:
 rm  032526N_ALL_f06_1_1_1_20.000_0
 rm  032526N_ALL_f07_1_1_1_20.000_1
 rm  032526N_ALL_f07_1_1_1_20.001_2


this can be done without stopping datarec.

Note that in few time the jobs shouldn't appear anymore to the slow control
web page of data processing .


(to be continued...)