Running Athena Full Chain on LCG
====================================
Athena full chain (generation, simulation, digitization, reconstruction and AOD production) step-by-step on LCG.

"Once you have your Digital Certificate (probably inside your Web Browser) you need to export it. How you do this will depend on your Web Browser but will be somewhere under preferences -> security -> certificates. You should end up with something like mycert.pfx. In order to use the Grid you need access to a local User Interface (UI) machine that has the LCG Middleware installed. Copy your certificate to this UI machine and under your home directory on the UI create a .globus directory."

0) To use lxplus as a UI:    (setup on lxcalc?)

> source /afs/cern.ch/project/gd/LCG-share/sl3/etc/profile.d/grid_env.sh  [or .csh]


1) PRepare the Certificate with the appropriate permission
> openssl pkcs12 -in mycert.pfx -clcerts -nokeys -out usercert.pem
(insert the master password)
> openssl pkcs12 -in mycert.pfx -nocerts -out userkey.pem
(master password, twice)

> chmod 444 $HOME/.globus/usercert.pem
> chmod 400 $HOME/.globus/userkey.pem


2) Generate your grid proxy:

> grid-proxy-init    (you have to give the GRID password phrase, that is the master password)


3) For each step, you need a shell script for your executable and a jdl file.

Generation
===========
Shell script (let's say, athena_gen.sh) :

#!/bin/bash
export ATLAS_RELEASE=11.0.4
source $VO_ATLAS_SW_DIR/software/$ATLAS_RELEASE/setup.sh
source $SITEROOT/dist/$ATLAS_RELEASE/Control/AthenaRunTime/*/cmt/setup.sh
cp $SITEROOT/dist/$ATLAS_RELEASE/InstallArea/share/PDGTABLE.MeV .
# Run the job:
athena.py jobOptions.pythia.py
# Save the generated events
lcg-cr -v --vo atlas -d destinationname -l lfn://user_pythia.pool.root file://`pwd`/pythia.pool.root

and the jdl (athena_gen.jdl) looks like:

############# Athena #################
Executable = "athena_gen.sh";
StdOutput = "athena_gen.out";
StdError = "athena_gen.err";
InputSandbox = {"athena_gen.sh","jobOptions.pythia.py"};
OutputSandbox = {"athena_gen.out", "athena_gen.err","CLIDDBout.txt"};
Requirements = Member("VO-atlas-release-$ATLAS_RELEASE",
 other.GlueHostApplicationSoftwareRunTimeEnvironment);
################################################

The jobOptions.pythia.py is the Pythia jobOption file.
In the lcg-cr part of the script, you can specify whichever destination SE you like
using the -d option. You can get a list of them with

 > lcg-infosites --vo atlas se


4) Now submit to the grid:

> edg-job-submit --vo atlas -o jobIDfile athena_gen.jdl


5) Watch your job's status using

> edg-job-status -i jobIDfile


6) And when it's finished download the output sandbox:

> edg-job-get-output -dir . -i jobIDfile


7) If all went well, you will have the stdout and stderr in a directory corresponding to your job ID,
but the root file ("pythia.pool.root") will be stored in whichever SE you specified in the script.
You can then use it as the input for the next stage.


8) So, for
Simulation
==========
your shell script, let's call it athena_sim.sh, might look like:

#!/bin/bash
export ATLAS_RELEASE=11.0.4
source $VO_ATLAS_SW_DIR/software/$ATLAS_RELEASE/setup.sh
source $SITEROOT/dist/$ATLAS_RELEASE/Control/AthenaRunTime/*/cmt/setup.sh
cp $SITEROOT/dist/$ATLAS_RELEASE/InstallArea/share/PDGTABLE.MeV .
cp $SITEROOT/dist/$ATLAS_RELEASE/InstallArea/share/bmagatlas03_test2.data .
mv bmagatlas03_test2.data fieldmap.dat
# Get the input
lcg-cp -v --vo atlas lfn://user_pythia.pool.root file://`pwd`/pythia.pool.root
# Insert the file into the FileCatalog
pool_insertFileToCatalog pythia.pool.root
# Run the job:
athena.py RomeSimulationConfig.py RomeGeo2G4.py > athena_sim.out
# Save the output events
lcg-cr -v --vo atlas -d destinationname -l lfn://user_g4hits.pool.root file://`pwd`/g4hits.pool.root

and your jdl will be:

############# Athena #################
Executable = "athena_sim.sh";
StdOutput = "athena_sim.out";
StdError = "athena_sim.err";
InputSandbox = {"athena_sim.sh","RomeSimulationConfig.py","RomeGeo2G4.py"};
OutputSandbox = {"athena_sim.out", "athena_sim.err", "CLIDDBout.txt"};
Requirements = Member("VO-atlas-release-$ATLAS_RELEASE",
 other.GlueHostApplicationSoftwareRunTimeEnvironment);
################################################

Repeat steps 3-7 for all the following stages of the chain - just make sure the names of the input and output files match up.
After the AOD is produced, you can copy it to your own machine (unless you shipped it there in the output sandbox) with

> lcg-cp --vo atlas lfn://user_aod.pool.root file://`pwd`/aod.pool.root

and analyze it as normal.

To find an lfn on the grid, use lcg-lr:

> lcg-lr --vo atlas lfn://user_esd.pool.root
srm://destinationname/dpm/name/home/atlas/generated/date/fileea4ae252-3865-4d62-8797-5af8471a285c

For reference, here are all the scripts, jdls and job options:
..........


References:

https://uimon.cern.ch/twiki/bin/view/Atlas/WorkBookStartingGrid