[Debian Wiki] Update of "BOINC/ServerGuide/AutoDockApp" by dhananjay
Debian Wiki
debian-www at lists.debian.org
Sun Aug 14 07:42:05 UTC 2011
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Debian Wiki" for change notification.
The "BOINC/ServerGuide/AutoDockApp" page has been changed by dhananjay:
http://wiki.debian.org/BOINC/ServerGuide/AutoDockApp?action=diff&rev1=5&rev2=6
Comment:
Elaborated the process.
'' THESE PAGES ARE STILL UNDER DEVELOPMENT ''
+ '''The scripts and files mentioned in this document can be obtained from [[DebianMed]] BOINC scripts git repository, available [[http://anonscm.debian.org/gitweb/?p=pkg-boinc/scripts.git|here]]''' ( server-examples subsection)
+
= BOINCed variant of the tutorial for running AutoDock on Debian and Ubuntu =
- We took a nice publicly available walk through for AutoDock and the MGLTools and modified that. There original is [[http://users.ox.ac.uk/~jesu1458/docker/|here]] which you may decide to walk through to increase your familiarity with the technology (biochemist or not). As you will see, that script was split in three sections. The first and last are left untouched. The middle one, i.e. the invocation of the tool, this is now modified for BOINC.
+ We took a nice publicly available walk through for AutoDock and the MGLTools and modified that. There original is [[http://users.ox.ac.uk/~jesu1458/docker/|here]] which you may decide to walk through to increase your familiarity with the technology (biochemist or not). As you will see, that script was split in three sections. The first and last are left almost untouched. The middle one, i.e. the invocation of the tool, this is now modified for BOINC.
- The scripts and files mentioned in this document can be obtained from DebianMed BOINC scripts git repo, avalable [[http://anonscm.debian.org/gitweb/?p=pkg-boinc/scripts.git|here]]
- == Step 1: Prepare the Server ==
+ The idea is to provide help for a smaller group of researchers to obtain computing resources from the masses. Also it will help to bridge some amount of gap that exists between general public and the research group.
- Follow the [[BOINC/ServerGuide]] description to create a working BOINC server web site. Edit the '''$installroot/$fileprojectname/html/project/project.inc''' to suite your requirements. Also consider addidng some content at '''$installroot/$fileprojectname/html/user/index.php''' about your project.
+ == BOINC Application ==
+ The BOINCed application is designed by splitting the docker into three parts,
+ 1. Pre Docking preparation.
+ 1. Docking of each ligand.
+ 1. Post Docking Summarization
- == Step 2: Prepare the application ==
- * Obtain the [[BOINC/ServerGuide/WrapperApp]] for each platform and sign them using your project keys. ( Look at [[BOINC/ServerGuide/AppDeployment#Use_the_Debian-provided_script_to_install_binaries_for_multiple_platforms]] for help )
- * Do any of the following to get the autodock application as an executing tool
- * Use Debian's packaged dynamic binaries of autodock
- This is problematic when the Linux distributions of contributors are too different and too diverse. This hence won't make any sense to a large volunteer run project. But it is fine for a BOINC project only running in the virtualised images on Macs, Windows ... and Linux.
- * Build statically linked binaries for each platform.
- {{{
- $ apt-get source autodocktools
- $ cd autodocktools-*/autodock
- $ ./configure LDFLAGS=-static && make
- }}}
- * Obtain obtain pre-built i686, x86_64 ones from the repository. (Look at http://anonscm.debian.org/gitweb/?p=pkg-boinc/scripts.git;a=tree;f=server-examples/autodock/boinc-app/static_binaries;hb=HEAD)
+ Corresponding scripts to each steps is in the git repository. Here, the Pre and Post steps need access to the full data, so computed in server. Whereas the docking part consists of isolated slots for each ligand. This, obviously is a perfect candidate to compute distributed.
+ The BOINC app by design should contain following files,
+ 1. Receptor Map files
+ 2. Ligand molecule description
+ 3. Docking parameter file.
+ 4. Autodock binary
+ 5. BOINC wrapper and job description file.
+
+ The application is designed using the [[BOINC/ServerGuide/WrapperApp]] framework. The static binary of autodock is used in the app.
+
+ Here, we present the steps to create such an application with ''boinc-server-maker'' and ''boinc-app-examples'' packages provided by debian. The steps illustrated should with some minor changes (Though difficult and lengthy) should work for a generic BOINC system.
+
+ The illustration assumes that reader have a working BOINC project in hand, the process required to make one is described at [[BOINC/ServerGuide]].
+
- == Step 3: Prepare the biology ==
+ === Prepare the Biology ===
This step was not changed with respect to the original ''docker'' script. A few file locations have been modified. This just slightly adapted version of its "upper half" is kept [[http://anonscm.debian.org/gitweb/?p=pkg-boinc/scripts.git;a=blob;f=server-examples/autodock/dock-pre.sh;hb=HEAD|here]] ([[http://anonscm.debian.org/gitweb/?p=pkg-boinc/scripts.git;a=blob;f=server-examples/autodock/dock-pre.sh;hb=HEAD|download]]).
* install Debian packages: autodocktools autodock autogrid
* follow the ''docker'' [[http://users.ox.ac.uk/~jesu1458/docker/|web tutorial]] for
- * download receptor from http://www.rcsb.org/pdb/download/downloadFile.do?fileFormat=pdb&compression=NO&structureId=2RH1
+ * download receptor from [[http://www.rcsb.org/pdb/download/downloadFile.do?fileFormat=pdb&compression=NO&structureId=2RH1|PDB]]
* reduce to first 5 transmembrane helices
* prepare docking with AutoDockTools (start runAdt instead of adt)
* add hydrogens
@@ -47, +54 @@
* split .mol2 ligand collection into many files
* name each correctly
- == Step 4: Application Deployment Continued ==
+ ==== dock-pre.sh ====
- === Checklist ===
+ The dock-pre expects all files needed in the working directory.
- If you have followed the biology part correctly, you should have following files at your disposal
{{{
+ user at host: $ tree
.
- |-- dock-pre.sh
+ |-- dock-pre.sh
+ |-- ligands.mol2
|-- receptor.gpf
- |-- receptor.pdbqt
+ `-- receptor.pdbqt
- `-- ligands.mol2
0 directories, 4 files
}}}
- === Prepare the maps and ligands ===
-
- Run dock-pre.sh on the directory, to create grid maps of receptor and prepare ligand. The directory after will look like this.
{{{
+ cat << EOFPRE >> dock-pre.sh
+ #!/bin/bash
+
+ set -e
+ origDir=$(pwd)
+ # Specify ligand set file name.
+
+ RECEPTOR=receptor
+
+ if [ ! -r $RECEPTOR.pdbqt -o ! -r $RECEPTOR.gpf ]; then
+ echo Receptor files could not be found!!!
+ exit -1
+ fi
+
+ echo Preparing the receptor...
+
+ echo autogrid4 -p $RECEPTOR.gpf -l $RECEPTOR.glg
+ autogrid4 -p $RECEPTOR.gpf -l $RECEPTOR.glg
+
+ echo "Done."
+
+
+ if [ -z "$LIGANDS" ]; then
+ LIGANDS=ligands
+ fi
+
+ if [ ! -r "$LIGANDS.mol2" ]; then
+ echo "Expected the file '$LIGANDS.mol2' to be located in current directory. This was not found and I don't know what to spilt."
+ exit -1
+ fi
+
+ mkdir $LIGANDS
+
+ echo "Preparing atomic coordinates of the ligands..."
+ cd $LIGANDS
+
+ csplit -ftmp -n4 -ks - '%^@.TRIPOS.MOLECULE%' '/^@.TRIPOS.MOLECULE/' '{*}' < $origDir/ligands.mol2
+
+ for f in *
+ do
+ entryName=$(grep ZINC $f)
+ if [ -z "$entryName" ]; then
+ continue
+ else
+ mv -v $f $entryName.mol2
+ fi
+ done
+
+ pythonPath=/usr/share/pyshared
+ # Prepare each ligand file for docking with AutoDock.
+ echo "Preparing ligands for docking..."
+ for f in *.mol2; do
+ echo $f
+ if [ -r $(echo $f|sed -e 's/.mol2$/.pdbqt/') ]; then continue; fi
+ python $pythonPath/AutoDockTools/Utilities24/prepare_ligand4.py -l $f
+ done
+
+
+ if [ ! -r "$RECEPTOR.pdbqt" ]; then
+ echo "Linking ../$RECEPTOR.pdbqt to local directory"
+ ln -f ../$RECEPTOR.pdbqt .
+ ln -f ../$RECEPTOR.maps.* .
+ ln -f ../$RECEPTOR.*.map .
+ fi
+
+ echo Starting the preparation of docking parameter files.
+ for f in *.pdbqt; do
+ base=$(basename $f .pdbqt)
+
+ if [ "$base" = "$RECEPTOR" ]; then
+ echo Test
+ continue
+ fi
+
+ echo "Preparing dpf for $base against $RECEPTOR"
+
+ python $pythonPath/AutoDockTools/Utilities24/prepare_dpf4.py -l "$base.pdbqt" -r "$RECEPTOR.pdbqt" \
+ -p ga_num_evals=1000000 \
+ -p ga_pop_size=100 \
+ -p ga_run=30
+
+ done
+
+
+ echo "Done."
+ echo "Find the ligand files in the $origDir/ligands subdirectory."
+ echo
+ echo "You can now create work units for BOINC."
+
+ exit 0
+ }}}
+
+ On running,
+ 1. Perfomrs intial grid calcuation using autogrid
+ 1. Splits the ligand files
+ 1. Generates the parameter files.
+
+ {{{
- user at host $ tree
+ user at host: $ tree
.
- |-- dock-pre.sh
+ |-- dock-pre.sh
|-- ligands
| |-- receptor.A.map
| |-- receptor.C.map
@@ -84, +186 @@
| |-- receptor.SA.map
| |-- ZINC00034176.mol2
| |-- ZINC00034176.pdbqt
+ | |-- and MORE...
- | |-- ZINC00034176_receptor.dpf
- | |-- ZINC00034177.mol2
- | |-- ZINC00034177.pdbqt
+ | `-- ZINC01818227.pdbqt
- | |-- ZINC00034177_receptor.dpf
- | `-- ZINC00557939.mol2 .... AND MORE...
- |
|-- ligands.mol2
|-- receptor.A.map
|-- receptor.C.map
@@ -104, +202 @@
|-- receptor.N.map
|-- receptor.OA.map
|-- receptor.pdbqt
- |-- receptor.SA.map
+ `-- receptor.SA.map
- }}}
+ }}}
+ === App Deployment ===
- === Craft the BOINC Appliv
- The docking needs the following files, most of them are ligand invariant, or they are same for every workunit prepared. The receptor maps, pdbqt molecule definitions for the receptor. The ligands vary per each workunit, we can include any number of ligands to be processed in one workunit, the scripts are designed to do that. But for the tutorial purposes we will restrict one ligand per workunit.
+ As the autodock binary is non native to BOINC, we use a WrapperApp to bridge them. The Wrapper is responsible for invocation, running, and reporting of the actual task.Deployment of a generic wrapper is described at BOINC/ServerGuide/WrapperApp.
- So each workunit consists following files.
- {{{
- 1. dock-client.sh
- 2. job.xml
- 3. autodock_bianary
- 4. $RECEPTOR.pdbqt
- 5. $RECEPTOR.*.* (Map files)
- 6. $RECEPTOR.dlg
- 7. $RECEPTOR.dpf
- 8. $LIGAND.pdbqt
- 9. $LIGAND_$RECEPTOR.dpf
- }}}
- 1-7 are workunit invariant, i.e same for every workunit. Or in BOINC sense, they are part of the app. They should staged in the app directory for update_scripts to parse, forming
-
+ ==== Autodock ====
+ Static binary of the autodock is used.
+ * The pre-built binaries for i686 and amd64 can be found [[http://anonscm.debian.org/gitweb/?p=pkg-boinc/scripts.git;a=tree;f=server-examples/autodock/boinc-app/static_binaries;hb=HEAD|here]]
+ * Or you can build one easily.
{{{
+ apt-get source autodocktools
+ cd autodocktools-*/autodock
+ ./configure LDFLAGS=-static && make
-
- `-- wrapper
- `-- wrapper_7.03_i686-pc-linux-gnu
- |-- autodock4
- |-- job.xml
- |-- receptor.A.map
- |-- receptor.C.map
- |-- receptor.d.map
- |-- receptor.e.map
- |-- receptor.glg
- |-- receptor.gpf
- |-- receptor.HD.map
- |-- receptor.maps.fld
- |-- receptor.maps.xyz
- |-- receptor.NA.map
- |-- receptor.N.map
- |-- receptor.OA.map
- |-- receptor.SA.map
- `-- wrapper_7.03_i686-pc-linux-gnu
}}}
- and the job description file for wrapper will be
+ ==== Wrapper ====
+ The job description file is used to pass information on the work to the wrapper, it contains details about the binary to call, the arguments to apps, the outputs etc etc.
+
+ AutoDockApp preforms docking, the corresponding command from docker.sh which the wrapper should be calling
{{{
- user at host $ cat <<EOF >> job.xml
+ autodock4 -p ligand_receptor.dpf -l result.dlg
+ }}}
+
+ Job description file is named job.xml, and it should be present in the app directory for wrapper to find and parse.
+
+ {{{
+ cat <<EOFJOB > job.xml
<job_desc>
<task>
- <application>dock-client.sh</application>
+ <application>autodock</application>
- <append_cmdline_args/>
+ <append_cmdline_args/>
- <stdout_filename>wu_log</stdout_filename>
</task>
</job_desc>
+ EOFJOB
+ }}}
+
+ Summerizing, AutoDockApp needs following files, Stage and sighn them them inside the $projectroot/apps directory.
+ {{{
+ apps
+ |
+ `-- wrapper
+ `-- wrapper_6.12_i686-pc-linux-gnu
+ |-- job.xml
+ |-- autodock
+ `-- wrapper_6.12_i686-pc-linux-gnu
+
+ and input files,
+ |-- receptor.A.map
+ |-- receptor.C.map
+ |-- receptor.d.map
+ |-- receptor.e.map
+ |-- receptor.HD.map
+ |-- receptor.maps.fld
+ |-- receptor.maps.xyz
+ |-- receptor.NA.map
+ |-- receptor.N.map
+ |-- receptor.OA.map
+ |-- receptor.SA.map
+ |-- ZINC00034176.pdbqt
+ `-- ZINC00034176_receptor.dpf
+
+ }}}
+ {{{
+ # Sign the bianries.
+
+ for f in apps/wrapper/*/; do
+ name = $(basename $f)
+ /path/to/sigh_executable $f /path/to/code_sign_private >> $name.sig
+ done
+ }}}
+
+ Add app defintion to project.xml, update the databse with new application included, add start the project.
+ {{{
+ bin/xadd && bin/update_versions
+ bin/start
+ }}}
+
+ === Work Generation ===
+
+ The BOINC app should be fed with the work to perfom, or in BOINC terminolgy a workunit. The workunit consists of input files and work and result defnitions. Thease defnitions are specified in a xml document called template. Theres seperate template for workunit or the input and result or output. Eventhoug the design of docking permits multiple ligands in a single workunit, Here we restrict each workunit to contain only one ligand to reduce complexity and also considering low end clients.
+
+ ==== Workunit Templates ====
+ The workunit has to define the input files, which is specified in workunit template,
+ {{{
+ cat <<EOFTEMP >> wu_template.xml
+
+ <file_info>
+ <number>0</number>
+ <sticky/>
+ <nodelete/>
+ </file_info>
+ <file_info>
+ <number>1</number>
+ <sticky/>
+ <nodelete/>
+ </file_info>
+ <file_info>
+ <number>2</number>
+ <sticky/>
+ <nodelete/>
+ </file_info>
+ <file_info>
+ <number>3</number>
+ <sticky/>
+ <nodelete/>
+ </file_info>
+ <file_info>
+ <number>4</number>
+ <sticky/>
+ <nodelete/>
+ </file_info>
+ <file_info>
+ <number>5</number>
+ <sticky/>
+ <nodelete/>
+ </file_info>
+ <file_info>
+ <number>6</number>
+ <sticky/>
+ <nodelete/>
+ </file_info>
+ <file_info>
+ <number>7</number>
+ <sticky/>
+ <nodelete/>
+ </file_info>
+ <file_info>
+ <number>8</number>
+ <sticky/>
+ <nodelete/>
+ </file_info>
+ <file_info>
+ <number>9</number>
+ <sticky/>
+ <nodelete/>
+ </file_info>
+ <file_info>
+ <number>10</number>
+ <sticky/>
+ <nodelete/>
+ </file_info>
+ <file_info>
+ <number>11</number>
+ <sticky/>
+ <nodelete/>
+ </file_info>
+ <file_info>
+ <number>12</number>
+ </file_info>
+ <file_info>
+ <number>13</number>
+ </file_info>
+
+ <workunit>
+
+ <file_ref>
+ <file_number>0</file_number>
+ <open_name>receptor.A.map</open_name>
+ <copy_file/>
+ </file_ref>
+ <file_ref>
+ <file_number>1</file_number>
+ <open_name>receptor.C.map</open_name>
+ <copy_file/>
+ </file_ref>
+ <file_ref>
+ <file_number>2</file_number>
+ <open_name>receptor.d.map</open_name>
+ <copy_file/>
+ </file_ref>
+ <file_ref>
+ <file_number>3</file_number>
+ <open_name>receptor.e.map</open_name>
+ <copy_file/>
+ </file_ref>
+ <file_ref>
+ <file_number>4</file_number>
+ <open_name>receptor.HD.map</open_name>
+ <copy_file/>
+ </file_ref>
+ <file_ref>
+ <file_number>5</file_number>
+ <open_name>receptor.maps.fld</open_name>
+ <copy_file/>
+ </file_ref>
+ <file_ref>
+ <file_number>6</file_number>
+ <open_name>receptor.maps.xyz</open_name>
+ <copy_file/>
+ </file_ref>
+ <file_ref>
+ <file_number>7</file_number>
+ <open_name>receptor.NA.map</open_name>
+ <copy_file/>
+ </file_ref>
+ <file_ref>
+ <file_number>8</file_number>
+ <open_name>receptor.N.map</open_name>
+ <copy_file/>
+ </file_ref>
+ <file_ref>
+ <file_number>9</file_number>
+ <open_name>receptor.OA.map</open_name>
+ <copy_file/>
+ </file_ref>
+ <file_ref>
+ <file_number>10</file_number>
+ <open_name>receptor.pdbqt</open_name>
+ <copy_file/>
+ </file_ref>
+ <file_ref>
+ <file_number>11</file_number>
+ <open_name>receptor.SA.map</open_name>
+ <copy_file/>
+ </file_ref>
+ <file_ref>
+ <file_number>12</file_number>
+ <open_name>ligand.pdbqt</open_name>
+ <copy_file/>
+ </file_ref>
+ <file_ref>
+ <file_number>13</file_number>
+ <open_name>ligand_receptor.dlg</open_name>
+ <copy_file/>
+ </file_ref>
+
+ <command_line>-p ligand_receptor.dpf -l ligand.dlg</command_line>
+ <rsc_fpops_bound>1.5e11</rsc_fpops_bound>
+ <rsc_fpops_est>1.5e11</rsc_fpops_est>
+
+ </workunit>
+ EOFTEMP
+ }}}
+
+ ''<sticky/>'' attribute specifies that the files are same amoung all workunits, so some bandwith can be saved.
+
+ Resut is obtianed as a dlg file, which should be transported back.
+
+ {{{
+ cat <<EOFTEMP >> reusult_template.xml
+ <file_info>
+ <name><OUTFILE_0/></name>
+ <generated_locally/>
+ <upload_when_present/>
+ <max_nbytes>5000000</max_nbytes>
+ <url><UPLOAD_URL/></url>
+ </file_info>
+
+ <result>
+ <file_ref>
+ <file_name><OUTFILE_0/></file_name>
+ <open_name>ligand.dlg</open_name>
+ <copy_file/>
+ </file_ref>
+ </result>
+ EOFTEMP
+ }}}
+
+ ==== dock-genwu.sh ====
+
+ The scripts used to genrate work, assumed to run inside the same direcotry as ''dock-pre.sh'', BOINC project variables should be edited to suite your setup.
+
+ {{{
+ cat <<EOFGEN >> dock-genwu.sh
+ #!/bin/bash
+ set -e
+
+ origDir=$(pwd)
+
+ # Define names.
+
+ RECEPTOR=receptor
+ if [ -z "$LIGANDS" ]; then
+ LIGANDS=ligands
+ fi
+
+ generated=Generated
+
+
+
+ # BOINC project details.
+ projectRoot=/tmp
+ app=wrapper
+ temp_wu=templates/wrapper_wu.xml
+ temp_res=template/wrapper_res.xml
+
+ downStage=$projectRoot/downloads
+
+ # USAGE
+
+ if [ -z $1 ]; then
+ cat <<EOF
+ Usage: dock-genwu.sh number-of-wu-needed
+ EOF
+ exit -1
+ fi
+
+ cd $LIGANDS
+ mkdir -p $generated
+ # Add receptor files to the staging area.
+ if [ ! -d $downStage ]; then
+ mkdir $downStage
+ fi
+ cp $RECEPTOR.pdbqt $RECEPTOR.maps.* $RECEPTOR.*.map $downStage
+
+
+ # Copy files for each ligand and generate corresponfing workunit.
+ count=0
+
+ for pf in *.dpf; do
+ if [[ $count -lt $1 ]]; then
+
+ base=${pf%%_*}
+
+ cp $base.pdbqt $pf $downStage
+
+ cd $projectRoot
+ ./bin/create_work --appname $app --wu_name $base --wu_template $temp_wu \
+ --result_template $temp_res $RECEPTOR.A.map $RECEPTOR.C.map $RECEPTOR.d.map $RECEPTOR.e.map $RECEPTOR.maps.fld \
+ $RECEPTOR.maps.xyz $RECEPTOR.NA.map $RECEPTOR.N.map $RECEPTOR.OA.map $RECEPTOR.pdbqt $RECEPTOR.SA.map \
+ $base.pdbqt ${base}_${RECEPTOR}.dpf
+ cd $orgDir
+ mv $base* $generated
+ rm $downStage/$base*
+ let count=$count+1
+
+ else
+ echo Specified number of workunits generated!!
+ fi
+ done
+
+ echo "Creation of work units completed."
+ exit 0
- }}}
+ }}}
+
+ Call it in
+ {{{
+ user at host : $ ./dock-genwu.sh <no-of-wu>
+ }}}
- ''<append_cmdline_args/>'' appends any command line arguments specified in the workunit template to be appended to the actual program. Please see the source of the dock client for details.
-
- Go ahead and application to boinc database.
- * Edit '''project.xml''' and add the wrapper application
- {{{
- <app>
- <name>wrapper</name>
- <user_friendly_name>Autodock test</user_friendly_name>
- </app>
- }}}
- * Run ''bin/xadd''
- * Sign the files.
- * Finally call '''bin/update_versions'''
-
-
- On the other hand, the files 8-9 vary for each workunit, or in BOINC terminology, they are inputs. The will be dealt with in next section.
-
- The middle part of the ''docker'' script is to be executed on the server. It resides [[http://anonscm.debian.org/gitweb/?p=pkg-boinc/scripts.git;a=blob;f=server-examples/autodock/dock-client.sh;hb=HEAD|here]] ([[http://anonscm.debian.org/gitweb/?p=pkg-boinc/scripts.git;a=blob_plain;f=server-examples/autodock/dock-client.sh;hb=HEAD|download]]) .
-
- === Step : Submit jobs to BOINC server ===
-
- This step involves the specification of work units. This is not completely trivial, since one job involves man files that need to be available at compute time. Some are invariant to the ligand, e.g. the receptor and the grid representing its force field, some are varying with every ligand, here especially the ligand itself.
-
- * script to submit work units
- * script to be executed to start autodock
More information about the pkg-boinc-commits
mailing list