[Neurodebian-devel] Could you enable fls-5.0-gpu package by default?
Michal Hocko
mstsxfx at gmail.com
Sun May 29 11:21:56 UTC 2016
On Sun, May 29, 2016 at 11:12:49AM +0200, Michael Hanke wrote:
> Hi,
>
> > I am trying to use CUDA features from the fls-5.0 source package.
> > fls-5.0-gpu is not built by default, though. I have tried to make it
> > compile and it doesn't look terribly hard to do it. I've ended up with
> > the (attached) two patches.
>
> There was no attachment -- please resend.
Sorry about that. See attached.
> > The first one: add_gpu_tools_pkg is "stolen" from the current debian
> > source package (5.0.7-4) and refreshed on top of 5.0.9-2.
> >
> > The second one: gpu-tools-fixes combines two compile fixes
> >
> > With the two patches and nvidia-kernel-dkms installed (Btw. shouldn't
> > this be recommended dependency?) I was able to run bedpostx successfully
> > on
> > 13:00.0 3D controller: NVIDIA Corporation GK110GL [Tesla K20m] (rev a1)
> >
> > Would it be possible to add the two patches and enable the -gpu package
> > automatically?
>
> Thanks a lot for composing patches! I will take a look at them as soon as I
> get my hands on them.
>
> As stated before, I have no hardware to test this one, but if these
> changes do not impair the rest of the package, I'd be happy to include
> them.
It should be pretty much self contained.
> For the sake of clarity:
>
> - you added nvidia-kernel-dkms as a _runtime_ dependency of the -gpu
> package only?
No I haven't altered the package but without the kernel module none of the gpu
enable programs started properly with:
modprobe: ERROR: ../libkmod/libkmod-module.c:816 kmod_module_insert_module() could not find module by name='nvidia_current'
modprobe: ERROR: could not insert 'nvidia_current': Function not implemented
cuda error at CUDA/init_gpu.cu:77. no CUDA-capable device is detected
ERROR: init_gpu: no CUDA-capable device is detected
So I think that this should be made a runtime dependency. I just wasn't
sure how to do this properly as somebody might be using the out of tree
driver...
> - you did not need to add additional build dependencies beyond the old
> patch?
yes, add_gpu_tools_pkg adds nvidia-cuda-toolkit as a build dependency
and nothing more was really needed. At least not in my build
environment.
> - 'gpu-tools-fixes' is something that should be submitted upstream as
> general fixes.
I guess so. nifti and newmat headers have to be specified in the build
env.
Please also note 5.0.7 needs one more fix to address sqrt nvcc linking
issue. This seems to be fixed in 5.0.9 version. I have attached this one
as well (workaround-nvidia-linkage-issue).
> If this is the case, I so no issues so far (not having seen the actual
> patches).
>
> Quick question upfront: Would you be willing and able to keep an eye on
> the GPU tools in FSL for the updates to come? Without being able to
> test, I am limited in my ability to fix things, should issues arise (and
> they usually do ;-).
Well, my free time is a really scarce resource (as for everybody I
guess). This is primarily motivated by a side project I am helping
with. I am not the direct user of the functionality. I guess the end
users who I am helping currently will be willing to help with testing
(CCed Antonin).
Thanks!
--
Michal Hocko
-------------- next part --------------
Index: fsl-5.0.9/debian/rules
===================================================================
--- fsl-5.0.9.orig/debian/rules
+++ fsl-5.0.9/debian/rules
@@ -52,8 +52,8 @@ FSL_LIBRARY_PATH=$(CURDIR)/lib
# install into this directory
DEB_DESTDIR=$(CURDIR)/debian/tmp
-# don't compile GPU
-COMPILE_GPU:=0
+# compile GPU only if building on x86 platform where nvcc present
+COMPILE_GPU:=$(shell which nvcc >/dev/null 2>&1 && echo 1 || echo 0)
# define common environment setup for all make calls
# complicated LD_LIBRARY expression ensures that no
Index: fsl-5.0.9/debian/templates/control
===================================================================
--- fsl-5.0.9.orig/debian/templates/control
+++ fsl-5.0.9/debian/templates/control
@@ -18,6 +18,7 @@ Build-Depends: debhelper (>= 7),
libnifti-dev,
libgiftiio-dev,
mesa-common-dev,
+ nvidia-cuda-toolkit [amd64 i386],
libexpat1-dev | libexpat-dev,
libglu1-mesa-dev | libglu-dev,
libgomp1
@@ -92,7 +93,7 @@ Depends: mozilla-firefox | www-browser,
Recommends: fsl-atlases,
fsl-#FSLMVERSION#-eddy-nonfree,
Suggests: fsl-feeds,
- fsl-#FSLMVERSION#-gpu,
+ fsl-#FSLMVERSION#-gpu (= ${binary:Version}),
octave | ${octave:Depends},
mriconvert | dicomnifti,
fsl-complete,
@@ -122,3 +123,19 @@ Description: analysis tools for FMRI, MR
FSL interoperates well with other brain imaging related software. This includes
Caret, FreeSurfer (cortical flattening and modelling). All FSL tools support
the NIfTI format.
+
+Package: fsl-#FSLMVERSION#-gpu
+Architecture: amd64 i386
+Depends: fsl-#FSLMVERSION#-core (= ${binary:Version}),
+ ${shlibs:Depends},
+ ${misc:Depends}
+Description: GPU-accelerated tools for FSL
+ FSL is a comprehensive library of image analysis and statistical tools
+ for fMRI, MRI and DTI brain imaging data. The suite consists of various
+ command line tools, as well as simple GUIs for its core analysis pipelines.
+ Among others, FSL offers implementations of standard GLM analysis,
+ white matter tractography, tissue segmentation, affine and non-linear
+ co-registration, and independent component analysis.
+ .
+ This package provides accelerated versions of some FSL commands to utilize
+ NVIDIA's CUDA toolkit.
Index: fsl-5.0.9/debian/control
===================================================================
--- fsl-5.0.9.orig/debian/control
+++ fsl-5.0.9/debian/control
@@ -18,6 +18,7 @@ Build-Depends: debhelper (>= 7),
libnifti-dev,
libgiftiio-dev,
mesa-common-dev,
+ nvidia-cuda-toolkit [amd64 i386],
libexpat1-dev | libexpat-dev,
libglu1-mesa-dev | libglu-dev,
libgomp1
@@ -92,7 +93,7 @@ Depends: mozilla-firefox | www-browser,
Recommends: fsl-atlases,
fsl-5.0-eddy-nonfree,
Suggests: fsl-feeds,
- fsl-5.0-gpu,
+ fsl-5.0-gpu (= ${binary:Version}),
octave | ${octave:Depends},
mriconvert | dicomnifti,
fsl-complete,
@@ -122,3 +123,19 @@ Description: analysis tools for FMRI, MR
FSL interoperates well with other brain imaging related software. This includes
Caret, FreeSurfer (cortical flattening and modelling). All FSL tools support
the NIfTI format.
+
+Package: fsl-5.0-gpu
+Architecture: amd64 i386
+Depends: fsl-5.0-core (= ${binary:Version}),
+ ${shlibs:Depends},
+ ${misc:Depends}
+Description: GPU-accelerated tools for FSL
+ FSL is a comprehensive library of image analysis and statistical tools
+ for fMRI, MRI and DTI brain imaging data. The suite consists of various
+ command line tools, as well as simple GUIs for its core analysis pipelines.
+ Among others, FSL offers implementations of standard GLM analysis,
+ white matter tractography, tissue segmentation, affine and non-linear
+ co-registration, and independent component analysis.
+ .
+ This package provides accelerated versions of some FSL commands to utilize
+ NVIDIA's CUDA toolkit.
-------------- next part --------------
Fix a missing include paths for newmath and nifti headers
they are not directly in /usr/include but rather sit in their specific directories.
LIBBEDPOSTX_CUDA will get those via USRINCFLAGS
Index: fsl-5.0.9/debian/externallibs.mk
===================================================================
--- fsl-5.0.9.orig/debian/externallibs.mk
+++ fsl-5.0.9/debian/externallibs.mk
@@ -45,3 +45,5 @@ INC_ZLIB = ${DEB_INCPATH}
#BOOSTDIR = /usr/local/boost
#LIB_BOOST = ${BOOSTDIR}
INC_BOOST = ${DEB_INCPATH}/boost
+
+INC_NIFTI = ${DEB_INCPATH}/nifti
Index: fsl-5.0.9/src/fdt/Makefile
===================================================================
--- fsl-5.0.9.orig/src/fdt/Makefile
+++ fsl-5.0.9/src/fdt/Makefile
@@ -7,7 +7,7 @@ endif
PROJNAME = fdt
-USRINCFLAGS = -I${INC_NEWMAT} -I${INC_NEWRAN} -I${INC_CPROB} -I${INC_PROB} -I${INC_BOOST} -I${INC_ZLIB}
+USRINCFLAGS = -I${INC_NEWMAT} -I${INC_NEWRAN} -I${INC_CPROB} -I${INC_PROB} -I${INC_BOOST} -I${INC_ZLIB} -I${INC_NIFTI}
USRLDFLAGS = -L${LIB_NEWMAT} -L${LIB_NEWRAN} -L${LIB_CPROB} -L${LIB_PROB} -L${LIB_ZLIB}
DLIBS = -lwarpfns -lbasisfield -lmeshclass -lbint -lnewimage -lutils -lmiscmaths -lnewmat -lfslio -lniftiio -lznz -lprob -lm -lz
@@ -167,7 +167,7 @@ ${EDDYCOMBINE}: ${EDDYCOMBINEOBJS}
${CXX} ${CXXFLAGS} ${LDFLAGS} -o $@ ${EDDYCOMBINEOBJS} ${DLIBS}
${LIBBEDPOSTX_CUDA}:
- ${CUDA}/bin/nvcc --shared --compiler-options '-fPIC' -o CUDA/libbedpostx_cuda.so CUDA/init_gpu.cu CUDA/samples.cu CUDA/diffmodels.cu CUDA/runmcmc.cu CUDA/xfibres_gpu.cu -O3 -gencode=arch=compute_20,code=\"sm_20,compute_20\" -gencode=arch=compute_30,code=\"sm_30,compute_30\" -gencode=arch=compute_35,code=\"sm_35,compute_35\" -gencode=arch=compute_50,code=\"sm_50,compute_50\" -lcudart -lcuda -lcurand -I. -L${CUDA}/lib64 -L${CUDA}/lib -ICUDA/options -I${CUDA}/include/thrust -I${FSLDIR}/extras/include/newmat -I${FSLDIR}/include -I${FSLDIR}/extras/include/boost -maxrregcount=64 -DBOOST_NOINLINE='__attribute__ ((noinline))'
+ ${CUDA}/bin/nvcc ${USRINCFLAGS} --shared --compiler-options '-fPIC' -o CUDA/libbedpostx_cuda.so CUDA/init_gpu.cu CUDA/samples.cu CUDA/diffmodels.cu CUDA/runmcmc.cu CUDA/xfibres_gpu.cu -O3 -gencode=arch=compute_20,code=\"sm_20,compute_20\" -gencode=arch=compute_30,code=\"sm_30,compute_30\" -gencode=arch=compute_35,code=\"sm_35,compute_35\" -gencode=arch=compute_50,code=\"sm_50,compute_50\" -lcudart -lcuda -lcurand -I. -L${CUDA}/lib64 -L${CUDA}/lib -ICUDA/options -I${CUDA}/include/thrust -I${FSLDIR}/extras/include/newmat -I${FSLDIR}/include -I${FSLDIR}/extras/include/boost -maxrregcount=64 -DBOOST_NOINLINE='__attribute__ ((noinline))'
@if [ ! -d ${FSLDEVDIR}/lib/ ] ; then ${MKDIR} ${FSLDEVDIR}/lib ; fi
${CP} -rf CUDA/libbedpostx_cuda.so ${FSLDEVDIR}/lib
-------------- next part --------------
Fix linkage issue:
ptxas warning : Too big maxrregcount value specified 64, will be ignored
ptxas fatal : Unresolved extern function 'sqrtf'
Makefile:158: recipe for target 'libbedpostx_cuda.so' failed
make[1]: *** [libbedpostx_cuda.so] Error 255
As per:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=FSL;a0b95f29.1503
"
Nvidia nvcc compiler contains a weird bug when using the function "sqrt" combined with a precision conversion in CUDA 6.0 and CUDA 6.5. That is causing the error:
ptxas fatal : Unresolved extern function 'sqrtf'
You can:
1. Use CUDA toolkit 7.0 or CUDA toolkit 5.5
2. or edit the file fsl/src/fdt/CUDA/diffmodels_utils.h
and change line 82:
#define d2lambda_gpu(d) (sqrt(double(d)))
with
#define d2lambda_gpu(d) (sqrt(d))
"
Index: fsl-5.0.7/src/fdt/CUDA/diffmodels_utils.h
===================================================================
--- fsl-5.0.7.orig/src/fdt/CUDA/diffmodels_utils.h
+++ fsl-5.0.7/src/fdt/CUDA/diffmodels_utils.h
@@ -76,10 +76,10 @@
#define FSMALL_gpu 0.001
//defined in diffmodels.h
-#define f2beta_gpu(f) (asin(double(sqrt(double(f)))))
+#define f2beta_gpu(f) (asin(double(sqrt(f))))
//defined in diffmodels.h
-#define d2lambda_gpu(d) (sqrt(double(d)))
+#define d2lambda_gpu(d) (sqrt(d))
//defined in diffmodels.h
//#define beta2f_gpu(beta) (pow(sin(double(beta)),2.0))
More information about the Neurodebian-devel
mailing list