libgd and libpng out-of-date?

[jsta]

I am having trouble running GLMr. It installs without error but I am getting the following error:

> devtools::install_github("GLEON/GLMr")
> GLMr::glm_version()

/usr/local/lib/R/site-library/GLMr/exec/nixglm: error while loading shared libraries: libgd.so.2: cannot open shared object file: No such file or directory
[1] 127

Update Summary
I think I was able to solve this issue by symlinking as follows:
$ ln -s /usr/lib/x86_64-linux-gnu/libgd.so.3 /usr/lib/x86_64-linux-gnu/libgd.so.2
$ ln -s /usr/lib/x86_64-linux-gnu/libpng16.so /usr/lib/x86_64-linux-gnu/libpng12.so.0

Misc

$ ldconfig -p | grep libgd

libgdbm_compat.so.3 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libgdbm_compat.so.3
libgdbm.so.3 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libgdbm.so.3
libgdal.so.20 (libc6,x86-64) => /usr/lib/libgdal.so.20
libgdal.so (libc6,x86-64) => /usr/lib/libgdal.so
libgd.so.3 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libgd.so.3
libgd.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libgd.so

$ ln -s /usr/lib/x86_64-linux-gnu/libgd.so.3 /usr/lib/x86_64-linux-gnu/libgd.so.2
$ Rscript -e "GLMr::glm_version()"

/usr/local/lib/R/site-library/GLMr/exec/nixglm: error while loading shared libraries: libpng12.so.0: cannot open shared object file: No such file or directory
[1] 127

$ ldconfig -p | grep libpng

libpng16.so.16 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libpng16.so.16
libpng16.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libpng16.so

$ ln -s /usr/lib/x86_64-linux-gnu/libpng16.so /usr/lib/x86_64-linux-gnu/libpng12.so.0
$ Rscript -e "GLMr::glm_version()"

 ------------------------------------------------
   |  General Lake Model (GLM)   Version 2.2.0rc5    |
   ------------------------------------------------

--help : show this blurb
--nml : get parameters from nmlfile
--xdisp : display temp/salt and selected others in x-window
--xdisp : like --xdisp, but use instead of plots.nml
--saveall : save plots to png files
--save-all-in-one : save all plots to png file
[1] 0

> sessionInfo()

R version 3.3.1 (2016-06-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux stretch/sid

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

loaded via a namespace (and not attached):
[1] GLMr_3.1.14

Docker Image

https://hub.docker.com/r/jsta/glm-test/

[lawinslow]

Yeah, it's linked against somewhat older libraries mainly because some of the computer clusters I use are on somewhat older Centos versions without the latest lib versions. I'm building it for *nix, but am by no means an expert.

Do you know if there is a more generic way to build against libraries so we don't have version issues like this?

[jsta]

I'm not sure. It might be solved by specifying compiler flags differently? This thread looks promising.

[lawinslow]

Hmm. Does your setup have libgd.so and libpng.so simlinks by default? We could link against those, but if they aren't common then it won't help.

[jsta]

I have looked both on my local system as well as within a docker container. Both have libgd.so. Neither have libpng.so. Docker has libpng16.so while my local system has libpng12.so.

[lawinslow]

So no major gain on libpng. Hmmm.

[lawinslow]

Ultimately, I wish we could move away from ifort and move to gfortran and just create an easily built package for *nix users.

[jsta]

Could this be dockerized? Would that violate ifort terms of service?

[lawinslow]

No clue. But I did setup something similar on a Travis build for GLM. General steps for installing, setting up, and using ifort to build GLM is here.

[bharris-fs]

@lawinslow: I just posted issue #122 about GLMr on Debian before noticing this issue, and I do hae libpng. Technically, I have libpng-dev, libpng-tools, libpng16-16, and libpng16-16:i386 installed.. I don't have either libpng or libpng12 available to me on Debian Stretch.

[jsta]

@bharris-fs Did you try my solution above symlinking libpng16 to libpng12?

$ ln -s /usr/lib/x86_64-linux-gnu/libgd.so.3 /usr/lib/x86_64-linux-gnu/libgd.so.2
$ ln -s /usr/lib/x86_64-linux-gnu/libpng16.so /usr/lib/x86_64-linux-gnu/libpng12.so.0

[bharris-fs]

@jsta No, I did not. When I do something like that, I tend to forget about it and then end up with a bit of a jumble. Besides, the computer I really plan to use with GLMr runs W7 or Windows 201 6 Server.
I agree the risk sounds small, for apparently libpng12 won't be installed on this system again.

[bharris-fs]

Okay, I got a free moment and tried:

root@here:/usr/lib/x86_64-linux-gnu# ln -s libgd.so.3 libgd.so.2
root@here:/usr/lib/x86_64-linux-gnu# ln -s libpng16.so libpng12.so.0
root@here:/usr/lib/x86_64-linux-gnu# ls -l libpng*
lrwxrwxrwx 1 root root     11 May  3 21:25 libpng12.so.0 -> libpng16.so
-rw-r--r-- 1 root root 328198 Jan  9  2017 libpng16.a
lrwxrwxrwx 1 root root     19 Jan  9  2017 libpng16.so -> libpng16.so.16.28.0
lrwxrwxrwx 1 root root     19 Sep 30  2017 libpng16.so.16 -> libpng16.so.16.28.0
-rw-r--r-- 1 root root 206768 Jan  9  2017 libpng16.so.16.28.0
lrwxrwxrwx 1 root root     10 Jan  9  2017 libpng.a -> libpng16.a
lrwxrwxrwx 1 root root     11 Jan  9  2017 libpng.so -> libpng16.so
root@here:/usr/lib/x86_64-linux-gnu# ls -l libgd.so*
lrwxrwxrwx 1 root root     14 Aug 31  2017 libgd.so -> libgd.so.3.0.4
lrwxrwxrwx 1 root root     10 May  3 21:16 libgd.so.2 -> libgd.so.3
lrwxrwxrwx 1 root root     14 Aug 31  2017 libgd.so.3 -> libgd.so.3.0.4
-rw-r--r-- 1 root root 406184 Aug 31  2017 libgd.so.3.0.4

Then I went back to the same GLMr session:

> example(run_glm)

rn_glm> sim_folder <- system.file('extdata', package = 'GLMr')

rn_glm> run_glm(sim_folder)
/usr/local/lib/R/site-library/GLMr/exec/nixglm: /usr/lib/x86_64-linux-gnu/libpng12.so.0: version `PNG12_0' not found (required by /usr/local/lib/R/site-library/GLMr/extbin/nixGLM/libgd.so.2)
[1] 1

rn_glm> ## Not run: 
rn_glm> ##D out_file <- file.path(sim_folder,'output.nc')
rn_glm> ##D nml_file <- file.path(sim_folder,'glm2.nml')
rn_glm> ##D library(glmtools)
rn_glm> ##D fig_path <- tempfile("temperature", fileext = '.png')
rn_glm> ##D plot_temp(file = out_file, fig_path = fig_path)
rn_glm> ##D cat('find plot here: '); cat(fig_path)
rn_glm> ## End(Not run)
rn_glm> 
rn_glm> 
rn_glm> 
> 

The good news? It got farther, but does this mean it also checked the version of "libpng12.so.0"?

@lawiinslow: There's maybe a bit of good news.: there's a libpng.so in the above that's linked to the current version of that library, so, at least on Debian, your hope might work. I thought it would not, because I simply looked at Debian packages, not what gets installed.

Any way around the version PNG12_0 problem?

[lawinslow]

Hmmm, there is most certainly way around the problem. Just not sure the easiest, or most universal option. My naive understanding is that its often easier just to build every binary locally on *nix systems because of the challenge of linked libraries. Its best to have everything just linked to local versions of dynamic libraries.

You can see it here. The issue you are seeing is that one of the dynamic libraries I've included in GLMr (libgd) is looking for a specific version of libpng.

Just thinking off the cuff here. More permanent options include:

1. Pull out the libgd library from GLMr and require users to have it installed themselves (this moves the potential linking problems up a level).

2. We could start building more versions of GLM for *nix with different lib requirements.

3. We could include more libraries within GLMr to try and obviate locally required libraries. Maybe this libpng is the only issue and if we included that originally linked version, we'd largely eliminate these issues.

4. Someone who knew this stuff better could perhaps improve GLM linking so we don't depend on specific versions (or outside libs at all). Better static linking?

We could try the above num 3 as a first cut. It might solve this issue as-is for a little while.

Thoughts?

[bharris-fs]

@lawinslow, yes, there can be advantages to building locally, I think.

From your four choices, here's my mildly-informed response:

1. I think this is the cleanest. Most of us on *nix are used to installing distro packages; as long as GLMr requires relatively common packages, I should think that would work. See rstan for the most complex installation process I know of, and it works. The biggest problem I can think of is that you won't necessarily know what version of libgd / libpng you'll find. You can probably reasonably set a dependency that requires at least a version X, but it's probably unrealistic to be able to get everyone to install X.Y.Z for GLMr.

2. That sounds a bit like a career! :-) Don't forget to update each of those different *nix systems each time they upgrade the pertinent packages.

3. That sounds a bit like an approach from the time before shared libraries: "I've got a libabc for this, a libabc for that, and a libabc for the other application--and they're all at different versions." Still, it might well work, and, as this is, so far, the only case like this I've seen, might not be much of a problem.

4. Sounds good, but I don't know enough to contribute on this one.

I'm curious: why is this a problem here? Is it because existing Fortran code in GLM needs something special? Is there anything to be done in figuring out why GLMr has this issue but most other R packages do not? You probably know. I don't, but GLMr looks tantalizingly close to something I could make use of right now.

Thanks for thinking about solutions together! Do you have more thoughts?

[lawinslow]

The issue here 99% lies in difficulty building GLM. It's not setup at the moment to just throw standard tooling at it and get a nicely built binary. If that option were available to us, then we could easily just have GLM built as part of the GLMr build process (something many R packages do) and directly link with the locally-available shared libraries. This is one of the reasons CRAN enforces a basically 100% open-source distributed codebase.

The primary culprit is the requirement to use the Intel Fortran compiler (ifort; instead of say gFortran). This is not free, or commonly available to be used for building on most systems. If we could fully build via gFortran, then there would be no problem. My understanding is the missing Fortran 2008 features in gFortran are what is holding back a fully open-source build process.

But because we must pre-compile GLM on a build machine with ifort, we then get these dependency issues. For Windows and Mac, it isn't quite as large an issue as we largely re-distribute the majority of linked dynamic libraries (though this has recently caused all sorts of issues with latest mac releases, sigh). Hence my thought # 3 above. Distributing more libraries isn't as much of an issue was it might sound either, considering the libraries will not be available on your path and therefore not conflict with anything else installed on the system (beyond when used for GLM).

[bharris-fs]

Thank you for the clear explanation. I should have figured it was something like that; there seems to be little reason to go to the complexity of included libraries if you don't have to.

I'd say that you should do exactly what you think you should do for GLM, which sounds like option 3. At this point, I'm still evaluating my alternatives, so don't regard anything I've written as equivalent to "If you add this capability, I will use GLMr."