Skip to content

Commit

Permalink
Build with static libs (#138)
Browse files Browse the repository at this point in the history 
* Update setup.py for static libs

Modification made to setup.py to allow static library linking for
g2c and the necessary libraries that g2c needs. This is done by
inspecting the list of symbols in the g2c static library file.

This can be controlled by setting env var USE_STATIC_LIBS="True".

This commit references issue #137

* Update README.md

[skip ci]

* Update setup.py

* More setup updates

More updates to setup.py for static library linking.  This update
allows for specifying the static library filenames explicitly
in the setup.cfg or by environment variable.

* Update setup.py for better handling of static libs

* More updates for setup.py for static libs

setup.py - move iteration for search for libraries to inside if usestaticlibs
because that is the only scenario when we need to.

setup.cfg - removed jpegturbo_lib line.  Not needed, just need jpeg_lib.

* Update setup.py

Using separate list, dep_libraries, when iterating over the static
library names.

---------

Co-authored-by: Eric Engle <EricEngle-NOAA@users.noreply.github.com>
  • Loading branch information
@EricEngle-NOAA
EricEngle-NOAA and EricEngle-NOAA authored Mar 13, 2024
1 parent 3288801 commit 365100b
Show file tree
Hide file tree
Showing 4 changed files with 157 additions and 44 deletions.
3 changes: 2 additions & 1 deletion MANIFEST.in
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ include VERSION
include create_docs.sh
include requirements.txt
graft docs
graft src/ext src/grib2io
graft src/ext
graft src/grib2io
graft tests
global-exclude *.py[cod] *.c *.so
11 changes: 10 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,18 @@ conda install -c conda-forge grib2io
pip install .
```

> [!NOTE]
> ### Building with static libraries
> The default behavior for building grib2io is to build against shared-object libraries. However, in production environments, it is beneficial to build against static library files. grib2io (v2.2.0+) allows for this type of build configuration. To build against static library files, set the environment variable, `USE_STATIC_LIBS="True"` before your build/install command. For example,
>
>```shell
>export USE_STATIC_LIBS="True"
>pip install .
>```
## Development
The intention of grib2io is to become the offical Python interface for the NCEP g2c library. Therefore, the development evolution of grib2io will mainly focus on how best to serve that purpose and its primary user's -- mainly meteorologists, physical scientists, and software developers supporting the missions within NOAA's National Weather Service (NWS) and National Centers for Environmental Prediction (NCEP), and other NOAA organizations.
The intention of grib2io is to become the offical Python interface for the NCEP g2c library. Therefore, the development evolution of grib2io will mainly focus on how best to serve that purpose and its primary users -- mainly meteorologists, physical scientists, and software developers supporting the missions within NOAA's National Weather Service (NWS) and National Centers for Environmental Prediction (NCEP), and other NOAA organizations.
## Disclaimer
Expand Down
27 changes: 19 additions & 8 deletions setup.cfg
View file
Original file line number Diff line number Diff line change
@@ -1,16 +1,13 @@
# ----------------------------------------------------------------------------------------
# setup.cfg.template for grib2io
# setup.cfg for grib2io
#
# Notes:
#
# 1) Rename this file to setup.cfg to set grib2io build options. Follow instructions
# below for editing.
#
# 2) Define the locations for the g2c library.
# Uncomment lines accordingly.
# ----------------------------------------------------------------------------------------

[directories]
[options]
#use_static_libs = True

[directories]
# ----------------------------------------------------------------------------------------
# Define paths for NCEPLIBS-g2c Library
#
Expand All @@ -20,3 +17,17 @@
# g2c_dir = /usr/local
# g2c_incdir =
# g2c_libdir =

[static_libs]
# ----------------------------------------------------------------------------------------
# Set the following variables if static linking to g2c.
#
# IMPORTANT: You do not need to specify the static library path for g2c here.
# ----------------------------------------------------------------------------------------
#g2c_lib =
#aec_lib =
#jasper_lib =
#jpeg_lib =
#openjpeg_lib =
#png_lib =
#z_lib =
160 changes: 126 additions & 34 deletions setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,58 @@
import numpy
import os
import platform
import subprocess
import sys

with open("VERSION","rt") as f:
VERSION = f.readline().strip()

libdirs = []
incdirs = []
libraries = ['g2c']


# ----------------------------------------------------------------------------------------
# find_library.
# ----------------------------------------------------------------------------------------
def find_library(name, dirs=None):
# This maps package names to library names used in the
# library filename.
pkgname_to_libname = {
'g2c': ['g2c'],
'aec': ['aec'],
'jasper': ['jasper'],
'jpeg': ['turbojpeg', 'jpeg'],
'openjpeg': ['openjp2'],
'png': ['png'],
'z': ['z'],}

def get_grib2io_version():
with open("VERSION","rt") as f:
ver = f.readline().strip()
return ver

def get_package_info(name, config, static=False):
pkg_dir = os.environ.get(name.upper()+'_DIR')
pkg_incdir = os.environ.get(name.upper()+'_INCDIR')
pkg_libdir = os.environ.get(name.upper()+'_LIBDIR')

if pkg_dir is None:
# Env var not set
pkg_dir = config.get('directories',name+'_dir',fallback=None)
if pkg_dir is None:
if static:
pkg_lib = config.get('static_libs',name+'_lib',fallback=None)
pkg_libdir = os.path.dirname(pkg_lib)
pkg_incdir = os.path.join(os.path.dirname(pkg_libdir),'include')
pkg_dir = os.path.dirname(pkg_libdir)

if pkg_dir is None:
for l in pkgname_to_libname[name]:
libname = os.path.dirname(find_library(l, static=static))
if libname is not None: break
pkg_libdir = libname
pkg_incdir = os.path.join(os.path.dirname(pkg_libdir),'include')

else:
# Env var was set
if os.path.exists(os.path.join(pkg_dir,'lib')):
pkg_libdir = os.path.join(pkg_dir,'lib')
elif os.path.exists(os.path.join(pkg_dir,'lib64')):
pkg_libdir = os.path.join(pkg_dir,'lib64')
if os.path.exists(os.path.join(pkg_dir,'include')):
pkg_incdir = os.path.join(pkg_dir,'include')
return (pkg_incdir, pkg_libdir)

def find_library(name, dirs=None, static=False):
_libext_by_platform = {"linux": ".so", "darwin": ".dylib"}
out = []

Expand All @@ -35,6 +73,7 @@ def find_library(name, dirs=None):

# For Linux and macOS (Apple Silicon), we have to search ourselves.
libext = _libext_by_platform[sys.platform]
if static: libext = '.a'
if dirs is None:
if os.environ.get("CONDA_PREFIX"):
dirs = [os.environ["CONDA_PREFIX"]]
Expand All @@ -45,7 +84,7 @@ def find_library(name, dirs=None):

out = []
for d in dirs:
libs = Path(d).rglob(f"lib*{name}{libext}")
libs = Path(d).rglob(f"lib{name}{libext}")
out.extend(libs)
if not out:
raise ValueError(f"""
Expand All @@ -57,6 +96,17 @@ def find_library(name, dirs=None):
""")
return out[0].absolute().resolve().as_posix()

# ----------------------------------------------------------------------------------------
# Main part of setup.py
# ----------------------------------------------------------------------------------------
VERSION = get_grib2io_version()

usestaticlibs = False
libraries = ['g2c']

extra_objects = []
incdirs = []
libdirs = []

# ----------------------------------------------------------------------------------------
# Build Cython sources
Expand All @@ -74,33 +124,75 @@ def find_library(name, dirs=None):
config.read(setup_cfg)

# ----------------------------------------------------------------------------------------
# Get NCEPLIBS-g2c library info.
# ----------------------------------------------------------------------------------------
if os.environ.get('G2C_DIR'):
g2c_dir = os.environ.get('G2C_DIR')
if os.path.exists(os.path.join(g2c_dir,'lib')):
g2c_libdir = os.path.join(g2c_dir,'lib')
elif os.path.exists(os.path.join(g2c_dir,'lib64')):
g2c_libdir = os.path.join(g2c_dir,'lib64')
g2c_incdir = os.path.join(g2c_dir,'include')
else:
g2c_dir = config.get('directories','g2c_dir',fallback=None)
if g2c_dir is None:
g2c_libdir = os.path.dirname(find_library('g2c'))
g2c_incdir = os.path.join(os.path.dirname(g2c_libdir),'include')
libdirs.append(g2c_libdir)
incdirs.append(g2c_incdir)

libdirs = list(set(libdirs))
# Check if static library linking is preferred.
# ----------------------------------------------------------------------------------------
if os.environ.get('USE_STATIC_LIBS'):
val = os.environ.get('USE_STATIC_LIBS')
if val not in {'True','False'}:
raise ValueError('Environment variable USE_STATIC_LIBS must be \'True\' or \'False\'')
usestaticlibs = True if val == 'True' else False
usestaticlibs = config.get('options', 'use_static_libs', fallback=usestaticlibs)

# ----------------------------------------------------------------------------------------
# Get g2c information
# ----------------------------------------------------------------------------------------
pkginfo = get_package_info(libraries[0], config, static=usestaticlibs)
incdirs.append(pkginfo[0])
libdirs.append(pkginfo[1])

# ----------------------------------------------------------------------------------------
# Perform work to determine required static library files.
# ----------------------------------------------------------------------------------------
if usestaticlibs:
staticlib = find_library('g2c', dirs=libdirs, static=True)
extra_objects.append(staticlib)
cmd = subprocess.run(['ar','-t',staticlib], stdout=subprocess.PIPE)
symbols = cmd.stdout.decode('utf-8')
if 'aec' in symbols:
libraries.append('aec')
if 'jpeg2000' in symbols:
libraries.append('jpeg')
libraries.append('jasper')
if 'openjpeg' in symbols:
libraries.append('openjpeg')
if 'png' in symbols:
libraries.append('png')
libraries.append('z')

# We already found g2c info, so iterate over libraries from [1:]
dep_libraries = [] if len(libraries) == 1 else libraries[1:]
for l in dep_libraries:
incdir, libdir = get_package_info(l, config, static=usestaticlibs)
incdirs.append(incdir)
libdirs.append(libdir)
if usestaticlibs:
l = pkgname_to_libname[l][0]
extra_objects.append(find_library(l, dirs=[libdir], static=usestaticlibs))

libraries = [] if usestaticlibs else list(set(libraries))
incdirs = list(set(incdirs))
incdirs.append(numpy.get_include())
libdirs = [] if usestaticlibs else list(set(libdirs))
extra_objects = list(set(extra_objects)) if usestaticlibs else []

print(f'Use static libs: {usestaticlibs}')
print(f'\t{incdirs = }')
print(f'\t{libdirs = }')
print(f'\t{extra_objects = }')

# ----------------------------------------------------------------------------------------
# Define extensions
# ----------------------------------------------------------------------------------------
g2clibext = Extension('grib2io.g2clib',[g2clib_pyx],include_dirs=incdirs,\
library_dirs=libdirs,libraries=libraries,runtime_library_dirs=libdirs)
redtoregext = Extension('grib2io.redtoreg',[redtoreg_pyx],include_dirs=[numpy.get_include()])
g2clibext = Extension('grib2io.g2clib',
[g2clib_pyx],
include_dirs = incdirs,
library_dirs = libdirs,
libraries = libraries,
runtime_library_dirs = libdirs,
extra_objects = extra_objects)
redtoregext = Extension('grib2io.redtoreg',
[redtoreg_pyx],
include_dirs = [numpy.get_include()])

# ----------------------------------------------------------------------------------------
# Create __config__.py
Expand Down

0 comments on commit 365100b

Please sign in to comment.