I'm wanting to upgrade Mapnik on a production system from 2.2.0 to 3.x to take advantage of improvements to data-driven rendering. Fortunately the install instructions for Ubuntu have been updated very recently (11 days ago as of this post), however I'm wondering what happens if I run the installation on a system with an existing install. Has anyone tried this?
We're using TileStache to render several map tile endpoints, most importantly some UTFGrid json files, and I am concerned that I will pollute the system— maybe irreparably!—by running the install.
Can anyone comment on what might happen if I run the latest Mapnik installation on a system with an existing, properly functioning install?
[Edit 8/22/2018]
I got through the transition on a test server and incorporated my prior edits into an thorough answer below.
Answer
I remembered I had a tiny Rackspace instance that I setup once upon a time to test code portability. It has a nearly identical version of our webstack, particularly the Ubuntu and Mapnik versions, so it made a reasonable test bed to experiment with moving from 2.2.x to 3.x.
In a nutshell, this was a complicated transition that involved sincere troubleshooting, research, stack resets, some code modifications to XML stylesheets, and although I got things working again, my confidence is a little weak. Were this a production system, downtime would have been considerable.
What follows is a list of the 'gotchas' I encountered trying to nail down the build dependencies (c++ compiler version, clang version, git version, etc.) and some configuration/usage issues I ran into after the fact. Hopefully consulting this list can help avoid some unnecessary downtime and hasten the transitional process..
In attempting to install and configure Mapnik on Ubuntu using the official instructions found here, under the heading "Install Mapnik from Source", I suggest augmenting those instructions with information provided below. FWIW this is relative to Ubuntu 12.04.5 LTS.
First, before installing or uninstalling anything Mapnik..
1) Compiler
Update the g++ compiler. Building Mapnik requires c14 compiler features, and you'll be unable to build if your compiler fails the dependencies/versioning test. In your terminal, type g++ --version and if your version is below 4.9 go ahead and update. I used the guide found here, specifically these commands:
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get install g++-4.9
sudo ln -f -s /usr/bin/g++-4.9 /usr/bin/g++
2) Fix Clang
I had an issue with clang versioning and fixed it by adapting the instructions found here under the heading "Source install of Mapnik Master (3.x)" I changed all instances of 3.6 to 3.8, and based on an error/warning I received on my fist install attempt, I added the --force-yes flag to the actual apt-get install instruction. These are the terminal commands I used to install the proper clang version:
CLANG_VERSION=3.8
sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test;
sudo add-apt-repository "deb http://llvm.org/apt/precise/ llvm-toolchain-precise-${CLANG_VERSION} main";
wget -O - http://llvm.org/apt/llvm-snapshot.gpg.key|sudo apt-key add -
sudo apt-get update -y
#### I had to add "--force-yes" to fix this install command:
#### sudo apt-get install -y clang-3.8;
sudo apt-get install -y --force-yes clang-3.8;
export CXX="clang++-3.8" && export CC="clang-3.8";
When you're finished, you should be able to type echo $CXX and get terminal output clang++-3.8 or echo $CC and get clang-3.8, respectively. If you don't get these responses, something may fail during a later configure/build step.
3) Upgrade Git
I initially ran into a git usage error related to use of a -C parameter the first time I executed the source bootstrap.sh command, which appears to establish some traits of the build environment. Googling for solutions, it seems I had two options, either modify the bootstrap.sh file by replacing instances of -C with --git-dir, as apparently these have equivalent functionality, or upgrade git.
You can run git --version in terminal—mine was 1.7.9.5 prior to upgrading. After upgrading, my version increased to 2.18.0, and the -C parameter issue disappeared. Upgrading was this easy:
sudo add-apt-repository ppa:git-core/ppa -y
sudo apt-get update
sudo apt-get install git -y
4) libgdal.la not found
I forgot where I encountered this issue, during either source bootstrap.sh or possibly the ./configure.. step. But I got the error:
# Error found:
# /home/buildnik/mapnik/mason_packages/linux-x86_64/libgdal/2.1.3/lib/libgdal.la not found
Based on the instructions under Set up build environment, manually installing libgdal1-dev like this fixed the issue:
apt-get install libgdal1-dev
I suspect you can do this before removing old Mapnik or starting any of the formal installation steps.
5) Honorable mention, severity unknown
When executing make test, I got errors like:
test/unit/datasource/postgis.cpp:87:
warning:
Can't run postgis.input tests - check postmaster is running and accessible
and..
ROR (Postgis Plugin: FATAL: role "root" does not exist
They might have been related to my installed PostGRESql version (9.1), which works fine. I never resolved these errors/warnings, but my Mapnik build/install seems to be working okay in spite of them. FWIW, elsewhere in the instructions, a note points Ubuntu 14.04 users toward pg 9.3.
6) Actually install Mapnik
At this point I uninstalled Mapnik, then started my fresh build/install. I'm not sure if it's 100% essential to remove your old Mapnik first, but I saw instructions recommending it, from Dane no less. So I removed Mapnik as my first step.
Everything went like this:
sudo apt-get purge libmapnik* mapnik-* python-mapnik
cd /home
mkdir buildnik
cd buildnik
git clone https://github.com/mapnik/mapnik mapnik --depth 10
cd mapnik
git submodule update --init
sudo apt-get install python zlib1g-dev clang make pkg-config curl
source bootstrap.sh
./configure CUSTOM_CXXFLAGS="-D_GLIBCXX_USE_CXX11_ABI=0" CXX=${CXX} CC=${CC}
make
make test
sudo make install
A few of these steps, particularly the make instruction, took quite awhile, probably because my server instance is tiny. But knowing this, at least don't expect just one or two minutes of downtime. Unless you play by wild-west rules, you'll want to do this upgrade during a reasonable maintenance window.
At this point I ran into issues importing Mapnik libs in Python. TileStache was throwing errors at Apache, which is how I noticed.
7) Install Mapnik Python bindings
First install the Mapnik Python bindings. It's super easy. Just make sure you have pip >= 1.4, which has something to do with the Python Wheels dependency. Use pip --version to check. Mine's pip 7.1.2, so I suspect pip version is a non-issue:
pip install mapnik
This will probably fix the issue.
8) Check ldconfig
You might not need to do this but the troubleshooting instructions call for a tweak to the LD config in the case of Python not finding Mapnik. I deviated slightly from the troubleshooting recommendation, based mostly on what I read here. Rather than add a path directly to /etc/ld.so.conf, I created a new file in /etc/ld.so.conf.d and added my path values there, like this:
cd /etc/ld.so.conf.d
touch niklib.conf
vi niklib.conf
Inside niklib.cof, add the following three lines:
# Required to link Mapnik libs
/usr/local/lib
/usr/local/lib64
After saving the file, update the environment with:
ldconfig -v
Note: If you dir /etc/ld.so.conf.d and see libc.conf, and if you vi /etc/ld.so.conf.d/libc.conf and see /usr/local/lib in that file, then you can skip this step #8, as your LD environment should already reference the necessary paths.
9) Mapnik XML design files (i.e. XML map stylesheets)
At this point I could launch Python and import Mapnik libs, but TileStache still wasn't rendering (most) tiles! By tailing the Apache log like this:
tail -f /var/log/apache2/error.log
I noticed errors popping up that looked like this:
failed to initialize projection with: '&google_mercator;'
This took awhile to understand and fix. It turns out default build-from-source configurations for Mapnik use a different XML library (rapidxml) than was used for some turnkey builds (libxml2), which I suspect a lot of people used around the 2.x release, most likely because those builds were promoted by one or more popular tutorials. The issue stems from so-called XML Entities, and it seems the (older?) Open Street Map (OMS) XML map designs used these XML entities, which further propagated this approach. You can find a longer discussion of this here.
As it happened, some of my own map designs were using XML entities to set the map srs/projection attributes, which were the source of my Apache errors. I got a lucky break by noticing that my UTFGrid tiles were actually rendering, and a close inspection of the map designs revealed a different manner of setting the map srs/projection. As such, I was able to make small changes to these XML files to eliminate the last of my show-stopping errors, bringing full functionality back online.
9a) What I did..
Basically I looked through my map files and changed every instance of something like this (notice the ENTITY tag):
]>
..to simply, this:
9b) What you may prefer..
Optionally, if you need to fix/check A LOT of XML designs, you might want to check out xmllint, which has an option to sanitize XML Entities, and which you can incorporate into automation and use to mass-render all your XML files, then overwrite them into place with a single command.
Try calling xmllint --version. If you don't get any feedback, you can install the utility like this:
apt-get install libxml2-utils
Once you've got it installed, look at a problematic XML file (one that contains an ENTITY tag), make a copy of it, and run xmllint on it like this to see how it morphs the code:
xmllint --noent parcels_bk.xml
It'll pump XML right into the console, and you'll see those ENTITY tags replaced by their actual values.
10) Honorable mention, inconsequential (mod_python version)
At this point I cleared my Apache log file, rebooted, then tailed the log to see if any more errors were appearing. Of course on serious systems it's better to make a backup of this file before clearing it:
> /var/log/apache2/error.log
reboot
tail -f /var/log/apache2/error.log
When I did this I saw the following (note mention of "version mismatch"):
[Tue Aug 21 19:12:15 2018] [notice] caught SIGTERM, shutting down
[Tue Aug 21 19:14:08 2018] [error] python_init: Python version mismatch, expected '2.7.2+', found '2.7.3'.
[Tue Aug 21 19:14:09 2018] [error] python_init: Python executable found '/usr/bin/python'.
[Tue Aug 21 19:14:09 2018] [error] python_init: Python path being used '/usr/lib/python2.7/:/usr/lib/python2.7/plat-
linux2:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload'.
[Tue Aug 21 19:14:09 2018] [notice] mod_python: Creating 8 session mutexes based on 150 max processes and 0 max threads.
[Tue Aug 21 19:14:09 2018] [notice] mod_python: using mutex_directory /tmp
[Tue Aug 21 19:14:09 2018] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.26 with Suhosin-Patch mod_python/3.3.1
Python/2.7.3 mod_wsgi/3.3 configured -- resuming normal operations
I was concerned this might have been related to my build or my bindings, etc., but turns out it's related to mod_python, which TileStache uses portage internet requests and responses through Apache. Since my install is working, I chose not to rock the boat any further and stopped there. ..I probably had this error/warning all along. But if you get this error and you're so inclined, it seems removing and reinstalling mod_python may be a solution:
apt-get remove libapache2-mod-python libapache2-mod-wsgi
apt-get build-dep libapache2-mod-python libapache2-mod-wsgi
That concludes my experience going from a working Mapnik 2.2.x to a working Mapnik 3.x on Ubuntu 12.04. Hopefully this helps someone transition quicker than I was able to and avoid some of the frustrations.
....finally, since these systems can be so very different: GOOD. LUCK.
No comments:
Post a Comment