Solving LMDB Version Incompatibility in PowerDNS: A Post-Mortem

Recently, I encountered a critical issue after a routine system upgrade: my PowerDNS instance was running on a newer version of liblmdb (1.0.0), while the existing database files were created with an older 0.9.x version. Here is how I navigated the transition, recovered the data, and cleaned up the environment.
The Challenge
When upgrading the base system, the liblmdb library transitioned to version 1.0.0. Because PowerDNS uses an LMDB backend, the discrepancy between the on-disk file format and the new library version created an immediate risk of data corruption. Simply restarting the service was not a safe option; we needed to ensure the data was compatible with the new ABI.
Step 1: Data Extraction
The biggest obstacle was the LMDB file structure. PowerDNS uses a sharded storage approach (multiple .lmdb files), and standard tools like mdb_dump expect a directory rather than a file path, resulting in the dreaded mdb_env_open failed, error 20: Not a directory.
To solve this, I performed a "staging" operation:

1. Created a temporary directory structure mimicking an LMDB environment.
   
2. Copied each individual shard (pdns.lmdb, pdns.lmdb-1, etc.) into this temporary space, renaming them to data.mdb.
   
3. Executed mdb_dump on the temporary folder to extract the data into a safe, text-based format.
   

Step 2: Verification and Cleanup
Once the data was safely backed up, I cleaned the environment. Over the years, the directory had accumulated legacy SQLite artifacts from a previous migration. Since the PowerDNS configuration (pdns.conf) was explicitly set to use the lmdb backend (launch=lmdb), I moved the unused SQLite files to an archive directory, significantly decluttering the system.
Step 3: Re-integration
With the service stopped, I ensured that all file permissions were correctly set to the bind user. Upon restarting the service, PowerDNS detected the existing shard files. Even though my configuration specified lmdb-shards=1, the engine intelligently detected the 64 shards on disk and adapted accordingly, successfully loading the backend without data loss.

Practical Implementation: The Migration Script
To automate the data extraction without hardcoding paths or shard counts, we can extract the database location directly from pdns.conf. The following sh script demonstrates how to safely dump all shards regardless of their number.

The Final Migration Tool
This sh script handles the entire lifecycle of the migration process:

1. Tool Discovery: Automatically finds and extracts legacy mdb_dump versions from your package backups (/usr/ports/packages/).
   
2. Data Integrity: Performs a "Stage-Dump-Load" operation, ensuring the new database format is created natively by the current system tools.
   
3. Safety First: Includes multiple sanity checks (root user verification, path safety, and non-destructive cleanup options).
   
4. Environment Awareness: Dynamically identifies PowerDNS ownership and permissions to ensure the service resumes seamlessly after migration.

#!/bin/sh

set -e

# Ensure script is run as root
if [ "$(id -u)" -ne 0 ]; then
    echo "This script must be run as root."
    exit 1
fi

# Configuration
CONFIG="/usr/local/etc/pdns/pdns.conf"
BACKUP_DIR="/tmp/pdns_lmdb_backup"
STAGING_DIR="/tmp/pdns_migrate/db"
LEGACY_DIR="/tmp/legacy_tools"
PKG_DIR="/usr/ports/packages/portmaster-backup"
SYSTEM_DUMP="/usr/local/bin/mdb_dump"
SYSTEM_LOAD="/usr/local/bin/mdb_load"

# Verify config exists
if [ ! -f "$CONFIG" ]; then
    echo "Error: Configuration file $CONFIG not found!"
    exit 1
fi

# 1. Detect available legacy packages
echo "Searching for lmdb packages in $PKG_DIR..."
PKGS=$(ls "$PKG_DIR"/lmdb-*.pkg 2>/dev/null)

if [ -z "$PKGS" ]; then
    echo "No legacy packages found. Using system default mdb_dump."
    MDB_DUMP_BIN="$SYSTEM_DUMP"
else
    echo "Found the following packages:"
    i=1
    set -- $PKGS
    for p in "$@"; do
        echo "$i) $(basename "$p")"
        i=$((i + 1))
    done

    printf "Select a package to extract mdb_dump from (or press Enter to use system default): "
    read -r choice

    if [ -n "$choice" ] && [ "$choice" -ge 1 ] && [ "$choice" -le "$#" ]; then
        SELECTED_PKG=$(eval echo \${$choice})
        echo "Extracting mdb_dump from $(basename "$SELECTED_PKG")..."
        
        mkdir -p "$LEGACY_DIR"
        tar -xf "$SELECTED_PKG" -C "$LEGACY_DIR" usr/local/bin/mdb_dump
        MDB_DUMP_BIN="$LEGACY_DIR/usr/local/bin/mdb_dump"
    else
        MDB_DUMP_BIN="$SYSTEM_DUMP"
    fi
fi

# Verify binaries exist and are executable
if [ ! -x "$MDB_DUMP_BIN" ]; then
    echo "Error: $MDB_DUMP_BIN is not executable."
    exit 1
fi
if [ ! -x "$SYSTEM_LOAD" ]; then
    echo "Error: $SYSTEM_LOAD not found."
    exit 1
fi

echo "Using: $($MDB_DUMP_BIN -V 2>&1 | head -n 1)"

# Extract DB path from pdns.conf
DB_PATH=$(grep "lmdb-filename" "$CONFIG" | cut -d'=' -f2 | tr -d ' ')
if [ -z "$DB_PATH" ]; then
    echo "Error: Could not determine lmdb-filename from $CONFIG"
    exit 1
fi
DB_DIR=$(dirname "$DB_PATH")

mkdir -p "$BACKUP_DIR" "$STAGING_DIR"

echo "Extracting database from $DB_DIR..."

# Process the main file and all shards
for file in "$DB_DIR"/pdns.lmdb*; do
    case "$file" in
        *-lock) continue ;; 
    esac

    rm -f "$STAGING_DIR/data.mdb"
    cp "$file" "$STAGING_DIR/data.mdb"

    DUMP_FILE="$BACKUP_DIR/$(basename "$file").dump"
    "$MDB_DUMP_BIN" -a "$STAGING_DIR" > "$DUMP_FILE"
    
    echo "Dumped: $(basename "$file") -> $(basename "$DUMP_FILE")"
done

# Recreate
printf "Ready to recreate database in $DB_DIR using new LMDB version? (y/n): "
read -r confirm
if [ "$confirm" = "y" ]; then
    # Security check: ensure DB_DIR is not root
    if [ -n "$DB_DIR" ] && [ "$DB_DIR" != "/" ] && [ "$DB_DIR" != "/etc" ]; then
        echo "Removing old database files from $DB_DIR..."
        rm -f "$DB_DIR"/pdns.lmdb*
    else
        echo "Safety check failed: Refusing to delete files in $DB_DIR"
        exit 1
    fi
    
    echo "Recreating database from dumps..."
    for dump in "$BACKUP_DIR"/*.dump; do
        target=$(basename "$dump" .dump)
        echo "Loading $dump -> $target"
        "$SYSTEM_LOAD" -a "$DB_DIR/$target" < "$dump"
    done
fi

# Cleanup
printf "Migration complete. Keep temporary files? (y/n): "
read -r keep_dumps
if [ "$keep_dumps" = "n" ]; then
    rm -rf "$BACKUP_DIR" "$STAGING_DIR" "$LEGACY_DIR"
    echo "Temporary files cleaned."
fi

# Apply permissions
PDNS_USER=$(grep "^setuid" "$CONFIG" | cut -d'=' -f2 | tr -d ' ')
PDNS_GROUP=$(grep "^setgid" "$CONFIG" | cut -d'=' -f2 | tr -d ' ')

[ -z "$PDNS_USER" ] && PDNS_USER=$(ls -ld "$DB_DIR" | awk '{print $3}')
[ -z "$PDNS_GROUP" ] && PDNS_GROUP=$(ls -ld "$DB_DIR" | awk '{print $4}')

echo "Applying permissions for $PDNS_USER:$PDNS_GROUP on $DB_DIR"
chown -R "$PDNS_USER:$PDNS_GROUP" "$DB_DIR"

# Service restart
echo "Restarting PowerDNS..."
service pdns restart

echo "Migration process finished."
tail -20 /var/log/messages | grep pdns

Lessons Learned

• Don't ignore the ldd output: Checking which shared libraries your binaries are actually linking against is the first step in diagnosing versioning conflicts.
  
• Staging is your best friend: When dealing with strict tool requirements (like mdb_dump expecting directories), manual staging is the safest way to perform a low-level migration.
  
• Clean house: If you have migrated from SQLite to LMDB in the past, don't leave the old files sitting in your production directory. They only increase the complexity of backups and debugging.
  

Final Thoughts
Upgrading database engines doesn't always have to be a "break-and-fix" cycle. By preparing a solid backup, understanding the underlying storage structure, and verifying the service logs post-restart, you can perform complex migrations while keeping your DNS infrastructure rock-solid.
Tags: #PowerDNS #LMDB #SystemAdministration #FreeBSD #DatabaseMigration #DevOps