TIL: Pyinstaller on Wine

Problem

Have a python script that has to be deployed to a Windows machine in a portable manner. Internet access is likely limited, installing Python runtime and pulling dependencies via Pip is out of the question; so is installing Docker. Trivial, right?

Tl;dr

I created a Docker container that packages 3.x python scripts into 32-bit Windows executables with Pyinstaller.

Development Chronicle

Had I been wise, I would have cut my losses and left halfways. In hindsight this is a niche tool with too many moving parts; far more, than there should be. By no means do I claim, that this isn’t overkill; yet morbid curiosity kept me from abandoning it. This is the chronicle of assembling all the pieces with duct tape and patience. Or maybe it’s just the story of vanilla flavoured hubris of ‘how hard can it be’-s and ‘what could go wrong’-s. Either way: follow me, Reader!

Python is portable out of the box, right?

Being new to Python, I thought virtual environment is a means of portability in Pythonland (Pythonia?). My naivety is rooted in my familiarity with Maven, where dependency hell takes a different form. After some light research it is now clear, that venvs are meant for something else; so much so, that they cannot be dropped into different directories, let alone expect them to bear runtimes of multiple platforms.

What about the portable distribution of Python? In retrospect that might have been a good alternative with a substantial amount of hand cranking. That’s an approach for next time.

There must be an existing packaging tool

Googling led me to Pyinstaller. It does exactly almost what I need: it used to support cross compilation, but it had always been flaky apparently, hence the feature had since been dropped. Yet I want to make Windows executables on linux.

No worries, Docker to the rescue! I am sure that I am not the first person to have come accross this problem. A solution must exist somewhere in the wild. The setup to look for is Ubuntu containerized with Wine and Python for Windows installed. Some exist, but none seem to work. To add insult to injury, they all seemed to have Python 2.x runtimes.

The prevalence of Python 2.x is easy to explain: unlike 3.x, it’s distributed with an msi installer lending itself easily to gui-less unattended installs. In contrast 3.x only has a gooey installer, which displays a dialog box even in silent non-interactive mode.

I roll my own

How hard could it be to roll my own? An afternoon, tops, right? Right?? It’s as easy as taking an Ubuntu base image, apt install wine, wine python-3.7.3.exe. Why wouldn’t this work… No seriously, why wouldn’t this work – as in it definitely doesn’t.

apt install wine-stable installs 64-bit wine; upon querying the version number a most helpful help message helps:

it looks like wine32 is missing, you should install it. multiarch needs to be enabled first. as root, please execute “dpkg –add-architecture i386 && apt-get update && apt-get install wine32

It LoOks LiKe blaBLabLa iS MisSinG… Ok! That’s better, a working copy of wine version 3.0. Unfortunately it doesn’t implement a number of dll calls, without which Python 3.5 install fine, but not 3.7. Yet only the latter is distributed officially. It’s a matter of mixing and matching the right version numbers, but that’s only feasible if we don’t want to stick to official sources. One sudden turn of events after another. Except Bobby Ewing is not gonna wake up.

Fortunately wine 4.0-rc1 is rumored to solve this particular issue. Only caveat is that it has to built from source. The saga continues…

BYORH – Bring Your Own Rabbit Hole

Deeper into the rabbit hole! Compile Wine 4.7 from sources. 64 bit wine doesn’t play nice with Python. Cross compiling for 32 bit target is virtually impossible even after some suggested voodoo. In a nutshell 32 bit and 64 bit development files cannot coexist on the same system. No worries: FROM i386/ubuntu.

Missing dependency drama: much to no surprise at all, the web is full of well meaning bad ideas. Gathering build time dependencies by hand (like animals) is one of them. Spellunking on google reveals various bits of ritual sacrifices, burning hoops to jump through, magic incantations, and endless apt install-s to copypaste. Don’t fall for any of them, the answer is deceivingly simple: enable source code repositories with

• sed -i '/deb-src/s/^# //' /etc/apt/sources.list && apt update and
• apt build-dep wine

That’s it. Yay \o/! Oh, you also need flex, bison, gcc, and build-essentials. Mkay? According to the readme it’s only a matter of

• ./configure
• make
• wait
• make coffee
• create spinny wheel emoticon for team slack
• still compiling
• make install

Chemical X

Wine is installed. Again. This time for real. If you can accept that installed lends itself to a wide variety of definitions. Copied to its place? Yes. Configured and ready to use? Not quite. Wine supports multiple configurations, or wine prefixes. Before wine can be put to use, winecfg has to be run. And winecfg is picky:

• Where is your x server? I can’t live without X! Give me X! Whaaaaa! Fail…

Needy much? Season 7 of the Wine saga continues with an unnecessary X server in a headless (an headless, if you are into french accents) Ubuntu image. This is not a novel problem, it has been done before in a number of different ways. Some work, some feel MacGyvered, some resemble a cul-de-sac.

The dead end: run a vnc server in Ubuntu, connect to it from host. If it seems like an overkill, it is because it is in fact just that. Being able to see what’s happening is only important for development, not for automation. when Mac Os Screen Share kept instantly disconnecting, I abandened this avenue. It has never really been that appealing anyway.

The MacGyver: run XQuartz on host, connect remotely. All you need to do is

• install XQuartz
• xhost + 127.0.0.1 on host to allow remote connections
• export DISPLAY=host.docker.internal:0 in the container

This works fine for development. It’s fine. Really. Todo! Replace later.

On to the next problem: Mono is missing! Install? Ok, Cancel! Gecko is missing! Install? Ok, Cancel! Wine is configured! Ok, cancel! Buttons everywhere, Click-click-click. Python install: next, next, install. More clicks. Problem for tomorrow me: automate the clicks!

Huzzah, finally wine python --version responds with Python 3.7.3 along with a plethora of wine related warning messages. If it was a bicycle, it would be the one from Dennis the Mennace: loud, and on training wheels. But it got you to Mr. Wilson’s house.

Automate the clickety-clackety

Xdotool is a handy choice for automating inputs to X. No error message can deter us now… “YoU nEEd to EnaBle XTesT”. Fine:

• defaults write org.x.X11 enable_test_extensions -boolean true

Automate all the clicks!

waitAndEnter() {
    title="$1"
    local window
    while [[ -z "$window" ]]; do
        sleep 10
        echo "waiting for $title..."
        set +e
        window=`xdotool search --name "$title"`
        set -e
    done

    echo "found: $window"

    xdotool windowfocus --sync $window
    xdotool key Return
}

Finally: it works! It’s alive!

It’s a lie… It only works when the button pressing automagic doohickey is run by hand. It fails when docker build drives it. It appears as though the container is different when created manually via docker commit-s from doing docker build. That’s nonesense. Rummaging through random github issues unrelated to Wine, I came upon a shiny piece of glimmering hope. Maybe a sneaky bastard of a background process doesn’t get to finish before docker moves on to the next build step. That pesky wineserver process, passive aggressive mother of all wine processes. Let’s add a busy wait until it finishes:

while (( $(ps | grep wineserver | grep -vc grep) != 0 )); do
    echo "waiting for wineserver to terminate..."
    sleep 5
done

Going headless

One last puzzle piece: replace XQuartz with Xvfb, X virtual frame buffer. One last bit of sleuthing: config still fails. Enter a suspicios X-related error message from wine: fixme:event:wait_for_withdrawn_state timed out. Lot’s of googling later came the realisation that this is a red hering. Nevertheless one that through trial, error, xdotool man page, and sheer luck lead to the solution. After adding --sync to xdotool windowfocus --sync $window, Python finally agrees to install.

Finally! Bob’s your (weird) uncle.