What goes into making an OnionShare release: Part 2
A few weeks ago I intended to make an OnionShare release, documenting the entire arduous process. I made a lot of progress, but then ran into endless problems getting the Flatpak packaging working and so decided to delay the release. Now I'm back at it. In this post I will finish tackling Flatpak and start tackling the Windows and macOS releases.
If you haven't read part 1 of this series, you might want to check it out now. I describe how I started making the release. I merged in translations from Weblate and made sure the correct translations were enabled for the desktop app and the documentation. After some struggling I got the Snapcraft release working--this involved upgrading the snap base from core20
to core22
so that I could upgrade the from PySide2 to PySide6. I then ran into a wall trying to get Flatpak working.
⚠️ WARNING: This blog post may make you want to smash your face against your keyboard due to the sheer volume of technical issues, error messages, and general injury to morale. I received feedback that the last blog post should have contained a similar warning, so I didn't want to repeat the same mistake.
Like part 1, this blog post is also stupidly long. And also like part 1, I didn't actually finish the release like I was hoping to, so expect a part 3. Here's a table of contents for this post:
- Finishing the Flatpak release
- Updating dependencies again
- Building OnionShare for Windows
- Building OnionShare for macOS
- Time to take a break
I'll start where I left off, with Flatpak.
Finishing the Flatpak release
When I left off a few weeks ago, I had made some progress with the Flatpak release, updating pyside6
and tor
. But I got seriously stuck on compiling obfs4proxy
, meek-client
, and snowflake-client
-- these are all Tor pluggable transports programmed in Go, and they're used to bypass censorship on networks where access to the Tor network is blocked.
When you need to add Go dependencies to Flatpak, you're supposed to use a script in the flatpak-builder-tools project to help you write the Flatpak manifest file, but it turns out that due to changes in the Go ecosystem, the script was entirely broken. I spend many hours completely rewriting it (here is my pull request), but still I ran into problems, and ultimately decided to just not update these dependencies for this release, leaving this as a future Micah problem.
I then found that the Flatpak package was failing to build because of some changes in Python with the onionshare-cli
(the command line version of OnionShare) and onionshare
(the desktop version) parts. It turns out, installing these Python projects like this was deprecated:
python3 setup.py install --prefix=${FLATPAK_DEST}
And I had to replace it with this:
pip3 install --prefix=${FLATPAK_DEST} --no-deps .
After this, I had tried building the Flatpak package, and hit the following error in the onionshare-cli
part:
$ flatpak-builder build --force-clean --jobs=$(nproc) --install-deps-from=flathub --install --user flatpak/org.onionshare.OnionShare.yaml
--snip--
ERROR: Could not find a version that satisfies the requirement poetry-core (from versions: none)
ERROR: No matching distribution found for poetry-core
--snip--
That's where I gave up. Now that I'm back, I have an idea.
Stopping using Poetry with Flatpak
OnionShare uses Poetry to manage its dependencies, but you can't use Poetry with Flatpak. Instead, in the Flatpak manifest file you have to define URLs and SHA256 checksums of every Python dependency, and install them from there.
Since I changed from directly running python3 setup.py install
to using pip3 install
, I think the problem is that pip3
is looking at the pyproject.toml
file for figuring out how to install it, and from there it can see that dependencies are managed by Poetry. I think the simplest solution is to just delete the pyproject.toml
file before running pip3 install
.
So I updated the onionshare-cli
part from this:
- name: onionshare-cli buildsystem: simple build-commands: - cd cli && pip3 install --prefix=${FLATPAK_DEST} --no-deps .
To this:
- name: onionshare-cli buildsystem: simple build-commands: - rm cli/pyproject.toml - cd cli && pip3 install --prefix=${FLATPAK_DEST} --no-deps .
Similarly, for the onionshare
part, I added rm desktop/pyproject.toml
in the build-commands
list.
Then I tried building it again:
$ flatpak-builder build --force-clean --install-deps-from=flathub --install --user flatpak/org.onionshare.OnionShare.yaml
--snip--
Installing app/org.onionshare.OnionShare/x86_64/master
Pruning cache
It worked without error this time. What happens when I run it?
$ flatpak run org.onionshare.OnionShare
Traceback (most recent call last):
File "/app/bin/onionshare", line 5, in <module>
from onionshare import main
File "/app/lib/python3.10/site-packages/onionshare/__init__.py", line 34, in <module>
from onionshare_cli.common import Common
File "/app/lib/python3.10/site-packages/onionshare_cli/__init__.py", line 30, in <module>
from .web import Web
File "/app/lib/python3.10/site-packages/onionshare_cli/web/__init__.py", line 21, in <module>
from .web import Web
File "/app/lib/python3.10/site-packages/onionshare_cli/web/web.py", line 26, in <module>
from packaging.version import Version
ModuleNotFoundError: No module named 'packaging'
Fixing the CLI Python packaging issues
The onionshare-cli
package is missing the packaging
Python dependency. So, I added that dependency to the CLI project:
cd cli poetry add packaging
And then, using the poetry-to-requirements.py
that I programmed and described in part 1, I rebuilt the onionshare-cli
dependencies:
cd ~/code/flatpak-builder-tools/pip/ ./flatpak-pip-generator $(../../onionshare/flatpak/poetry-to-requirements.py ../../onionshare/cli/pyproject.toml) ../flatpak-json2yaml.py ./python3-modules.json mv python3-modules.yml onionshare-cli.yaml
I then updated the Flatpak manifest to use the dependencies in onionshare-cli.yaml
, and rebuilt and installed the Flatpak package, and tried running it again. This time, it failed with another import:
$ flatpak run org.onionshare.OnionShare
Traceback (most recent call last):
File "/app/bin/onionshare", line 5, in <module>
from onionshare import main
File "/app/lib/python3.10/site-packages/onionshare/__init__.py", line 34, in <module>
from onionshare_cli.common import Common
File "/app/lib/python3.10/site-packages/onionshare_cli/__init__.py", line 31, in <module>
from .onion import TorErrorProtocolError, TorTooOldEphemeral, TorTooOldStealth, Onion
File "/app/lib/python3.10/site-packages/onionshare_cli/onion.py", line 27, in <module>
import nacl.public
File "/app/lib/python3.10/site-packages/nacl/public.py", line 16, in <module>
import nacl.bindings
File "/app/lib/python3.10/site-packages/nacl/bindings/__init__.py", line 16, in <module>
from nacl.bindings.crypto_aead import (
File "/app/lib/python3.10/site-packages/nacl/bindings/crypto_aead.py", line 17, in <module>
from nacl._sodium import ffi, lib
ModuleNotFoundError: No module named 'nacl._sodium'
It looks like for some reason the flatpak-builder-tools PIP generator script isn't working to properly install PyNaCl, a cryptography library that OnionShare uses.
While I was writing my first blog post about this release, I opened a bug report in the flatpak-builder-tools repo explaining that the Poetry lockfile generator was broken. Longtime OnionShare developer Miguel Jacq posted a comment on that bug report pointing to a pull request in the flatpak-builder-tools repo which actually fixed the issue, but hadn't yet been approved and merged. (Welcome to the world of open source software development.)
Since I was having trouble with the PIP generator, I figured maybe the patched Poetry lockfile generator would work. So, in my flatpak-builder-tools/poetry
folder I patched flatpak-poetry-generator.py
to match what's in that PR, and then I used it to generate the dependencies:
cd flatpak-builder-tools/poetry ./flatpak-poetry-generator.py --production ../../onionshare/cli/poetry.lock ../flatpak-json2yaml.py generated-poetry-sources.json mv generated-poetry-sources.yml onionshare-cli.yaml
Now, onionshare-cli.yaml
has a block of code that should define all of the dependencies, and it starts like this:
name: poetry-deps buildsystem: simple build-commands: - pip3 install --no-index --find-links="file://${PWD}" --prefix=${FLATPAK_DEST} bidict blinker brotli certifi cffi charset-normalizer click colorama cython dnspython eventlet exceptiongroup flask flask-compress flask-socketio gevent gevent-websocket greenlet idna importlib-metadata iniconfig itsdangerous jinja2 markupsafe packaging pluggy psutil pycparser pynacl pysocks pytest python-engineio python-socketio requests setuptools six stem tomli unidecode urllib3 waitress werkzeug zipp zope-event zope-interface sources: - type: file url: https://files.pythonhosted.org/packages/b5/82/ce0b6380f35f49d3fe687979a324c342cfa3588380232f3801db9dd62f9e/bidict-0.22.1-py3-none-any.whl sha256: 6ef212238eb884b664f28da76f33f1d28b260f665fc737b413b287d5487d1e7b --snip--
I added this to the Flatpak manifest file, replacing the dependencies for the onionshare-cli
, and tried to build the Flatpak package again. This time, it failed to finish building:
$ flatpak-builder build --force-clean --install-deps-from=flathub --install --user flatpak/org.onionshare.OnionShare.yaml
--snip--
Processing ./gevent-23.9.0.post1.tar.gz
Installing build dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [4 lines of output]
Looking in links: file:///run/build/poetry-deps
Processing ./setuptools-68.1.2-py3-none-any.whl
ERROR: Could not find a version that satisfies the requirement Cython>=3.0.2 (from versions: none)
ERROR: No matching distribution found for Cython>=3.0.2
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Error: module poetry-deps: Child process exited with code 1
It needs Cython>=3.0.2
, and somehow the Poetry lockfile generator didn't detect this. No worries, I'll just add it. Back in the cli
folder I added it by running:
$ poetry add Cython
Using version ^3.0.2 for cython
Updating dependencies
Resolving dependencies... Downloading https://files.pythonhosted.org/packages/03/e9/9cc0c4f0d8a566089d0962Resolving dependencies... Downloading https://files.pythonhosted.org/packages/03/e9/9cc0c4f0d8a566089d0962Resolving dependencies... Downloading https://files.pythonhosted.org/packages/03/e9/9cc0c4f0d8a566089d0962Resolving dependencies... (0.5s)
Package operations: 1 install, 0 updates, 0 removals
• Installing cython (3.0.2)
Then I used the Poetry lockfile generator to rebuild the deps, added them to the Flatpak manifest file, and tried building it again. And it failed again. Here's the relevant error message this time:
Processing ./gevent-23.9.0.post1.tar.gz
Installing build dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [8 lines of output]
Looking in links: file:///run/build/poetry-deps
Processing ./setuptools-68.1.2-py3-none-any.whl
Processing ./Cython-3.0.2-py2.py3-none-any.whl
Processing ./cffi-1.15.1.tar.gz
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
ERROR: Could not find a version that satisfies the requirement greenlet>=3.0rc1 (from versions: 2.0.2)
ERROR: No matching distribution found for greenlet>=3.0rc1
[end of output]
Ugh. Okay instead of tediously documenting each little thing I try, I'm going to take a break from blogging and focus on just fixing this problem...
🎵 Debugging noises... 🎵
Got it working! Ultimately, I had to add the following Poetry dependencies to the onionshare-cli
project:
packaging = "^23.1"
gevent = "^23.9.1"
wheel = "^0.41.2"
cffi = "^1.15.1"
cython = "^3.0.2"
Also, after running poetry add
to add dependencies, for some reason greenlet
kept getting upgraded to version 3.0.0rc3. But when I run poetry update
, it would then downgrade to the latest stable version, 2.0.2, which is what I want. I don't understand the byzantine network of dependency relationships that Poetry is unraveling, but at least I got it working.
And I also had to modify the output from the Poetry lockfile generator. It kept crashing while trying to installing the brotlicffi
package with an error message that included this:
********************************************************************************
Requirements should be satisfied by a PEP 517 installer.
If you are using pip, you can try `pip install --use-pep517`.
********************************************************************************
So, I modified the onionshare-cli
part to first explicitly install brotlicffi
by running this:
pip3 install --no-index --find-links="file://${PWD}" --prefix=${FLATPAK_DEST} --use-pep517 brotlicffi
And then continue on with installing the rest of the Python packages like normal. Here's what the onionshare-cli
part of the Flatpak manifest file looked like:
- name: onionshare-cli buildsystem: simple build-commands: - rm cli/pyproject.toml - cd cli && pip3 install --prefix=${FLATPAK_DEST} --no-deps . sources: - type: dir path: .. # - type: git # url: https://github.com/onionshare/onionshare.git # tag: v2.6 modules: - name: poetry-deps buildsystem: simple build-commands: - pip3 install --no-index --find-links="file://${PWD}" --prefix=${FLATPAK_DEST} --use-pep517 brotlicffi - pip3 install --no-index --find-links="file://${PWD}" --prefix=${FLATPAK_DEST} bidict blinker brotli certifi cffi charset-normalizer click colorama cython dnspython eventlet exceptiongroup flask flask-compress flask-socketio gevent gevent-websocket greenlet greenlet h11 idna importlib-metadata iniconfig itsdangerous jinja2 markupsafe packaging pluggy psutil pycparser pynacl pysocks pytest python-engineio python-socketio requests setuptools simple-websocket six stem tomli unidecode urllib3 waitress werkzeug wheel wsproto zipp zope-event zope-interface sources: - type: file url: https://files.pythonhosted.org/packages/b5/82/ce0b6380f35f49d3fe687979a324c342cfa3588380232f3801db9dd62f9e/bidict-0.22.1-py3-none-any.whl sha256: 6ef212238eb884b664f28da76f33f1d28b260f665fc737b413b287d5487d1e7b - type: file url: https://files.pythonhosted.org/packages/0d/f1/5f39e771cd730d347539bb74c6d496737b9d5f0a53bc9fdbf3e170f1ee48/blinker-1.6.2-py3-none-any.whl sha256: c3d739772abb7bc2860abf5f2ec284223d9ad5c76da018234f6f50d6f31ab1f0 --snip--
And now, remarkably, I can build the Flatpak package and it actually runs!
$ flatpak-builder build --force-clean --install-deps-from=flathub --install --user flatpak/org.onionshare.OnionShare.yaml
--snip--
Installing app/org.onionshare.OnionShare/x86_64/master
Pruning cache
$ flatpak run org.onionshare.OnionShare
╭───────────────────────────────────────────╮
│ * ▄▄█████▄▄ * │
│ ▄████▀▀▀████▄ * │
│ ▀▀█▀ ▀██▄ │
│ * ▄█▄ ▀██▄ │
│ ▄█████▄ ███ -+- │
│ ███ ▀█████▀ │
│ ▀██▄ ▀█▀ │
│ * ▀██▄ ▄█▄▄ * │
│ * ▀████▄▄▄████▀ │
│ ▀▀█████▀▀ │
│ -+- * │
│ ▄▀▄ ▄▀▀ █ │
│ █ █ ▀ ▀▄ █ │
│ █ █ █▀▄ █ ▄▀▄ █▀▄ ▀▄ █▀▄ ▄▀▄ █▄▀ ▄█▄ │
│ ▀▄▀ █ █ █ ▀▄▀ █ █ ▄▄▀ █ █ ▀▄█ █ ▀▄▄ │
│ │
│ v2.6.1 │
│ │
│ https://onionshare.org/ │
╰───────────────────────────────────────────╯
I tried out a few things. I connected to Tor, I shared some files with myself and downloaded them using Tor Browser, and I tested connecting to Tor using a bridge. It all works! I think, finally, I have finished Flatpak packaging.
Updating dependencies again
I already did this in part 1, but it's been a few weeks and a lot of new versions of dependencies have been released. So I went ahead and updated all of the Poetry deps like this:
cd cli poetry update cd ../desktop poetry update cd ../docs poetry update
This is especially important because OnionShare depends on gevent
which just had a critical privilege escalation vulnerability (though we confirmed that OnionShare itself isn't vulnerable to it).
Building OnionShare for Windows
Making a release is always painful, but this one has been much more painful than normal. With my Flatpak woes out of the way, I'm hoping for smooth sailing going forward. In order to reduce the pain, I've tried to automate as much of the Windows and macOS build process as I can using a GitHub Actions workflow. Basically, when I push new commits to a PR on GitHub, a robot goes to work trying to build binaries for me for 64-bit Windows, Intel macOS, and also Snapcraft and Flatpak for Linux.
After I pushed my latest commit to my PR branch, the GitHub Actions workflow was triggered. It successfully built Snapcraft and Flatpak binaries, but it failed at building Windows and macOS binaries.
When I click on the failed build-win64
job for details, I see that it failed on the "Build snowflake" step:
Debugging Windows build in GitHub Actions
Let me back up and explain what's going on. In the OnionShare git repo there's a file called .github/workflows/build.yml
that defines code that tells VMs how to build these binaries. Here's the beginning of the code for the build-win64
job:
build-win64: runs-on: windows-latest steps: - uses: actions/checkout@v3 - name: Install python uses: actions/setup-python@v4 with: python-version: '3.10.11' - name: Install poetry run: C:\hostedtoolcache\windows\Python\3.10.11\x64\python -m pip install poetry - name: Restore cache - poetry uses: actions/cache@v3 with: path: ~\AppData\Local\pypoetry\Cache\virtualenvs key: ${{ runner.os }}-win64-poetry-${{ hashFiles('desktop/poetry.lock') }} - name: Install poetry dependencies run: | cd desktop C:\hostedtoolcache\windows\Python\3.10.11\x64\Scripts\poetry install C:\hostedtoolcache\windows\Python\3.10.11\x64\Scripts\poetry env list --full-path - name: Restore cache - tor uses: actions/cache@v3 with: path: desktop\build\tor key: ${{ runner.os }}-win64-tor-${{ hashFiles('desktop/scripts/get-tor.py') }} - name: Get tor binaries from Tor Browser (64-bit) run: cd desktop && C:\hostedtoolcache\windows\Python\3.10.11\x64\Scripts\poetry run python .\scripts\get-tor.py win64
Basically, GitHub Actions starts a Windows VM that's preloaded with development tools and then runs these steps one at a time. First it checks out the git repo, then installs Python 3.10.11, then using that version of Python it installs Poetry. It then installs the Poetry dependencies, and runs the desktop/scripts/get-tor.py
Python script to download Tor Browser for Windows and extract the binaries.
Before various steps this job also restores some caches. This saves time. For example, if I push a new commit and desktop/poetry.lock
hasn't changed, then it just uses the Poetry virtualenv from the previous run instead of having to re-download and install all of the Poetry deps.
This is the step that it's failing on:
- name: Build snowflake shell: pwsh run: | if ((Test-Path -Path 'desktop\onionshare\resources\tor\snowflake-client.exe') -eq $True) { Write-Output "snowflake already built" } else { cd desktop .\scripts\build-pt-snowflake.ps1 }
This is a small PowerShell script. It basically checks to see if snowflake-client.exe
already exists (if the it was cached from a previous job) and if so outputs a message that it's already built. Otherwise, it runs the scripts/build-pt-snowflake.ps1
PowerShell script to compile it.
Here's the content of the build-pt-snowflake.ps1
script:
$env:SNOWFLAKE_TAG = 'v2.6.0' New-Item -ItemType Directory -Force -Path .\build\snowflake cd .\build\snowflake git clone https://git.torproject.org/pluggable-transports/snowflake.git cd snowflake git checkout $SNOWFLAKE_TAG go build .\client Move-Item -Path .\client.exe -Destination ..\..\..\onionshare\resources\tor\snowflake-client.exe
This creates a folder, clones the Snowflake source code into it, checks out the correct branch for the version we want to build, compiles it with Go, and then moves the binary.
And here's the error message from this step:
Directory: D:\a\onionshare\onionshare\desktop\build
Mode LastWriteTime Length Name
---- ------------- ------ ----
d---- 9/28/2023 1:45 AM snowflake
Cloning into 'snowflake'...
warning: redirecting to https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake.git/
Your branch is up to date with 'origin/main'.
go: errors parsing go.mod:
D:\a\onionshare\onionshare\desktop\build\snowflake\snowflake\go.mod:5: unknown directive: toolchain
Move-Item: D:\a\onionshare\onionshare\desktop\scripts\build-pt-snowflake.ps1:9
Line |
9 | Move-Item -Path .\client.exe -Destination ..\..\..\onionshare\resourc …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Cannot find path 'D:\a\onionshare\onionshare\desktop\build\snowflake\snowflake\client.exe' because it does not
| exist.
Error: Process completed with exit code 1.
It looks like there was an error trying to compile Snowflake, and so moving the binary failed because the binary was never created.
I don't currently have a Windows VM for OnionShare development, so I'm going to create a new one and see if I can reproduce this issue.
Setting up Windows for OnionShare development
I downloaded the Windows 11 ISO from Microsoft. In case you don't know, Microsoft lets you download ISOs for Windows which you can then install in VMs or on computers, for free. When you're installing it, you can just say you don't have a license key, and it lets you continue. You can use Windows as much as you want without paying for it, but certain features are disabled, such as the ability to change your desktop wallpaper. For the purposes of software development, you don't need any of those features.
I installed VirtualBox, created a new Windows 11 VM, and booted to it, installing Windows 11 Pro. After setting up an account, I installed all of the updates, so now I have an up-to-date Windows 11 box:
Then, I follow the instructions in desktop/README.md
to set up the development environment, specifically following the Windows instructions. This includes:
- Installing Git for Windows.
- Cloning the OnionShare git repo, and checking out the
release-2.6.1
branch. - Installing Python for Windows.
- Installing Poetry and the Poetry dependencies. In PowerShell:
pip install poetry cd code\onionshare\desktop poetry install
- Installing Microsoft C++ Build Tools, specifically "Desktop development with C++".
- Installing 7-Zip and gpg4win, and also adding both of these to the path so I can run them from PowerShell.
- Running the
get-tor.py
script to download Tor Browser and extract the binaries:cd code\onionshare\desktop poetry run python .\scripts\get-tor.py win64
- Installing Go.
The next step in the instructions is to run the PowerShell scripts that build obfs4proxy, snowflake, and meek. This is also the step where the GitHub Actions job failed. So, I'll start with obfs4proxy:
When I tried running the obfs4proxy build script I got the following error:
PS C:\Users\dev\code\onionshare\desktop> .\scripts\build-pt-obfs4proxy.ps1
.\scripts\build-pt-obfs4proxy.ps1 : File
C:\Users\dev\code\onionshare\desktop\scripts\build-pt-obfs4proxy.ps1 cannot be loaded because running
scripts is disabled on this system. For more information, see about_Execution_Policies at
https:/go.microsoft.com/fwlink/?LinkID=135170.
At line:1 char:1
+ .\scripts\build-pt-obfs4proxy.ps1
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : SecurityError: (:) [], PSSecurityException
+ FullyQualifiedErrorId : UnauthorizedAccess
This is because by default in Windows, PowerShell scripts can't just get executed. To fix this, I opened a PowerShell window as an administrator and ran Set-ExecutionPolicy -ExecutionPolicy Bypass
:
PS C:\Windows\system32> Get-ExecutionPolicy
Restricted
PS C:\Windows\system32> Set-ExecutionPolicy -ExecutionPolicy Bypass
Execution Policy Change
The execution policy helps protect you from scripts that you do not trust. Changing the execution policy might expose
you to the security risks described in the about_Execution_Policies help topic at
https:/go.microsoft.com/fwlink/?LinkID=135170. Do you want to change the execution policy?
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "N"): Y
PS C:\Windows\system32>
Then, back in my original PowerShell window, I ran .\scripts\build-pt-obfs4proxy.ps1
again, and this time it worked: it downloaded the obfs4proxy source, compiled it, and then moved the binary into the correct place in the OnionShare folder.
Now, let's try building snowflake, which is the one that failed in GitHub Actions. I ran ./scripts/build-pt-snowflake.sh
... and it worked fine. Okay, I'll have to figure out what's going on in GitHub Actions still. In the meantime, I also ran ./scripts/build-pt-meek.sh
to build meek, and that worked too.
And that's it, the development environment is set up. I tried running OnionShare with poetry run onionshare -v
, and it worked!
Right now this is just running OnionShare from the Python source tree--there's another step involved for turning it into a .exe
executable and building the installer. But before doing that, I need to figure out how to fix the GitHub Actions job.
Debugging Windows build in GitHub Actions some more
Taking a closer look at the error message from the GitHub Actions log, this is where it's failing to build snowflake:
go: errors parsing go.mod:
D:\a\onionshare\onionshare\desktop\build\snowflake\snowflake\go.mod:5: unknown directive: toolchain
Here's the top of desktop\build\snowflake\snowflake\go.mod
:
module gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/v2 go 1.21 toolchain go1.21.1
The build-win64
job uses the windows-latest
GitHub runner, which is Windows Server 2022. Looking up the documentation, it comes with Go 1.20.8 installed. My guess is I can fix this by installing the latest version of Go in the workflow.
I did this by adding the following step to .github/workflows/build.yml
:
- name: Install Go >=1.21.1 uses: actions/setup-go@v4 with: go-version: '>=1.21.1' - run: go version
I committed my code, pushed it to the release-2.6.1
branch, and GitHub Actions re-ran. This time, the build-win64
job finished! (The build-mac-intel
job failed, but I'll work on that in a minute.)
Testing the Windows binary from GitHub Actions
This workflow also generated some artifacts, including a win64-build
one:
I downloaded the artifact, win64-build.zip
, and unzipped it. Inside is a file called onionshare-win64.zip
, which I copied to my VM and unzipped there. Here's the folder with the OnionShare build:
I double-clicked on onionshare.exe
, and it worked! An OnionShare window opened. I went ahead and connected to Tor, created an anonymous drop box, installed Tor Browser, and loaded my drop box to make sure it all worked.
And for good measure, I also tested the CLI version of OnionShare in Windows:
Great, Windows works! I'll still need to code sign and package it for distribution, but before I do that I want to create a signed git tag, and before I do that I want to make sure that the release is completely ready. I'll come back to Windows in a bit, but for now it's time to move on to macOS.
Building OnionShare for macOS
Just like the Windows release, the GitHub Actions workflow also builds a binary for macOS using the build-mac-intel
job. At the moment this just builds the binary for Intel Macs. It can't yet make binaries for Apple Silicon Macs (M1 and M2 ARM64 processors).
I just looked it up, and according to GitHub's roadmap they'll be adding support for Apple Silicon GitHub Action runners in Q4 of 2023. This means that for now I'll need to manually build the Apple Silicon binaries from my own Apple Silicon Mac, but I should be able to use the Intel binaries built by the workflow.
Anyway, back to the build-mac-intel
job. Let's see why it was failing.
It failed on the "Build OnionShare" step, which is the final step before compressing the binary and saving it as a GitHub Actions artifact. There are 38,000 log messages though and it's not immediately clear exactly what caused it to fail.
I'm going to set up a macOS development environment for OnionShare and see if I can manually reproduce the problem.
Setting up macOS for OnionShare development
I have an old early 2015 MacBook Pro (with an Intel processor) that I use as my dedicated macOS software release laptop. Just like for Windows, the instructions for setting up the macOS dev environment are in desktop/README.md
.
I've already set up a dev environment on this computer in the past, but I'm going to update it for this release:
- I powered this computer on for the first time in many months and installed macOS updates. It runs macOS 12.7, the latest version that Apple still supports for this hardware. I also installed Homebrew updates.
- This computer has Python 3.10.8 installed, so while I'm at it I updated it to the latest version, Python 3.11.5. Then I installed Poetry.
- I switched to the
release-2.6.1
branch of in myonionshare
source tree. - I installed the latest dependencies with Poetry.
- I got the latest -Tor binaries by running:
cd desktop poetry run python ./scripts/get-tor.py macos
- I upgraded the version of Go I had installed, and then built all of the pluggable transports by running:
./scripts/build-pt-obfs4proxy.sh ./scripts/build-pt-snowflake.sh ./scripts/build-pt-meek.sh
And that's it, the development environment is set up. I tried running OnionShare with poetry run onionshare -v
, and it worked!
Building the macOS app bundle
The GitHub Actions job failed on the "Build OnionShare" step, which is defined as this:
- name: Build OnionShare run: | cd desktop /Library/Frameworks/Python.framework/Versions/3.10/bin/poetry run python ./setup-freeze.py build /Library/Frameworks/Python.framework/Versions/3.10/bin/poetry run python ./setup-freeze.py bdist_mac /Library/Frameworks/Python.framework/Versions/3.10/bin/poetry run python ./scripts/build-macos.py cleanup-build
Or more simply:
poetry run python ./setup-freeze.py build poetry run python ./setup-freeze.py bdist_mac poetry run python ./scripts/build-macos.py cleanup-build
The Windows and Mac versions of OnionShare use software called cx_Freeze to "freeze" the Python code into executable binaries, .exe
files in Windows and Mach-O files in macOS. The build
step freezes the Python into binaries and the bdist_mac
step creates a macOS app bundle (a .app
folder, the kind that you drag into Applications). And the last ./scripts/build-macos.py cleanup-build
script deletes a bunch of unused stuff in from the app bundle, making the final file size much smaller.
I will try to manually run each of these steps on my old Mac and see if I can reproduce the problem:
% poetry run python ./setup-freeze.py build
running build
running build_py
creating build/lib
creating build/lib/onionshare
copying onionshare/tor_settings_tab.py -> build/lib/onionshare
--snip--
copying /Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/PySide6/Qt/plugins/sqldrivers/libqsqlodbc.dylib -> /Users/user/code/onionshare/desktop/build/exe.macosx-10.9-universal2-3.11/lib/PySide6/Qt/plugins/sqldrivers/libqsqlodbc.dylib
copying /usr/local/opt/libiodbc/lib/libiodbc.2.dylib -> /Users/user/code/onionshare/desktop/build/exe.macosx-10.9-universal2-3.11/lib/libiodbc.2.dylib
error: [Errno 2] No such file or directory: '/usr/local/opt/libiodbc/lib/libiodbc.2.dylib'
Aha, an error! It's trying to copy /usr/local/opt/libiodbc/lib/libiodbc.2.dylib
into the build folder but failing because that file doesn't exist. Maybe I can just install it and then try again?
I ran brew search libiodbc
and found that there's a package with that name, so I tried installing it:
brew install libiodbc
Afer installing it, I confirmed that the file /usr/local/opt/libiodbc/lib/libiodbc.2.dylib
exists. So, I ran the build script again:
% poetry run python ./setup-freeze.py build
running build
--snip--
copying /Applications/Postgres.app/Contents/Versions/14/lib/libpq.5.dylib -> /Users/user/code/onionshare/desktop/build/exe.macosx-10.9-universal2-3.11/lib/libpq.5.dylib
error: [Errno 2] No such file or directory: '/Applications/Postgres.app/Contents/Versions/14/lib/libpq.5.dylib'
Another missing dependency. Ahh, and this must be why I made this step in the build-macos-intel
job:
- name: Install cx_Freeze/PySide6 build dependencies run: | brew install libiodbc cd ~/Downloads curl -O -L https://github.com/PostgresApp/PostgresApp/releases/download/v2.5.12/Postgres-2.5.12-14.dmg hdiutil attach Postgres-2.5.12-14.dmg cp -r /Volumes/Postgres-2.5.12-14/Postgres.app /Applications/ hdiutil detach /Volumes/Postgres-2.5.12-14
It looks like I need to install the Postgres app, which PySide6 pulls a library out of. And hey, now that I look at it, that step also includes brew install libiodbc
already--this must not be the first time I've debugged this specific issue.
Looking at https://github.com/PostgresApp/PostgresApp/releases I see that the latest version of the Postgres app, using Postgres 14, is 2.6.5. So I went ahead and updated that:
- name: Install cx_Freeze/PySide6 build dependencies run: | brew install libiodbc cd ~/Downloads curl -O -L https://github.com/PostgresApp/PostgresApp/releases/download/v2.6.5/Postgres-2.6.5-14.dmg hdiutil attach Postgres-2.6.5-14.dmg cp -r /Volumes/Postgres-2.6.5-14/Postgres.app /Applications/ hdiutil detach /Volumes/Postgres-2.6.5-14
I installed this version of Postgres on my build machine, and ran the build script again. This time it succeeded!
So I moved on to the bdist_mac
command, which should build the app bundle:
% poetry run python ./setup-freeze.py bdist_mac
running bdist_mac
running build_exe
creating directory /Users/user/code/onionshare/desktop/build/exe.macosx-10.9-universal2-3.11/lib
copying /Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/cx_Freeze/bases/lib/Python -> /Users/user/code/onionshare/desktop/build/exe.macosx-10.9-universal2-3.11/lib/Python
--snip--
Resolved rpath:
/Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/PySide6
/Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/PySide6/Qt/lib
Loaded libraries:
@rpath/libshiboken6.abi3.6.5.dylib -> None
@rpath/QtCore.framework/Versions/A/QtCore -> /Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/PySide6/Qt/lib/QtCore.framework/Versions/A/QtCore
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit -> /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration -> /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
/usr/lib/libc++.1.dylib -> /usr/lib/libc++.1.dylib
/usr/lib/libSystem.B.dylib -> /usr/lib/libSystem.B.dylib
error: [Errno 2] No such file or directory: '/Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/libpyside6.abi3.6.5.dylib'
Ok, another error message. This time it's complaining that a library in the app bundle, OnionShare.app/Contents/MacOS/libpyside6.abi3.6.5.dylib
, doesn't exist. I checked to see if that file exists anywhere in the app bundle, and it does in three different places in fact:
% find build/OnionShare.app | grep libpyside6.abi3.6.5.dylib
build/OnionShare.app/Contents/Resources/lib/PySide6/libpyside6.abi3.6.5.dylib
build/OnionShare.app/Contents/Resources/lib/libpyside6.abi3.6.5.dylib
build/OnionShare.app/Contents/Resources/libpyside6.abi3.6.5.dylib
🎵 Debugging noises... 🎵
Ok I have a lead. The setup-freeze.py
file is a cx_Freeze setup script, and looking in it I have various platform-specific modifications depending on which platform I'm trying to freeze:
if platform.system() == "Windows": include_msvcr = True gui_base = "Win32GUI" # gui_base = None exec_icon = os.path.join("onionshare", "resources", "onionshare.ico") elif platform.system() == "Darwin": import PySide6 import shiboken6 include_msvcr = False gui_base = None exec_icon = None include_files += [ ( os.path.join(PySide6.__path__[0], "libpyside6.abi3.6.5.dylib"), "libpyside6.abi3.6.5.dylib", ), ( os.path.join(shiboken6.__path__[0], "libshiboken6.abi3.6.5.dylib"), "libshiboken6.abi3.6.5.dylib", ), ]
When platform.system() == "Darwin"
(e.g. when this script is being run on macOS), it adds some extra files to be included. I can't remember why I originally added this hack (probably it was required to get the build to succeed) but I'm going to try just deleting it and see if that helps. I've updated the macOS specific code to just be:
elif platform.system() == "Darwin": include_msvcr = False gui_base = None exec_icon = None
And then I'm running the bdist_mac
command again... and it finished without an errors!
Now I'll run the last command:
% poetry run python ./scripts/build-macos.py cleanup-build
> Delete unused Qt Frameworks
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/lib/PySide6/Qt/lib/QtMultimediaQuick.framework
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/lib/PySide6/Qt/lib/QtQuickControls2.framework
--snip--
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/lib/PySide6/QtWebEngineQuick.pyi
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/lib/PySide6/Qt/lib/QtWebEngineQuickDelegatesQml.framework
> Move files around so Apple will notarize
Traceback (most recent call last):
File "/Users/user/code/onionshare/desktop/./scripts/build-macos.py", line 364, in <module>
main()
File "/Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/code/onionshare/desktop/./scripts/build-macos.py", line 186, in cleanup_build
os.rename(
IsADirectoryError: [Errno 21] Is a directory: '/Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/lib' -> '/Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/Resources/lib'
The build-macos.py cleanup-build
script does a few things:
- Deletes unused Qt frameworks to save space. This seems to finish successfully.
- Moves files around because this was necessary to notarize the app bundle (part of the final code signing step).
- Deletes more unusued PySide6 files to save even more space.
Here's the beginning of the moving-files-around part of the code:
print("> Move files around so Apple will notarize") # https://github.com/marcelotduarte/cx_Freeze/issues/594 # https://gist.github.com/TechnicalPirate/259a9c24878fcad948452cb148af2a2c#file-custom_bdist_mac-py-L415 # Move lib from MacOS into Resources os.rename( f"{app_path}/Contents/MacOS/lib", f"{app_path}/Contents/Resources/lib", ) run( ["ln", "-s", "../Resources/lib"], cwd=f"{app_path}/Contents/MacOS", )
The cx_Freeze bug that was referenced was just closed 3 weeks ago! So probably this moving files around stuff is no longer necessary. I'll try just ripping it all out and see what happens... First I ran the bdist_mac
command again to make a fresh version of the app bundle, and then I ran the cleanup script again, this time without the code to move files around:
% poetry run python ./scripts/build-macos.py cleanup-build
> Delete unused Qt Frameworks
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/lib/PySide6/Qt/lib/QtMultimediaQuick.framework
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/lib/PySide6/Qt/lib/QtQuickControls2.framework
--snip--
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/lib/PySide6/QtWebEngineQuick.pyi
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/lib/PySide6/Qt/lib/QtWebEngineQuickDelegatesQml.framework
> Delete more unused PySide6 stuff to save space
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/Resources/lib/PySide6/Designer.app
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/Resources/lib/PySide6/examples
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/Resources/lib/PySide6/glue
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/Resources/lib/PySide6/include
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/Resources/lib/PySide6/lupdate
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/Resources/lib/PySide6/libpyside6.abi3.6.4.dylib
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/Resources/lib/PySide6/Qt/qml
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/Resources/lib/shiboken6/libshiboken6.abi3.6.4.dylib
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/Resources/lib/PySide6/Assistant.app
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/Resources/lib/PySide6/Linguist.app
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/Resources/lib/PySide6/libpyside6qml.abi3.6.4.dylib
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/Resources/lib/PySide6/lrelease
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/Resources/lib/PySide6/qmlformat
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/Resources/lib/PySide6/qmllint
Deleted: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/Resources/lib/PySide6/qmlls
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtBluetooth
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtConcurrent
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtDesigner
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtNetworkAuth
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtNfc
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtOpenGL
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtOpenGLWidgets
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtPositioning
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtQuick3D
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtQuick3DRuntimeRender
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtQuick3DUtils
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtShaderTools
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtStateMachine
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtSvgWidgets
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtWebChannel
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtWebEngineCore
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtWebEngineQuick
Cannot delete, filename not found: /Users/user/code/onionshare/desktop/build/OnionShare.app/Contents/MacOS/QtXml
> Freed 847 mb
Nice, it succeeded this time! However, I'm using a newer version of PySide6 that I was in the last release, and it seems that many of the files it's trying to delete don't exist anymore. So I edited the build-macos.py
script to remove the code that was giving errors.
I should more thoroughly check to see if there are other files I can delete from the app bundle too. But first, I'll try running the app bundle:
% ./build/OnionShare.app/Contents/MacOS/onionshare -v
zsh: segmentation fault ./build/OnionShare.app/Contents/MacOS/onionshare -v
And it crashes... Okay, let's try again. I rebuilt the app bundle, but this time didn't run the cleanup script. Can I run the app bundle now without it crashing? ...nope. It crashes even before running the cleanup script.
I pushed the changes that I've made so far to the release-2.6.1
branch and the build-mac-intel
job actually finished and created a binary in GitHub Actions, though I wouldn't be surprised if the binary it made crashes just like this one.
I'm going to need to spend some more time fixing this, but for now I think it's time to take a break.
Time to take a break
I've made a lot more solid progress, but this post is getting long. Here's what I want to make sure to get done next time:
- Finish making the macOS app bundle work on my old Mac, and make sure the GitHub Actions workflow creates a working Mac app bundle too.
- Create a working macOS app bundle on my Apple Silicon Mac too.
- Update the version of Python in the GitHub Actions workflow to 3.11.5--I realized I haven't done that yet.
- Once I think everything is ready, do a final test of the binaries created by GitHub Actions, including the Snapcraft and Flatpak packages.
- Get the other OnionShare devs to review my pull request and merge it into
main
. - Create the PGP-signed git tag for
v2.6.1
. - Final packaging and code signing for the Windows release, using the new HARICA smart card that I've never used before.
- Final packaging and code signing for the macOS releases, both Intel and the new Apple Silicon version.
- Publish the release, which includes:
- Gather all the binaries for the different platform and create detached PGP signatures for them all.
- Create a new release on GitHub and upload all of the binaries and their PGP sigs.
- Publish the release to PyPI, so people can install the CLI app that way.
- Update the app in Homebrew, and also figure out how this is going to work with separate Intel and Apple Silicon versions.
- Update the onionshare.org website, including uploading all of the binaries and sigs to https://onionshare.org/dist/, updating the documentation website, and updating https://onionshare.org/latest-version.txt, which the Windows and macOS versions use to detect the latest version and prompt for updates.
That feels like more than enough to save for a future blog post.