What goes into making an OnionShare release: Part 1

Posted September 11, 2023 in onionshare code

In the nine years (!) that I've been working on OnionShare, a growing community of contributors have taken on more and more of the work, but I'm still the only one who has actually made any releases. I'm hoping to change that. Even though OnionShare is established open source software, making a release is an extremely cumbersome process. This blog post (and the ones after) documents all the work I'm doing to make the OnionShare 2.6.1 release. This way others who will take over making releases in the future (and anyone interested in releasing open source desktop software) can see what goes into it.

As you'll soon see, it's bonkers how much work it can be. Though to be fair, this release is especially bad. In the course of making it, I've ended up creating a pull request and opening an issue on an upstream project, flatpak-builder-tools. I would love to find secure ways to streamline and automate more of it.

This blog post is stupidly long (and it's only part 1!), so here's a table of contents:

Background on OnionShare desktop releases

Each release involves publishing binaries for Windows, macOS, and Linux:

  • The Windows version for OnionShare 2.6.1 will be 64-bit. All previous Windows versions have been 32-bit, but after upgrading to PySide6, which doesn't support 32-bit Windows, we've decided to abandon support for 32-bit Windows.
  • The macOS version will be for both Intel and Apple Silicon. This will be the first release that supports Apple Silicon Macs--in a future release, we might make a single universal2 binary, but for now they will be separate.
  • For Linux, rather than supporting a myriad of different distros, we're just making releases for Snapcraft and Flatpak, which can be installed in any distro. While Snapcraft and Flatpak packaging can be extremely challenging (as you'll see below), there's a whole different set of challenges of supporting every version of Ubuntu, Debian, Fedora, Arch, and so--especially when you rely on dependencies that aren't packaged in these OSes, or that are only packaged in newer versions (e.g. Ubuntu 23.04) but not older versions (e.g. Ubuntu 20.04).

This blog post is just releases for the desktop version of OnionShare--the iPhone and Android versions have their own separate processes and release schedules.

Another complication is code signing:

  • The Windows .exe files, and the .msi installer, must be digitally signed with a code signing certificate from a trusted Certificate Authority. In previous versions of OnionShare, I've used the Polish CA Certum to get my code signing certificate, since it was inexpensive for open source projects. For this release, we're switching to HARICA, a CA run by Greek universities. The code signing keys for both Certum and HARICA are stored on physical USB smart cards.
  • The macOS app bundle (.app folders) must be code signed using a valid Apple Developer key. I'll be using my same personal Apple Developer account that I've signed previous OnionShare releases with.
  • All of the source and binary packages published to https://onionshare.org/dist/ are also code signed using a PGP key. I've been using my own personal PGP key to sign these, and I will continue to do that for this release. In the future, we might create a new OnionShare PGP signing key that can be shared amongst the people making the releases. Also, I use my PGP key to sign the git tag for the release.

Recently, the new nonprofit Science & Design, founded by Glenn Sorrentino (who designed the beautiful OnionShare UX!), has taken on the role of fiscal sponsor for OnionShare. They're applying for grants and now we have a little bit of funding to actually pay OnionShare developers to keep the project alive and vibrant. The new HARICA Windows code signing key is in Science & Design's name.

OnionShare is complicated software with a lot of dependencies, all of which must be included in each package. The dependencies include:

  • PySide6, which is Python support for the Qt GUI library. OnionShare 2.6 used PySide2, but in this version we've upgraded it to PySide6. In the past (when OnionShare used PyQt5) I've had to build Qt 5 from source, but the process for PySide is simpler, as pre-built binaries can be installed from PyPI.
  • tor and libevent, which is a dependency of tor. The Windows and Mac versions download Tor Browser and extract the binaries. The Snapcraft and Flatpak versions build these from source. They're pretty simple to build from source.
  • obfs4proxy, meek-client, and snowflake-client. These are Tor pluggable transports, required for bypassing censorship when the connection to Tor is blocked. These are all implemented in Go, and they get built from source. This isn't too bad in Windows, macOS, or Snapcraft, but as you'll see, it can be a nightmare in Flatpak.
  • All of the Python dependencies for both the CLI and desktop version, which are managed by Poetry but all get packaged in different ways depending on the platform.

The process of making a release is, basically, to set up a development environment and run the appropriate build scripts on each platform. Since there are several different platforms, in the past I've mostly done this using VMs. However, we've started automating this using a GitHub Actions workflow that builds binaries various platforms. This way I can download those binaries, make sure they work, and code sign them offline.

I have a dedicated older MacBook (with an Intel processor) that I use for making macOS releases. This is the only computer that has copies of my Apple Developer signing keys. I plan on setting up a new Windows 11 VM to make this Windows release--for code signing, I'll use USB passthrough so I can use the smart card with my code signing key in the VM. I also have a newer MacBook Pro with an Apple Silicon processor that I'll need to use (at least somewhat) to make the Apple Silicon release, and an x86-64 computer running Ubuntu, which I can use for making the Snapcraft and Flatpak releases.

OnionShare is not only cross-platform, it's also multilingual. A major task each time I make a release is localization: making sure that all of the languages OnionShare has been translated into it make it into this release, and doing the same for the documentation.

The release process is already meticulously documented in the RELEASE.md file in the git repo--these are the steps that I follow myself each time I make a release. But also, I always end up tweaking and updating this file with each release, and I'm sure I'll do with this release.

Preparing the release

At the top of RELEASE.md (this links to the version of this file when I started the release process) I've documented several steps to take to prepare the release, including:

  • Updating the version string in several files (cli/pyproject.toml, cli/onionshare_cli/resources/version.txt, desktop/pyproject.toml, and others), and update the CHANGELOG.md file
  • Ensuring the documentation is up-to-date
  • Ensuring the localization is up-to-date (OnionShare and its documentation is translated into several languages using Weblate--when people submit new translations, Weblate creates git commits that must be merged into the project)
  • Make sure the Snapcraft packaging works
  • Make sure the Flatpak packaging works

I'm going to have to make some code changes, so I'm starting by creating a new git branch specifically for this release:

git branch release-2.6.1
git checkout release-2.6.1

Here's my pull request for this release: https://github.com/onionshare/onionshare/pull/1749

Next, I'm going through each of the files listed to update the version string to 2.6.1, though in this case I had actually already done this. (I had started making this release months earlier, but then got incredibly busy and never finished.)

Updating the release instructions

RELEASE.md is really a living document. I tend to make changes every time I make a release. As I'm writing this blog post, I noticed that RELEASE.md includes instructions to update to the latest version of Tor, but it doesn't include instructions on updating all of the Python dependencies, which is something I do with each release. So, I'm modifying it.

I'm adding instructions that basically say to change to the cli, desktop, and docs folders and run poetry update. The cli folder has the code for the command line version of OnionShare, the desktop folder has the code for the desktop version, and the docs folder has the code for the documentation website hosted at https://docs.onionshare.org/. Each of these is a separate Python project with dependencies managed by Poetry.

I updated these Python dependencies and made a commit.

RELEASE.md already had a section about updating Tor and also the Tor pluggable transports (tools to bypasses censorship in cases where Tor is blocked) that are built-in to OnionShare, meek, obfs4proxy, and snowflake. Starting with this release, we no longer need to manually update the version of Tor anymore--the desktop/scripts/get-tor.py script, which downloads Tor Browser and extracts the Tor binary from it, had recently been updated to always download the latest version.

On my Mac, I ran the script/get-tor.py script, to download the macOS version of Tor Browser, and to extract the tor and libevent binaries:

$ cd desktop
$ poetry run python scripts/get-tor.py macos
Imported Tor GPG key: ['EF6E286DDA85EA2A4BA7DE684E2C6E8793298290']
Downloading https://dist.torproject.org/torbrowser/12.5.3/TorBrowser-12.5.3-macos_ALL.dmg
Downloading https://dist.torproject.org/torbrowser/12.5.3/TorBrowser-12.5.3-macos_ALL.dmg.asc
Tor Browser verification successful!
Checksumming whole disk (unknown partition : 0)…
..................................................................................................................................................................................
  whole disk (unknown partition : 0): verified   CRC32 $30C00646
verified   CRC32 $8925F2DE
/dev/disk4                                              /Volumes/Tor Browser
Traceback (most recent call last):
  File "/Users/user/code/onionshare/desktop/scripts/get-tor.py", line 343, in <module>
    main()
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/code/onionshare/desktop/scripts/get-tor.py", line 333, in main
    get_tor_macos(gpg, torkey, platform_url, platform_filename, expected_platform_sig)
  File "/Users/user/code/onionshare/desktop/scripts/get-tor.py", line 161, in get_tor_macos
    shutil.copyfile(
  File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/shutil.py", line 256, in copyfile
    with open(src, 'rb') as fsrc:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/Volumes/Tor Browser/Tor Browser.app/Contents/MacOS/Tor/tor.real'

Hmm, I've hit my first problem. The error is: FileNotFoundError: [Errno 2] No such file or directory: '/Volumes/Tor Browser/Tor Browser.app/Contents/MacOS/Tor/tor.real'. Hold on a sec while I look into this...

🎵 Debugging noises... 🎵

In the macOS version of Tor Browser 12.5.3, which is currently the latest version, it turns out that the file Tor Browser.app/Contents/MacOS/Tor/tor.real has been renamed to Tor Browser.app/Contents/MacOS/Tor/tor. The script was trying to copy the invalid path to the Tor binary and was crashing with an error, so I fixed the get-tor.py script to use the new filename.

This is the type of small problem that I typically hit while making a release, and that I go ahead and fix in order to finish the release.

I commited my changes to RELEASE.md.

I also updated the pluggable transports for Windows and macOS. The desktop/scripts folder includes the follow build scripts:

  • build-pt-meek.ps1
  • build-pt-meek.sh
  • build-pt-obfs4proxy.ps1
  • build-pt-obfs4proxy.sh
  • build-pt-snowflake.ps1
  • build-pt-snowflake.sh

The .ps1 files are PowerShell scripts, for doing the Windows builds, and the .sh files are shell scripts, for macOS (and Linux, for dev purposes) builds. Each of these files starts out with a git tag to build. For example, build-pt-meek.sh starts like this:

#!/bin/bash
MEEK_TAG=v0.37.0

I edited all of these, updated them the latest versions, and commited my changes: I updated meek to 0.38.0 and snowflake to 2.6.0--obfs4proxy was still using the latest version.

Updating the change log

Each time I make a release I write an update to CHANGELOG.md to list the new major things have changed. The simplest way to keep track of all of this is via Github milestones. I look at all of the closed issue in the 2.6.1 milestone, and then make sure that the change log includes all of these things in the 2.6.1 section. In this case, the changes are:

  • Release updates: Automate builds with CI, make just 64-bit Windows release, make both Intel and Apple Silicon macOS releases
  • Upgrade dependencies, including Tor, meek, and snowflake
  • Bug fix: Restore the primary_action mode settings in a tab after OnionShare reconnects to Tor
  • Bug fix: Fix issue with auto-connecting to Tor with persistent tabs open
  • Bug fix: Fix packaging issue where Windows version of OnionShare conflicts with Windows version of Dangerzone

Ensuring the documentation is up-to-date

This release doesn't add any new features, so fortunately I don't have to spend any time documenting them all. It's mostly bug fixes, updating dependencies, and revamping how releases are made (thanks to automated builds with a GitHub Actions workflow).

Ensuring the localization is up-to-date

Volunteers on Weblate (often with the help of Localization Lab) translate all of the strings in the OnionShare desktop app into many different languages. Likewise, they also translate the documentation, as well as the strings in the mobile apps. Right now there's no consistent list of languages that OnionShare supports. Basically whenever I make a release, if OnionShare has been at least 90% translated into a language, then we include that language in the release. Otherwise, we don't.

Updating the OnionShare desktop strings

After volunteers translate the English strings into other languages, Weblate makes commits with those new strings into its own git repo at https://hosted.weblate.org/projects/onionshare/translations/. I need to make sure that my local onionshare folder has this as a git remote called weblate, like this:

git remote add weblate https://hosted.weblate.org/projects/onionshare/translations/

Then I pull in all of the latest localization changes from Weblate:

git pull weblate main

That's all I need to do to update the translations for the desktop app. However, I still need to determine which languages have been at least 90% translated so I can know whether or not to include them in the language changing drop-down menu in OnionShare's Settings tab.

Checking for languages with that are at least 90% translated

To check this, I need to run the script docs/check-weblate.py, passing in my Weblate API key. This is a script that uses the Weblate API to determine the percentage of the strings that have been translated into each language, for both the desktop app and the documentation:

$ cd docs
$ poetry run ./check-weblate.py $WEBLATE_API_KEY
GET https://hosted.weblate.org/api/projects/onionshare/languages/
Traceback (most recent call last):
  File "/Users/user/code/onionshare/docs/./check-weblate.py", line 146, in <module>
    asyncio.run(main())
  File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/user/code/onionshare/docs/./check-weblate.py", line 113, in main
    languages[obj["code"]] = obj["language"]
                             ~~~^^^^^^^^^^^^
KeyError: 'language'

And... I've hit another problem! Let me take a minute to debug this.

🎵 Debugging noises... 🎵

Okay, fixed. It turns out the Weblate API has slightly changed since the last time I made a release (it's been almost a year). Making a request to https://hosted.weblate.org/api/projects/onionshare/languages/ returns a list of objects, with each object representing a language that was included in the OnionShare project. Each language object includes the name of the language, like Uyghur, and the language code, like ug. The key for name of the language used to be language, and the key for the language code was code. It turns out, the name of the language is now using the key name instead of language. So, I fixed it by changing this line:

languages[obj["code"]] = obj["language"]

To this:

languages[obj["code"]] = obj["name"]

Okay, the check-weblate.py script should work now, though it takes a long time to finish running. The OnionShare project has 70 languages listed in Weblate, so it makes 70 HTTP requests for the OnionShare desktop app strings, one for each language, and also another 70 HTTP requests for each of the nine pages of OnionShare documentation. In order to avoid hammering the Weblate server, it waits one second between HTTP requests, meaning that it it spends 700 seconds (or 11 minutes and 40 seconds) just waiting between HTTP requests.

Here's what happens when I run it:

$ poetry run ./check-weblate.py $WEBLATE_API_KEY
GET https://hosted.weblate.org/api/projects/onionshare/languages/
GET https://hosted.weblate.org/api/translations/onionshare/translations/af/
GET https://hosted.weblate.org/api/translations/onionshare/translations/sq/
GET https://hosted.weblate.org/api/translations/onionshare/translations/am/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ar/
GET https://hosted.weblate.org/api/translations/onionshare/translations/hy/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ay/
GET https://hosted.weblate.org/api/translations/onionshare/translations/be/
GET https://hosted.weblate.org/api/translations/onionshare/translations/bn/
GET https://hosted.weblate.org/api/translations/onionshare/translations/bs/
GET https://hosted.weblate.org/api/translations/onionshare/translations/bg/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ca/
GET https://hosted.weblate.org/api/translations/onionshare/translations/zh_Hans/
GET https://hosted.weblate.org/api/translations/onionshare/translations/zh_Hant/
GET https://hosted.weblate.org/api/translations/onionshare/translations/hr/
GET https://hosted.weblate.org/api/translations/onionshare/translations/cs/
GET https://hosted.weblate.org/api/translations/onionshare/translations/da/
GET https://hosted.weblate.org/api/translations/onionshare/translations/nl/
GET https://hosted.weblate.org/api/translations/onionshare/translations/en/
GET https://hosted.weblate.org/api/translations/onionshare/translations/eo/ | error 404
GET https://hosted.weblate.org/api/translations/onionshare/translations/fil/
GET https://hosted.weblate.org/api/translations/onionshare/translations/fi/
GET https://hosted.weblate.org/api/translations/onionshare/translations/fr/
GET https://hosted.weblate.org/api/translations/onionshare/translations/gl/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ka/
GET https://hosted.weblate.org/api/translations/onionshare/translations/de/
GET https://hosted.weblate.org/api/translations/onionshare/translations/el/
GET https://hosted.weblate.org/api/translations/onionshare/translations/gu/
GET https://hosted.weblate.org/api/translations/onionshare/translations/he/
GET https://hosted.weblate.org/api/translations/onionshare/translations/hi/
GET https://hosted.weblate.org/api/translations/onionshare/translations/hu/
GET https://hosted.weblate.org/api/translations/onionshare/translations/is/
GET https://hosted.weblate.org/api/translations/onionshare/translations/id/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ga/
GET https://hosted.weblate.org/api/translations/onionshare/translations/it/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ja/
GET https://hosted.weblate.org/api/translations/onionshare/translations/km/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ko/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ckb/
GET https://hosted.weblate.org/api/translations/onionshare/translations/lt/
GET https://hosted.weblate.org/api/translations/onionshare/translations/lg/
GET https://hosted.weblate.org/api/translations/onionshare/translations/mk/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ms/
GET https://hosted.weblate.org/api/translations/onionshare/translations/nb_NO/
GET https://hosted.weblate.org/api/translations/onionshare/translations/om/ | error 404
GET https://hosted.weblate.org/api/translations/onionshare/translations/fa/
GET https://hosted.weblate.org/api/translations/onionshare/translations/pl/
GET https://hosted.weblate.org/api/translations/onionshare/translations/pt_BR/
GET https://hosted.weblate.org/api/translations/onionshare/translations/pt_PT/
GET https://hosted.weblate.org/api/translations/onionshare/translations/pa/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ro/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ru/
GET https://hosted.weblate.org/api/translations/onionshare/translations/sr_Latn/
GET https://hosted.weblate.org/api/translations/onionshare/translations/sn/
GET https://hosted.weblate.org/api/translations/onionshare/translations/si/
GET https://hosted.weblate.org/api/translations/onionshare/translations/sk/
GET https://hosted.weblate.org/api/translations/onionshare/translations/sl/
GET https://hosted.weblate.org/api/translations/onionshare/translations/es/
GET https://hosted.weblate.org/api/translations/onionshare/translations/sw/
GET https://hosted.weblate.org/api/translations/onionshare/translations/sv/
GET https://hosted.weblate.org/api/translations/onionshare/translations/tl/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ta/
GET https://hosted.weblate.org/api/translations/onionshare/translations/te/
GET https://hosted.weblate.org/api/translations/onionshare/translations/bo/
GET https://hosted.weblate.org/api/translations/onionshare/translations/tr/
GET https://hosted.weblate.org/api/translations/onionshare/translations/tk/ | error 404
GET https://hosted.weblate.org/api/translations/onionshare/translations/uk/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ug/
GET https://hosted.weblate.org/api/translations/onionshare/translations/vi/
GET https://hosted.weblate.org/api/translations/onionshare/translations/wo/
GET https://hosted.weblate.org/api/translations/onionshare/translations/yo/
GET https://hosted.weblate.org/api/translations/onionshare/doc-advanced/af/
GET https://hosted.weblate.org/api/translations/onionshare/doc-advanced/sq/
--snip--
GET https://hosted.weblate.org/api/translations/onionshare/doc-tor/wo/
GET https://hosted.weblate.org/api/translations/onionshare/doc-tor/yo/

Traceback (most recent call last):
  File "/Users/user/code/onionshare/docs/./check-weblate.py", line 146, in <module>
    asyncio.run(main())
  File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/user/code/onionshare/docs/./check-weblate.py", line 136, in main
    await app_percent_output(90, 101)
  File "/Users/user/code/onionshare/docs/./check-weblate.py", line 53, in app_percent_output
    app_translations[lang_code] >= percent_min
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: 'eo'

Eek, I've hit another issue. And unfortunately I hit it after making all of those HTTP requests. This means that once I fix it, I'll have to make them all over again. Alright hang on while I fix this...

🎵 Debugging noises... 🎵

Fixed it. The problem here is that Esperanto (the language with code eo) was listed in the OnionShare weblate project, however no one started translating the OnionShare app itself into that language. That's why, if you look back at the output, the Weblate API returned 404 when making the eo request:

GET https://hosted.weblate.org/api/translations/onionshare/translations/eo/ | error 404

The solution here is to just make the check-weblate.py script less brittle by making sure a language exists before checking the percentage it's been translated. I do that by replacing this if statement:

if (
    app_translations[lang_code] >= percent_min
    and app_translations[lang_code] < percent_max
):

With this one:

if (
    lang_code in app_translations
    and app_translations[lang_code] >= percent_min
    and app_translations[lang_code] < percent_max
):

Alright, the third time's a charm:

$ poetry run ./check-weblate.py $WEBLATE_API_KEY
GET https://hosted.weblate.org/api/projects/onionshare/languages/
GET https://hosted.weblate.org/api/translations/onionshare/translations/af/
--snip--
GET https://hosted.weblate.org/api/translations/onionshare/doc-tor/wo/
GET https://hosted.weblate.org/api/translations/onionshare/doc-tor/yo/

App translations >= 90%
=======================
Afrikaans (af), 100.0%
Albanian (sq), 99.2%
Arabic (ar), 100.0%
Belarusian (be), 100.0%
Catalan (ca), 100.0%
Chinese (Simplified) (zh_Hans), 100.0%
Chinese (Traditional) (zh_Hant), 100.0%
Croatian (hr), 98.4%
Czech (cs), 100.0%
English (en), 100.0%
Finnish (fi), 100.0%
French (fr), 100.0%
German (de), 100.0%
Greek (el), 100.0%
Icelandic (is), 100.0%
Italian (it), 91.7%
Japanese (ja), 100.0%
Lithuanian (lt), 99.2%
Norwegian Bokmål (nb_NO), 90.2%
Persian (fa), 98.8%
Polish (pl), 100.0%
Portuguese (Brazil) (pt_BR), 99.6%
Russian (ru), 99.2%
Shona (sn), 98.8%
Spanish (es), 100.0%
Swahili (sw), 99.2%
Swedish (sv), 99.2%
Turkish (tr), 100.0%
Ukrainian (uk), 100.0%
Vietnamese (vi), 100.0%

App translations >= 50%
=======================
Bengali (bn), 88.2%
Danish (da), 67.9%
Dutch (nl), 73.8%
Galician (gl), 82.4%
Indonesian (id), 68.3%
Irish (ga), 53.5%
Khmer (Central) (km), 83.2%
Kurdish (Central) (ckb), 64.0%
Portuguese (Portugal) (pt_PT), 84.7%
Romanian (ro), 50.3%
Serbian (latin) (sr_Latn), 75.7%
Slovak (sk), 64.0%

App translations >= 0%
=======================
Amharic (am), 1.1%
Armenian (hy), 0.0%
Aymara (ay), 0.0%
Bosnian (bs), 0.0%
Bulgarian (bg), 35.1%
Filipino (fil), 0.0%
Georgian (ka), 1.9%
Gujarati (gu), 4.2%
Hebrew (he), 9.3%
Hindi (hi), 39.8%
Hungarian (hu), 28.5%
Korean (ko), 28.1%
Luganda (lg), 0.0%
Macedonian (mk), 2.7%
Malay (ms), 5.0%
Punjabi (pa), 1.1%
Sinhala (si), 0.7%
Slovenian (sl), 10.5%
Tagalog (tl), 0.0%
Tamil (ta), 0.0%
Telugu (te), 48.4%
Tibetan (bo), 0.0%
Uyghur (ug), 0.0%
Wolof (wo), 0.0%
Yoruba (yo), 8.5%

Docs translations >= 90%
========================
English (en), 100%
French (fr), 100%
Greek (el), 100%
Polish (pl), 100%
Spanish (es), 100%
Turkish (tr), 100%
Ukrainian (uk), 100%
Vietnamese (vi), 100%

Docs translations >= 50%
========================
Afrikaans (af), 81%
Chinese (Simplified) (zh_Hans), 59%
Chinese (Traditional) (zh_Hant), 59%
Czech (cs), 59%
Finnish (fi), 69%
German (de), 79%
Italian (it), 76%
Japanese (ja), 60%
Khmer (Central) (km), 77%
Norwegian Bokmål (nb_NO), 76%
Portuguese (Brazil) (pt_BR), 84%
Portuguese (Portugal) (pt_PT), 54%
Russian (ru), 79%
Swahili (sw), 74%
Swedish (sv), 65%

Docs translations >= 0%
========================
Arabic (ar), 40%
Belarusian (be), 33%
Bengali (bn), 29%
Bulgarian (bg), 16%
Catalan (ca), 36%
Croatian (hr), 30%
Dutch (nl), 43%
Filipino (fil), 13%
Galician (gl), 27%
Icelandic (is), 22%
Indonesian (id), 15%
Irish (ga), 22%
Korean (ko), 43%
Kurdish (Central) (ckb), 39%
Lithuanian (lt), 5%
Serbian (latin) (sr_Latn), 31%
Slovak (sk), 30%

Excellent, now I know which languages to enable in the OnionShare app, as well as which languages to enable in the documentation.

Enabling languages in the OnionShare app

I'll add the newly translated languages to OnionShare in a minute, but first I will update the internal lists of country names, each list of countries translated into a different language.

OnionShare has a censorship circumvention workflow that can attempt to automatically connect even you're in a place that blocks access to Tor. For this to work though, it needs to know what country you're in to guess which censorship circumvention technique is most likely to work. It could automatically detect your country based on your IP address, or alternatively you can specify what country you're in. If your language is set to, say, Vietnamese, then when you select your country, the list of country names should be written in Vietnamese. That's why OnionShare includes these country name lists.

To update the translations of country names, I ran this:

poetry run python ./scripts/countries-update-list.py

Great. Next, I want to enable the correct languages. From the check-weblate.py output above, I can tell that I should enable the following languages in the OnionShare desktop app for this release:

  • Afrikaans (af), 100.0%
  • Albanian (sq), 99.2%
  • Arabic (ar), 100.0%
  • Belarusian (be), 100.0%
  • Catalan (ca), 100.0%
  • Chinese (Simplified) (zh_Hans), 100.0%
  • Chinese (Traditional) (zh_Hant), 100.0%
  • Croatian (hr), 98.4%
  • Czech (cs), 100.0%
  • English (en), 100.0%
  • Finnish (fi), 100.0%
  • French (fr), 100.0%
  • German (de), 100.0%
  • Greek (el), 100.0%
  • Icelandic (is), 100.0%
  • Italian (it), 91.7%
  • Japanese (ja), 100.0%
  • Lithuanian (lt), 99.2%
  • Norwegian BokmÃ¥l (nb_NO), 90.2%
  • Persian (fa), 98.8%
  • Polish (pl), 100.0%
  • Portuguese (Brazil) (pt_BR), 99.6%
  • Russian (ru), 99.2%
  • Shona (sn), 98.8%
  • Spanish (es), 100.0%
  • Swahili (sw), 99.2%
  • Swedish (sv), 99.2%
  • Turkish (tr), 100.0%
  • Ukrainian (uk), 100.0%
  • Vietnamese (vi), 100.0%

I do that by editing the self.available_locales dictionary in cli/onionshare_cli/settings.py. In OnionShare 2.6 there were only 10 languages enabled, but in version 2.6.1 there will be 30! The translators have been quite busy. Here's the new self.available_locales that I've added:

# Dictionary of available languages in this version of OnionShare,
# mapped to the language name, in that language
self.available_locales = {
    "af": "Afrikaans",  # Afrikaans
    "sq": "Shqip",  # Albanian
    "ar": "العربية",  # Arabic
    "be": "Беларуская",  # Belarusian
    # "bn": "বাংলা",  # Bengali
    "ca": "Català",  # Catalan
    "zh_Hant": "正體中文 (繁體)",  # Traditional Chinese
    "zh_Hans": "中文 (简体)",  # Simplified Chinese
    "hr": "Hrvatski",  # Croatian
    "cs": "čeština",  # Czech
    # "da": "Dansk",  # Danish
    # "nl": "Nederlands",  # Dutch
    "en": "English",  # English
    "fi": "Suomi",  # Finnish
    "fr": "Français",  # French
    # "gl": "Galego",  # Galician
    "de": "Deutsch",  # German
    "el": "Ελληνικά",  # Greek
    "is": "Íslenska",  # Icelandic
    # "id": "Bahasa Indonesia",  # Indonesian
    # "ga": "Gaeilge",  # Irish
    "it": "Italiano",  # Italian
    "ja": "日本語",  # Japanese
    # "ckb": "Soranî",  # Kurdish (Central)
    "lt": "Lietuvių Kalba",  # Lithuanian
    "nb_NO": "Norsk Bokmål",  # Norwegian Bokmål
    "fa": "فارسی",  # Persian
    "pl": "Polski",  # Polish
    "pt_BR": "Português (Brasil)",  # Portuguese Brazil
    # "pt_PT": "Português (Portugal)",  # Portuguese Portugal
    # "ro": "Română",  # Romanian
    "ru": "Русский",  # Russian
    "sn": "chiShona",  # Shona
    # "sr_Latn": "Srpska (latinica)",  # Serbian (latin)
    # "sk": "Slovenčina",  # Slovak
    "es": "Español",  # Spanish
    "sw": "Kiswahili",  # Swahili
    "sv": "Svenska",  # Swedish
    # "te": "తెలుగు",  # Telugu
    "tr": "Türkçe",  # Turkish
    "uk": "Українська",  # Ukrainian
    "vi": "Tiếng Việt",  # Vietnamese
}

This dictionary maps language codes to languages names, in that language. For most of these languages I had to look up how to write their names here. Note that several languages in this code block are commented out--these are languages that have once been included in OnionShare, but that I'm not enabling in this release since they aren't at least 90% translated this time around.

After making this change, I decided to test it out. I made sure to follow the instructions in desktop/README.md to make sure my computer has a local development environment, and then I ran OnionShare from the source tree:

poetry run onionshare -v

After connecting to Tor, I opened the Settings tab and sure enough, all of these languages are listed in the language dropdown:

The OnionShare language dropdown

I ran OnionShare again, opened Settings, and changed my language to Tiếng Việt (Vietnamese). It prompted me (in Vietnamese) to restart OnionShare, and so I did. I then opened it again, and there we have it: OnionShare in Vietnamese.

OnionShare in Vietnamese

Enabling languages in the documentation

From the check-weblate.py output above, I can tell that I should enable the following languages in the documentation for this release:

  • English (en), 100%
  • French (fr), 100%
  • Greek (el), 100%
  • Polish (pl), 100%
  • Spanish (es), 100%
  • Turkish (tr), 100%
  • Ukrainian (uk), 100%
  • Vietnamese (vi), 100%

I do that by editing docs/source/conf.py and updating the languages list of tuples. This time, the documentation will be translated into 8 languages. Here's the new languages list:

languages = [
    ("English", "en"),  # English
    ("Français", "fr"),  # French
    # ("Deutsch", "de"),  # German
    ("Ελληνικά", "el"),  # Greek
    # ("Italiano", "it"),  # Italian
    # ("日本語", "ja"),  # Japanese
    # ("ភាសាខ្មែរ", "km"),  # Khmer (Central)
    # ("Norsk Bokmål", "nb_NO"),  # Norwegian Bokmål
    ("Polish", "pl"),  # Polish
    # ("Portuguese (Brazil)", "pt_BR"),  # Portuguese (Brazil))
    # ("Русский", "ru"),  # Russian
    ("Español", "es"),  # Spanish
    # ("Svenska", "sv"),  # Swedish
    ("Türkçe", "tr"),  # Turkish
    ("Українська", "uk"),  # Ukrainian
    ("Tiếng Việt", "vi"),  # Vietnamese
]

Unfortunately, in this release I'm commenting out documentation translations for Japanese, Khmer, and Swedish, since they didn't make the 90% threshold.

Next, I also need to edit docs/build.sh and update the LOCALES variable to include language codes for this same list of enabled languages. In this case it's:

LOCALES="en fr el pl es tr uk vi"

Finally, I build the new documentation by running this from the docs folder:

poetry run ./build.sh

This builds the static documentation website (hosted at https://docs.onionshare.org/) for version 2.6.1, and stored it in docs/build/docs/2.6.1/. Here's a screenshot of the the documentation, viewed locally:

Documentation in English

In that screenshot I had clicked the menu button in the bottom left, which shows which languages the documentation is translated into, and lets you switch. Here's a screenshot of the same documentation page, but this time in Vietnamese:

Documentation in English

I've committed all of these changes, and with that, localization for 2.6.1 is finished!

Making sure Snapcraft packaging works

To make sure the Snapcraft package works, I need to switch to my computer running Ubuntu--in my case, I'm running Ubuntu 23.04.

To make the Snapcraft release, I need to update all of the dependencies in the Snapcraft YAML file, snap/snapcraft.yaml, and then build and install a snap, to make sure it all works as expected.

Updating dependencies

I'll start by updating the versions of tor, libevent, obfs4, snowflake-client, and meek-client.

Here's the existing tor part:

tor:
  source: https://dist.torproject.org/tor-0.4.7.12.tar.gz
  source-checksum: sha256/3b5d969712c467851bd028f314343ef15a97ea457191e93ffa97310b05b9e395
  source-type: tar
  plugin: autotools
  autotools-configure-parameters:
    - "--with-libevent-dir=$SNAPCRAFT_PART_INSTALL/../../libevent/install/usr/local"
  build-packages:
    - libssl-dev
    - zlib1g-dev
  after: [libevent]

I want to upgrade to the latest version of Tor, so I'm going to https://dist.torproject.org/ to check to see what that is. At the moment, it looks like the latest release is tor-0.4.8.5.tar.gz. So on my computer, I'll download this file, along with it's checksum, and its PGP signature:

wget https://dist.torproject.org/tor-0.4.8.5.tar.gz
wget https://dist.torproject.org/tor-0.4.8.5.tar.gz.sha256sum
wget https://dist.torproject.org/tor-0.4.8.5.tar.gz.sha256sum.asc

I'm then verifying the checksum's signature:

$ gpg --verify tor-0.4.8.5.tar.gz.sha256sum.asc 
gpg: assuming signed data in 'tor-0.4.8.5.tar.gz.sha256sum'
gpg: Signature made Wed 30 Aug 2023 06:14:29 AM PDT
gpg:                using RSA key B74417EDDF22AC9F9E90F49142E86A2A11F48D36
gpg: Good signature from "David Goulet <[email protected]>" [unknown]
gpg:                 aka "David Goulet <[email protected]>" [unknown]
gpg:                 aka "David Goulet <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: B744 17ED DF22 AC9F 9E90  F491 42E8 6A2A 11F4 8D36
gpg: Signature made Wed 30 Aug 2023 07:14:27 AM PDT
gpg:                using EDDSA key 514102454D0A87DB0767A1EBBE6A0531C18A9179
gpg: Good signature from "Alexander Færøy <[email protected]>" [unknown]
gpg:                 aka "Alexander Færøy <[email protected]>" [unknown]
gpg:                 aka "Alexander Færøy <[email protected]>" [unknown]
gpg:                 aka "Alexander Færøy <[email protected]>" [unknown]
gpg:                 aka "Alexander Færøy <[email protected]>" [unknown]
gpg:                 aka "Alexander Færøy <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 1C1B C007 A9F6 07AA 8152  C040 BEA7 B180 B149 1921
     Subkey fingerprint: 5141 0245 4D0A 87DB 0767  A1EB BE6A 0531 C18A 9179

Then I'm making sure the SHA256 checksum matches:

$ sha256sum --check tor-0.4.8.5.tar.gz.sha256sum
tor-0.4.8.5.tar.gz: OK

It does, so now I'm updating snapcraft.yaml to include the new version of Tor, along with its checksum:

tor:
  source: https://dist.torproject.org/tor-0.4.8.5.tar.gz
  source-checksum: sha256/6957cfd14a29eee7555c52f8387a46f2ce2f5fe7dadf93547f1bc74b1657e119

Next, I'll do something similar for libevent. Here's the libevent part:

libevent:
  source: https://github.com/libevent/libevent/releases/download/release-2.1.12-stable/libevent-2.1.12-stable.tar.gz
  source-checksum: sha256/92e6de1be9ec176428fd2367677e61ceffc2ee1cb119035037a27d346b0403bb
  source-type: tar
  plugin: autotools

Looking at https://github.com/libevent/libevent/releases/ it seems that 2.1.12-stable is still the latest stable version, so I'll leave this part as is.

Next, I'll upgrade obfs4, snowflake-client, and meek-client. These ones are slightly simpler because they're pulled from git, and I just need to update the git tags--the same tags I used in the desktop/scripts/build-pt-* scripts. I'm leaving obfs4 with the source tag obfs4proxy-0.0.14, I'm upgrading snowflake-client to use the source tag v2.6.0, and I'm upgrading meek-client to use the source tag v0.38.0.

I then turn my attention to the onionshare part. While the CLI and desktop version of OnionShare use Poetry to keep track of Python dependencies, I had issues making this work with Snapcraft, so instead I redefine those dependencies in a requirements.txt files. Without going into the nitty gritty details, I basically need to look at all of the dependencies in cli/pyproject.toml and desktop/pyproject.toml and make sure the requirements.txt file in the override-pull section of the onionshare part matches them. Here's my new override-pull section:

override-pull: |
  snapcraftctl pull
  rm pyproject.toml poetry.lock
  cat > requirements.txt << EOF
  # onionshare_cli
  click
  flask==2.3.2
  flask-compress==1.13
  flask-socketio==5.3.4
  psutil
  pysocks
  requests[socks]
  unidecode
  urllib3
  eventlet
  setuptools
  pynacl
  colorama
  gevent-websocket
  stem==1.8.1
  waitress
  werkzeug==2.3.4
  # onionshare
  qrcode
  EOF

Trying to update from PySide2 to PySide6

One of the major changes between OnionShare 2.6 and 2.6.1 is that we've upgraded Qt for Python (the GUI framework that OnionShare uses) from PySide2 to PySide6. PySide2 brings Qt 5.x support to Python, and PySide6 brings Qt 6.x support. The main reason for upgrading is because we wanted to support Apple Silicon Macs: PySide2 does not publish arm64 binaries, which PySide6 does.

In the stage-packages section of the onionshare part, the YAML file lists Ubuntu packages that are required for the app to run, including:

  • python3-pyside2.qtcore
  • python3-pyside2.qtgui
  • python3-pyside2.qtwidgets

Can I just change these pyside2s to pyside6s? Nope. I searched Ubuntu packages and it looks like the equivalent PySide6 packages aren't available. Okay, I will delete these python3-pyside2.* packages from stage-packages, and add PySide6==6.5.2 to the requirements.txt file in the override-pull section, to see if that works.

In order to test, I need to build the snap locally. To do that, first I need to install snapcraft (the tool for building snaps, as opposed to snap, which is the tool for installing already-built snaps) on my computer:

sudo snap install snapcraft --classic

Then I can build it by running:

snapcraft

This takes a long time to run. It launches a new VM, downloads the source code for all of the OnionShare dependencies, compiles them all, and builds OnionShare itself. It's quicker on subsequent runs too. Eventually, it finishes with:

--snip--
Priming onionshare 
+ snapcraftctl prime
This part is missing libraries that cannot be satisfied with any available stage-packages known to snapcraft:
- libQt6Quick3DHelpersImpl.so.6
- libQt6Quick3DSpatialAudio.so.6
- libmysqlclient.so.21
- libxcb-cursor.so.0
- libxkbfile.so.1
These dependencies can be satisfied via additional parts or content sharing. Consider validating configured filesets if this dependency was built.
Snapping |                                                                                                
Snapped onionshare_2.6.1_amd64.snap

Hmm, there's a warning that it's missing Qt6 libraries. That's probably not good.

🎵 Debugging noises... 🎵

The base of this snap is core20, which is based on Ubuntu 20.04 LTS. I searched Ubuntu 20.04 packages for "libqt6" and there no results, but there are some "libqt6" packages in Ubuntu 22.04. If I can't get this to work, I might need to upgrade the snap from core20 to core22, which could come with its own set of problems. Alternatively, I could try compiling Qt6 inside the snap to get the libraries, instead of installing them from the package manager.

Still, I'll install the snap I created and then run it to see what happens.

$ sudo snap install ./onionshare_2.6.1_amd64.snap --devmode
onionshare 2.6.1 installed
$ /snap/bin/onionshare
Warning: Schema “org.gnome.system.locale” has path “/system/locale/”.  Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy” has path “/system/proxy/”.  Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.http” has path “/system/proxy/http/”.  Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.https” has path “/system/proxy/https/”.  Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.ftp” has path “/system/proxy/ftp/”.  Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.socks” has path “/system/proxy/socks/”.  Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Traceback (most recent call last):
  File "/snap/onionshare/x3/bin/onionshare", line 5, in <module>
    from onionshare import main
  File "/snap/onionshare/x3/lib/python3.8/site-packages/onionshare/__init__.py", line 34, in <module>
    from onionshare_cli.common import Common
  File "/snap/onionshare/x3/lib/python3.8/site-packages/onionshare_cli/__init__.py", line 30, in <module>
    from .web import Web
  File "/snap/onionshare/x3/lib/python3.8/site-packages/onionshare_cli/web/__init__.py", line 21, in <module>
    from .web import Web
  File "/snap/onionshare/x3/lib/python3.8/site-packages/onionshare_cli/web/web.py", line 26, in <module>
    from packaging.version import Version
ModuleNotFoundError: No module named 'packaging'

Okay, I'll add packaging to the requirements.txt and try again:

snapcraft # this takes awhile to finish
sudo snap install ./onionshare_2.6.1_amd64.snap --devmode
/snap/bin/onionshare

This time it fails with the error:

python3: symbol lookup error: /snap/onionshare/x4/lib/python3.8/site-packages/PySide6/Qt/plugins/platforms/../../lib/libQt6WaylandClient.so.6: undefined symbol: wl_proxy_marshal_flags

Okay yeah, I expected that it wouldn't work. I'm going to need to get Qt6 installed in this snap.

Updating from core20 to core22

🎵 Debugging noises continue... 🎵

In order to install the Qt6 libraries from Ubuntu packages, I'm going to upgrade this snap to use core22, since Ubuntu 22.04 has Qt6 packages with names like libqt6core6 and libqt6gui6.

While I'm performing this upgrade, I'll be following this official guide on migrating from core20 to core22. Here's what I'm changing:

  • I'm changing base: core20 to base: core22.
  • In all of the parts with an override-pull section, I'm replacing snapcraftctl pull with craftctl default.
  • In my override-build sections, I'm changing the environment variable $SNAPCRAFT_PART_INSTALL to $CRAFT_PART_INSTALL.
  • The onionshare app uses the gnome-3-38 extension, but I'm removing it altogether--if it's required, I can try the gnome extension (which supports core22) or the kde-neon extension

After making these changes, I'm trying again:

$ snapcraft
Traceback (most recent call last):
  File "/snap/snapcraft/9542/bin/snapcraft", line 8, in <module>
    sys.exit(run())
  File "/snap/snapcraft/9542/lib/python3.8/site-packages/snapcraft/cli.py", line 255, in run
    _run_dispatcher(dispatcher, global_args)
  File "/snap/snapcraft/9542/lib/python3.8/site-packages/snapcraft/cli.py", line 228, in _run_dispatcher
    dispatcher.run()
  File "/snap/snapcraft/9542/lib/python3.8/site-packages/craft_cli/dispatcher.py", line 448, in run
    return self._loaded_command.run(self._parsed_command_args)
  File "/snap/snapcraft/9542/lib/python3.8/site-packages/snapcraft/commands/lifecycle.py", line 265, in run
    super().run(parsed_args)
  File "/snap/snapcraft/9542/lib/python3.8/site-packages/snapcraft/commands/lifecycle.py", line 138, in run
    parts_lifecycle.run(self.name, parsed_args)
  File "/snap/snapcraft/9542/lib/python3.8/site-packages/snapcraft/parts/lifecycle.py", line 216, in run
    _run_command(
  File "/snap/snapcraft/9542/lib/python3.8/site-packages/snapcraft/parts/lifecycle.py", line 262, in _run_command
    _run_in_provider(project, command_name, parsed_args)
  File "/snap/snapcraft/9542/lib/python3.8/site-packages/snapcraft/parts/lifecycle.py", line 487, in _run_in_provider
    providers.ensure_provider_is_available(provider)
  File "/snap/snapcraft/9542/lib/python3.8/site-packages/snapcraft/providers.py", line 148, in ensure_provider_is_available
    LXDProvider.ensure_provider_is_available()
  File "/snap/snapcraft/9542/lib/python3.8/site-packages/craft_providers/lxd/lxd_provider.py", line 70, in ensure_provider_is_available
    ensure_lxd_is_ready()
  File "/snap/snapcraft/9542/lib/python3.8/site-packages/craft_providers/lxd/installer.py", line 132, in ensure_lxd_is_ready
    raise errors.LXDError(
craft_providers.lxd.errors.LXDError: LXD requires additional permissions.
Ensure that the user is in the 'lxd' group.
Visit https://linuxcontainers.org/lxd/getting-started-cli/ for instructions on installing and configuring LXD for your operating system.

It turns out, Snapcraft is moving away from VMs to build snaps and towards Linux containers (which honestly is way nicer--this makes it simpler to build snaps inside of VMs, without needing nested VMs). I ran into this a little while I was getting snaps to build in GitHub Actions. Based on this error message, it looks like I need to add my user on my Ubuntu system to the lxd group. I do that by running this:

sudo usermod -a -G lxd $USER

Next, I'm saving all my work and rebooting my Ubuntu computer. After booting back in, I tried again, and realized I had to run this command before it will work:

lxd init --auto

After that, running snapcraft is finally building the OnionShare snap, this time in a container instead of in a VM. Here's the output:

$ snapcraft
Launching instance...
Executed: pull launcher
Executed: pull libevent
Executed: pull meek-client
Executed: pull obfs4
Executed: pull snowflake-client
Executed: pull tor
Executed: pull onionshare-cli
Executed: pull onionshare
Executed: build launcher
Executed: build libevent
Executed: build meek-client
Executed: build obfs4
Executed: build snowflake-client
Executed: skip pull libevent (already ran)
Executed: skip build libevent (already ran)
Executed: stage libevent (required to build 'tor')
Executed: build tor
Executed: skip pull meek-client (already ran)
Executed: skip build meek-client (already ran)
Executed: stage meek-client (required to build 'onionshare-cli')
Executed: skip pull obfs4 (already ran)
Executed: skip build obfs4 (already ran)
Executed: stage obfs4 (required to build 'onionshare-cli')
Executed: skip pull tor (already ran)
Executed: skip build tor (already ran)
Executed: stage tor (required to build 'onionshare-cli')
Executed: skip pull snowflake-client (already ran)
Executed: skip build snowflake-client (already ran)
Executed: stage snowflake-client (required to build 'onionshare-cli')
Executed: build onionshare-cli
Executed: skip pull onionshare-cli (already ran)
Executed: skip build onionshare-cli (already ran)
Executed: stage onionshare-cli (required to build 'onionshare')
Executed: build onionshare
Executed: stage launcher
Executed: skip stage libevent (already ran)
Executed: skip stage meek-client (already ran)
Executed: skip stage obfs4 (already ran)
Executed: skip stage snowflake-client (already ran)
Executed: skip stage tor (already ran)
Executed: skip stage onionshare-cli (already ran)
Executed: stage onionshare
Executed: prime launcher
Executed: prime libevent
Executed: prime meek-client
Executed: prime obfs4
Executed: prime snowflake-client
Executed: prime tor
Executed: prime onionshare-cli
Executed: prime onionshare
Executed parts lifecycle
Generated snap metadata
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtMultimedia.abi3.so'
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtMultimediaWidgets.abi3.so'
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtQml.abi3.so'
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtQuick.abi3.so'
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtSpatialAudio.abi3.so'
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtTextToSpeech.abi3.so'
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtWebEngineCore.abi3.so'
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtWebEngineQuick.abi3.so'
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtWebEngineWidgets.abi3.so'
Lint warnings:
- library: lib/python3.10/site-packages/PySide6/Qt/lib/libQt6WebEngineCore.so.6: missing dependency 'libxkbfile.so.1'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/lib/libQt6WebEngineQuick.so.6: missing dependency 'libxkbfile.so.1'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/lib/libQt6WebEngineWidgets.so.6: missing dependency 'libxkbfile.so.1'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/lib/libQt6XcbQpa.so.6: missing dependency 'libxcb-cursor.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/libexec/QtWebEngineProcess: missing dependency 'libxkbfile.so.1'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/designer/libqwebengineview.so: missing dependency 'libxkbfile.so.1'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/multimedia/libgstreamermediaplugin.so: missing dependency 'libgstallocators-1.0.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/multimedia/libgstreamermediaplugin.so: missing dependency 'libgstapp-1.0.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/multimedia/libgstreamermediaplugin.so: missing dependency 'libgstaudio-1.0.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/multimedia/libgstreamermediaplugin.so: missing dependency 'libgstbase-1.0.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/multimedia/libgstreamermediaplugin.so: missing dependency 'libgstgl-1.0.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/multimedia/libgstreamermediaplugin.so: missing dependency 'libgstpbutils-1.0.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/multimedia/libgstreamermediaplugin.so: missing dependency 'libgstreamer-1.0.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/multimedia/libgstreamermediaplugin.so: missing dependency 'libgstvideo-1.0.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/platforms/libqxcb.so: missing dependency 'libxcb-cursor.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/sqldrivers/libqsqlmysql.so: missing dependency 'libmysqlclient.so.21'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/xcbglintegrations/libqxcb-egl-integration.so: missing dependency 'libxcb-cursor.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/xcbglintegrations/libqxcb-glx-integration.so: missing dependency 'libxcb-cursor.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/qml/QtQuick3D/Helpers/impl/libqtquick3dhelpersimplplugin.so: missing dependency 'libQt6Quick3DHelpersImpl.so.6'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/qml/QtQuick3D/SpatialAudio/libquick3dspatialaudioplugin.so: missing dependency 'libQt6Quick3DSpatialAudio.so.6'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/qml/QtWebEngine/libqtwebenginequickplugin.so: missing dependency 'libxkbfile.so.1'. (https://snapcraft.io/docs/linters-library)
- library: libQt6MultimediaWidgets.so.6: unused library 'lib/python3.10/site-packages/PySide6/Qt/lib/libQt6MultimediaWidgets.so.6'. (https://snapcraft.io/docs/linters-library)
- library: libQt6Quick3DGlslParser.so.6: unused library 'lib/python3.10/site-packages/PySide6/Qt/lib/libQt6Quick3DGlslParser.so.6'. (https://snapcraft.io/docs/linters-library)
- library: libQt6Quick3DIblBaker.so.6: unused library 'lib/python3.10/site-packages/PySide6/Qt/lib/libQt6Quick3DIblBaker.so.6'. (https://snapcraft.io/docs/linters-library)
- library: libEGL_mesa.so.0: unused library 'usr/lib/x86_64-linux-gnu/libEGL_mesa.so.0.0.0'. (https://snapcraft.io/docs/linters-library)
- library: libGLX_mesa.so.0: unused library 'usr/lib/x86_64-linux-gnu/libGLX_mesa.so.0.0.0'. (https://snapcraft.io/docs/linters-library)
- library: libcolordprivate.so.2: unused library 'usr/lib/x86_64-linux-gnu/libcolordprivate.so.2.0.5'. (https://snapcraft.io/docs/linters-library)
- library: libdconf.so.1: unused library 'usr/lib/x86_64-linux-gnu/libdconf.so.1.0.0'. (https://snapcraft.io/docs/linters-library)
- library: libexslt.so.0: unused library 'usr/lib/x86_64-linux-gnu/libexslt.so.0.8.20'. (https://snapcraft.io/docs/linters-library)
- library: libgdk_pixbuf_xlib-2.0.so.0: unused library 'usr/lib/x86_64-linux-gnu/libgdk_pixbuf_xlib-2.0.so.0.4000.2'. (https://snapcraft.io/docs/linters-library)
- library: libicuio.so.70: unused library 'usr/lib/x86_64-linux-gnu/libicuio.so.70.1'. (https://snapcraft.io/docs/linters-library)
- library: libicutest.so.70: unused library 'usr/lib/x86_64-linux-gnu/libicutest.so.70.1'. (https://snapcraft.io/docs/linters-library)
- library: libodbccr.so.2: unused library 'usr/lib/x86_64-linux-gnu/libodbccr.so.2.0.0'. (https://snapcraft.io/docs/linters-library)
- library: libpulse-mainloop-glib.so.0: unused library 'usr/lib/x86_64-linux-gnu/libpulse-mainloop-glib.so.0.0.6'. (https://snapcraft.io/docs/linters-library)
- library: libpulse-simple.so.0: unused library 'usr/lib/x86_64-linux-gnu/libpulse-simple.so.0.1.1'. (https://snapcraft.io/docs/linters-library)
- library: librsvg-2.so.2: unused library 'usr/lib/x86_64-linux-gnu/librsvg-2.so.2.48.0'. (https://snapcraft.io/docs/linters-library)
- library: libssl3.so: unused library 'usr/lib/x86_64-linux-gnu/libssl3.so'. (https://snapcraft.io/docs/linters-library)
- library: libevent_core-2.1.so.7: unused library 'usr/local/lib/libevent_core-2.1.so.7.0.1'. (https://snapcraft.io/docs/linters-library)
- library: libevent_extra-2.1.so.7: unused library 'usr/local/lib/libevent_extra-2.1.so.7.0.1'. (https://snapcraft.io/docs/linters-library)
- library: libevent_openssl-2.1.so.7: unused library 'usr/local/lib/libevent_openssl-2.1.so.7.0.1'. (https://snapcraft.io/docs/linters-library)
- library: libevent_pthreads-2.1.so.7: unused library 'usr/local/lib/libevent_pthreads-2.1.so.7.0.1'. (https://snapcraft.io/docs/linters-library)
Created snap package onionshare_2.6.1_amd64.snap

I installed this snap and tried running it, and this time it crashed with the error:

Failed to create wl_display (No such file or directory)
qt.qpa.plugin: Could not load the Qt platform plugin "wayland" in "" even though it was found.
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vkkhrdisplay, vnc, wayland-egl, wayland, xcb.

Aborted (core dumped)

It looks like I need to install some more dependencies for this to work. The lint warnings give some useful hints as to what they might be. Look at this line:

- library: lib/python3.10/site-packages/PySide6/Qt/lib/libQt6WebEngineCore.so.6: missing dependency 'libxkbfile.so.1'. (https://snapcraft.io/docs/linters-library)

Qt6 is missing the dependency libxkbfile.so.1. In Ubuntu you can search for what package contains that file like this:

$ apt-file search libxkbfile.so.1
libxkbfile1: /usr/lib/x86_64-linux-gnu/libxkbfile.so.1
libxkbfile1: /usr/lib/x86_64-linux-gnu/libxkbfile.so.1.0.2

This means that if I install the package libxkbfile1, it should come with the library I need. Do the same for all of the lint warnings saying I was missing a dependency, I found that I needed to add the following packages to stage-packages:

  • libgstreamer1.0-0
  • libgstreamer1.0-dev
  • libgstreamer-gl1.0-0
  • libgstreamer-plugins-base1.0-0
  • libmysqlclient21
  • libxcb-cursor0
  • libxkbfile1
  • qml6-module-qtquick3d-spatialaudio

I tried to build another snap:

$ snapcraft
Launching instance...
Executed: skip pull launcher (already ran)
Executed: skip pull libevent (already ran)
Executed: skip pull meek-client (already ran)
Executed: skip pull obfs4 (already ran)
Executed: skip pull snowflake-client (already ran)
Executed: skip pull tor (already ran)
Executed: skip pull onionshare-cli (already ran)
Stage package not found in part 'onionshare': qml6-module-qtquick3d-spatialaudio.
Failed to execute pack in instance.
Full execution log: '/home/user/.local/state/snapcraft/log/snapcraft-20230905-172355.921802.log'

After some invesigation, I found that qml6-module-qtquick3d-spatialaudio is in the Ubuntu universe repo, not the main one. I don't think OnionShare actually uses that one though, so I decided to just remove it and try again. This time there are fewer lint warnings about missing dependencies:

Lint warnings:                                                                                                                                                                                                        
- library: lib/python3.10/site-packages/PySide6/Qt/qml/QtQuick3D/Helpers/impl/libqtquick3dhelpersimplplugin.so: missing dependency 'libQt6Quick3DHelpersImpl.so.6'. (https://snapcraft.io/docs/linters-library)       
- library: lib/python3.10/site-packages/PySide6/Qt/qml/QtQuick3D/SpatialAudio/libquick3dspatialaudioplugin.so: missing dependency 'libQt6Quick3DSpatialAudio.so.6'. (https://snapcraft.io/docs/linters-library)

Now let's try running it:

$ /snap/bin/onionshare
╭───────────────────────────────────────────╮
│    *            ▄▄█████▄▄            *    │
│               ▄████▀▀▀████▄     *         │
│              ▀▀█▀       ▀██▄              │
│      *      ▄█▄          ▀██▄             │
│           ▄█████▄         ███        -+-  │
│             ███         ▀█████▀           │
│             ▀██▄          ▀█▀             │
│         *    ▀██▄       ▄█▄▄     *        │
│ *             ▀████▄▄▄████▀               │
│                 ▀▀█████▀▀                 │
│             -+-                     *     │
│   ▄▀▄               ▄▀▀ █                 │
│   █ █     ▀         ▀▄  █                 │
│   █ █ █▀▄ █ ▄▀▄ █▀▄  ▀▄ █▀▄ ▄▀▄ █▄▀ ▄█▄   │
│   ▀▄▀ █ █ █ ▀▄▀ █ █ ▄▄▀ █ █ ▀▄█ █   ▀▄▄   │
│                                           │
│                  v2.6.1                   │
│                                           │
│          https://onionshare.org/          │
╰───────────────────────────────────────────╯

Failed to create wl_display (No such file or directory)
qt.qpa.plugin: Could not load the Qt platform plugin "wayland" in "" even though it was found.
Gtk-Message: 19:24:23.370: Failed to load module "canberra-gtk-module"
Gtk-Message: 19:24:23.370: Failed to load module "canberra-gtk-module"

And it worked!

OnionShare running in Snapcraft

I successfully connected to Tor, and I quickly tested it, and everything appears to work.

But there's one more thing I want to do. In the previous version of the snap package, I had an extra launcher component that set some environment variables before running onionshare or onionshare-cli, but I don't think that's necessary anymore. Instead I can simplify things just by updating the PATH and LD_LIBRARY_PATH to add $SNAP/usr/local paths to them. So, I've deleted the whole launcher part and updated the two apps to no longer use the launcher, and to update environment variables :

apps:
  onionshare:
    common-id: org.onionshare.OnionShare
    command: bin/onionshare
    plugs:
      - desktop
      - home
      - network
      - network-bind
      - removable-media
    environment:
      LANG: C.UTF-8
      PATH: $SNAP/bin:$SNAP/usr/bin:$SNAP/usr/local/bin:$PATH
      LD_LIBRARY_PATH: $LD_LIBRARY_PATH:$SNAP/usr/local/lib

  cli:
    common-id: org.onionshare.OnionShareCli
    command: bin/onionshare-cli
    plugs:
      - home
      - network
      - network-bind
      - removable-media
    environment:
      LANG: C.UTF-8
      PATH: $SNAP/bin:$SNAP/usr/bin:$SNAP/usr/local/bin:$PATH
      LD_LIBRARY_PATH: $LD_LIBRARY_PATH:$SNAP/usr/local/lib

I built another snap, installed it, and tested it one more time, and it worked! Time to commit my code.

Making sure the Flatpak packaging works

With Snapcraft done, let's take a look at Flatpak. The Flatpak manifest file is in flatpak/org.onionshare.OnionShare.yaml. To update the Flatpak package I basically need to upgrade all of the dependencies listed in the manifest, including the URLs to download them from and their sha256 checksums.

But first, I'm going to make sure I have flatpak and flatpak-builder installed, and make sure the Flathub repository is added:

sudo apt install flatpak flatpak-builder
flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo

Updating pyside6

I'll start with the the pyside6 module, since that's at the top (luckily, I had already upgraded the Flatpak packaging from using PySide2 to PySide6). Here's the code for the PySide6 module:

- name: pyside6
  buildsystem: simple
  build-commands: []
  modules:
    - name: pyside6-essentials
      only-arches:
        - x86_64
      buildsystem: simple
      build-commands:
        - pip3 install --verbose --exists-action=i --no-index --find-links="file://${PWD}"
          --prefix=${FLATPAK_DEST} "pyside6-essentials" --no-build-isolation
      sources:
        - type: file
          url: https://files.pythonhosted.org/packages/e5/96/f43cdcb397f8a8cff6991ef8109385cc5ad9b0ad78c6dc2988b3b776fe49/PySide6_Essentials-6.4.2-cp37-abi3-manylinux_2_28_x86_64.whl
          sha256: 8c3d37cca6e27f6da12b50b20e741d593ccc857bdcdb82d97f8f7c8bfe53639a
      modules:
        - name: shiboken6
          buildsystem: simple
          build-commands:
            - pip3 install --verbose --exists-action=i --no-index --find-links="file://${PWD}"
              --prefix=${FLATPAK_DEST} "shiboken6" --no-build-isolation
          sources:
            - type: file
              url: https://files.pythonhosted.org/packages/24/f6/f1fe9220a616789a1c6b1b73670d8b1dec882ac730a8b534f963b3f26182/shiboken6-6.4.2-cp37-abi3-manylinux_2_28_x86_64.whl
              sha256: 0616c1a12d1e51e680595b3940b986275c1df952a751416a0730a59e5b90105f

This essentially installs two Python modules, PySide6-Essentials and shiboken6 from PyPI--these are binary packages that are pre-compiled for x86_64 architectures. Flatpak downloads the whl files from the URLs provided and verifies the sha256 checksums, and runs the build commands to install them. Since pyside6-essentials has shiboken6 as a dependency, shiboken6 is listed under modules under pyside6-essentials.

To update this, I just need to check PyPI for the latest versions of both of these packages and update the URL and sha256 checksums. The latest version of pyside6-essentials is 6.5.2, and the latest version of shiboken6 is also 6.5.2. You can find links to the package files, along with their checksums, on the PyPI pages for those packages under "Download files".

So I updated those. Just to make it clear, here's the diff:

diff --git a/flatpak/org.onionshare.OnionShare.yaml b/flatpak/org.onionshare.OnionShare.yaml
index b455cb71..9d0f0cb5 100644
--- a/flatpak/org.onionshare.OnionShare.yaml
+++ b/flatpak/org.onionshare.OnionShare.yaml
@@ -35,8 +35,8 @@ modules:
             --prefix=${FLATPAK_DEST} "pyside6-essentials" --no-build-isolation
         sources:
           - type: file
-            url: https://files.pythonhosted.org/packages/e5/96/f43cdcb397f8a8cff6991ef8109385cc5ad9b0ad78c6dc2988b3b776fe49/PySide6_Essentials-6.4.2-cp37-abi3-manylinux_2_28_x86_64.whl
-            sha256: 8c3d37cca6e27f6da12b50b20e741d593ccc857bdcdb82d97f8f7c8bfe53639a
+            url: https://files.pythonhosted.org/packages/d0/de/9a089e91c2e0fe4f122218bba4f9dbde46338659f412739bd9db1ed9df4f/PySide6_Essentials-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl
+            sha256: 1620e82b38714a1570b142c01694d0415a25526517b24620ff9b00c9f76cfca9
         modules:
           - name: shiboken6
             buildsystem: simple
@@ -45,8 +45,8 @@ modules:
                 --prefix=${FLATPAK_DEST} "shiboken6" --no-build-isolation
             sources:
               - type: file
-                url: https://files.pythonhosted.org/packages/24/f6/f1fe9220a616789a1c6b1b73670d8b1dec882ac730a8b534f963b3f26182/shiboken6-6.4.2-cp37-abi3-manylinux_2_28_x86_64.whl
-                sha256: 0616c1a12d1e51e680595b3940b986275c1df952a751416a0730a59e5b90105f
+                url: https://files.pythonhosted.org/packages/55/44/d8c366dd4f069166ab9890acb44d004c5e6122714e44c169273dcbbca897/shiboken6-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl
+                sha256: 3fbc35ff3c19e7d39433671bfc1be3d7fa9d071bfdd0ffe1c2a4d27acd6cf6a5
   - name: tor
     buildsystem: autotools
     sources:

Updating tor

The process to upgrade the version of tor installed in the tor module is similar, but this time the buildsystem is set to autotools, which means it will basically run ./configure, make, and make install to compile it from source. The tor module has its own modules section which lists libevent a dependency of tor.

To update it, I just need to update the tor source package URL and sha256 hash, along with the libevent source package URL and sha256 hash. I already did this same thing for the Snapcraft package, so I'm just copying the URLs and sha256 checksums from snapcraft.yaml.

Trying to update obfs4proxy, meek-client, and snowflake-client

These three pluggable transports are all written in Go, and they all have dependencies of their own. This is where I'm going to start relying on flatpak-builder-tools, a collection of scripts that make this work much simpler.

In another folder, I clone the repo:

git clone https://github.com/flatpak/flatpak-builder-tools.git
cd flatpak-builder-tools

This project includes Go Get Generator in the go-get folder, with unfortunately rather convoluted instructions. For each of these go dependencies, I need to:

  • Create a new Flatpak manifest file just for the one go dependency, with network access available during the build, and run the go get command
  • Run flatpak-builder with the --keep-build-dirs flag, which will download all of the dependencies for the go project and keep them when it's done building
  • Run the flatpak-go-get-generator.py script to create a Flatpak manifest file that actually includes all of these dependencies (their git repo URLs and commit IDs)--though, it will generate the manifest in JSON format
  • Convert the manifest it generates from JSON into YAML, and copy and it paste it into the OnionShare Flatpak manifest YAML file

I know from my Snapcraft work that there are no new versions of obfs4proxy, but I do need to upgrade meek-client and snowflake-client. Let's start with meek-client.

I'm starting by making a new file called meek-client.yaml and copying and pasting code from the Go Get Generator readme, except changing the go get build command to point to git.torproject.org/pluggable-transports/meek.git/[email protected].

But then when I tried running the flatpak-builder command, I kept get errors. It's been a long time since I last did this and I don't quite remember how I got it working...

🎵 Debugging noises... 🎵

After much trial and error--including learning that the go get syntax itself is now deprecated in favor of go install (I'm not much of a Go programmer)--I got flatpak-builder to work with this manifest file:

app-id: com.example.meek-client
runtime: org.freedesktop.Platform
runtime-version: '21.08'
sdk: org.freedesktop.Sdk
sdk-extensions:
  - org.freedesktop.Sdk.Extension.golang
modules:
  - name: meek-client
    buildsystem: simple
    build-options:
      append-path: /usr/lib/sdk/golang/bin
      env:
        GOBIN: /app/bin
        GO111MODULE: on
        GOPATH: /run/build/meek-client
      build-args:
        - --share=network
    build-commands:
      - go install git.torproject.org/pluggable-transports/meek.git/[email protected]

I built it with flatpak-builder by running:

$ flatpak-builder build --force-clean --install-deps-from=flathub --keep-build-dirs ./meek-client.yaml
Dependency Sdk: org.freedesktop.Sdk 21.08
Updating org.freedesktop.Sdk/x86_64/21.08

Nothing to do.
Dependency Runtime: org.freedesktop.Platform 21.08
Updating org.freedesktop.Platform/x86_64/21.08

Nothing to do.
Dependency Extension: org.freedesktop.Sdk.Extension.golang 21.08
Updating org.freedesktop.Sdk.Extension.golang/x86_64/21.08

Nothing to do.
Downloading sources
Initializing build dir
Committing stage init to cache
Starting build of com.example.meek-client
========================================================================
Building module meek-client in /home/user/code/flatpak-builder-tools/go-get/.flatpak-builder/build/meek-client-1
========================================================================
Running: go install git.torproject.org/pluggable-transports/meek.git/[email protected]
go: downloading git.torproject.org/pluggable-transports/meek.git v0.38.0
go: downloading git.torproject.org/pluggable-transports/goptlib.git v1.1.0
go: downloading golang.org/x/net v0.0.0-20220909164309-bea034e7d591
go: downloading github.com/refraction-networking/utls v1.1.5
go: downloading golang.org/x/crypto v0.0.0-20220829220503-c86fa9a7ed90
go: downloading github.com/klauspost/compress v1.15.9
go: downloading github.com/andybalholm/brotli v1.0.4
go: downloading golang.org/x/text v0.3.7
go: downloading golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10
debugedit: /home/user/code/flatpak-builder-tools/go-get/.flatpak-builder/rofiles/rofiles-Y6o1sc/files/bin/meek-client: DWARF version 0 unhandled
compressing debuginfo in: /home/user/code/flatpak-builder-tools/go-get/.flatpak-builder/rofiles/rofiles-Y6o1sc/files/bin/meek-client
processing: /home/user/code/flatpak-builder-tools/go-get/.flatpak-builder/rofiles/rofiles-Y6o1sc/files/bin/meek-client
[25] .debug_abbrev compressed -> .zdebug_abbrev (307 => 289 94.14%)
[26] .debug_line compressed -> .zdebug_line (529246 => 485044 91.65%)
[27] .debug_frame compressed -> .zdebug_frame (104484 => 84780 81.14%)
[28] .debug_gdb_scripts NOT compressed, wouldn't be smaller
[29] .debug_info compressed -> .zdebug_info (875910 => 787285 89.88%)
[30] .debug_loc compressed -> .zdebug_loc (671185 => 536563 79.94%)
[31] .debug_ranges compressed -> .zdebug_ranges (177308 => 148978 84.02%)
[9] Updating section string table
stripping /home/user/code/flatpak-builder-tools/go-get/.flatpak-builder/rofiles/rofiles-Y6o1sc/files/bin/meek-client to /home/user/code/flatpak-builder-tools/go-get/.flatpak-builder/rofiles/rofiles-Y6o1sc/files/lib/debug/bin/meek-client.debug
Committing stage build-meek-client to cache
Cleaning up
Committing stage cleanup to cache
Finishing app
Using meek-client as command
Please review the exported files and the metadata
Committing stage finish to cache
Pruning cache

This built this simple Flatpak package, which basically ran go install to download meek-client version 0.38.0, along with all of its dependencies, and then compile them. Now that I've done this, I should be able to use the flatpak-go-get-generator.py script to create the up-to-date meek-client module for me.

Finally, let's see the magic work:

$ python3 flatpak-go-get-generator.py .flatpak-builder/build/meek-client/
Traceback (most recent call last):
  File "/home/user/code/flatpak-builder-tools/go-get/flatpak-go-get-generator.py", line 92, in <module>
    main()
  File "/home/user/code/flatpak-builder-tools/go-get/flatpak-go-get-generator.py", line 82, in main
    source_list = sources(args.build_dir)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/code/flatpak-builder-tools/go-get/flatpak-go-get-generator.py", line 68, in sources
    return list(map(repo_source, repo_paths(build_dir)))
                                 ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/code/flatpak-builder-tools/go-get/flatpak-go-get-generator.py", line 38, in repo_paths
    for domain in domains:
  File "/usr/lib/python3.11/pathlib.py", line 932, in iterdir
    for name in os.listdir(self):
                ^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '.flatpak-builder/build/meek-client/src'

Hmm. While I can see that my .flatpak-builder/build/meek-client folder does indeed have all of the dependencies for meek-client downloaded, there's not a src folder in sight. What's going on?

🎵 Debugging noises get louder... 🎵

So it turns out, due to changes in the go ecosystem, the Go Get Generator is broken beyond repair. It's supposed to, basically, download and build your Go package from source, and then look at all of the dependencies that it had to download and compile the list of Flatpak modules based on the git repos and the specific commits it used. But modern versions of Go doesn't seem to git clone all of the dependencies anymore.

Debugging flatpak-builder-tools

Since I need to figure out how to finish making this Flatpak package, and the Go Get Generator in flatpak-builder-tools is broken, I decided to program my own replacement. I forked the flatpak-builder-tools repo, deleted the broken go-get folder, created a new go folder, and wrote my own new script, flatpak-go-deps.py. Here's my pull request to flatpak-builder-tools project.

It's not merged upstream at the time of writing (it's still a draft PR), but in the meantime you can see the new code I contributed, including the readme, at https://github.com/micahflee/flatpak-builder-tools/tree/fix-go/go. The script I wrote, at the moment, is 282 lines of code.

I'm going to start over with meek-client, but this time using my new script:

$ ./flatpak-go-deps.py git.torproject.org/pluggable-transports/meek.git/[email protected]
go: creating new go.mod: module tempmod
Cloning into 'src/meek-client'...
warning: redirecting to https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/meek.git/
remote: Enumerating objects: 2676, done.
remote: Counting objects: 100% (658/658), done.
remote: Compressing objects: 100% (281/281), done.
remote: Total 2676 (delta 372), reused 658 (delta 372), pack-reused 2018
Receiving objects: 100% (2676/2676), 549.97 KiB | 527.00 KiB/s, done.
Resolving deltas: 100% (1546/1546), done.
Note: switching to 'v0.38.0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 3be00b7 programVersion = "0.38.0"

build-commands:
- . /usr/lib/sdk/golang/enable.sh; export GOPATH=$PWD; export GO111MODULE=off; go
  install git.torproject.org/pluggable-transports/meek.git/meek.git
build-options:
  env:
    GOBIN: /app/bin/
buildsystem: simple
name: meek-client
sources:
- dest: src/git/torproject/org/pluggable-transports/goptlib/git
  tag: v1.1.0
  type: git
  url: https://git.torproject.org/pluggable-transports/goptlib.git.git
- dest: src/github/com/andybalholm/brotli
  tag: v1.0.4
  type: git
  url: https://github.com/andybalholm/brotli.git
- dest: src/github/com/klauspost/compress
  tag: v1.15.9
  type: git
  url: https://github.com/klauspost/compress.git
- dest: src/github/com/refraction-networking/utls
  tag: v1.1.5
  type: git
  url: https://github.com/refraction-networking/utls.git
- dest: src/golang/org/x/crypto
  tag: v0.0.0-20220829220503-c86fa9a7ed90
  type: git
  url: https://golang.org/x/crypto.git
- dest: src/golang/org/x/net
  tag: v0.0.0-20220909164309-bea034e7d591
  type: git
  url: https://golang.org/x/net.git
- dest: src/golang/org/x/sys
  tag: v0.0.0-20220728004956-3c1f35247d10
  type: git
  url: https://golang.org/x/sys.git
- dest: src/golang/org/x/term
  tag: v0.0.0-20210927222741-03fcf44c2211
  type: git
  url: https://golang.org/x/term.git
- dest: src/golang/org/x/text
  tag: v0.3.7
  type: git
  url: https://golang.org/x/text.git
- dest: src/golang/org/x/tools
  tag: v0.0.0-20180917221912-90fa682c2a6e
  type: git
  url: https://golang.org/x/tools.git

I then copy and paste the YAML into my OnionShare Flatpak manifest--though I'm reordering it slightly so name is at the top. This is what it looks like:

- name: meek-client
  build-commands:
  - . /usr/lib/sdk/golang/enable.sh; export GOPATH=$PWD; export GO111MODULE=off; go
    install git.torproject.org/pluggable-transports/meek.git/meek.git
  build-options:
    env:
      GOBIN: /app/bin/
  buildsystem: simple
  sources:
  - dest: src/git/torproject/org/pluggable-transports/goptlib/git
    tag: v1.1.0
    type: git
    url: https://git.torproject.org/pluggable-transports/goptlib.git.git
  - dest: src/github/com/andybalholm/brotli
    tag: v1.0.4
    type: git
    url: https://github.com/andybalholm/brotli.git
  - dest: src/github/com/klauspost/compress
    tag: v1.15.9
    type: git
    url: https://github.com/klauspost/compress.git
  - dest: src/github/com/refraction-networking/utls
    tag: v1.1.5
    type: git
    url: https://github.com/refraction-networking/utls.git
  - dest: src/golang/org/x/crypto
    tag: v0.0.0-20220829220503-c86fa9a7ed90
    type: git
    url: https://golang.org/x/crypto.git
  - dest: src/golang/org/x/net
    tag: v0.0.0-20220909164309-bea034e7d591
    type: git
    url: https://golang.org/x/net.git
  - dest: src/golang/org/x/sys
    tag: v0.0.0-20220728004956-3c1f35247d10
    type: git
    url: https://golang.org/x/sys.git
  - dest: src/golang/org/x/term
    tag: v0.0.0-20210927222741-03fcf44c2211
    type: git
    url: https://golang.org/x/term.git
  - dest: src/golang/org/x/text
    tag: v0.3.7
    type: git
    url: https://golang.org/x/text.git
  - dest: src/golang/org/x/tools
    tag: v0.0.0-20180917221912-90fa682c2a6e
    type: git
    url: https://golang.org/x/tools.git

Next, I'm doing the same with snowflake-client, and also obfs4proxy again for good measure:

./flatpak-go-deps.py git.torproject.org/pluggable-transports/snowflake.git/[email protected]
./flatpak-go-deps.py gitlab.com/yawning/obfs4.git/[email protected]

The one change I needed to do was add the command mv /app/bin/client /app/bin/snowflake-client to the end of build-commands in snowflake-client, since by default the binary it creates is just called client.

This looks like it should work, but I still need to test it to confirm that it actually works. (Spoiler: It doesn't.) But before I can test it by building the Flatpak package, I'm going to finish updating the rest of the Flatpak manifest files, specifically updating the Python dependencies.

Trying to update Python dependencies

Here's my documentation from RELEASE.md for how to go about updating all of the Python dependencies, for both onionshare-cli and onionshare, in the Flatpak manifest file:

pip3 install toml requirements-parser

# clone flatpak-build-tools
git clone https://github.com/flatpak/flatpak-builder-tools.git

# get onionshare-cli dependencies
cd poetry
./flatpak-poetry-generator.py ../../onionshare/cli/poetry.lock
cd ..

# get onionshare dependencies
cd pip
./flatpak-pip-generator $(python3 -c 'import toml; print("\n".join(toml.loads(open("../../onionshare/desktop/pyproject.toml").read())["tool"]["poetry"]["dependencies"]))' |grep -vi onionshare_cli |grep -vi python | grep -vi pyside6 | grep -vi cx_freeze |tr "\n" " ")
cd ..

# convert to yaml
./flatpak-json2yaml.py -o onionshare-cli.yml poetry/generated-poetry-sources.json
./flatpak-json2yaml.py -o onionshare.yml pip/python3-modules.json

Hopefully this will just work with minimal fuss.

I'm going to start with the onionsharea-cli dependencies:

$ ./flatpak-poetry-generator.py ../../onionshare/cli/poetry.lock
Scanning "../../onionshare/cli/poetry.lock" 
Traceback (most recent call last):
  File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 166, in <module>
    main()
  File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 139, in main
    dep_names = get_dep_names(parsed_lockfile, include_devel=include_devel)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 112, in get_dep_names
    package["category"] == "dev"
    ~~~~~~~^^^^^^^^^^^^
KeyError: 'category'

Minimal fuss, it turns out, was too much to hope for. 🎵 Debugging noises... 🎵

Another flatpak-builder-tools rabbit hole

I popped open flatpak-poetry-generator.py to see what the problem is. The exception happened in this function:

def get_dep_names(parsed_lockfile: dict, include_devel: bool = True) -> list:
    """Gets the list of dependency names.

    Args:
        parsed_lockfile (dict): The dictionary of the parsed lockfile.
        include_devel (bool): Include dev dependencies, defaults to True.

    Returns (list): The dependency names.

    """
    dep_names = []
    for section, packages in parsed_lockfile.items():
        if section == "package":
            for package in packages:
                if (
                    package["category"] == "dev"
                    and include_devel
                    and not package["optional"]
                    or package["category"] == "main"
                    and not package["optional"]
                ):
                    dep_names.append(package["name"])
    return dep_names

For debugging purposes, I added the following line to the beginning of the for loop that's looping through packages (before the if statement):

print(json.dumps(package, indent=2))

And I re-ran the script. This is the package that it choked on:

{
  "name": "bidict",
  "version": "0.22.1",
  "description": "The bidirectional mapping library for Python.",
  "optional": false,
  "python-versions": ">=3.7",
  "files": [
    {
      "file": "bidict-0.22.1-py3-none-any.whl",
      "hash": "sha256:6ef212238eb884b664f28da76f33f1d28b260f665fc737b413b287d5487d1e7b"
    },
    {
      "file": "bidict-0.22.1.tar.gz",
      "hash": "sha256:1e0f7f74e4860e6d0943a05d4134c63a2fad86f3d4732fb265bd79e4e856d81d"
    }
  ],
  "extras": {
    "docs": [
      "furo",
      "sphinx",
      "sphinx-copybutton"
    ],
    "lint": [
      "pre-commit"
    ],
    "test": [
      "hypothesis",
      "pytest",
      "pytest-benchmark[histogram]",
      "pytest-cov",
      "pytest-xdist",
      "sortedcollections",
      "sortedcontainers",
      "sphinx"
    ]
  }
}

This package doesn't have a category key. Looking at the if statement, it seems that this basically says append the package to the list of dependencies if it's the dev category, include_devel is true, and the package isn't optional, or if it's in the main category and it's not optional. For this package, category doesn't seem to be set, so I'll modify the if statement like this:

if (
    ("category" not in package and not package["optional"])
    or (
        package["category"] == "dev"
        and include_devel
        and not package["optional"]
    )
    or (package["category"] == "main" and not package["optional"])
):

Now if category isn't set, it adds it to the list anyway. I also added some extra parenthesis to make the logic more clear. Let's see if this did this trick...

$ ./flatpak-poetry-generator.py ../../onionshare/cli/poetry.lock
Scanning "../../onionshare/cli/poetry.lock" 
Traceback (most recent call last):
  File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 168, in <module>
    main()
  File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 157, in main
    sources = get_module_sources(parsed_lockfile, include_devel=include_devel)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 67, in get_module_sources
    package["category"] == "dev"
    ~~~~~~~^^^^^^^^^^^^
KeyError: 'category'

It's a similar error but it's on a different line of code, from this function:

def get_module_sources(parsed_lockfile: dict, include_devel: bool = True) -> list:
    """Gets the list of sources from a toml parsed lockfile.

    Args:
        parsed_lockfile (dict): The dictionary of the parsed lockfile.
        include_devel (bool): Include dev dependencies, defaults to True.

    Returns (list): The sources.

    """
    sources = []
    hash_re = re.compile(r"(sha1|sha224|sha384|sha256|sha512|md5):([a-f0-9]+)")
    for section, packages in parsed_lockfile.items():
        if section == "package":
            for package in packages:
                if (
                    package["category"] == "dev"
                    and include_devel
                    and not package["optional"]
                    or package["category"] == "main"
                    and not package["optional"]
                ):
                    # Check for old metadata format (poetry version < 1.0.0b2)
                    if "hashes" in parsed_lockfile["metadata"]:
                        hashes = parsed_lockfile["metadata"]["hashes"][package["name"]]
                    # Else new metadata format
                    else:
                        hashes = []
                        for package_name in parsed_lockfile["metadata"]["files"]:
                            if package_name == package["name"]:
--snip--

I made the same change there and ran it again:

$ ./flatpak-poetry-generator.py ../../onionshare/cli/poetry.lock
Scanning "../../onionshare/cli/poetry.lock" 
Traceback (most recent call last):
  File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 170, in <module>
    main()
  File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 159, in main
    sources = get_module_sources(parsed_lockfile, include_devel=include_devel)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 81, in get_module_sources
    for package_name in parsed_lockfile["metadata"]["files"]:
                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyError: 'files'

Okay... this time it's crashing because parsed_lockfile["metadata"] doesn't have a files key. You can see this line of code in the code block above. Notice the comments:

  • # Check for old metadata format (poetry version < 1.0.0b2)
  • # Else new metadata format

This makes me think that it's possible there's even yet another new Poetry metadata format that my current poetry.lock file is using, but that flatpak-poetry-generator.py doesn't know about yet, and this is why it's crashing. When I open cli/poetry.lock, it includes a comment at the top saying:

# This file is automatically @generated by Poetry 1.5.1 and should not be changed by hand.

"This is too much work," I thought to myself.

So I decided to shift gears. I opened a bug report in the flatpak-builder-tools repo about this bug I've encountered. I'm going to stop trying to use flatpak-poetry-generator.py and instead just use the Flatpak PIP Generator for this.

Adding Poetry to requirements.txt script

The Flatpak PIP Generator let's you pass in a Python requirements.txt file as input and it generates a Flatpak manifest for those Python dependencies, in JSON format. Instead of trying to deal with poetry.lock files, I decided to write a script that converts the pyproject.toml files (where all of my Poetry dependencies are defined) into requirements.txt files. Here's the flatpak/poetry-to-requirements.py I just wrote:

#!/usr/bin/env python3
import toml
import click


def format_version(dep, version):
    if version == "*":
        return dep
    # If it's a dictionary, assume it's in the format {extras = ["socks"], version = "*"}
    elif isinstance(version, dict) and "version" in version:
        version = version["version"]
        if version == "*":
            return dep
        elif version.startswith("^"):
            return f"{dep}>={version[1:]}.0"
        elif version.startswith((">=", "<=", "!=", "==", "<", ">")):
            return f"{dep}{version}"
        else:
            return f"{dep}=={version}"
    elif version.startswith("^"):
        return f"{dep}>={version[1:]}.0"
    elif version.startswith((">=", "<=", "!=", "==", "<", ">")):
        return f"{dep}{version}"
    else:
        return f"{dep}=={version}"


@click.command()
@click.argument("pyproject_filename")
def poetry_to_requirements(pyproject_filename):
    """Convert poetry dependencies in a pyproject.toml to requirements format."""
    with open(pyproject_filename, "r") as f:
        data = toml.load(f)

    dependencies = data.get("tool", {}).get("poetry", {}).get("dependencies", {})

    requirements = []

    for dep, version in dependencies.items():
        if dep == "python" or dep == "onionshare_cli":
            continue

        formatted = format_version(dep, version)
        if formatted:
            requirements.append(formatted)

    for req in requirements:
        print(req)


if __name__ == "__main__":
    poetry_to_requirements()

Here's the output when I run it on cli/pyproject.toml:

$ ./poetry-to-requirements.py ../cli/pyproject.toml 
click
flask==2.3.2
flask-compress>=1.13.0
flask-socketio==5.3.4
psutil
pysocks
requests
unidecode
urllib3
eventlet
setuptools
pynacl
colorama
gevent-websocket
stem==1.8.1
waitress>=2.1.2.0
werkzeug>=2.3.4

And here's the output when I run it on desktop/pyproject.toml:

$ ./poetry-to-requirements.py ../desktop/pyproject.toml 
PySide6==6.5.2
qrcode
werkzeug
python-gnupg

Excellent. So, now I'm going to use this new script, along with flatpak-pip-generator from flatpak-builder-tools:

$ cd flatpak-builder-tools/pip
$ ./flatpak-pip-generator $(../../onionshare/flatpak/poetry-to-requirements.py ../../onionshare/cli/pyproject.toml)
========================================================================
Downloading sources
========================================================================
Running: "pip3 download --exists-action=i --dest /tmp/pip-generator-python3-modules7s7_v9xw -r /tmp/requirements.gnj3l2ng"
Collecting click
  Using cached click-8.1.7-py3-none-any.whl (97 kB)
Collecting flask==2.3.2
  Using cached Flask-2.3.2-py3-none-any.whl (96 kB)
--snip--
Generating dependencies for stem
Generating dependencies for waitress
Generating dependencies for werkzeug

Output saved to python3-modules.json

It created the file python3-modules.json (in JSON format), so I'm going to use the script that comes with flatpak-builder-tools to convert it to YAML:

../flatpak-json2yaml.py ./python3-modules.json
mv python3-modules.yml onionshare-cli.yaml

Now onionshare-cli.yaml should have the CLI dependencies. While I'm at it, I'll do the same thing for the GUI version:

$ ./flatpak-pip-generator $(../../onionshare/flatpak/poetry-to-requirements.py ../../onionshare/desktop/pyproject.toml)
========================================================================
Downloading sources
========================================================================
Running: "pip3 download --exists-action=i --dest /tmp/pip-generator-python3-modulesjl9ykpn9 -r /tmp/requirements.05kucdaf"
Collecting PySide6==6.5.2
  Downloading PySide6-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl (6.7 kB)
Collecting qrcode
  Downloading qrcode-7.4.2-py3-none-any.whl (46 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.2/46.2 kB 958.3 kB/s eta 0:00:00
Collecting werkzeug
  Using cached werkzeug-2.3.7-py3-none-any.whl (242 kB)
Collecting python-gnupg
  Downloading python_gnupg-0.5.1-py2.py3-none-any.whl (20 kB)
Collecting shiboken6==6.5.2
  Downloading shiboken6-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl (174 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 174.2/174.2 kB 4.0 MB/s eta 0:00:00
Collecting PySide6-Essentials==6.5.2
  Downloading PySide6_Essentials-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl (81.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 81.2/81.2 MB 10.0 MB/s eta 0:00:00
Collecting PySide6-Addons==6.5.2
  Downloading PySide6_Addons-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl (126.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 126.3/126.3 MB 7.9 MB/s eta 0:00:00
Collecting typing-extensions
  Downloading typing_extensions-4.7.1-py3-none-any.whl (33 kB)
Collecting pypng
  Downloading pypng-0.20220715.0-py3-none-any.whl (58 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.1/58.1 kB 3.7 MB/s eta 0:00:00
Collecting MarkupSafe>=2.1.1
  Using cached MarkupSafe-2.1.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28 kB)
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/PySide6-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/PySide6_Addons-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/PySide6_Essentials-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/shiboken6-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/qrcode-7.4.2-py3-none-any.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/werkzeug-2.3.7-py3-none-any.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/python_gnupg-0.5.1-py2.py3-none-any.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/MarkupSafe-2.1.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/pypng-0.20220715.0-py3-none-any.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/typing_extensions-4.7.1-py3-none-any.whl
Successfully downloaded PySide6 PySide6-Addons PySide6-Essentials shiboken6 qrcode werkzeug python-gnupg MarkupSafe pypng typing-extensions
========================================================================
Downloading arch independent packages
========================================================================
Traceback (most recent call last):
  File "/home/user/code/flatpak-builder-tools/pip/./flatpak-pip-generator", line 291, in <module>
    url = get_tar_package_url_pypi(name, version)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/code/flatpak-builder-tools/pip/./flatpak-pip-generator", line 91, in get_tar_package_url_pypi
    raise Exception(err)
Exception: Failed to get shiboken6-6.5.2 source from https://pypi.org/pypi/shiboken6/6.5.2/json

Ahh yes, I knew there was a reason why I manually included PySide6 in the Flatpak manifest earlier--it's because you can't download architecture-independent versions of it from PyPI--it's only available for x86_64. I'll handle this by just grepping PySide6 out of the requirements.txt file:

./flatpak-pip-generator $(../../onionshare/flatpak/poetry-to-requirements.py ../../onionshare/desktop/pyproject.toml | grep -v PySide6)

That worked and generated a new python3-modules.json. Now I'll convert it to YAML too.

../flatpak-json2yaml.py ./python3-modules.json
mv python3-modules.yml onionshare-desktop.yaml

Now that I have onionshare-desktop.yaml and onionshare-cli.yaml, I'm opening these files and copying and pasting the content into my Flatpak manifest for the onionshare and onionshare-cli modules. I'm finally ready to test it!

I updated RELEASE.md to explain all of these new steps, including using the new poetry-to-requirements.py script I wrote, and commited my changes.

Testing Flatpak

Alright, let's build the Flatpak package:

$ flatpak-builder build --force-clean --install-deps-from=flathub --install --user flatpak/org.onionshare.OnionShare.yaml
Dependency Sdk: org.kde.Sdk 6.4
Installing org.kde.Sdk/x86_64/6.4 from flathub
Info: org.kde.Sdk is end-of-life, with reason: We strongly recommend moving to the latest stable version of the Plaform and SDK
Info: org.kde.Sdk.Locale is end-of-life, with reason: We strongly recommend moving to the latest stable version of the Plaform and SDK
Installing runtime/org.freedesktop.Platform.GL.default/x86_64/22.08
Installing runtime/org.freedesktop.Platform.GL.default/x86_64/22.08-extra
^C

I pressed CTRL-C to cancel early because I noticed this warning:

Info: org.kde.Sdk is end-of-life, with reason: We strongly recommend moving to the latest stable version of the Plaform and SDK

When I run flatpak search org.kde.Sdk I see that the latest version of the org.kde.Sdk runtime is 6.5, but my Flatpak manifest file is using 6.4. I updated it to 6.5 and then tried again:

flatpak-builder build --force-clean --install-deps-from=flathub --install --user flatpak/org.onionshare.OnionShare.yaml

Great, it's not showing the warning when I use the new runtime. However, when it got to the "downloading sources" step, it crashed with this error:

Initialized empty Git repository in /home/user/code/onionshare/.flatpak-builder/git/https_filippo.io_edwards25519.git-3QRMA2/
remote: 404 page not found
fatal: repository 'https://filippo.io/edwards25519.git/' not found
Failed to download sources: module obfs4proxy: Child process exited with code 128

Unfortunately it looks like there's an issue with my Go dependencies--which basically means there's an issue with the flatpak-go-deps.py script I wrote in my flatpak-builder-tools PR. 🎵 Debugging noises... 🎵

Fixing flatpak-go-deps.py script

Here's the source from the Flatpak manifest that the flatpak-builder command above choked on:

- dest: src/filippo/io/edwards25519
  tag: v1.0.0-rc.1.0.20210721174708-390f27c3be20
  type: git
  url: https://filippo.io/edwards25519.git

After spending a lot of time looking into how Go package resolution works, and how Go figures out the git URLs for packages, I realized that I needed to update my flatpak-go-deps.py script to make HTTP requests and parse the responses in order to accurately discover git URLs.

It turns out, even though there's a Go package called filippo.io/edwards25519, and Go package names tend to map to git repo URLs, this isn't actually always the case, and https://filippo.io/edwards25519.git is not a valid git repo URL. I needed to make my code make an HTTP requests to https://filippo.io/edwards25519/?go-get=1 and then use BeautifulSoup to parse the response for a go-import meta tag, and that includes the real git URL, which in this case is https://github.com/FiloSottile/edwards25519.

But that wasn't the only problem. There were many more, including:

  • The GitLab server that hosts code for meek-client and snowflake-client, git.torproject.org, didn't seem to work properly with ?go-get=1 requests, so I had to make an exception for that.
  • I discovered that sometimes Go packages are pinned to git tags, but other times they're pinned to individual commits. In the block above it says the tag is v1.0.0-rc.1.0.20210721174708-390f27c3be20, but that's not a real git tag. Instead, it's pinned to a commit, and the short version of the commit ID is 390f27c3be20.
  • I realized this would be much more stable if I just use commit IDs instead of tags for everything, so I started to make the code git clone every repo and check out the correct tag, so I can look up the commit IDs.
  • But then I realized that this takes forever to run and has to download gigabytes of source code, so I streamlined it by using the GitHub API for github.com sources, and the GitLab API for gitlab.com sources. This made it way faster, but I also started hitting GitHub API rate limits, so I added support for passing in a GitHub token to avoid the rate limits.
  • There's a lot more to this particular rabbit hole too, but I'll spare the rest of the details... In short, I still haven't gotten it working all the way. 😭

In all, I spent about 6 hours (!?) fighting with my flatpak-go-deps.py script.

Instead of continuing the suffering, I decided that I just won't update the pluggable transports in the Flatpak version of this release. The versions running in OnionShare 2.6 don't have any security issues, so this will be a future me problem.

After all that work, I went ahead and replaced the obfs4proxy, meek-client, and snowflake-client sections of the Flatpak manifest file with the versions from OnionShare 2.6.

Giving up on Go dependencies, and finishing Flatpak packaging

And now, finally, the Flatpak package actually builds:

$ flatpak-builder build --force-clean --install-deps-from=flathub --install --user flatpak/org.onionshare.OnionShare.yaml
--snip--
Installing app/org.onionshare.OnionShare/x86_64/master
Pruning cache

Here's what happens when I try running it:

$ flatpak run org.onionshare.OnionShare
Traceback (most recent call last):
  File "/app/bin/onionshare", line 33, in <module>
    sys.exit(load_entry_point('onionshare==2.6.1', 'console_scripts', 'onionshare')())
  File "/app/bin/onionshare", line 22, in importlib_load_entry_point
    for entry_point in distribution(dist_name).entry_points
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 969, in distribution
    return Distribution.from_name(distribution_name)
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 548, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for onionshare

Looking at the flatpak-builder logs, the onionshare-cli and onionshare Python packages actually failed with errors. Here are the logs for the onionshare-cli part:

========================================================================
Building module onionshare-cli in /home/user/code/onionshare/.flatpak-builder/build/onionshare-cli-1
========================================================================
Running: cd cli && python3 setup.py install --prefix=${FLATPAK_DEST}
running install
/usr/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
/usr/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://github.com/pypa/setuptools/issues/917 for details.
        ********************************************************************************

!!
  self.initialize_options()
Checking .pth file support in /app/lib/python3.10/site-packages/
/usr/bin/python3 -E -c pass
TEST FAILED: /app/lib/python3.10/site-packages/ does NOT support .pth files
bad install directory or PYTHONPATH

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.  The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /app/lib/python3.10/site-packages/

and your PYTHONPATH environment variable currently contains:

    ''

Here are some of your options for correcting the problem:

* You can choose a different installation directory, i.e., one that is
  on PYTHONPATH or supports .pth files

* You can add the installation directory to the PYTHONPATH environment
  variable.  (It must then also be on PYTHONPATH whenever you run
  Python and want to use the package(s) you are installing.)

* You can set up the installation directory to support ".pth" files by
  using one of the approaches described here:

  https://setuptools.pypa.io/en/latest/deprecated/easy_install.html#custom-installation-locations


Please make the appropriate changes for your system and try again.

Here's the beginning of the onionshare-cli module in the Flatpak manifest:

- name: onionshare-cli
  buildsystem: simple
  build-commands:
    - cd cli && python3 setup.py install --prefix=${FLATPAK_DEST}

According to the error message, running setup.py directly is now deprecated. So, I replaced that build command with:

cd cli && pip3 install --prefix=${FLATPAK_DEST} --no-deps .

The onionshare module (the desktop version of the app) similarly was running setup.py directly, so I replaced that one too.

Then, I tried building the Flatpak package again:

$ flatpak-builder build --force-clean --jobs=$(nproc) --install-deps-from=flathub --install --user flatpak/org.onionshare.OnionShare.yaml
--snip--
========================================================================
Building module onionshare-cli in /home/user/code/onionshare/.flatpak-builder/build/onionshare-cli-1
========================================================================
Running: cd cli && pip3 install --prefix=${FLATPAK_DEST} --no-deps .
Processing /run/build/onionshare-cli/cli
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [7 lines of output]
      WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6294e1bbe0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/poetry-core/
      WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6294e1bf10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/poetry-core/
      WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6294e1bfd0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/poetry-core/
      WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6294e503a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/poetry-core/
      WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6294e50550>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/poetry-core/
      ERROR: Could not find a version that satisfies the requirement poetry-core (from versions: none)
      ERROR: No matching distribution found for poetry-core
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Error: module onionshare-cli: Child process exited with code 1

🎵 Debugging noises drowning out all other sounds... 🎵

It's getting late on a Sunday night, so I think this is a good time to commit my code so far and then give up for the time being.

Pushing back the release date

In a few days, I'm going to Portugal for the Global Gathering Feira where I'll get to hang out with many human rights activists who specialize in internet freedom that I haven't seen in forever, and also meet new ones! During the event, I will do a two-hour project showcase for OnionShare, where people can come by and ask questions about the project. I'll also do a separate project showcase for my upcoming book, Hacks, Leaks, and Revelations: The Art of Analyzing Hacked and Leaked Data.

Last week I decided to sit down and make the OnionShare 2.6.1 release before the Global Gathering. So far I've spent (...checks notes...) 22 hours working on this release (and writing this blog post along with it), and I still haven't even finished Linux packaging yet. It's clear that I'm not going to make my goal of finishing the release before going to Portugal. Instead, I'll finish it when I get back.

I've pushed all of my code:

If you're interested, feel free to fix all of these issues I'm running into while I'm gone! I will happily review your work and merge it into my PR.

And in the meantime, keep an eye out for part 2 of this post, where I will hopefully actually finish the release. I plan on documenting the rest of the process, including how to make polished, code-signed Windows and macOS packages, update the onionshare.org website and the documentation, publish the Homebrew package, and so on.