What goes into making an OnionShare release: Part 1
In the nine years (!) that I've been working on OnionShare, a growing community of contributors have taken on more and more of the work, but I'm still the only one who has actually made any releases. I'm hoping to change that. Even though OnionShare is established open source software, making a release is an extremely cumbersome process. This blog post (and the ones after) documents all the work I'm doing to make the OnionShare 2.6.1 release. This way others who will take over making releases in the future (and anyone interested in releasing open source desktop software) can see what goes into it.
As you'll soon see, it's bonkers how much work it can be. Though to be fair, this release is especially bad. In the course of making it, I've ended up creating a pull request and opening an issue on an upstream project, flatpak-builder-tools. I would love to find secure ways to streamline and automate more of it.
This blog post is stupidly long (and it's only part 1!), so here's a table of contents:
- Background on OnionShare desktop releases
- Preparing the release
- Updating the release instructions
- Updating the change log
- Ensuring the documentation is up-to-date
- Ensuring the localization is up-to-date
- Making sure Snapcraft packaging works
- Making sure the Flatpak packaging works
- Debugging flatpak-builder-tools
- Pushing back the release date
Background on OnionShare desktop releases
Each release involves publishing binaries for Windows, macOS, and Linux:
- The Windows version for OnionShare 2.6.1 will be 64-bit. All previous Windows versions have been 32-bit, but after upgrading to PySide6, which doesn't support 32-bit Windows, we've decided to abandon support for 32-bit Windows.
- The macOS version will be for both Intel and Apple Silicon. This will be the first release that supports Apple Silicon Macs--in a future release, we might make a single universal2 binary, but for now they will be separate.
- For Linux, rather than supporting a myriad of different distros, we're just making releases for Snapcraft and Flatpak, which can be installed in any distro. While Snapcraft and Flatpak packaging can be extremely challenging (as you'll see below), there's a whole different set of challenges of supporting every version of Ubuntu, Debian, Fedora, Arch, and so--especially when you rely on dependencies that aren't packaged in these OSes, or that are only packaged in newer versions (e.g. Ubuntu 23.04) but not older versions (e.g. Ubuntu 20.04).
This blog post is just releases for the desktop version of OnionShare--the iPhone and Android versions have their own separate processes and release schedules.
Another complication is code signing:
- The Windows
.exefiles, and the.msiinstaller, must be digitally signed with a code signing certificate from a trusted Certificate Authority. In previous versions of OnionShare, I've used the Polish CA Certum to get my code signing certificate, since it was inexpensive for open source projects. For this release, we're switching to HARICA, a CA run by Greek universities. The code signing keys for both Certum and HARICA are stored on physical USB smart cards. - The macOS app bundle (
.appfolders) must be code signed using a valid Apple Developer key. I'll be using my same personal Apple Developer account that I've signed previous OnionShare releases with. - All of the source and binary packages published to https://onionshare.org/dist/ are also code signed using a PGP key. I've been using my own personal PGP key to sign these, and I will continue to do that for this release. In the future, we might create a new OnionShare PGP signing key that can be shared amongst the people making the releases. Also, I use my PGP key to sign the git tag for the release.
Recently, the new nonprofit Science & Design, founded by Glenn Sorrentino (who designed the beautiful OnionShare UX!), has taken on the role of fiscal sponsor for OnionShare. They're applying for grants and now we have a little bit of funding to actually pay OnionShare developers to keep the project alive and vibrant. The new HARICA Windows code signing key is in Science & Design's name.
OnionShare is complicated software with a lot of dependencies, all of which must be included in each package. The dependencies include:
- PySide6, which is Python support for the Qt GUI library. OnionShare 2.6 used PySide2, but in this version we've upgraded it to PySide6. In the past (when OnionShare used PyQt5) I've had to build Qt 5 from source, but the process for PySide is simpler, as pre-built binaries can be installed from PyPI.
torandlibevent, which is a dependency oftor. The Windows and Mac versions download Tor Browser and extract the binaries. The Snapcraft and Flatpak versions build these from source. They're pretty simple to build from source.obfs4proxy,meek-client, andsnowflake-client. These are Tor pluggable transports, required for bypassing censorship when the connection to Tor is blocked. These are all implemented in Go, and they get built from source. This isn't too bad in Windows, macOS, or Snapcraft, but as you'll see, it can be a nightmare in Flatpak.- All of the Python dependencies for both the CLI and desktop version, which are managed by Poetry but all get packaged in different ways depending on the platform.
The process of making a release is, basically, to set up a development environment and run the appropriate build scripts on each platform. Since there are several different platforms, in the past I've mostly done this using VMs. However, we've started automating this using a GitHub Actions workflow that builds binaries various platforms. This way I can download those binaries, make sure they work, and code sign them offline.
I have a dedicated older MacBook (with an Intel processor) that I use for making macOS releases. This is the only computer that has copies of my Apple Developer signing keys. I plan on setting up a new Windows 11 VM to make this Windows release--for code signing, I'll use USB passthrough so I can use the smart card with my code signing key in the VM. I also have a newer MacBook Pro with an Apple Silicon processor that I'll need to use (at least somewhat) to make the Apple Silicon release, and an x86-64 computer running Ubuntu, which I can use for making the Snapcraft and Flatpak releases.
OnionShare is not only cross-platform, it's also multilingual. A major task each time I make a release is localization: making sure that all of the languages OnionShare has been translated into it make it into this release, and doing the same for the documentation.
The release process is already meticulously documented in the RELEASE.md file in the git repo--these are the steps that I follow myself each time I make a release. But also, I always end up tweaking and updating this file with each release, and I'm sure I'll do with this release.
Preparing the release
At the top of RELEASE.md (this links to the version of this file when I started the release process) I've documented several steps to take to prepare the release, including:
- Updating the version string in several files (
cli/pyproject.toml,cli/onionshare_cli/resources/version.txt,desktop/pyproject.toml, and others), and update theCHANGELOG.mdfile - Ensuring the documentation is up-to-date
- Ensuring the localization is up-to-date (OnionShare and its documentation is translated into several languages using Weblate--when people submit new translations, Weblate creates git commits that must be merged into the project)
- Make sure the Snapcraft packaging works
- Make sure the Flatpak packaging works
I'm going to have to make some code changes, so I'm starting by creating a new git branch specifically for this release:
git branch release-2.6.1 git checkout release-2.6.1
Here's my pull request for this release: https://github.com/onionshare/onionshare/pull/1749
Next, I'm going through each of the files listed to update the version string to 2.6.1, though in this case I had actually already done this. (I had started making this release months earlier, but then got incredibly busy and never finished.)
Updating the release instructions
RELEASE.md is really a living document. I tend to make changes every time I make a release. As I'm writing this blog post, I noticed that RELEASE.md includes instructions to update to the latest version of Tor, but it doesn't include instructions on updating all of the Python dependencies, which is something I do with each release. So, I'm modifying it.
I'm adding instructions that basically say to change to the cli, desktop, and docs folders and run poetry update. The cli folder has the code for the command line version of OnionShare, the desktop folder has the code for the desktop version, and the docs folder has the code for the documentation website hosted at https://docs.onionshare.org/. Each of these is a separate Python project with dependencies managed by Poetry.
I updated these Python dependencies and made a commit.
RELEASE.md already had a section about updating Tor and also the Tor pluggable transports (tools to bypasses censorship in cases where Tor is blocked) that are built-in to OnionShare, meek, obfs4proxy, and snowflake. Starting with this release, we no longer need to manually update the version of Tor anymore--the desktop/scripts/get-tor.py script, which downloads Tor Browser and extracts the Tor binary from it, had recently been updated to always download the latest version.
On my Mac, I ran the script/get-tor.py script, to download the macOS version of Tor Browser, and to extract the tor and libevent binaries:
$ cd desktop
$ poetry run python scripts/get-tor.py macos
Imported Tor GPG key: ['EF6E286DDA85EA2A4BA7DE684E2C6E8793298290']
Downloading https://dist.torproject.org/torbrowser/12.5.3/TorBrowser-12.5.3-macos_ALL.dmg
Downloading https://dist.torproject.org/torbrowser/12.5.3/TorBrowser-12.5.3-macos_ALL.dmg.asc
Tor Browser verification successful!
Checksumming whole disk (unknown partition : 0)…
..................................................................................................................................................................................
whole disk (unknown partition : 0): verified CRC32 $30C00646
verified CRC32 $8925F2DE
/dev/disk4 /Volumes/Tor Browser
Traceback (most recent call last):
File "/Users/user/code/onionshare/desktop/scripts/get-tor.py", line 343, in <module>
main()
File "/Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/Library/Caches/pypoetry/virtualenvs/onionshare-aqknF-N0-py3.11/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/user/code/onionshare/desktop/scripts/get-tor.py", line 333, in main
get_tor_macos(gpg, torkey, platform_url, platform_filename, expected_platform_sig)
File "/Users/user/code/onionshare/desktop/scripts/get-tor.py", line 161, in get_tor_macos
shutil.copyfile(
File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/shutil.py", line 256, in copyfile
with open(src, 'rb') as fsrc:
^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/Volumes/Tor Browser/Tor Browser.app/Contents/MacOS/Tor/tor.real'
Hmm, I've hit my first problem. The error is: FileNotFoundError: [Errno 2] No such file or directory: '/Volumes/Tor Browser/Tor Browser.app/Contents/MacOS/Tor/tor.real'. Hold on a sec while I look into this...
🎵 Debugging noises... 🎵
In the macOS version of Tor Browser 12.5.3, which is currently the latest version, it turns out that the file Tor Browser.app/Contents/MacOS/Tor/tor.real has been renamed to Tor Browser.app/Contents/MacOS/Tor/tor. The script was trying to copy the invalid path to the Tor binary and was crashing with an error, so I fixed the get-tor.py script to use the new filename.
This is the type of small problem that I typically hit while making a release, and that I go ahead and fix in order to finish the release.
I commited my changes to RELEASE.md.
I also updated the pluggable transports for Windows and macOS. The desktop/scripts folder includes the follow build scripts:
build-pt-meek.ps1build-pt-meek.shbuild-pt-obfs4proxy.ps1build-pt-obfs4proxy.shbuild-pt-snowflake.ps1build-pt-snowflake.sh
The .ps1 files are PowerShell scripts, for doing the Windows builds, and the .sh files are shell scripts, for macOS (and Linux, for dev purposes) builds. Each of these files starts out with a git tag to build. For example, build-pt-meek.sh starts like this:
#!/bin/bash MEEK_TAG=v0.37.0
I edited all of these, updated them the latest versions, and commited my changes: I updated meek to 0.38.0 and snowflake to 2.6.0--obfs4proxy was still using the latest version.
Updating the change log
Each time I make a release I write an update to CHANGELOG.md to list the new major things have changed. The simplest way to keep track of all of this is via Github milestones. I look at all of the closed issue in the 2.6.1 milestone, and then make sure that the change log includes all of these things in the 2.6.1 section. In this case, the changes are:
- Release updates: Automate builds with CI, make just 64-bit Windows release, make both Intel and Apple Silicon macOS releases
- Upgrade dependencies, including Tor, meek, and snowflake
- Bug fix: Restore the primary_action mode settings in a tab after OnionShare reconnects to Tor
- Bug fix: Fix issue with auto-connecting to Tor with persistent tabs open
- Bug fix: Fix packaging issue where Windows version of OnionShare conflicts with Windows version of Dangerzone
Ensuring the documentation is up-to-date
This release doesn't add any new features, so fortunately I don't have to spend any time documenting them all. It's mostly bug fixes, updating dependencies, and revamping how releases are made (thanks to automated builds with a GitHub Actions workflow).
Ensuring the localization is up-to-date
Volunteers on Weblate (often with the help of Localization Lab) translate all of the strings in the OnionShare desktop app into many different languages. Likewise, they also translate the documentation, as well as the strings in the mobile apps. Right now there's no consistent list of languages that OnionShare supports. Basically whenever I make a release, if OnionShare has been at least 90% translated into a language, then we include that language in the release. Otherwise, we don't.
Updating the OnionShare desktop strings
After volunteers translate the English strings into other languages, Weblate makes commits with those new strings into its own git repo at https://hosted.weblate.org/projects/onionshare/translations/. I need to make sure that my local onionshare folder has this as a git remote called weblate, like this:
git remote add weblate https://hosted.weblate.org/projects/onionshare/translations/
Then I pull in all of the latest localization changes from Weblate:
git pull weblate main
That's all I need to do to update the translations for the desktop app. However, I still need to determine which languages have been at least 90% translated so I can know whether or not to include them in the language changing drop-down menu in OnionShare's Settings tab.
Checking for languages with that are at least 90% translated
To check this, I need to run the script docs/check-weblate.py, passing in my Weblate API key. This is a script that uses the Weblate API to determine the percentage of the strings that have been translated into each language, for both the desktop app and the documentation:
$ cd docs
$ poetry run ./check-weblate.py $WEBLATE_API_KEY
GET https://hosted.weblate.org/api/projects/onionshare/languages/
Traceback (most recent call last):
File "/Users/user/code/onionshare/docs/./check-weblate.py", line 146, in <module>
asyncio.run(main())
File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/Users/user/code/onionshare/docs/./check-weblate.py", line 113, in main
languages[obj["code"]] = obj["language"]
~~~^^^^^^^^^^^^
KeyError: 'language'
And... I've hit another problem! Let me take a minute to debug this.
🎵 Debugging noises... 🎵
Okay, fixed. It turns out the Weblate API has slightly changed since the last time I made a release (it's been almost a year). Making a request to https://hosted.weblate.org/api/projects/onionshare/languages/ returns a list of objects, with each object representing a language that was included in the OnionShare project. Each language object includes the name of the language, like Uyghur, and the language code, like ug. The key for name of the language used to be language, and the key for the language code was code. It turns out, the name of the language is now using the key name instead of language. So, I fixed it by changing this line:
languages[obj["code"]] = obj["language"]
To this:
languages[obj["code"]] = obj["name"]
Okay, the check-weblate.py script should work now, though it takes a long time to finish running. The OnionShare project has 70 languages listed in Weblate, so it makes 70 HTTP requests for the OnionShare desktop app strings, one for each language, and also another 70 HTTP requests for each of the nine pages of OnionShare documentation. In order to avoid hammering the Weblate server, it waits one second between HTTP requests, meaning that it it spends 700 seconds (or 11 minutes and 40 seconds) just waiting between HTTP requests.
Here's what happens when I run it:
$ poetry run ./check-weblate.py $WEBLATE_API_KEY
GET https://hosted.weblate.org/api/projects/onionshare/languages/
GET https://hosted.weblate.org/api/translations/onionshare/translations/af/
GET https://hosted.weblate.org/api/translations/onionshare/translations/sq/
GET https://hosted.weblate.org/api/translations/onionshare/translations/am/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ar/
GET https://hosted.weblate.org/api/translations/onionshare/translations/hy/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ay/
GET https://hosted.weblate.org/api/translations/onionshare/translations/be/
GET https://hosted.weblate.org/api/translations/onionshare/translations/bn/
GET https://hosted.weblate.org/api/translations/onionshare/translations/bs/
GET https://hosted.weblate.org/api/translations/onionshare/translations/bg/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ca/
GET https://hosted.weblate.org/api/translations/onionshare/translations/zh_Hans/
GET https://hosted.weblate.org/api/translations/onionshare/translations/zh_Hant/
GET https://hosted.weblate.org/api/translations/onionshare/translations/hr/
GET https://hosted.weblate.org/api/translations/onionshare/translations/cs/
GET https://hosted.weblate.org/api/translations/onionshare/translations/da/
GET https://hosted.weblate.org/api/translations/onionshare/translations/nl/
GET https://hosted.weblate.org/api/translations/onionshare/translations/en/
GET https://hosted.weblate.org/api/translations/onionshare/translations/eo/ | error 404
GET https://hosted.weblate.org/api/translations/onionshare/translations/fil/
GET https://hosted.weblate.org/api/translations/onionshare/translations/fi/
GET https://hosted.weblate.org/api/translations/onionshare/translations/fr/
GET https://hosted.weblate.org/api/translations/onionshare/translations/gl/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ka/
GET https://hosted.weblate.org/api/translations/onionshare/translations/de/
GET https://hosted.weblate.org/api/translations/onionshare/translations/el/
GET https://hosted.weblate.org/api/translations/onionshare/translations/gu/
GET https://hosted.weblate.org/api/translations/onionshare/translations/he/
GET https://hosted.weblate.org/api/translations/onionshare/translations/hi/
GET https://hosted.weblate.org/api/translations/onionshare/translations/hu/
GET https://hosted.weblate.org/api/translations/onionshare/translations/is/
GET https://hosted.weblate.org/api/translations/onionshare/translations/id/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ga/
GET https://hosted.weblate.org/api/translations/onionshare/translations/it/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ja/
GET https://hosted.weblate.org/api/translations/onionshare/translations/km/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ko/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ckb/
GET https://hosted.weblate.org/api/translations/onionshare/translations/lt/
GET https://hosted.weblate.org/api/translations/onionshare/translations/lg/
GET https://hosted.weblate.org/api/translations/onionshare/translations/mk/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ms/
GET https://hosted.weblate.org/api/translations/onionshare/translations/nb_NO/
GET https://hosted.weblate.org/api/translations/onionshare/translations/om/ | error 404
GET https://hosted.weblate.org/api/translations/onionshare/translations/fa/
GET https://hosted.weblate.org/api/translations/onionshare/translations/pl/
GET https://hosted.weblate.org/api/translations/onionshare/translations/pt_BR/
GET https://hosted.weblate.org/api/translations/onionshare/translations/pt_PT/
GET https://hosted.weblate.org/api/translations/onionshare/translations/pa/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ro/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ru/
GET https://hosted.weblate.org/api/translations/onionshare/translations/sr_Latn/
GET https://hosted.weblate.org/api/translations/onionshare/translations/sn/
GET https://hosted.weblate.org/api/translations/onionshare/translations/si/
GET https://hosted.weblate.org/api/translations/onionshare/translations/sk/
GET https://hosted.weblate.org/api/translations/onionshare/translations/sl/
GET https://hosted.weblate.org/api/translations/onionshare/translations/es/
GET https://hosted.weblate.org/api/translations/onionshare/translations/sw/
GET https://hosted.weblate.org/api/translations/onionshare/translations/sv/
GET https://hosted.weblate.org/api/translations/onionshare/translations/tl/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ta/
GET https://hosted.weblate.org/api/translations/onionshare/translations/te/
GET https://hosted.weblate.org/api/translations/onionshare/translations/bo/
GET https://hosted.weblate.org/api/translations/onionshare/translations/tr/
GET https://hosted.weblate.org/api/translations/onionshare/translations/tk/ | error 404
GET https://hosted.weblate.org/api/translations/onionshare/translations/uk/
GET https://hosted.weblate.org/api/translations/onionshare/translations/ug/
GET https://hosted.weblate.org/api/translations/onionshare/translations/vi/
GET https://hosted.weblate.org/api/translations/onionshare/translations/wo/
GET https://hosted.weblate.org/api/translations/onionshare/translations/yo/
GET https://hosted.weblate.org/api/translations/onionshare/doc-advanced/af/
GET https://hosted.weblate.org/api/translations/onionshare/doc-advanced/sq/
--snip--
GET https://hosted.weblate.org/api/translations/onionshare/doc-tor/wo/
GET https://hosted.weblate.org/api/translations/onionshare/doc-tor/yo/
Traceback (most recent call last):
File "/Users/user/code/onionshare/docs/./check-weblate.py", line 146, in <module>
asyncio.run(main())
File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/Users/user/code/onionshare/docs/./check-weblate.py", line 136, in main
await app_percent_output(90, 101)
File "/Users/user/code/onionshare/docs/./check-weblate.py", line 53, in app_percent_output
app_translations[lang_code] >= percent_min
~~~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: 'eo'
Eek, I've hit another issue. And unfortunately I hit it after making all of those HTTP requests. This means that once I fix it, I'll have to make them all over again. Alright hang on while I fix this...
🎵 Debugging noises... 🎵
Fixed it. The problem here is that Esperanto (the language with code eo) was listed in the OnionShare weblate project, however no one started translating the OnionShare app itself into that language. That's why, if you look back at the output, the Weblate API returned 404 when making the eo request:
GET https://hosted.weblate.org/api/translations/onionshare/translations/eo/ | error 404
The solution here is to just make the check-weblate.py script less brittle by making sure a language exists before checking the percentage it's been translated. I do that by replacing this if statement:
if ( app_translations[lang_code] >= percent_min and app_translations[lang_code] < percent_max ):
With this one:
if ( lang_code in app_translations and app_translations[lang_code] >= percent_min and app_translations[lang_code] < percent_max ):
Alright, the third time's a charm:
$ poetry run ./check-weblate.py $WEBLATE_API_KEY
GET https://hosted.weblate.org/api/projects/onionshare/languages/
GET https://hosted.weblate.org/api/translations/onionshare/translations/af/
--snip--
GET https://hosted.weblate.org/api/translations/onionshare/doc-tor/wo/
GET https://hosted.weblate.org/api/translations/onionshare/doc-tor/yo/
App translations >= 90%
=======================
Afrikaans (af), 100.0%
Albanian (sq), 99.2%
Arabic (ar), 100.0%
Belarusian (be), 100.0%
Catalan (ca), 100.0%
Chinese (Simplified) (zh_Hans), 100.0%
Chinese (Traditional) (zh_Hant), 100.0%
Croatian (hr), 98.4%
Czech (cs), 100.0%
English (en), 100.0%
Finnish (fi), 100.0%
French (fr), 100.0%
German (de), 100.0%
Greek (el), 100.0%
Icelandic (is), 100.0%
Italian (it), 91.7%
Japanese (ja), 100.0%
Lithuanian (lt), 99.2%
Norwegian Bokmål (nb_NO), 90.2%
Persian (fa), 98.8%
Polish (pl), 100.0%
Portuguese (Brazil) (pt_BR), 99.6%
Russian (ru), 99.2%
Shona (sn), 98.8%
Spanish (es), 100.0%
Swahili (sw), 99.2%
Swedish (sv), 99.2%
Turkish (tr), 100.0%
Ukrainian (uk), 100.0%
Vietnamese (vi), 100.0%
App translations >= 50%
=======================
Bengali (bn), 88.2%
Danish (da), 67.9%
Dutch (nl), 73.8%
Galician (gl), 82.4%
Indonesian (id), 68.3%
Irish (ga), 53.5%
Khmer (Central) (km), 83.2%
Kurdish (Central) (ckb), 64.0%
Portuguese (Portugal) (pt_PT), 84.7%
Romanian (ro), 50.3%
Serbian (latin) (sr_Latn), 75.7%
Slovak (sk), 64.0%
App translations >= 0%
=======================
Amharic (am), 1.1%
Armenian (hy), 0.0%
Aymara (ay), 0.0%
Bosnian (bs), 0.0%
Bulgarian (bg), 35.1%
Filipino (fil), 0.0%
Georgian (ka), 1.9%
Gujarati (gu), 4.2%
Hebrew (he), 9.3%
Hindi (hi), 39.8%
Hungarian (hu), 28.5%
Korean (ko), 28.1%
Luganda (lg), 0.0%
Macedonian (mk), 2.7%
Malay (ms), 5.0%
Punjabi (pa), 1.1%
Sinhala (si), 0.7%
Slovenian (sl), 10.5%
Tagalog (tl), 0.0%
Tamil (ta), 0.0%
Telugu (te), 48.4%
Tibetan (bo), 0.0%
Uyghur (ug), 0.0%
Wolof (wo), 0.0%
Yoruba (yo), 8.5%
Docs translations >= 90%
========================
English (en), 100%
French (fr), 100%
Greek (el), 100%
Polish (pl), 100%
Spanish (es), 100%
Turkish (tr), 100%
Ukrainian (uk), 100%
Vietnamese (vi), 100%
Docs translations >= 50%
========================
Afrikaans (af), 81%
Chinese (Simplified) (zh_Hans), 59%
Chinese (Traditional) (zh_Hant), 59%
Czech (cs), 59%
Finnish (fi), 69%
German (de), 79%
Italian (it), 76%
Japanese (ja), 60%
Khmer (Central) (km), 77%
Norwegian Bokmål (nb_NO), 76%
Portuguese (Brazil) (pt_BR), 84%
Portuguese (Portugal) (pt_PT), 54%
Russian (ru), 79%
Swahili (sw), 74%
Swedish (sv), 65%
Docs translations >= 0%
========================
Arabic (ar), 40%
Belarusian (be), 33%
Bengali (bn), 29%
Bulgarian (bg), 16%
Catalan (ca), 36%
Croatian (hr), 30%
Dutch (nl), 43%
Filipino (fil), 13%
Galician (gl), 27%
Icelandic (is), 22%
Indonesian (id), 15%
Irish (ga), 22%
Korean (ko), 43%
Kurdish (Central) (ckb), 39%
Lithuanian (lt), 5%
Serbian (latin) (sr_Latn), 31%
Slovak (sk), 30%
Excellent, now I know which languages to enable in the OnionShare app, as well as which languages to enable in the documentation.
Enabling languages in the OnionShare app
I'll add the newly translated languages to OnionShare in a minute, but first I will update the internal lists of country names, each list of countries translated into a different language.
OnionShare has a censorship circumvention workflow that can attempt to automatically connect even you're in a place that blocks access to Tor. For this to work though, it needs to know what country you're in to guess which censorship circumvention technique is most likely to work. It could automatically detect your country based on your IP address, or alternatively you can specify what country you're in. If your language is set to, say, Vietnamese, then when you select your country, the list of country names should be written in Vietnamese. That's why OnionShare includes these country name lists.
To update the translations of country names, I ran this:
poetry run python ./scripts/countries-update-list.py
Great. Next, I want to enable the correct languages. From the check-weblate.py output above, I can tell that I should enable the following languages in the OnionShare desktop app for this release:
- Afrikaans (af), 100.0%
- Albanian (sq), 99.2%
- Arabic (ar), 100.0%
- Belarusian (be), 100.0%
- Catalan (ca), 100.0%
- Chinese (Simplified) (zh_Hans), 100.0%
- Chinese (Traditional) (zh_Hant), 100.0%
- Croatian (hr), 98.4%
- Czech (cs), 100.0%
- English (en), 100.0%
- Finnish (fi), 100.0%
- French (fr), 100.0%
- German (de), 100.0%
- Greek (el), 100.0%
- Icelandic (is), 100.0%
- Italian (it), 91.7%
- Japanese (ja), 100.0%
- Lithuanian (lt), 99.2%
- Norwegian Bokmål (nb_NO), 90.2%
- Persian (fa), 98.8%
- Polish (pl), 100.0%
- Portuguese (Brazil) (pt_BR), 99.6%
- Russian (ru), 99.2%
- Shona (sn), 98.8%
- Spanish (es), 100.0%
- Swahili (sw), 99.2%
- Swedish (sv), 99.2%
- Turkish (tr), 100.0%
- Ukrainian (uk), 100.0%
- Vietnamese (vi), 100.0%
I do that by editing the self.available_locales dictionary in cli/onionshare_cli/settings.py. In OnionShare 2.6 there were only 10 languages enabled, but in version 2.6.1 there will be 30! The translators have been quite busy. Here's the new self.available_locales that I've added:
# Dictionary of available languages in this version of OnionShare, # mapped to the language name, in that language self.available_locales = { "af": "Afrikaans", # Afrikaans "sq": "Shqip", # Albanian "ar": "العربية", # Arabic "be": "Беларуская", # Belarusian # "bn": "বাংলা", # Bengali "ca": "Català", # Catalan "zh_Hant": "正體中文 (繁體)", # Traditional Chinese "zh_Hans": "中文 (简体)", # Simplified Chinese "hr": "Hrvatski", # Croatian "cs": "čeština", # Czech # "da": "Dansk", # Danish # "nl": "Nederlands", # Dutch "en": "English", # English "fi": "Suomi", # Finnish "fr": "Français", # French # "gl": "Galego", # Galician "de": "Deutsch", # German "el": "Ελληνικά", # Greek "is": "Íslenska", # Icelandic # "id": "Bahasa Indonesia", # Indonesian # "ga": "Gaeilge", # Irish "it": "Italiano", # Italian "ja": "日本語", # Japanese # "ckb": "Soranî", # Kurdish (Central) "lt": "Lietuvių Kalba", # Lithuanian "nb_NO": "Norsk Bokmål", # Norwegian Bokmål "fa": "فارسی", # Persian "pl": "Polski", # Polish "pt_BR": "Português (Brasil)", # Portuguese Brazil # "pt_PT": "Português (Portugal)", # Portuguese Portugal # "ro": "Română", # Romanian "ru": "Русский", # Russian "sn": "chiShona", # Shona # "sr_Latn": "Srpska (latinica)", # Serbian (latin) # "sk": "Slovenčina", # Slovak "es": "Español", # Spanish "sw": "Kiswahili", # Swahili "sv": "Svenska", # Swedish # "te": "తెలుగు", # Telugu "tr": "Türkçe", # Turkish "uk": "Українська", # Ukrainian "vi": "Tiếng Việt", # Vietnamese }
This dictionary maps language codes to languages names, in that language. For most of these languages I had to look up how to write their names here. Note that several languages in this code block are commented out--these are languages that have once been included in OnionShare, but that I'm not enabling in this release since they aren't at least 90% translated this time around.
After making this change, I decided to test it out. I made sure to follow the instructions in desktop/README.md to make sure my computer has a local development environment, and then I ran OnionShare from the source tree:
poetry run onionshare -v
After connecting to Tor, I opened the Settings tab and sure enough, all of these languages are listed in the language dropdown:
I ran OnionShare again, opened Settings, and changed my language to Tiếng Việt (Vietnamese). It prompted me (in Vietnamese) to restart OnionShare, and so I did. I then opened it again, and there we have it: OnionShare in Vietnamese.
Enabling languages in the documentation
From the check-weblate.py output above, I can tell that I should enable the following languages in the documentation for this release:
- English (en), 100%
- French (fr), 100%
- Greek (el), 100%
- Polish (pl), 100%
- Spanish (es), 100%
- Turkish (tr), 100%
- Ukrainian (uk), 100%
- Vietnamese (vi), 100%
I do that by editing docs/source/conf.py and updating the languages list of tuples. This time, the documentation will be translated into 8 languages. Here's the new languages list:
languages = [ ("English", "en"), # English ("Français", "fr"), # French # ("Deutsch", "de"), # German ("Ελληνικά", "el"), # Greek # ("Italiano", "it"), # Italian # ("日本語", "ja"), # Japanese # ("ភាសាខ្មែរ", "km"), # Khmer (Central) # ("Norsk Bokmål", "nb_NO"), # Norwegian Bokmål ("Polish", "pl"), # Polish # ("Portuguese (Brazil)", "pt_BR"), # Portuguese (Brazil)) # ("Русский", "ru"), # Russian ("Español", "es"), # Spanish # ("Svenska", "sv"), # Swedish ("Türkçe", "tr"), # Turkish ("Українська", "uk"), # Ukrainian ("Tiếng Việt", "vi"), # Vietnamese ]
Unfortunately, in this release I'm commenting out documentation translations for Japanese, Khmer, and Swedish, since they didn't make the 90% threshold.
Next, I also need to edit docs/build.sh and update the LOCALES variable to include language codes for this same list of enabled languages. In this case it's:
LOCALES="en fr el pl es tr uk vi"
Finally, I build the new documentation by running this from the docs folder:
poetry run ./build.sh
This builds the static documentation website (hosted at https://docs.onionshare.org/) for version 2.6.1, and stored it in docs/build/docs/2.6.1/. Here's a screenshot of the the documentation, viewed locally:
In that screenshot I had clicked the menu button in the bottom left, which shows which languages the documentation is translated into, and lets you switch. Here's a screenshot of the same documentation page, but this time in Vietnamese:
I've committed all of these changes, and with that, localization for 2.6.1 is finished!
Making sure Snapcraft packaging works
To make sure the Snapcraft package works, I need to switch to my computer running Ubuntu--in my case, I'm running Ubuntu 23.04.
To make the Snapcraft release, I need to update all of the dependencies in the Snapcraft YAML file, snap/snapcraft.yaml, and then build and install a snap, to make sure it all works as expected.
Updating dependencies
I'll start by updating the versions of tor, libevent, obfs4, snowflake-client, and meek-client.
Here's the existing tor part:
tor: source: https://dist.torproject.org/tor-0.4.7.12.tar.gz source-checksum: sha256/3b5d969712c467851bd028f314343ef15a97ea457191e93ffa97310b05b9e395 source-type: tar plugin: autotools autotools-configure-parameters: - "--with-libevent-dir=$SNAPCRAFT_PART_INSTALL/../../libevent/install/usr/local" build-packages: - libssl-dev - zlib1g-dev after: [libevent]
I want to upgrade to the latest version of Tor, so I'm going to https://dist.torproject.org/ to check to see what that is. At the moment, it looks like the latest release is tor-0.4.8.5.tar.gz. So on my computer, I'll download this file, along with it's checksum, and its PGP signature:
wget https://dist.torproject.org/tor-0.4.8.5.tar.gz wget https://dist.torproject.org/tor-0.4.8.5.tar.gz.sha256sum wget https://dist.torproject.org/tor-0.4.8.5.tar.gz.sha256sum.asc
I'm then verifying the checksum's signature:
$ gpg --verify tor-0.4.8.5.tar.gz.sha256sum.asc
gpg: assuming signed data in 'tor-0.4.8.5.tar.gz.sha256sum'
gpg: Signature made Wed 30 Aug 2023 06:14:29 AM PDT
gpg: using RSA key B74417EDDF22AC9F9E90F49142E86A2A11F48D36
gpg: Good signature from "David Goulet <[email protected]>" [unknown]
gpg: aka "David Goulet <[email protected]>" [unknown]
gpg: aka "David Goulet <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: B744 17ED DF22 AC9F 9E90 F491 42E8 6A2A 11F4 8D36
gpg: Signature made Wed 30 Aug 2023 07:14:27 AM PDT
gpg: using EDDSA key 514102454D0A87DB0767A1EBBE6A0531C18A9179
gpg: Good signature from "Alexander Færøy <[email protected]>" [unknown]
gpg: aka "Alexander Færøy <[email protected]>" [unknown]
gpg: aka "Alexander Færøy <[email protected]>" [unknown]
gpg: aka "Alexander Færøy <[email protected]>" [unknown]
gpg: aka "Alexander Færøy <[email protected]>" [unknown]
gpg: aka "Alexander Færøy <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 1C1B C007 A9F6 07AA 8152 C040 BEA7 B180 B149 1921
Subkey fingerprint: 5141 0245 4D0A 87DB 0767 A1EB BE6A 0531 C18A 9179
Then I'm making sure the SHA256 checksum matches:
$ sha256sum --check tor-0.4.8.5.tar.gz.sha256sum
tor-0.4.8.5.tar.gz: OK
It does, so now I'm updating snapcraft.yaml to include the new version of Tor, along with its checksum:
tor: source: https://dist.torproject.org/tor-0.4.8.5.tar.gz source-checksum: sha256/6957cfd14a29eee7555c52f8387a46f2ce2f5fe7dadf93547f1bc74b1657e119
Next, I'll do something similar for libevent. Here's the libevent part:
libevent: source: https://github.com/libevent/libevent/releases/download/release-2.1.12-stable/libevent-2.1.12-stable.tar.gz source-checksum: sha256/92e6de1be9ec176428fd2367677e61ceffc2ee1cb119035037a27d346b0403bb source-type: tar plugin: autotools
Looking at https://github.com/libevent/libevent/releases/ it seems that 2.1.12-stable is still the latest stable version, so I'll leave this part as is.
Next, I'll upgrade obfs4, snowflake-client, and meek-client. These ones are slightly simpler because they're pulled from git, and I just need to update the git tags--the same tags I used in the desktop/scripts/build-pt-* scripts. I'm leaving obfs4 with the source tag obfs4proxy-0.0.14, I'm upgrading snowflake-client to use the source tag v2.6.0, and I'm upgrading meek-client to use the source tag v0.38.0.
I then turn my attention to the onionshare part. While the CLI and desktop version of OnionShare use Poetry to keep track of Python dependencies, I had issues making this work with Snapcraft, so instead I redefine those dependencies in a requirements.txt files. Without going into the nitty gritty details, I basically need to look at all of the dependencies in cli/pyproject.toml and desktop/pyproject.toml and make sure the requirements.txt file in the override-pull section of the onionshare part matches them. Here's my new override-pull section:
override-pull: | snapcraftctl pull rm pyproject.toml poetry.lock cat > requirements.txt << EOF # onionshare_cli click flask==2.3.2 flask-compress==1.13 flask-socketio==5.3.4 psutil pysocks requests[socks] unidecode urllib3 eventlet setuptools pynacl colorama gevent-websocket stem==1.8.1 waitress werkzeug==2.3.4 # onionshare qrcode EOF
Trying to update from PySide2 to PySide6
One of the major changes between OnionShare 2.6 and 2.6.1 is that we've upgraded Qt for Python (the GUI framework that OnionShare uses) from PySide2 to PySide6. PySide2 brings Qt 5.x support to Python, and PySide6 brings Qt 6.x support. The main reason for upgrading is because we wanted to support Apple Silicon Macs: PySide2 does not publish arm64 binaries, which PySide6 does.
In the stage-packages section of the onionshare part, the YAML file lists Ubuntu packages that are required for the app to run, including:
python3-pyside2.qtcorepython3-pyside2.qtguipython3-pyside2.qtwidgets
Can I just change these pyside2s to pyside6s? Nope. I searched Ubuntu packages and it looks like the equivalent PySide6 packages aren't available. Okay, I will delete these python3-pyside2.* packages from stage-packages, and add PySide6==6.5.2 to the requirements.txt file in the override-pull section, to see if that works.
In order to test, I need to build the snap locally. To do that, first I need to install snapcraft (the tool for building snaps, as opposed to snap, which is the tool for installing already-built snaps) on my computer:
sudo snap install snapcraft --classic
Then I can build it by running:
snapcraft
This takes a long time to run. It launches a new VM, downloads the source code for all of the OnionShare dependencies, compiles them all, and builds OnionShare itself. It's quicker on subsequent runs too. Eventually, it finishes with:
--snip--
Priming onionshare
+ snapcraftctl prime
This part is missing libraries that cannot be satisfied with any available stage-packages known to snapcraft:
- libQt6Quick3DHelpersImpl.so.6
- libQt6Quick3DSpatialAudio.so.6
- libmysqlclient.so.21
- libxcb-cursor.so.0
- libxkbfile.so.1
These dependencies can be satisfied via additional parts or content sharing. Consider validating configured filesets if this dependency was built.
Snapping |
Snapped onionshare_2.6.1_amd64.snap
Hmm, there's a warning that it's missing Qt6 libraries. That's probably not good.
🎵 Debugging noises... 🎵
The base of this snap is core20, which is based on Ubuntu 20.04 LTS. I searched Ubuntu 20.04 packages for "libqt6" and there no results, but there are some "libqt6" packages in Ubuntu 22.04. If I can't get this to work, I might need to upgrade the snap from core20 to core22, which could come with its own set of problems. Alternatively, I could try compiling Qt6 inside the snap to get the libraries, instead of installing them from the package manager.
Still, I'll install the snap I created and then run it to see what happens.
$ sudo snap install ./onionshare_2.6.1_amd64.snap --devmode
onionshare 2.6.1 installed
$ /snap/bin/onionshare
Warning: Schema “org.gnome.system.locale” has path “/system/locale/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy” has path “/system/proxy/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.http” has path “/system/proxy/http/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.https” has path “/system/proxy/https/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.ftp” has path “/system/proxy/ftp/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.socks” has path “/system/proxy/socks/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Traceback (most recent call last):
File "/snap/onionshare/x3/bin/onionshare", line 5, in <module>
from onionshare import main
File "/snap/onionshare/x3/lib/python3.8/site-packages/onionshare/__init__.py", line 34, in <module>
from onionshare_cli.common import Common
File "/snap/onionshare/x3/lib/python3.8/site-packages/onionshare_cli/__init__.py", line 30, in <module>
from .web import Web
File "/snap/onionshare/x3/lib/python3.8/site-packages/onionshare_cli/web/__init__.py", line 21, in <module>
from .web import Web
File "/snap/onionshare/x3/lib/python3.8/site-packages/onionshare_cli/web/web.py", line 26, in <module>
from packaging.version import Version
ModuleNotFoundError: No module named 'packaging'
Okay, I'll add packaging to the requirements.txt and try again:
snapcraft # this takes awhile to finish sudo snap install ./onionshare_2.6.1_amd64.snap --devmode /snap/bin/onionshare
This time it fails with the error:
python3: symbol lookup error: /snap/onionshare/x4/lib/python3.8/site-packages/PySide6/Qt/plugins/platforms/../../lib/libQt6WaylandClient.so.6: undefined symbol: wl_proxy_marshal_flags
Okay yeah, I expected that it wouldn't work. I'm going to need to get Qt6 installed in this snap.
Updating from core20 to core22
🎵 Debugging noises continue... 🎵
In order to install the Qt6 libraries from Ubuntu packages, I'm going to upgrade this snap to use core22, since Ubuntu 22.04 has Qt6 packages with names like libqt6core6 and libqt6gui6.
While I'm performing this upgrade, I'll be following this official guide on migrating from core20 to core22. Here's what I'm changing:
- I'm changing
base: core20tobase: core22. - In all of the parts with an
override-pullsection, I'm replacingsnapcraftctl pullwithcraftctl default. - In my
override-buildsections, I'm changing the environment variable$SNAPCRAFT_PART_INSTALLto$CRAFT_PART_INSTALL. - The
onionshareapp uses thegnome-3-38extension, but I'm removing it altogether--if it's required, I can try thegnomeextension (which supportscore22) or thekde-neonextension
After making these changes, I'm trying again:
$ snapcraft
Traceback (most recent call last):
File "/snap/snapcraft/9542/bin/snapcraft", line 8, in <module>
sys.exit(run())
File "/snap/snapcraft/9542/lib/python3.8/site-packages/snapcraft/cli.py", line 255, in run
_run_dispatcher(dispatcher, global_args)
File "/snap/snapcraft/9542/lib/python3.8/site-packages/snapcraft/cli.py", line 228, in _run_dispatcher
dispatcher.run()
File "/snap/snapcraft/9542/lib/python3.8/site-packages/craft_cli/dispatcher.py", line 448, in run
return self._loaded_command.run(self._parsed_command_args)
File "/snap/snapcraft/9542/lib/python3.8/site-packages/snapcraft/commands/lifecycle.py", line 265, in run
super().run(parsed_args)
File "/snap/snapcraft/9542/lib/python3.8/site-packages/snapcraft/commands/lifecycle.py", line 138, in run
parts_lifecycle.run(self.name, parsed_args)
File "/snap/snapcraft/9542/lib/python3.8/site-packages/snapcraft/parts/lifecycle.py", line 216, in run
_run_command(
File "/snap/snapcraft/9542/lib/python3.8/site-packages/snapcraft/parts/lifecycle.py", line 262, in _run_command
_run_in_provider(project, command_name, parsed_args)
File "/snap/snapcraft/9542/lib/python3.8/site-packages/snapcraft/parts/lifecycle.py", line 487, in _run_in_provider
providers.ensure_provider_is_available(provider)
File "/snap/snapcraft/9542/lib/python3.8/site-packages/snapcraft/providers.py", line 148, in ensure_provider_is_available
LXDProvider.ensure_provider_is_available()
File "/snap/snapcraft/9542/lib/python3.8/site-packages/craft_providers/lxd/lxd_provider.py", line 70, in ensure_provider_is_available
ensure_lxd_is_ready()
File "/snap/snapcraft/9542/lib/python3.8/site-packages/craft_providers/lxd/installer.py", line 132, in ensure_lxd_is_ready
raise errors.LXDError(
craft_providers.lxd.errors.LXDError: LXD requires additional permissions.
Ensure that the user is in the 'lxd' group.
Visit https://linuxcontainers.org/lxd/getting-started-cli/ for instructions on installing and configuring LXD for your operating system.
It turns out, Snapcraft is moving away from VMs to build snaps and towards Linux containers (which honestly is way nicer--this makes it simpler to build snaps inside of VMs, without needing nested VMs). I ran into this a little while I was getting snaps to build in GitHub Actions. Based on this error message, it looks like I need to add my user on my Ubuntu system to the lxd group. I do that by running this:
sudo usermod -a -G lxd $USER
Next, I'm saving all my work and rebooting my Ubuntu computer. After booting back in, I tried again, and realized I had to run this command before it will work:
lxd init --auto
After that, running snapcraft is finally building the OnionShare snap, this time in a container instead of in a VM. Here's the output:
$ snapcraft
Launching instance...
Executed: pull launcher
Executed: pull libevent
Executed: pull meek-client
Executed: pull obfs4
Executed: pull snowflake-client
Executed: pull tor
Executed: pull onionshare-cli
Executed: pull onionshare
Executed: build launcher
Executed: build libevent
Executed: build meek-client
Executed: build obfs4
Executed: build snowflake-client
Executed: skip pull libevent (already ran)
Executed: skip build libevent (already ran)
Executed: stage libevent (required to build 'tor')
Executed: build tor
Executed: skip pull meek-client (already ran)
Executed: skip build meek-client (already ran)
Executed: stage meek-client (required to build 'onionshare-cli')
Executed: skip pull obfs4 (already ran)
Executed: skip build obfs4 (already ran)
Executed: stage obfs4 (required to build 'onionshare-cli')
Executed: skip pull tor (already ran)
Executed: skip build tor (already ran)
Executed: stage tor (required to build 'onionshare-cli')
Executed: skip pull snowflake-client (already ran)
Executed: skip build snowflake-client (already ran)
Executed: stage snowflake-client (required to build 'onionshare-cli')
Executed: build onionshare-cli
Executed: skip pull onionshare-cli (already ran)
Executed: skip build onionshare-cli (already ran)
Executed: stage onionshare-cli (required to build 'onionshare')
Executed: build onionshare
Executed: stage launcher
Executed: skip stage libevent (already ran)
Executed: skip stage meek-client (already ran)
Executed: skip stage obfs4 (already ran)
Executed: skip stage snowflake-client (already ran)
Executed: skip stage tor (already ran)
Executed: skip stage onionshare-cli (already ran)
Executed: stage onionshare
Executed: prime launcher
Executed: prime libevent
Executed: prime meek-client
Executed: prime obfs4
Executed: prime snowflake-client
Executed: prime tor
Executed: prime onionshare-cli
Executed: prime onionshare
Executed parts lifecycle
Generated snap metadata
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtMultimedia.abi3.so'
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtMultimediaWidgets.abi3.so'
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtQml.abi3.so'
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtQuick.abi3.so'
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtSpatialAudio.abi3.so'
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtTextToSpeech.abi3.so'
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtWebEngineCore.abi3.so'
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtWebEngineQuick.abi3.so'
Unable to determine library dependencies for 'lib/python3.10/site-packages/PySide6/QtWebEngineWidgets.abi3.so'
Lint warnings:
- library: lib/python3.10/site-packages/PySide6/Qt/lib/libQt6WebEngineCore.so.6: missing dependency 'libxkbfile.so.1'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/lib/libQt6WebEngineQuick.so.6: missing dependency 'libxkbfile.so.1'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/lib/libQt6WebEngineWidgets.so.6: missing dependency 'libxkbfile.so.1'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/lib/libQt6XcbQpa.so.6: missing dependency 'libxcb-cursor.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/libexec/QtWebEngineProcess: missing dependency 'libxkbfile.so.1'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/designer/libqwebengineview.so: missing dependency 'libxkbfile.so.1'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/multimedia/libgstreamermediaplugin.so: missing dependency 'libgstallocators-1.0.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/multimedia/libgstreamermediaplugin.so: missing dependency 'libgstapp-1.0.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/multimedia/libgstreamermediaplugin.so: missing dependency 'libgstaudio-1.0.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/multimedia/libgstreamermediaplugin.so: missing dependency 'libgstbase-1.0.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/multimedia/libgstreamermediaplugin.so: missing dependency 'libgstgl-1.0.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/multimedia/libgstreamermediaplugin.so: missing dependency 'libgstpbutils-1.0.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/multimedia/libgstreamermediaplugin.so: missing dependency 'libgstreamer-1.0.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/multimedia/libgstreamermediaplugin.so: missing dependency 'libgstvideo-1.0.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/platforms/libqxcb.so: missing dependency 'libxcb-cursor.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/sqldrivers/libqsqlmysql.so: missing dependency 'libmysqlclient.so.21'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/xcbglintegrations/libqxcb-egl-integration.so: missing dependency 'libxcb-cursor.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/plugins/xcbglintegrations/libqxcb-glx-integration.so: missing dependency 'libxcb-cursor.so.0'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/qml/QtQuick3D/Helpers/impl/libqtquick3dhelpersimplplugin.so: missing dependency 'libQt6Quick3DHelpersImpl.so.6'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/qml/QtQuick3D/SpatialAudio/libquick3dspatialaudioplugin.so: missing dependency 'libQt6Quick3DSpatialAudio.so.6'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/qml/QtWebEngine/libqtwebenginequickplugin.so: missing dependency 'libxkbfile.so.1'. (https://snapcraft.io/docs/linters-library)
- library: libQt6MultimediaWidgets.so.6: unused library 'lib/python3.10/site-packages/PySide6/Qt/lib/libQt6MultimediaWidgets.so.6'. (https://snapcraft.io/docs/linters-library)
- library: libQt6Quick3DGlslParser.so.6: unused library 'lib/python3.10/site-packages/PySide6/Qt/lib/libQt6Quick3DGlslParser.so.6'. (https://snapcraft.io/docs/linters-library)
- library: libQt6Quick3DIblBaker.so.6: unused library 'lib/python3.10/site-packages/PySide6/Qt/lib/libQt6Quick3DIblBaker.so.6'. (https://snapcraft.io/docs/linters-library)
- library: libEGL_mesa.so.0: unused library 'usr/lib/x86_64-linux-gnu/libEGL_mesa.so.0.0.0'. (https://snapcraft.io/docs/linters-library)
- library: libGLX_mesa.so.0: unused library 'usr/lib/x86_64-linux-gnu/libGLX_mesa.so.0.0.0'. (https://snapcraft.io/docs/linters-library)
- library: libcolordprivate.so.2: unused library 'usr/lib/x86_64-linux-gnu/libcolordprivate.so.2.0.5'. (https://snapcraft.io/docs/linters-library)
- library: libdconf.so.1: unused library 'usr/lib/x86_64-linux-gnu/libdconf.so.1.0.0'. (https://snapcraft.io/docs/linters-library)
- library: libexslt.so.0: unused library 'usr/lib/x86_64-linux-gnu/libexslt.so.0.8.20'. (https://snapcraft.io/docs/linters-library)
- library: libgdk_pixbuf_xlib-2.0.so.0: unused library 'usr/lib/x86_64-linux-gnu/libgdk_pixbuf_xlib-2.0.so.0.4000.2'. (https://snapcraft.io/docs/linters-library)
- library: libicuio.so.70: unused library 'usr/lib/x86_64-linux-gnu/libicuio.so.70.1'. (https://snapcraft.io/docs/linters-library)
- library: libicutest.so.70: unused library 'usr/lib/x86_64-linux-gnu/libicutest.so.70.1'. (https://snapcraft.io/docs/linters-library)
- library: libodbccr.so.2: unused library 'usr/lib/x86_64-linux-gnu/libodbccr.so.2.0.0'. (https://snapcraft.io/docs/linters-library)
- library: libpulse-mainloop-glib.so.0: unused library 'usr/lib/x86_64-linux-gnu/libpulse-mainloop-glib.so.0.0.6'. (https://snapcraft.io/docs/linters-library)
- library: libpulse-simple.so.0: unused library 'usr/lib/x86_64-linux-gnu/libpulse-simple.so.0.1.1'. (https://snapcraft.io/docs/linters-library)
- library: librsvg-2.so.2: unused library 'usr/lib/x86_64-linux-gnu/librsvg-2.so.2.48.0'. (https://snapcraft.io/docs/linters-library)
- library: libssl3.so: unused library 'usr/lib/x86_64-linux-gnu/libssl3.so'. (https://snapcraft.io/docs/linters-library)
- library: libevent_core-2.1.so.7: unused library 'usr/local/lib/libevent_core-2.1.so.7.0.1'. (https://snapcraft.io/docs/linters-library)
- library: libevent_extra-2.1.so.7: unused library 'usr/local/lib/libevent_extra-2.1.so.7.0.1'. (https://snapcraft.io/docs/linters-library)
- library: libevent_openssl-2.1.so.7: unused library 'usr/local/lib/libevent_openssl-2.1.so.7.0.1'. (https://snapcraft.io/docs/linters-library)
- library: libevent_pthreads-2.1.so.7: unused library 'usr/local/lib/libevent_pthreads-2.1.so.7.0.1'. (https://snapcraft.io/docs/linters-library)
Created snap package onionshare_2.6.1_amd64.snap
I installed this snap and tried running it, and this time it crashed with the error:
Failed to create wl_display (No such file or directory)
qt.qpa.plugin: Could not load the Qt platform plugin "wayland" in "" even though it was found.
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.
Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vkkhrdisplay, vnc, wayland-egl, wayland, xcb.
Aborted (core dumped)
It looks like I need to install some more dependencies for this to work. The lint warnings give some useful hints as to what they might be. Look at this line:
- library: lib/python3.10/site-packages/PySide6/Qt/lib/libQt6WebEngineCore.so.6: missing dependency 'libxkbfile.so.1'. (https://snapcraft.io/docs/linters-library)
Qt6 is missing the dependency libxkbfile.so.1. In Ubuntu you can search for what package contains that file like this:
$ apt-file search libxkbfile.so.1
libxkbfile1: /usr/lib/x86_64-linux-gnu/libxkbfile.so.1
libxkbfile1: /usr/lib/x86_64-linux-gnu/libxkbfile.so.1.0.2
This means that if I install the package libxkbfile1, it should come with the library I need. Do the same for all of the lint warnings saying I was missing a dependency, I found that I needed to add the following packages to stage-packages:
libgstreamer1.0-0libgstreamer1.0-devlibgstreamer-gl1.0-0libgstreamer-plugins-base1.0-0libmysqlclient21libxcb-cursor0libxkbfile1qml6-module-qtquick3d-spatialaudio
I tried to build another snap:
$ snapcraft
Launching instance...
Executed: skip pull launcher (already ran)
Executed: skip pull libevent (already ran)
Executed: skip pull meek-client (already ran)
Executed: skip pull obfs4 (already ran)
Executed: skip pull snowflake-client (already ran)
Executed: skip pull tor (already ran)
Executed: skip pull onionshare-cli (already ran)
Stage package not found in part 'onionshare': qml6-module-qtquick3d-spatialaudio.
Failed to execute pack in instance.
Full execution log: '/home/user/.local/state/snapcraft/log/snapcraft-20230905-172355.921802.log'
After some invesigation, I found that qml6-module-qtquick3d-spatialaudio is in the Ubuntu universe repo, not the main one. I don't think OnionShare actually uses that one though, so I decided to just remove it and try again. This time there are fewer lint warnings about missing dependencies:
Lint warnings:
- library: lib/python3.10/site-packages/PySide6/Qt/qml/QtQuick3D/Helpers/impl/libqtquick3dhelpersimplplugin.so: missing dependency 'libQt6Quick3DHelpersImpl.so.6'. (https://snapcraft.io/docs/linters-library)
- library: lib/python3.10/site-packages/PySide6/Qt/qml/QtQuick3D/SpatialAudio/libquick3dspatialaudioplugin.so: missing dependency 'libQt6Quick3DSpatialAudio.so.6'. (https://snapcraft.io/docs/linters-library)
Now let's try running it:
$ /snap/bin/onionshare
╭───────────────────────────────────────────╮
│ * ▄▄█████▄▄ * │
│ ▄████▀▀▀████▄ * │
│ ▀▀█▀ ▀██▄ │
│ * ▄█▄ ▀██▄ │
│ ▄█████▄ ███ -+- │
│ ███ ▀█████▀ │
│ ▀██▄ ▀█▀ │
│ * ▀██▄ ▄█▄▄ * │
│ * ▀████▄▄▄████▀ │
│ ▀▀█████▀▀ │
│ -+- * │
│ ▄▀▄ ▄▀▀ █ │
│ █ █ ▀ ▀▄ █ │
│ █ █ █▀▄ █ ▄▀▄ █▀▄ ▀▄ █▀▄ ▄▀▄ █▄▀ ▄█▄ │
│ ▀▄▀ █ █ █ ▀▄▀ █ █ ▄▄▀ █ █ ▀▄█ █ ▀▄▄ │
│ │
│ v2.6.1 │
│ │
│ https://onionshare.org/ │
╰───────────────────────────────────────────╯
Failed to create wl_display (No such file or directory)
qt.qpa.plugin: Could not load the Qt platform plugin "wayland" in "" even though it was found.
Gtk-Message: 19:24:23.370: Failed to load module "canberra-gtk-module"
Gtk-Message: 19:24:23.370: Failed to load module "canberra-gtk-module"
And it worked!
I successfully connected to Tor, and I quickly tested it, and everything appears to work.
But there's one more thing I want to do. In the previous version of the snap package, I had an extra launcher component that set some environment variables before running onionshare or onionshare-cli, but I don't think that's necessary anymore. Instead I can simplify things just by updating the PATH and LD_LIBRARY_PATH to add $SNAP/usr/local paths to them. So, I've deleted the whole launcher part and updated the two apps to no longer use the launcher, and to update environment variables :
apps: onionshare: common-id: org.onionshare.OnionShare command: bin/onionshare plugs: - desktop - home - network - network-bind - removable-media environment: LANG: C.UTF-8 PATH: $SNAP/bin:$SNAP/usr/bin:$SNAP/usr/local/bin:$PATH LD_LIBRARY_PATH: $LD_LIBRARY_PATH:$SNAP/usr/local/lib cli: common-id: org.onionshare.OnionShareCli command: bin/onionshare-cli plugs: - home - network - network-bind - removable-media environment: LANG: C.UTF-8 PATH: $SNAP/bin:$SNAP/usr/bin:$SNAP/usr/local/bin:$PATH LD_LIBRARY_PATH: $LD_LIBRARY_PATH:$SNAP/usr/local/lib
I built another snap, installed it, and tested it one more time, and it worked! Time to commit my code.
Making sure the Flatpak packaging works
With Snapcraft done, let's take a look at Flatpak. The Flatpak manifest file is in flatpak/org.onionshare.OnionShare.yaml. To update the Flatpak package I basically need to upgrade all of the dependencies listed in the manifest, including the URLs to download them from and their sha256 checksums.
But first, I'm going to make sure I have flatpak and flatpak-builder installed, and make sure the Flathub repository is added:
sudo apt install flatpak flatpak-builder flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo
Updating pyside6
I'll start with the the pyside6 module, since that's at the top (luckily, I had already upgraded the Flatpak packaging from using PySide2 to PySide6). Here's the code for the PySide6 module:
- name: pyside6 buildsystem: simple build-commands: [] modules: - name: pyside6-essentials only-arches: - x86_64 buildsystem: simple build-commands: - pip3 install --verbose --exists-action=i --no-index --find-links="file://${PWD}" --prefix=${FLATPAK_DEST} "pyside6-essentials" --no-build-isolation sources: - type: file url: https://files.pythonhosted.org/packages/e5/96/f43cdcb397f8a8cff6991ef8109385cc5ad9b0ad78c6dc2988b3b776fe49/PySide6_Essentials-6.4.2-cp37-abi3-manylinux_2_28_x86_64.whl sha256: 8c3d37cca6e27f6da12b50b20e741d593ccc857bdcdb82d97f8f7c8bfe53639a modules: - name: shiboken6 buildsystem: simple build-commands: - pip3 install --verbose --exists-action=i --no-index --find-links="file://${PWD}" --prefix=${FLATPAK_DEST} "shiboken6" --no-build-isolation sources: - type: file url: https://files.pythonhosted.org/packages/24/f6/f1fe9220a616789a1c6b1b73670d8b1dec882ac730a8b534f963b3f26182/shiboken6-6.4.2-cp37-abi3-manylinux_2_28_x86_64.whl sha256: 0616c1a12d1e51e680595b3940b986275c1df952a751416a0730a59e5b90105f
This essentially installs two Python modules, PySide6-Essentials and shiboken6 from PyPI--these are binary packages that are pre-compiled for x86_64 architectures. Flatpak downloads the whl files from the URLs provided and verifies the sha256 checksums, and runs the build commands to install them. Since pyside6-essentials has shiboken6 as a dependency, shiboken6 is listed under modules under pyside6-essentials.
To update this, I just need to check PyPI for the latest versions of both of these packages and update the URL and sha256 checksums. The latest version of pyside6-essentials is 6.5.2, and the latest version of shiboken6 is also 6.5.2. You can find links to the package files, along with their checksums, on the PyPI pages for those packages under "Download files".
So I updated those. Just to make it clear, here's the diff:
diff --git a/flatpak/org.onionshare.OnionShare.yaml b/flatpak/org.onionshare.OnionShare.yaml index b455cb71..9d0f0cb5 100644 --- a/flatpak/org.onionshare.OnionShare.yaml +++ b/flatpak/org.onionshare.OnionShare.yaml @@ -35,8 +35,8 @@ modules: --prefix=${FLATPAK_DEST} "pyside6-essentials" --no-build-isolation sources: - type: file - url: https://files.pythonhosted.org/packages/e5/96/f43cdcb397f8a8cff6991ef8109385cc5ad9b0ad78c6dc2988b3b776fe49/PySide6_Essentials-6.4.2-cp37-abi3-manylinux_2_28_x86_64.whl - sha256: 8c3d37cca6e27f6da12b50b20e741d593ccc857bdcdb82d97f8f7c8bfe53639a + url: https://files.pythonhosted.org/packages/d0/de/9a089e91c2e0fe4f122218bba4f9dbde46338659f412739bd9db1ed9df4f/PySide6_Essentials-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl + sha256: 1620e82b38714a1570b142c01694d0415a25526517b24620ff9b00c9f76cfca9 modules: - name: shiboken6 buildsystem: simple @@ -45,8 +45,8 @@ modules: --prefix=${FLATPAK_DEST} "shiboken6" --no-build-isolation sources: - type: file - url: https://files.pythonhosted.org/packages/24/f6/f1fe9220a616789a1c6b1b73670d8b1dec882ac730a8b534f963b3f26182/shiboken6-6.4.2-cp37-abi3-manylinux_2_28_x86_64.whl - sha256: 0616c1a12d1e51e680595b3940b986275c1df952a751416a0730a59e5b90105f + url: https://files.pythonhosted.org/packages/55/44/d8c366dd4f069166ab9890acb44d004c5e6122714e44c169273dcbbca897/shiboken6-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl + sha256: 3fbc35ff3c19e7d39433671bfc1be3d7fa9d071bfdd0ffe1c2a4d27acd6cf6a5 - name: tor buildsystem: autotools sources:
Updating tor
The process to upgrade the version of tor installed in the tor module is similar, but this time the buildsystem is set to autotools, which means it will basically run ./configure, make, and make install to compile it from source. The tor module has its own modules section which lists libevent a dependency of tor.
To update it, I just need to update the tor source package URL and sha256 hash, along with the libevent source package URL and sha256 hash. I already did this same thing for the Snapcraft package, so I'm just copying the URLs and sha256 checksums from snapcraft.yaml.
Trying to update obfs4proxy, meek-client, and snowflake-client
These three pluggable transports are all written in Go, and they all have dependencies of their own. This is where I'm going to start relying on flatpak-builder-tools, a collection of scripts that make this work much simpler.
In another folder, I clone the repo:
git clone https://github.com/flatpak/flatpak-builder-tools.git cd flatpak-builder-tools
This project includes Go Get Generator in the go-get folder, with unfortunately rather convoluted instructions. For each of these go dependencies, I need to:
- Create a new Flatpak manifest file just for the one go dependency, with network access available during the build, and run the
go getcommand - Run
flatpak-builderwith the--keep-build-dirsflag, which will download all of the dependencies for the go project and keep them when it's done building - Run the
flatpak-go-get-generator.pyscript to create a Flatpak manifest file that actually includes all of these dependencies (their git repo URLs and commit IDs)--though, it will generate the manifest in JSON format - Convert the manifest it generates from JSON into YAML, and copy and it paste it into the OnionShare Flatpak manifest YAML file
I know from my Snapcraft work that there are no new versions of obfs4proxy, but I do need to upgrade meek-client and snowflake-client. Let's start with meek-client.
I'm starting by making a new file called meek-client.yaml and copying and pasting code from the Go Get Generator readme, except changing the go get build command to point to git.torproject.org/pluggable-transports/meek.git/[email protected].
But then when I tried running the flatpak-builder command, I kept get errors. It's been a long time since I last did this and I don't quite remember how I got it working...
🎵 Debugging noises... 🎵
After much trial and error--including learning that the go get syntax itself is now deprecated in favor of go install (I'm not much of a Go programmer)--I got flatpak-builder to work with this manifest file:
app-id: com.example.meek-client runtime: org.freedesktop.Platform runtime-version: '21.08' sdk: org.freedesktop.Sdk sdk-extensions: - org.freedesktop.Sdk.Extension.golang modules: - name: meek-client buildsystem: simple build-options: append-path: /usr/lib/sdk/golang/bin env: GOBIN: /app/bin GO111MODULE: on GOPATH: /run/build/meek-client build-args: - --share=network build-commands: - go install git.torproject.org/pluggable-transports/meek.git/[email protected]
I built it with flatpak-builder by running:
$ flatpak-builder build --force-clean --install-deps-from=flathub --keep-build-dirs ./meek-client.yaml Dependency Sdk: org.freedesktop.Sdk 21.08 Updating org.freedesktop.Sdk/x86_64/21.08 Nothing to do. Dependency Runtime: org.freedesktop.Platform 21.08 Updating org.freedesktop.Platform/x86_64/21.08 Nothing to do. Dependency Extension: org.freedesktop.Sdk.Extension.golang 21.08 Updating org.freedesktop.Sdk.Extension.golang/x86_64/21.08 Nothing to do. Downloading sources Initializing build dir Committing stage init to cache Starting build of com.example.meek-client ======================================================================== Building module meek-client in /home/user/code/flatpak-builder-tools/go-get/.flatpak-builder/build/meek-client-1 ======================================================================== Running: go install git.torproject.org/pluggable-transports/meek.git/[email protected] go: downloading git.torproject.org/pluggable-transports/meek.git v0.38.0 go: downloading git.torproject.org/pluggable-transports/goptlib.git v1.1.0 go: downloading golang.org/x/net v0.0.0-20220909164309-bea034e7d591 go: downloading github.com/refraction-networking/utls v1.1.5 go: downloading golang.org/x/crypto v0.0.0-20220829220503-c86fa9a7ed90 go: downloading github.com/klauspost/compress v1.15.9 go: downloading github.com/andybalholm/brotli v1.0.4 go: downloading golang.org/x/text v0.3.7 go: downloading golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10 debugedit: /home/user/code/flatpak-builder-tools/go-get/.flatpak-builder/rofiles/rofiles-Y6o1sc/files/bin/meek-client: DWARF version 0 unhandled compressing debuginfo in: /home/user/code/flatpak-builder-tools/go-get/.flatpak-builder/rofiles/rofiles-Y6o1sc/files/bin/meek-client processing: /home/user/code/flatpak-builder-tools/go-get/.flatpak-builder/rofiles/rofiles-Y6o1sc/files/bin/meek-client [25] .debug_abbrev compressed -> .zdebug_abbrev (307 => 289 94.14%) [26] .debug_line compressed -> .zdebug_line (529246 => 485044 91.65%) [27] .debug_frame compressed -> .zdebug_frame (104484 => 84780 81.14%) [28] .debug_gdb_scripts NOT compressed, wouldn't be smaller [29] .debug_info compressed -> .zdebug_info (875910 => 787285 89.88%) [30] .debug_loc compressed -> .zdebug_loc (671185 => 536563 79.94%) [31] .debug_ranges compressed -> .zdebug_ranges (177308 => 148978 84.02%) [9] Updating section string table stripping /home/user/code/flatpak-builder-tools/go-get/.flatpak-builder/rofiles/rofiles-Y6o1sc/files/bin/meek-client to /home/user/code/flatpak-builder-tools/go-get/.flatpak-builder/rofiles/rofiles-Y6o1sc/files/lib/debug/bin/meek-client.debug Committing stage build-meek-client to cache Cleaning up Committing stage cleanup to cache Finishing app Using meek-client as command Please review the exported files and the metadata Committing stage finish to cache Pruning cache
This built this simple Flatpak package, which basically ran go install to download meek-client version 0.38.0, along with all of its dependencies, and then compile them. Now that I've done this, I should be able to use the flatpak-go-get-generator.py script to create the up-to-date meek-client module for me.
Finally, let's see the magic work:
$ python3 flatpak-go-get-generator.py .flatpak-builder/build/meek-client/
Traceback (most recent call last):
File "/home/user/code/flatpak-builder-tools/go-get/flatpak-go-get-generator.py", line 92, in <module>
main()
File "/home/user/code/flatpak-builder-tools/go-get/flatpak-go-get-generator.py", line 82, in main
source_list = sources(args.build_dir)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/code/flatpak-builder-tools/go-get/flatpak-go-get-generator.py", line 68, in sources
return list(map(repo_source, repo_paths(build_dir)))
^^^^^^^^^^^^^^^^^^^^^
File "/home/user/code/flatpak-builder-tools/go-get/flatpak-go-get-generator.py", line 38, in repo_paths
for domain in domains:
File "/usr/lib/python3.11/pathlib.py", line 932, in iterdir
for name in os.listdir(self):
^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '.flatpak-builder/build/meek-client/src'
Hmm. While I can see that my .flatpak-builder/build/meek-client folder does indeed have all of the dependencies for meek-client downloaded, there's not a src folder in sight. What's going on?
🎵 Debugging noises get louder... 🎵
So it turns out, due to changes in the go ecosystem, the Go Get Generator is broken beyond repair. It's supposed to, basically, download and build your Go package from source, and then look at all of the dependencies that it had to download and compile the list of Flatpak modules based on the git repos and the specific commits it used. But modern versions of Go doesn't seem to git clone all of the dependencies anymore.
Debugging flatpak-builder-tools
Since I need to figure out how to finish making this Flatpak package, and the Go Get Generator in flatpak-builder-tools is broken, I decided to program my own replacement. I forked the flatpak-builder-tools repo, deleted the broken go-get folder, created a new go folder, and wrote my own new script, flatpak-go-deps.py. Here's my pull request to flatpak-builder-tools project.
It's not merged upstream at the time of writing (it's still a draft PR), but in the meantime you can see the new code I contributed, including the readme, at https://github.com/micahflee/flatpak-builder-tools/tree/fix-go/go. The script I wrote, at the moment, is 282 lines of code.
I'm going to start over with meek-client, but this time using my new script:
$ ./flatpak-go-deps.py git.torproject.org/pluggable-transports/meek.git/[email protected]
go: creating new go.mod: module tempmod
Cloning into 'src/meek-client'...
warning: redirecting to https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/meek.git/
remote: Enumerating objects: 2676, done.
remote: Counting objects: 100% (658/658), done.
remote: Compressing objects: 100% (281/281), done.
remote: Total 2676 (delta 372), reused 658 (delta 372), pack-reused 2018
Receiving objects: 100% (2676/2676), 549.97 KiB | 527.00 KiB/s, done.
Resolving deltas: 100% (1546/1546), done.
Note: switching to 'v0.38.0'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at 3be00b7 programVersion = "0.38.0"
build-commands:
- . /usr/lib/sdk/golang/enable.sh; export GOPATH=$PWD; export GO111MODULE=off; go
install git.torproject.org/pluggable-transports/meek.git/meek.git
build-options:
env:
GOBIN: /app/bin/
buildsystem: simple
name: meek-client
sources:
- dest: src/git/torproject/org/pluggable-transports/goptlib/git
tag: v1.1.0
type: git
url: https://git.torproject.org/pluggable-transports/goptlib.git.git
- dest: src/github/com/andybalholm/brotli
tag: v1.0.4
type: git
url: https://github.com/andybalholm/brotli.git
- dest: src/github/com/klauspost/compress
tag: v1.15.9
type: git
url: https://github.com/klauspost/compress.git
- dest: src/github/com/refraction-networking/utls
tag: v1.1.5
type: git
url: https://github.com/refraction-networking/utls.git
- dest: src/golang/org/x/crypto
tag: v0.0.0-20220829220503-c86fa9a7ed90
type: git
url: https://golang.org/x/crypto.git
- dest: src/golang/org/x/net
tag: v0.0.0-20220909164309-bea034e7d591
type: git
url: https://golang.org/x/net.git
- dest: src/golang/org/x/sys
tag: v0.0.0-20220728004956-3c1f35247d10
type: git
url: https://golang.org/x/sys.git
- dest: src/golang/org/x/term
tag: v0.0.0-20210927222741-03fcf44c2211
type: git
url: https://golang.org/x/term.git
- dest: src/golang/org/x/text
tag: v0.3.7
type: git
url: https://golang.org/x/text.git
- dest: src/golang/org/x/tools
tag: v0.0.0-20180917221912-90fa682c2a6e
type: git
url: https://golang.org/x/tools.git
I then copy and paste the YAML into my OnionShare Flatpak manifest--though I'm reordering it slightly so name is at the top. This is what it looks like:
- name: meek-client build-commands: - . /usr/lib/sdk/golang/enable.sh; export GOPATH=$PWD; export GO111MODULE=off; go install git.torproject.org/pluggable-transports/meek.git/meek.git build-options: env: GOBIN: /app/bin/ buildsystem: simple sources: - dest: src/git/torproject/org/pluggable-transports/goptlib/git tag: v1.1.0 type: git url: https://git.torproject.org/pluggable-transports/goptlib.git.git - dest: src/github/com/andybalholm/brotli tag: v1.0.4 type: git url: https://github.com/andybalholm/brotli.git - dest: src/github/com/klauspost/compress tag: v1.15.9 type: git url: https://github.com/klauspost/compress.git - dest: src/github/com/refraction-networking/utls tag: v1.1.5 type: git url: https://github.com/refraction-networking/utls.git - dest: src/golang/org/x/crypto tag: v0.0.0-20220829220503-c86fa9a7ed90 type: git url: https://golang.org/x/crypto.git - dest: src/golang/org/x/net tag: v0.0.0-20220909164309-bea034e7d591 type: git url: https://golang.org/x/net.git - dest: src/golang/org/x/sys tag: v0.0.0-20220728004956-3c1f35247d10 type: git url: https://golang.org/x/sys.git - dest: src/golang/org/x/term tag: v0.0.0-20210927222741-03fcf44c2211 type: git url: https://golang.org/x/term.git - dest: src/golang/org/x/text tag: v0.3.7 type: git url: https://golang.org/x/text.git - dest: src/golang/org/x/tools tag: v0.0.0-20180917221912-90fa682c2a6e type: git url: https://golang.org/x/tools.git
Next, I'm doing the same with snowflake-client, and also obfs4proxy again for good measure:
./flatpak-go-deps.py git.torproject.org/pluggable-transports/snowflake.git/[email protected] ./flatpak-go-deps.py gitlab.com/yawning/obfs4.git/[email protected]
The one change I needed to do was add the command mv /app/bin/client /app/bin/snowflake-client to the end of build-commands in snowflake-client, since by default the binary it creates is just called client.
This looks like it should work, but I still need to test it to confirm that it actually works. (Spoiler: It doesn't.) But before I can test it by building the Flatpak package, I'm going to finish updating the rest of the Flatpak manifest files, specifically updating the Python dependencies.
Trying to update Python dependencies
Here's my documentation from RELEASE.md for how to go about updating all of the Python dependencies, for both onionshare-cli and onionshare, in the Flatpak manifest file:
pip3 install toml requirements-parser # clone flatpak-build-tools git clone https://github.com/flatpak/flatpak-builder-tools.git # get onionshare-cli dependencies cd poetry ./flatpak-poetry-generator.py ../../onionshare/cli/poetry.lock cd .. # get onionshare dependencies cd pip ./flatpak-pip-generator $(python3 -c 'import toml; print("\n".join(toml.loads(open("../../onionshare/desktop/pyproject.toml").read())["tool"]["poetry"]["dependencies"]))' |grep -vi onionshare_cli |grep -vi python | grep -vi pyside6 | grep -vi cx_freeze |tr "\n" " ") cd .. # convert to yaml ./flatpak-json2yaml.py -o onionshare-cli.yml poetry/generated-poetry-sources.json ./flatpak-json2yaml.py -o onionshare.yml pip/python3-modules.json
Hopefully this will just work with minimal fuss.
I'm going to start with the onionsharea-cli dependencies:
$ ./flatpak-poetry-generator.py ../../onionshare/cli/poetry.lock
Scanning "../../onionshare/cli/poetry.lock"
Traceback (most recent call last):
File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 166, in <module>
main()
File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 139, in main
dep_names = get_dep_names(parsed_lockfile, include_devel=include_devel)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 112, in get_dep_names
package["category"] == "dev"
~~~~~~~^^^^^^^^^^^^
KeyError: 'category'
Minimal fuss, it turns out, was too much to hope for. 🎵 Debugging noises... 🎵
Another flatpak-builder-tools rabbit hole
I popped open flatpak-poetry-generator.py to see what the problem is. The exception happened in this function:
def get_dep_names(parsed_lockfile: dict, include_devel: bool = True) -> list: """Gets the list of dependency names. Args: parsed_lockfile (dict): The dictionary of the parsed lockfile. include_devel (bool): Include dev dependencies, defaults to True. Returns (list): The dependency names. """ dep_names = [] for section, packages in parsed_lockfile.items(): if section == "package": for package in packages: if ( package["category"] == "dev" and include_devel and not package["optional"] or package["category"] == "main" and not package["optional"] ): dep_names.append(package["name"]) return dep_names
For debugging purposes, I added the following line to the beginning of the for loop that's looping through packages (before the if statement):
print(json.dumps(package, indent=2))
And I re-ran the script. This is the package that it choked on:
{ "name": "bidict", "version": "0.22.1", "description": "The bidirectional mapping library for Python.", "optional": false, "python-versions": ">=3.7", "files": [ { "file": "bidict-0.22.1-py3-none-any.whl", "hash": "sha256:6ef212238eb884b664f28da76f33f1d28b260f665fc737b413b287d5487d1e7b" }, { "file": "bidict-0.22.1.tar.gz", "hash": "sha256:1e0f7f74e4860e6d0943a05d4134c63a2fad86f3d4732fb265bd79e4e856d81d" } ], "extras": { "docs": [ "furo", "sphinx", "sphinx-copybutton" ], "lint": [ "pre-commit" ], "test": [ "hypothesis", "pytest", "pytest-benchmark[histogram]", "pytest-cov", "pytest-xdist", "sortedcollections", "sortedcontainers", "sphinx" ] } }
This package doesn't have a category key. Looking at the if statement, it seems that this basically says append the package to the list of dependencies if it's the dev category, include_devel is true, and the package isn't optional, or if it's in the main category and it's not optional. For this package, category doesn't seem to be set, so I'll modify the if statement like this:
if ( ("category" not in package and not package["optional"]) or ( package["category"] == "dev" and include_devel and not package["optional"] ) or (package["category"] == "main" and not package["optional"]) ):
Now if category isn't set, it adds it to the list anyway. I also added some extra parenthesis to make the logic more clear. Let's see if this did this trick...
$ ./flatpak-poetry-generator.py ../../onionshare/cli/poetry.lock
Scanning "../../onionshare/cli/poetry.lock"
Traceback (most recent call last):
File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 168, in <module>
main()
File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 157, in main
sources = get_module_sources(parsed_lockfile, include_devel=include_devel)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 67, in get_module_sources
package["category"] == "dev"
~~~~~~~^^^^^^^^^^^^
KeyError: 'category'
It's a similar error but it's on a different line of code, from this function:
def get_module_sources(parsed_lockfile: dict, include_devel: bool = True) -> list: """Gets the list of sources from a toml parsed lockfile. Args: parsed_lockfile (dict): The dictionary of the parsed lockfile. include_devel (bool): Include dev dependencies, defaults to True. Returns (list): The sources. """ sources = [] hash_re = re.compile(r"(sha1|sha224|sha384|sha256|sha512|md5):([a-f0-9]+)") for section, packages in parsed_lockfile.items(): if section == "package": for package in packages: if ( package["category"] == "dev" and include_devel and not package["optional"] or package["category"] == "main" and not package["optional"] ): # Check for old metadata format (poetry version < 1.0.0b2) if "hashes" in parsed_lockfile["metadata"]: hashes = parsed_lockfile["metadata"]["hashes"][package["name"]] # Else new metadata format else: hashes = [] for package_name in parsed_lockfile["metadata"]["files"]: if package_name == package["name"]: --snip--
I made the same change there and ran it again:
$ ./flatpak-poetry-generator.py ../../onionshare/cli/poetry.lock
Scanning "../../onionshare/cli/poetry.lock"
Traceback (most recent call last):
File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 170, in <module>
main()
File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 159, in main
sources = get_module_sources(parsed_lockfile, include_devel=include_devel)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/code/flatpak-builder-tools/poetry/./flatpak-poetry-generator.py", line 81, in get_module_sources
for package_name in parsed_lockfile["metadata"]["files"]:
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyError: 'files'
Okay... this time it's crashing because parsed_lockfile["metadata"] doesn't have a files key. You can see this line of code in the code block above. Notice the comments:
# Check for old metadata format (poetry version < 1.0.0b2)# Else new metadata format
This makes me think that it's possible there's even yet another new Poetry metadata format that my current poetry.lock file is using, but that flatpak-poetry-generator.py doesn't know about yet, and this is why it's crashing. When I open cli/poetry.lock, it includes a comment at the top saying:
# This file is automatically @generated by Poetry 1.5.1 and should not be changed by hand.
"This is too much work," I thought to myself.
So I decided to shift gears. I opened a bug report in the flatpak-builder-tools repo about this bug I've encountered. I'm going to stop trying to use flatpak-poetry-generator.py and instead just use the Flatpak PIP Generator for this.
Adding Poetry to requirements.txt script
The Flatpak PIP Generator let's you pass in a Python requirements.txt file as input and it generates a Flatpak manifest for those Python dependencies, in JSON format. Instead of trying to deal with poetry.lock files, I decided to write a script that converts the pyproject.toml files (where all of my Poetry dependencies are defined) into requirements.txt files. Here's the flatpak/poetry-to-requirements.py I just wrote:
#!/usr/bin/env python3 import toml import click def format_version(dep, version): if version == "*": return dep # If it's a dictionary, assume it's in the format {extras = ["socks"], version = "*"} elif isinstance(version, dict) and "version" in version: version = version["version"] if version == "*": return dep elif version.startswith("^"): return f"{dep}>={version[1:]}.0" elif version.startswith((">=", "<=", "!=", "==", "<", ">")): return f"{dep}{version}" else: return f"{dep}=={version}" elif version.startswith("^"): return f"{dep}>={version[1:]}.0" elif version.startswith((">=", "<=", "!=", "==", "<", ">")): return f"{dep}{version}" else: return f"{dep}=={version}" @click.command() @click.argument("pyproject_filename") def poetry_to_requirements(pyproject_filename): """Convert poetry dependencies in a pyproject.toml to requirements format.""" with open(pyproject_filename, "r") as f: data = toml.load(f) dependencies = data.get("tool", {}).get("poetry", {}).get("dependencies", {}) requirements = [] for dep, version in dependencies.items(): if dep == "python" or dep == "onionshare_cli": continue formatted = format_version(dep, version) if formatted: requirements.append(formatted) for req in requirements: print(req) if __name__ == "__main__": poetry_to_requirements()
Here's the output when I run it on cli/pyproject.toml:
$ ./poetry-to-requirements.py ../cli/pyproject.toml
click
flask==2.3.2
flask-compress>=1.13.0
flask-socketio==5.3.4
psutil
pysocks
requests
unidecode
urllib3
eventlet
setuptools
pynacl
colorama
gevent-websocket
stem==1.8.1
waitress>=2.1.2.0
werkzeug>=2.3.4
And here's the output when I run it on desktop/pyproject.toml:
$ ./poetry-to-requirements.py ../desktop/pyproject.toml
PySide6==6.5.2
qrcode
werkzeug
python-gnupg
Excellent. So, now I'm going to use this new script, along with flatpak-pip-generator from flatpak-builder-tools:
$ cd flatpak-builder-tools/pip
$ ./flatpak-pip-generator $(../../onionshare/flatpak/poetry-to-requirements.py ../../onionshare/cli/pyproject.toml)
========================================================================
Downloading sources
========================================================================
Running: "pip3 download --exists-action=i --dest /tmp/pip-generator-python3-modules7s7_v9xw -r /tmp/requirements.gnj3l2ng"
Collecting click
Using cached click-8.1.7-py3-none-any.whl (97 kB)
Collecting flask==2.3.2
Using cached Flask-2.3.2-py3-none-any.whl (96 kB)
--snip--
Generating dependencies for stem
Generating dependencies for waitress
Generating dependencies for werkzeug
Output saved to python3-modules.json
It created the file python3-modules.json (in JSON format), so I'm going to use the script that comes with flatpak-builder-tools to convert it to YAML:
../flatpak-json2yaml.py ./python3-modules.json mv python3-modules.yml onionshare-cli.yaml
Now onionshare-cli.yaml should have the CLI dependencies. While I'm at it, I'll do the same thing for the GUI version:
$ ./flatpak-pip-generator $(../../onionshare/flatpak/poetry-to-requirements.py ../../onionshare/desktop/pyproject.toml)
========================================================================
Downloading sources
========================================================================
Running: "pip3 download --exists-action=i --dest /tmp/pip-generator-python3-modulesjl9ykpn9 -r /tmp/requirements.05kucdaf"
Collecting PySide6==6.5.2
Downloading PySide6-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl (6.7 kB)
Collecting qrcode
Downloading qrcode-7.4.2-py3-none-any.whl (46 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.2/46.2 kB 958.3 kB/s eta 0:00:00
Collecting werkzeug
Using cached werkzeug-2.3.7-py3-none-any.whl (242 kB)
Collecting python-gnupg
Downloading python_gnupg-0.5.1-py2.py3-none-any.whl (20 kB)
Collecting shiboken6==6.5.2
Downloading shiboken6-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl (174 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 174.2/174.2 kB 4.0 MB/s eta 0:00:00
Collecting PySide6-Essentials==6.5.2
Downloading PySide6_Essentials-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl (81.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 81.2/81.2 MB 10.0 MB/s eta 0:00:00
Collecting PySide6-Addons==6.5.2
Downloading PySide6_Addons-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl (126.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 126.3/126.3 MB 7.9 MB/s eta 0:00:00
Collecting typing-extensions
Downloading typing_extensions-4.7.1-py3-none-any.whl (33 kB)
Collecting pypng
Downloading pypng-0.20220715.0-py3-none-any.whl (58 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.1/58.1 kB 3.7 MB/s eta 0:00:00
Collecting MarkupSafe>=2.1.1
Using cached MarkupSafe-2.1.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28 kB)
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/PySide6-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/PySide6_Addons-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/PySide6_Essentials-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/shiboken6-6.5.2-cp37-abi3-manylinux_2_28_x86_64.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/qrcode-7.4.2-py3-none-any.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/werkzeug-2.3.7-py3-none-any.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/python_gnupg-0.5.1-py2.py3-none-any.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/MarkupSafe-2.1.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/pypng-0.20220715.0-py3-none-any.whl
Saved /tmp/pip-generator-python3-modulesjl9ykpn9/typing_extensions-4.7.1-py3-none-any.whl
Successfully downloaded PySide6 PySide6-Addons PySide6-Essentials shiboken6 qrcode werkzeug python-gnupg MarkupSafe pypng typing-extensions
========================================================================
Downloading arch independent packages
========================================================================
Traceback (most recent call last):
File "/home/user/code/flatpak-builder-tools/pip/./flatpak-pip-generator", line 291, in <module>
url = get_tar_package_url_pypi(name, version)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/code/flatpak-builder-tools/pip/./flatpak-pip-generator", line 91, in get_tar_package_url_pypi
raise Exception(err)
Exception: Failed to get shiboken6-6.5.2 source from https://pypi.org/pypi/shiboken6/6.5.2/json
Ahh yes, I knew there was a reason why I manually included PySide6 in the Flatpak manifest earlier--it's because you can't download architecture-independent versions of it from PyPI--it's only available for x86_64. I'll handle this by just grepping PySide6 out of the requirements.txt file:
./flatpak-pip-generator $(../../onionshare/flatpak/poetry-to-requirements.py ../../onionshare/desktop/pyproject.toml | grep -v PySide6)
That worked and generated a new python3-modules.json. Now I'll convert it to YAML too.
../flatpak-json2yaml.py ./python3-modules.json mv python3-modules.yml onionshare-desktop.yaml
Now that I have onionshare-desktop.yaml and onionshare-cli.yaml, I'm opening these files and copying and pasting the content into my Flatpak manifest for the onionshare and onionshare-cli modules. I'm finally ready to test it!
I updated RELEASE.md to explain all of these new steps, including using the new poetry-to-requirements.py script I wrote, and commited my changes.
Testing Flatpak
Alright, let's build the Flatpak package:
$ flatpak-builder build --force-clean --install-deps-from=flathub --install --user flatpak/org.onionshare.OnionShare.yaml
Dependency Sdk: org.kde.Sdk 6.4
Installing org.kde.Sdk/x86_64/6.4 from flathub
Info: org.kde.Sdk is end-of-life, with reason: We strongly recommend moving to the latest stable version of the Plaform and SDK
Info: org.kde.Sdk.Locale is end-of-life, with reason: We strongly recommend moving to the latest stable version of the Plaform and SDK
Installing runtime/org.freedesktop.Platform.GL.default/x86_64/22.08
Installing runtime/org.freedesktop.Platform.GL.default/x86_64/22.08-extra
^C
I pressed CTRL-C to cancel early because I noticed this warning:
Info: org.kde.Sdk is end-of-life, with reason: We strongly recommend moving to the latest stable version of the Plaform and SDK
When I run flatpak search org.kde.Sdk I see that the latest version of the org.kde.Sdk runtime is 6.5, but my Flatpak manifest file is using 6.4. I updated it to 6.5 and then tried again:
flatpak-builder build --force-clean --install-deps-from=flathub --install --user flatpak/org.onionshare.OnionShare.yaml
Great, it's not showing the warning when I use the new runtime. However, when it got to the "downloading sources" step, it crashed with this error:
Initialized empty Git repository in /home/user/code/onionshare/.flatpak-builder/git/https_filippo.io_edwards25519.git-3QRMA2/
remote: 404 page not found
fatal: repository 'https://filippo.io/edwards25519.git/' not found
Failed to download sources: module obfs4proxy: Child process exited with code 128
Unfortunately it looks like there's an issue with my Go dependencies--which basically means there's an issue with the flatpak-go-deps.py script I wrote in my flatpak-builder-tools PR. 🎵 Debugging noises... 🎵
Fixing flatpak-go-deps.py script
Here's the source from the Flatpak manifest that the flatpak-builder command above choked on:
- dest: src/filippo/io/edwards25519 tag: v1.0.0-rc.1.0.20210721174708-390f27c3be20 type: git url: https://filippo.io/edwards25519.git
After spending a lot of time looking into how Go package resolution works, and how Go figures out the git URLs for packages, I realized that I needed to update my flatpak-go-deps.py script to make HTTP requests and parse the responses in order to accurately discover git URLs.
It turns out, even though there's a Go package called filippo.io/edwards25519, and Go package names tend to map to git repo URLs, this isn't actually always the case, and https://filippo.io/edwards25519.git is not a valid git repo URL. I needed to make my code make an HTTP requests to https://filippo.io/edwards25519/?go-get=1 and then use BeautifulSoup to parse the response for a go-import meta tag, and that includes the real git URL, which in this case is https://github.com/FiloSottile/edwards25519.
But that wasn't the only problem. There were many more, including:
- The GitLab server that hosts code for
meek-clientandsnowflake-client,git.torproject.org, didn't seem to work properly with?go-get=1requests, so I had to make an exception for that. - I discovered that sometimes Go packages are pinned to git tags, but other times they're pinned to individual commits. In the block above it says the tag is
v1.0.0-rc.1.0.20210721174708-390f27c3be20, but that's not a real git tag. Instead, it's pinned to a commit, and the short version of the commit ID is390f27c3be20. - I realized this would be much more stable if I just use commit IDs instead of tags for everything, so I started to make the code git clone every repo and check out the correct tag, so I can look up the commit IDs.
- But then I realized that this takes forever to run and has to download gigabytes of source code, so I streamlined it by using the GitHub API for github.com sources, and the GitLab API for gitlab.com sources. This made it way faster, but I also started hitting GitHub API rate limits, so I added support for passing in a GitHub token to avoid the rate limits.
- There's a lot more to this particular rabbit hole too, but I'll spare the rest of the details... In short, I still haven't gotten it working all the way. 😭
In all, I spent about 6 hours (!?) fighting with my flatpak-go-deps.py script.
Instead of continuing the suffering, I decided that I just won't update the pluggable transports in the Flatpak version of this release. The versions running in OnionShare 2.6 don't have any security issues, so this will be a future me problem.
After all that work, I went ahead and replaced the obfs4proxy, meek-client, and snowflake-client sections of the Flatpak manifest file with the versions from OnionShare 2.6.
Giving up on Go dependencies, and finishing Flatpak packaging
And now, finally, the Flatpak package actually builds:
$ flatpak-builder build --force-clean --install-deps-from=flathub --install --user flatpak/org.onionshare.OnionShare.yaml
--snip--
Installing app/org.onionshare.OnionShare/x86_64/master
Pruning cache
Here's what happens when I try running it:
$ flatpak run org.onionshare.OnionShare
Traceback (most recent call last):
File "/app/bin/onionshare", line 33, in <module>
sys.exit(load_entry_point('onionshare==2.6.1', 'console_scripts', 'onionshare')())
File "/app/bin/onionshare", line 22, in importlib_load_entry_point
for entry_point in distribution(dist_name).entry_points
File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 969, in distribution
return Distribution.from_name(distribution_name)
File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 548, in from_name
raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for onionshare
Looking at the flatpak-builder logs, the onionshare-cli and onionshare Python packages actually failed with errors. Here are the logs for the onionshare-cli part:
========================================================================
Building module onionshare-cli in /home/user/code/onionshare/.flatpak-builder/build/onionshare-cli-1
========================================================================
Running: cd cli && python3 setup.py install --prefix=${FLATPAK_DEST}
running install
/usr/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!
********************************************************************************
Please avoid running ``setup.py`` directly.
Instead, use pypa/build, pypa/installer or other
standards-based tools.
See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
********************************************************************************
!!
self.initialize_options()
/usr/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!
********************************************************************************
Please avoid running ``setup.py`` and ``easy_install``.
Instead, use pypa/build, pypa/installer or other
standards-based tools.
See https://github.com/pypa/setuptools/issues/917 for details.
********************************************************************************
!!
self.initialize_options()
Checking .pth file support in /app/lib/python3.10/site-packages/
/usr/bin/python3 -E -c pass
TEST FAILED: /app/lib/python3.10/site-packages/ does NOT support .pth files
bad install directory or PYTHONPATH
You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from. The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
/app/lib/python3.10/site-packages/
and your PYTHONPATH environment variable currently contains:
''
Here are some of your options for correcting the problem:
* You can choose a different installation directory, i.e., one that is
on PYTHONPATH or supports .pth files
* You can add the installation directory to the PYTHONPATH environment
variable. (It must then also be on PYTHONPATH whenever you run
Python and want to use the package(s) you are installing.)
* You can set up the installation directory to support ".pth" files by
using one of the approaches described here:
https://setuptools.pypa.io/en/latest/deprecated/easy_install.html#custom-installation-locations
Please make the appropriate changes for your system and try again.
Here's the beginning of the onionshare-cli module in the Flatpak manifest:
- name: onionshare-cli buildsystem: simple build-commands: - cd cli && python3 setup.py install --prefix=${FLATPAK_DEST}
According to the error message, running setup.py directly is now deprecated. So, I replaced that build command with:
cd cli && pip3 install --prefix=${FLATPAK_DEST} --no-deps .
The onionshare module (the desktop version of the app) similarly was running setup.py directly, so I replaced that one too.
Then, I tried building the Flatpak package again:
$ flatpak-builder build --force-clean --jobs=$(nproc) --install-deps-from=flathub --install --user flatpak/org.onionshare.OnionShare.yaml
--snip--
========================================================================
Building module onionshare-cli in /home/user/code/onionshare/.flatpak-builder/build/onionshare-cli-1
========================================================================
Running: cd cli && pip3 install --prefix=${FLATPAK_DEST} --no-deps .
Processing /run/build/onionshare-cli/cli
Installing build dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [7 lines of output]
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6294e1bbe0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/poetry-core/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6294e1bf10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/poetry-core/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6294e1bfd0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/poetry-core/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6294e503a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/poetry-core/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6294e50550>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/poetry-core/
ERROR: Could not find a version that satisfies the requirement poetry-core (from versions: none)
ERROR: No matching distribution found for poetry-core
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Error: module onionshare-cli: Child process exited with code 1
🎵 Debugging noises drowning out all other sounds... 🎵
It's getting late on a Sunday night, so I think this is a good time to commit my code so far and then give up for the time being.
Pushing back the release date
In a few days, I'm going to Portugal for the Global Gathering Feira where I'll get to hang out with many human rights activists who specialize in internet freedom that I haven't seen in forever, and also meet new ones! During the event, I will do a two-hour project showcase for OnionShare, where people can come by and ask questions about the project. I'll also do a separate project showcase for my upcoming book, Hacks, Leaks, and Revelations: The Art of Analyzing Hacked and Leaked Data.
Last week I decided to sit down and make the OnionShare 2.6.1 release before the Global Gathering. So far I've spent (...checks notes...) 22 hours working on this release (and writing this blog post along with it), and I still haven't even finished Linux packaging yet. It's clear that I'm not going to make my goal of finishing the release before going to Portugal. Instead, I'll finish it when I get back.
I've pushed all of my code:
- The code for this release is in the
release-2.6.1branch, and here's the associated work-in-progress pull request: https://github.com/onionshare/onionshare/pull/1749 - My work-in-progress pull request to fix Go support in flatpak-builder-tools is here: https://github.com/flatpak/flatpak-builder-tools/pull/369
If you're interested, feel free to fix all of these issues I'm running into while I'm gone! I will happily review your work and merge it into my PR.
And in the meantime, keep an eye out for part 2 of this post, where I will hopefully actually finish the release. I plan on documenting the rest of the process, including how to make polished, code-signed Windows and macOS packages, update the onionshare.org website and the documentation, publish the Homebrew package, and so on.
Subscribe to feed