bump(main/python): 3.13

[thunder-coding]

Requires some additional work, so don't get too excited.

DEFINITELY NOT READY FOR REVIEW. EXPECT A LOT OF SILLY COMMITS TO HAUNT YOU IN YOUR NOTIFICATIONS IF YOU ARE REVIEWING THIS PR AT THIS STAGE

The following is the significant part that needs to be done first:

• Move away from Ubuntu's python when building cross python. We need to bootstrap same version of build-python first. Thankfully I had done this a few days ago for an unrelated project, so I have the build scripts with me for an older version that works, I just need to port it for Python 3.13
• Get things building
• Get pip packages to build
• Full test suite locally on Termux + termux-docker, like I do for nodejs

Since nobody seems to have touched this, I am also thinking of taking up maintainership of python. Will try to keep this up to date with full test coverage

Packages that need to be rebuilt and tested:

Rebuild Part 1 #27768

Package	Status	Additional comments
packages/2ping	👍	-
packages/libllvm	-	lldb subpackage
packages/lilypond	👍	fixed builds due to hardcoded python binary

Rebuild Part 2 #27777

Package	Status	Additional comments
packages/xcb-proto	👍	python-xcbgen subpackage
packages/asciidoc	👍	-

Package	Status	Additional comments
packages/borgbackup	-	-
packages/dbus-python	-	-
packages/electrum	-	-
packages/fetchmail	-	-
packages/gobject-introspection	-	-
packages/gst-python	-	-
packages/libarrow-cpp	-	-
packages/libncnn	-	-
packages/libplacebo	-	-
packages/libtorrent-rasterbar	-	-
packages/libunbound	-	-
packages/libxml2	-	-
packages/manim	-	-
packages/matplotlib	-	-
packages/nala	-	-
packages/panda3d	-	-
packages/pybind11	-	-
packages/pycairo	-	-
packages/pygobject	-	-
packages/python-apsw	-	-
packages/python-apt	-	-
packages/python-bcrypt	-	-
packages/python-brotli	-	-
packages/python-cmake	-	-
packages/python-contourpy	-	-
packages/python-cryptography	-	-
packages/python-greenlet	-	-
packages/python-grpcio	-	-
packages/python-lameenc	-	-
packages/python-libsass	-	-
packages/python-llvmlite	-	-
packages/python-lxml	-	-
packages/python-msgpack	-	-
packages/python-numpy	-	-
packages/python-onnxruntime	-	-
packages/python-pillow	-	-
packages/python-pip	-	-
packages/python-pycryptodomex	-	-
packages/python-sabyenc3	-	-
packages/python-scipy	-	-
packages/python-skia-pathops	-	-
packages/python-tflite-runtime	-	-
packages/python-tldp	-	-
packages/python-torch	-	-
packages/python-torchaudio	-	-
packages/python-torchcodec	-	-
packages/python-torchvision	-	-
packages/python-xlib	-	-
packages/python-yt-dlp	-	-
packages/ranger	-	-
packages/rdiff-backup	-	-
packages/recoll	-	-
packages/ruff	-	-
packages/seafile-client	-	-
packages/speechd	-	-
packages/tinysparql	-	-
packages/vapoursynth	-	-
packages/yt-dlp-ejs	-	-
x11-packages/audiotube	-	-
x11-packages/gjots2	-	-
x11-packages/gnumeric	-	-
x11-packages/godot	-	-
x11-packages/ibus	-	-
x11-packages/kitty	-	-
x11-packages/meld	-	-
x11-packages/opencv	-	-
x11-packages/orca	-	-
x11-packages/pyatspi	-	-
x11-packages/pyqt5	-	-
x11-packages/python-opencv-python	-	-
x11-packages/python-pyqtwebengine	-	-
x11-packages/python-qscintilla	-	-
x11-packages/xkeyboard-config	-	-

Additional packages that have python3.xx/site-packages as found using apt-file:

Package	Status	Additional comments
cinnamon	-	-
codelldb	-	-
frida	-	frida-python subpackage
gnucash	-	-
gnumeric	-	gnumeric-python subpackage
ledger	-	-
libsearpc	-	-
libxml2	-	libxml2-python subpackage
libarrow-cpp	-	python-pyarrow subpackage
python-pynvim	-	-
python-ruff	-	-
python-xapp	-	-
libunbound	-	pyunbound subpackage
udocker	-	-
xapp	-	-

List generated using modified version of `scripts/generate-apt-packages-list.sh`:

#!/usr/bin/env bash
set -euo pipefail

# This script generates a list of packages with their versions and other details to make it easier for parsing by check-repository-consistency.js
# Outputs scripts/apt-packages-list-<arch>.txt for each architecture
#
# Format of each line:
# <package_name> <repo-name> <version> <may_have_staticsplit>
#
# Usage:
# ./scripts/generate-apt-packages-list.sh "/path/to/output_dir"
#
# The script will generate "/path/to/output_dir/apt-packages-list-<arch>.txt"
# for aarch64, arm, i686 and x86_64

if [[ "$#" != 1 ]]; then
	echo 'Usage:'
	echo './scripts/generate-apt-packages-list.sh "/path/to/output_dir"'
	exit 1
fi


TERMUX_PACKAGES_DIR="$(realpath "$(dirname "$(realpath "$0")")/..")"
OUTPUT_DIR="$1"

readarray -t repo_paths <<< "$(jq --raw-output 'del(.pkg_format) | keys | .[]' "$TERMUX_PACKAGES_DIR/repo.json")"

for arch in "aarch64" "arm" "i686" "x86_64"; do
	# Note that this is loop for generating the list of packages is being parallelized for each architecture
	for repo_path in "${repo_paths[@]}"; do
		repo_name="$(jq --raw-output ".\"$repo_path\".name" "$TERMUX_PACKAGES_DIR/repo.json")"
		for pkg_path in "$TERMUX_PACKAGES_DIR/$repo_path"/*; do
			(
				APT_VERSION=
				export TERMUX_PKG_REVISION=0
				export TERMUX_PKG_NO_STATICSPLIT=false
				set +euo pipefail
				. "$pkg_path/build.sh" &> /dev/null || :
				if [ -n "$TERMUX_PKG_PYTHON_COMMON_DEPS" ] || [[ "$TERMUX_ON_DEVICE_BUILD" = "false" && -n "$TERMUX_PKG_PYTHON_BUILD_DEPS" ]] || [[ "$TERMUX_ON_DEVICE_BUILD" = "true" && -n "$TERMUX_PKG_PYTHON_TARGET_DEPS" ]]; then
					# Enable python setting
					TERMUX_PKG_SETUP_PYTHON=true
				fi
				if [[ "$TERMUX_PKG_SETUP_PYTHON" = "true" ]]; then
					echo "$pkg_path"
				fi
				set -euo pipefail
			)
		done
	done > "$OUTPUT_DIR/apt-packages-list-$arch.txt" & # Parallelize each architecture
done

wait

[robertkirkman]

For convenient reference, here is the series of commands which need to work consistently (no matter how many times they are repeated, even if repeated many times) when this is removed:

ssh -p 8022 192.168.12.185 # termux device ip address
rm -rf output
exit
rm -rf output
scripts/run-docker.sh ./build-package.sh -I -f python
scp -r -P 8022 output/ 192.168.12.185:~
ssh -p 8022 192.168.12.185
cd output/
apt reinstall ./*.deb
python -c "import _random"
exit

[robertkirkman]

Since nobody seems to have touched this, I am also thinking of taking up maintainership of python.

Wonderful! I had been planning to do this, but only after finishing my current python PR. My current python PR is still not finished, so you can feel free to work on this.

Is there any specific reason this isn't the newest version, 3.14.2?

[thunder-coding]

Is there any specific reason this isn't the newest version, 3.14.2?

I assume that python 3.14 might have some breaking changes that might not be suitable for everyone yet. Will upgrade to 3.13 now, then will target 3.14 in the next few months. I'm targeting python 3.13 by mid-January.

[robertkirkman]

Could you explain more about the justification for this change?

In my PR

• tree-wide: port debpython to termux  #23652

I have on-device build of Python implemented, it does require Python to already be installed, but that applies to all Termux packages, because if Python is not installed already when building Termux packages, then buildorder.py does not work so no packages can be built.

[thunder-coding]

Currently my priority is not on-device builds. I want to first fix docker builds. I'll be reverting this once testing is done on-device and on termux-docker for CI builds of python. And that was supposed to be TERMUX_PKG_ON_DEVICE_BUILD_NOT_SUPPORTED=true

[robertkirkman]

Note also:

• false is the default value of TERMUX_PKG_ON_DEVICE_BUILD_NOT_SUPPORTED, so this line as written doesn't do anything, which might not be what you were initially intending
• TERMUX_PKG_ON_DEVICE_BUILD_NOT_SUPPORTED=true is intended specifically for packages that edit files during the build that will not be tracked by the same package in the package manager, for example, qt6-qtwebengine is TERMUX_PKG_ON_DEVICE_BUILD_NOT_SUPPORTED=true because it edits the file $TERMUX_PREFIX/lib/librt.so during the build, which is a file normally tracked by the package ndk-sysroot. Does that situation apply to this python in some way?

termux-packages/x11-packages/qt6-qtwebengine/build.sh

Line 106 in b9da78b

rm $TERMUX_PREFIX/lib/lib{{pthread,resolv}.a,rt.so}

[robertkirkman]

Currently my priority is not on-device builds. I want to first fix docker builds. I'll be reverting this once testing is done

Thanks for explaining, I didn't see your response before posting my next message. I just wanted to let you know that TERMUX_PKG_ON_DEVICE_BUILD_NOT_SUPPORTED=true is not intended to be used with packages that are just untested or harmlessly broken for building on-device - it is intended specifically for packages that have been tested and found to be harmfully broken for building on-device, which is a detail that is unfortunately not clearly documented in Termux.

[robertkirkman]

Python 3.13 Upstream Android support (PEP738) test notes

• Appears to be working successfully (after working around the issue involving pip described above)
• pip install may download native libraries prebuilt for Android from some PyPi packages rather than building them all from source
• Whether some may include ADB-isms (like attempts to access absolute path /data/local/tmp), APK-isms (like attempts to access ANativeActivity or ANativeWindow), or Android-8-isms (like binaries built targeting a higher API level than 24 which then attempt to call functions like getloadavg()) remains unknown, but no cases of any of those have yet been observed
• PyPi packages containing native libraries prebuilt for Android identified using this list: https://beeware.org/mobile-wheels/

Test 1: pip install pypdfium2

• Worked without errors
• Important: testing indicates that pypdfium2 has implemented Android native library support compatible with Python 3.12, meaning that some, but not all, PyPi packages already contain Android native libraries that are successfully deployed to Termux currently, without necessarily requiring Python 3.13. Because of that, the behavior of pip install pypdfium2 is the same regardless of whether this PR is applied.

Test 2: pip install optree

• Worked without errors
• Testing indicates that, unlike the case of pypdfium2, optree leverages PEP738 to deploy its Android native library, meaning that pip install optree does behave differently in Python 3.13 from how it does in Python 3.12.
• Building wheel for optree did not print; instead, Downloading optree-0.18.0-cp313-cp313-android_24_x86_64.whl printed
• A $PREFIX/lib/python3.13/site-packages/optree/_C.cpython-313-x86_64-linux-android.so was installed
• file $PREFIX/lib/python3.13/site-packages/optree/_C.cpython-313-x86_64-linux-android.so prints this:
  • ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, for Android 24, built by NDK r27d (13750724), BuildID[sha1]=84f99286f68a16ce87232652f5558339de6ddb65, stripped
• Since it was built by NDK r27d, not NDK r28 or newer, this suggests that a possible side effect could be incompatibility of some of these PEP738 wheels with 16KB-page-size-only devices.
• optree basic program tested successfully:

#!/usr/bin/env python3

import optree

tree = {'b': (2, [3, 4]), 'a': 1, 'c': 5, 'd': 6}
optree.tree_flatten(tree)

Summary

• Appears to be working, no Termux-specific interventions appear to be necessary at this time, unless a particular PyPi package is found to be bugged
• Some PyPi packages may temporarily lose compatibility with 16KB-page-size-only devices until their respective upstreams upgrade to NDK r28 or newer; this seems minor, since there are very few 16KB-page-size-only devices.

[thunder-coding]

Thanks for the brief. I havent started testing yet. Just pushed the changed I had for proper cross-compilation I has been holding since some time. Will be busy this month with some stuff. Wont be moving ahead without running tests for Python

[robertkirkman]

Oh I'm sorry I forgot to tell you that this line needs to be outside of the if [[ ! -d "$_PYTHON_FOLDER" ]]; then block otherwise there will be an error when repeated builds are performed

I noticed that yesterday and I thought I told you but I forgot to