Fixed app restart on Android 10, Python output redirect to logcat (flet-dev#187)

* Make setenv calls awaitable in Python init

Refactored the setenv function to return a Future<void> and updated all calls to setenv to use await. This ensures environment variables are set in the correct order before proceeding.

* Ensure single CPython interpreter per process

Introduces a guard to prevent repeated initialization and finalization of the CPython interpreter, addressing crashes on second launch with CPython 3.12. Also ensures the current native thread is registered with the GIL when running Python code, improving stability when re-entering Python from new Dart isolates or threads.

* Remove 'Warning:' prefix from debug message

Updated the debugPrint statement to remove the 'Warning:' prefix when unable to load libpyjni.so. This change streamlines log output for consistency.

* Revert "Ensure single CPython interpreter per process"

This reverts commit 3f4adc0.

* Refactor debug logging and Python runtime init

Replaces direct debugPrint calls with a private _debug function for consistent logging. Adds a check to only initialize the Python runtime if it is not already active, and removes unconditional Py_Finalize call to avoid finalizing an already active runtime.

* Add logcat forwarding for Python stdout/stderr

Introduces a Python initialization script that redirects stdout and stderr to Android logcat using a custom writer and logging handler. Adds a Dart function to inject this script into the Python interpreter, ensuring Python logs are visible in Android logcat. Also adds debug messages to trace script execution.

* Ensure GIL is held for all Python C API calls

Introduced a _withGIL helper to wrap all calls to the Python C API with GIL acquisition and release. This change improves thread safety and correctness when interacting with the Python interpreter from Dart, especially in isolate contexts.

* Refactor GIL usage in Python execution and error handling

Moved _withGIL calls to wrap larger code blocks instead of individual function calls in runPythonProgramInIsolate and getPythonError. This simplifies the code and reduces redundant GIL management, improving readability and maintainability.

* Add AppLifecycleListener to finalize Python on detach

Replaces flutter/foundation.dart with flutter/widgets.dart and adds an AppLifecycleListener to call Py_FinalizeEx when the app detaches. This ensures proper cleanup of the Python interpreter when the app lifecycle ends.

* Move Py_FinalizeEx call to end of isolate function

Removed AppLifecycleListener-based finalization and now explicitly call cpython.Py_FinalizeEx() at the end of runPythonProgramInIsolate. This ensures Python finalization occurs after the program finishes, improving resource management.

* Comment out Py_FinalizeEx call in runPythonProgramInIsolate

The calls to cpython.Py_FinalizeEx() and its debug log have been commented out in runPythonProgramInIsolate. This may be to prevent issues related to finalizing the Python interpreter in this context.

* Ensure Python interpreter finalization after execution

Refactored runPythonProgramInIsolate to always finalize the Python interpreter in a finally block, ensuring proper cleanup and GIL management after script or module execution.

* Add safe interpreter finalization for CPython

Introduced the _finalizeInterpreter function to safely finalize the CPython interpreter without releasing the GIL after Py_FinalizeEx, preventing fatal errors. Updated runPythonProgramInIsolate to use this new function for proper cleanup.

* Fix memory deallocation for native UTF-8 strings

Replaced incorrect use of Py_DecRef with malloc.free for freeing native UTF-8 strings allocated with toNativeUtf8. This ensures proper memory management and prevents potential memory leaks.

* Update interpreter finalization logic for Android

Refactored _finalizeInterpreter to remove unnecessary GIL release after Py_FinalizeEx. Interpreter is no longer finalized in runPythonProgramInIsolate to prevent crashes when re-initializing extension modules on Android.

* Comment out GIL management in _withGIL function

The GIL acquisition and release logic in _withGIL has been commented out, and the function now directly executes the action. Interpreter finalization code in runPythonProgramInIsolate remains commented out, with updated comments explaining the rationale.

* Enable GIL management and interpreter finalization

Uncommented code to properly acquire and release the Python GIL in _withGIL, and to finalize the interpreter after running a Python program in an isolate. This ensures thread safety and clean interpreter state between runs.

* Improve CPython isolate handling and logcat setup

Ensures logcat forwarding is idempotent across Dart isolate restarts and avoids finalizing the CPython interpreter between runs to prevent native crashes. Also refines isolate management to prevent killing isolates that may leave the interpreter in a bad state, and improves error handling for logcat forwarding setup.

* Refactor Python execution to use sub-interpreters

Each Python invocation now runs in a fresh sub-interpreter to reduce cross-run leakage and prevent GIL deadlocks. The logcat initialization script is now idempotent and only runs once per process. The unused _finalizeInterpreter function was removed.

* Add process termination support for Android plugin

Introduces a 'terminate' method to the Android plugin and Dart interface, allowing the app to kill its process for a clean restart. Also disables GIL management in _withGIL due to potential issues with embedded CPython state after Dart isolate restarts.

* Enable GIL management in _withGIL function

Uncommented and restored the logic to acquire and release the Python GIL in the _withGIL function, ensuring thread safety when interacting with the Python interpreter.

* Refactor CPython FFI execution and error handling

Simplifies the CPython FFI execution flow by removing sub-interpreter usage and the custom GIL management helper. Updates error handling, logging, and resource management to use debugPrint and proper finalization. Also improves asynchronous execution and logcat forwarding setup.

* Replace debugPrint with spDebug and add log utility

Introduced a new spDebug function in src/log.dart to standardize debug logging with a '[serious_python]' prefix. Updated all debugPrint calls in serious_python_android.dart and cpython.dart to use spDebug for consistent log output.

* Add debug logs for CPython loading and initialization

Added debug statements to log when CPython is loaded and when Python is already initialized, aiding in troubleshooting and execution flow visibility.

* Add debug print for missing asset hash file

Adds a debugPrint statement to log when the asset hash file is not found during asset extraction, aiding in debugging missing hash issues.

* Add debug prints for asset hash extraction and writing

Added debugPrint statements to log asset hash extraction, hash file writing, and conditional hash file operations in extractAssetOrFile. This improves traceability and debugging of asset handling.

* Refactor asset hash reading in extractAssetOrFile

Moved asset hash reading outside of the checkHash block to ensure it is always attempted. Removed redundant debug prints and simplified hash file writing logic.

* Remove terminate method from AndroidPlugin

Eliminated the handling of the 'terminate' method call, including the delayed process termination logic, from the AndroidPlugin class.

* Remove unused imports from AndroidPlugin.java

Deleted unused imports for Handler, Looper, and Process to clean up the code and improve maintainability.

* Remove terminate method from SeriousPythonAndroid

The terminate() override and its related comments were removed from SeriousPythonAndroid. This simplifies the class and removes the process termination logic previously handled via methodChannel.

* Bump version to 0.9.7 and update changelogs

Release 0.9.7 for all serious_python packages. This update fixes app restart on Android 10 and redirects Python output to logcat. All relevant pubspec, build, and podspec files have been updated to reflect the new version.

Author: FeodorFitsner

src/serious_python/CHANGELOG.md

@@ -1,3 +1,8 @@
+## 0.9.7
+
+* Fix app restart on Android 10.
+* Redirect Python output to logcat.
+
 ## 0.9.6
 
 * Make zipDirectory call asynchronous.

src/serious_python/pubspec.yaml

@@ -2,7 +2,7 @@ name: serious_python
 description: A cross-platform plugin for adding embedded Python runtime to your Flutter apps.
 homepage: https://flet.dev
 repository: https://github.com/cfms-dev/serious-python
-version: 0.9.6
+version: 0.9.7
 
 platforms:
   ios:

src/serious_python_android/CHANGELOG.md

@@ -1,3 +1,8 @@
+## 0.9.7
+
+* Fix app restart on Android 10.
+* Redirect Python output to logcat.
+
 ## 0.9.6
 
 * Make zipDirectory call asynchronous.

src/serious_python_android/android/build.gradle

@@ -1,5 +1,5 @@
 group 'com.flet.serious_python_android'
-version '0.9.6'
+version '0.9.7'
 
 def python_version = '3.14.2'
 

src/serious_python_android/lib/serious_python_android.dart

@@ -8,6 +8,7 @@ import 'package:path/path.dart' as p;
 import 'package:serious_python_platform_interface/serious_python_platform_interface.dart';
 
 import 'src/cpython.dart';
+import 'src/log.dart';
 
 /// An implementation of [SeriousPythonPlatform] that uses method channels.
 class SeriousPythonAndroid extends SeriousPythonPlatform {
@@ -43,13 +44,13 @@ class SeriousPythonAndroid extends SeriousPythonPlatform {
           .invokeMethod<String>('loadLibrary', {'libname': 'pyjni'});
       await setenv("FLET_JNI_READY", "1");
     } catch (e) {
-      debugPrint("Unable to load libpyjni.so library: $e");
+      spDebug("Unable to load libpyjni.so library: $e");
     }
 
     // unpack python bundle
     final nativeLibraryDir =
         await methodChannel.invokeMethod<String>('getNativeLibraryDir');
-    debugPrint("getNativeLibraryDir: $nativeLibraryDir");
+    spDebug("getNativeLibraryDir: $nativeLibraryDir");
 
     var bundlePath = "$nativeLibraryDir/libpythonbundle.so";
     var sitePackagesZipPath = "$nativeLibraryDir/libpythonsitepackages.so";
@@ -59,7 +60,7 @@ class SeriousPythonAndroid extends SeriousPythonPlatform {
     }
     var pythonLibPath =
         await extractFileZip(bundlePath, targetPath: "python_bundle");
-    debugPrint("pythonLibPath: $pythonLibPath");
+    spDebug("pythonLibPath: $pythonLibPath");
 
     var programDirPath = p.dirname(appPath);
 
@@ -73,7 +74,7 @@ class SeriousPythonAndroid extends SeriousPythonPlatform {
     if (await File(sitePackagesZipPath).exists()) {
       var sitePackagesPath = await extractFileZip(sitePackagesZipPath,
           targetPath: "python_site_packages");
-      debugPrint("sitePackagesPath: $sitePackagesPath");
+      spDebug("sitePackagesPath: $sitePackagesPath");
       moduleSearchPaths.add(sitePackagesPath);
     }
 

src/serious_python_android/lib/src/cpython.dart

@@ -3,14 +3,46 @@ import 'dart:ffi';
 import 'dart:isolate';
 
 import 'package:ffi/ffi.dart';
-import 'package:flutter/foundation.dart';
 import 'package:path/path.dart' as p;
 
 import 'gen.dart';
+import 'log.dart';
 
 export 'gen.dart';
 
 CPython? _cpython;
+String? _logcatForwardingError;
+const _logcatInitScript = r'''
+import sys, logging
+
+# Make this init idempotent across Dart isolate restarts.
+if not getattr(sys, "__serious_python_logcat_configured__", False):
+    sys.__serious_python_logcat_configured__ = True
+
+    from ctypes import cdll
+    liblog = cdll.LoadLibrary("liblog.so")
+    ANDROID_LOG_INFO = 4
+
+    def _log_to_logcat(msg, level=ANDROID_LOG_INFO):
+        if not msg:
+            return
+        if isinstance(msg, bytes):
+            msg = msg.decode("utf-8", errors="replace")
+        liblog.__android_log_write(level, b"serious_python", msg.encode("utf-8"))
+
+    class _LogcatWriter:
+        def write(self, msg):
+            _log_to_logcat(msg.strip())
+        def flush(self):
+            pass
+
+    sys.stdout = sys.stderr = _LogcatWriter()
+    handler = logging.StreamHandler(sys.stderr)
+    handler.setFormatter(logging.Formatter("%(levelname)s %(message)s"))
+    root = logging.getLogger()
+    root.handlers[:] = [handler]
+    root.setLevel(logging.DEBUG)
+''';
 
 CPython getCPython(String dynamicLibPath) {
   return _cpython ??= _cpython = CPython(DynamicLibrary.open(dynamicLibPath));
@@ -46,21 +78,35 @@ Future<String> runPythonProgramInIsolate(List<Object> arguments) async {
   var programDirPath = p.dirname(pythonProgramPath);
   var programModuleName = p.basenameWithoutExtension(pythonProgramPath);
 
-  debugPrint("dynamicLibPath: $dynamicLibPath");
-  debugPrint("programDirPath: $programDirPath");
-  debugPrint("programModuleName: $programModuleName");
+  spDebug("dynamicLibPath: $dynamicLibPath");
+  spDebug("programDirPath: $programDirPath");
+  spDebug("programModuleName: $programModuleName");
 
   final cpython = getCPython(dynamicLibPath);
+  spDebug("CPython loaded");
+  if (cpython.Py_IsInitialized() != 0) {
+    spDebug("Python already initialized, skipping execution.");
+    sendPort.send("");
+    return "";
+  }
+
   cpython.Py_Initialize();
-  debugPrint("after Py_Initialize()");
+  spDebug("after Py_Initialize()");
 
   var result = "";
 
+  final logcatSetupError = _setupLogcatForwarding(cpython);
+  if (logcatSetupError != null) {
+    cpython.Py_Finalize();
+    sendPort.send(logcatSetupError);
+    return logcatSetupError;
+  }
+
   if (script != "") {
     // run script
     final scriptPtr = script.toNativeUtf8();
     int sr = cpython.PyRun_SimpleString(scriptPtr.cast<Char>());
-    debugPrint("PyRun_SimpleString for script result: $sr");
+    spDebug("PyRun_SimpleString for script result: $sr");
     malloc.free(scriptPtr);
     if (sr != 0) {
       result = getPythonError(cpython);
@@ -76,7 +122,7 @@ Future<String> runPythonProgramInIsolate(List<Object> arguments) async {
   }
 
   cpython.Py_Finalize();
-  debugPrint("after Py_Finalize()");
+  spDebug("after Py_Finalize()");
 
   sendPort.send(result);
 
@@ -94,7 +140,7 @@ String getPythonError(CPython cpython) {
   cpython.Py_DecRef(tracebackModuleNamePtr.cast());
 
   if (tracebackModulePtr != nullptr) {
-    //debugPrint("Traceback module loaded");
+    //spDebug("Traceback module loaded");
 
     final formatFuncName = "format_exception".toNativeUtf8();
     final pFormatFunc = cpython.PyObject_GetAttrString(
@@ -127,3 +173,20 @@ String getPythonError(CPython cpython) {
     return "Error loading traceback module.";
   }
 }
+
+String? _setupLogcatForwarding(CPython cpython) {
+  if (_logcatForwardingError != null) {
+    return _logcatForwardingError;
+  }
+
+  final setupPtr = _logcatInitScript.toNativeUtf8();
+  final result = cpython.PyRun_SimpleString(setupPtr.cast<Char>());
+  malloc.free(setupPtr);
+
+  if (result != 0) {
+    _logcatForwardingError = getPythonError(cpython);
+    return _logcatForwardingError;
+  }
+
+  return null;
+}

src/serious_python_android/lib/src/log.dart

@@ -0,0 +1,10 @@
+import 'package:flutter/foundation.dart';
+
+void spDebug(String message) {
+  if (message.startsWith('[serious_python]')) {
+    debugPrint(message);
+  } else {
+    debugPrint('[serious_python] $message');
+  }
+}
+

src/serious_python_android/pubspec.yaml

@@ -2,7 +2,7 @@ name: serious_python_android
 description: Android implementation of the serious_python plugin
 homepage: https://flet.dev
 repository: https://github.com/flet-dev/serious-python
-version: 0.9.6
+version: 0.9.7
 
 environment:
   sdk: ">=3.0.0 <4.0.0"

src/serious_python_darwin/CHANGELOG.md

@@ -1,3 +1,8 @@
+## 0.9.7
+
+* Fix app restart on Android 10.
+* Redirect Python output to logcat.
+
 ## 0.9.6
 
 * Make zipDirectory call asynchronous.

src/serious_python_darwin/darwin/serious_python_darwin.podspec

@@ -4,7 +4,7 @@
 #
 Pod::Spec.new do |s|
   s.name             = 'serious_python_darwin'
-  s.version          = '0.9.6'
+  s.version          = '0.9.7'
   s.summary          = 'A cross-platform plugin for adding embedded Python runtime to your Flutter apps.'
   s.description      = <<-DESC
   A cross-platform plugin for adding embedded Python runtime to your Flutter apps.