Python Agent Packaging and Usage Guide

About 1598 wordsAbout 5 min

This document describes the packaging process and usage of the Python Agent in the M9A project, for reference and adaptation by other projects.

Overview

M9A's Python Agent is an automated task executor developed based on MaaFramework's Python bindings. The project adopts the following technical solutions:

Virtual Environment Management: Automatically creates and uses .venv virtual environment on Linux and in development mode
Dependency Management: Supports both local whl files and online mirror sources for installation
Configuration System: Manages pip sources, hot updates, and other options through JSON configuration files
Hot Update Mechanism: Supports online updates of resource files
Cross-Platform Support: Supports Windows, Linux, and macOS platforms

Project Structure

Source Code Structure

project/
├── .github/workflows/
│   └── install.yml              # CI/CD packaging workflow
├── agent/
│   ├── main.py                  # Agent main entry file
│   ├── utils/                   # Utility modules (logging, config, hot update, etc.)
│   └── custom/                  # Custom business logic
├── assets/                      # Resource files (images, configs, etc.)
├── tools/ci/
│   ├── setup_embed_python.py    # Setup embedded Python
│   ├── download_deps.py         # Download Python dependencies
│   ├── install.py               # Packaging script (Full version)
│   └── install_cli.py           # Packaging script (Lite version)
└── requirements.txt             # Python dependency list

Packaged Structure

install/                     # Packaging output directory (generated by CI/CD)
├── agent/
│   ├── main.py              # Agent main entry file
│   ├── utils/               # Utility modules (logging, config, hot update, etc.)
│   └── custom/              # Custom business logic
├── python/                  # Embedded Python environment (Windows/macOS)
│   ├── python.exe           # Windows
│   └── bin/python3          # macOS
├── deps/                    # Python dependency whl files
│   ├── package1.whl
│   ├── package2.whl
│   └── ...
├── resource/                # Resource folder from Assets
└── interface.json           # Version information

Agent Main Entry File Analysis

Core Functional Modules

agent/main.py contains the following core functionalities:

1. Virtual Environment Management

def ensure_venv_and_relaunch_if_needed():
    """
    Ensures venv exists, and if not already running in the script-managed venv,
    relaunches the script within it. Supports Linux, Windows, and macOS platforms.
    """

Workflow:

Check if currently running in a virtual environment (determined by sys.prefix != sys.base_prefix)
If not in a virtual environment:
- Check if .venv directory exists, create it if not
- Relaunch the script using Python from the virtual environment
- Exit the current process, returning the subprocess exit code
If already in the virtual environment, continue execution

Platform Differences:

Windows: Uses .venv/Scripts/python.exe
Linux/macOS: Prefers .venv/bin/python3, falls back to .venv/bin/python

Trigger Conditions:

Linux System: Always enabled (uses virtual environment even in production)
Development Mode (interface.json version is "DEBUG"): Enabled on all platforms
Windows/macOS Production: Not enabled (uses embedded Python)

2. Dependency Installation Mechanism

def install_requirements(req_file="requirements.txt", pip_config=None) -> bool:

Installation Strategy (by priority):

Local whl File Installation (highest priority)
- Check if .whl files exist in the deps/ directory
- Install from local using pip install --find-links deps/ --no-index
- Advantages: Offline availability, fast, version control
Online Mirror Source Installation (fallback when local fails)
- Use configured primary mirror source (default: Tsinghua mirror)
- Add backup mirror source (default: USTC mirror)
- Command example: pip install -i <primary> --extra-index-url <backup>
pip Global Configuration Installation (when no mirror configured)
- Use user's local pip configuration
- Suitable for environments with custom pip configuration

Important Parameters:

--no-warn-script-location: Suppress script location warnings
--break-system-packages: Allow installation in system Python (required for Linux)

3. Configuration System

The project uses JSON configuration files to manage various settings, located in the config/ directory:

pip_config.json

{
  "enable_pip_install": true,
  "mirror": "https://pypi.tuna.tsinghua.edu.cn/simple",
  "backup_mirror": "https://mirrors.ustc.edu.cn/pypi/simple"
}

hot_update.json

{
  "enable_hot_update": true
}

Configuration file features:

Automatically creates default configuration on first run
Uses default configuration when reading fails, without interrupting the program
Users can manually modify configuration files

4. Version Detection and Hot Update

def agent(is_dev_mode=False):
    # Version check
    version_info = check_resource_version()

    # Hot update (optimized based on manifest timestamp)
    manifest_result = check_manifest_updates()
    if manifest_result["has_any_update"]:
        update_result = check_and_update_resources()

Hot Update Process:

Check the timestamp of remote manifest files
Compare with local cache to determine if there are updates
Download only changed resource files
Update local manifest cache

CI/CD Packaging Workflow

Packaging Workflow Overview

The GitHub Actions workflow (.github/workflows/install.yml) defines the complete packaging process:

jobs:
  meta:      # Version number and release type determination
  windows:   # Windows platform packaging
  linux:     # Linux platform packaging
  macos:     # macOS platform packaging
  changelog: # Generate changelog
  release:   # Release to GitHub Releases

Platform-Specific Packaging Steps

Windows Platform

steps:
  # 1. Download MaaFramework
  - name: Download MaaFramework
    uses: robinraju/release-downloader@v1
    with:
      repository: MaaXYZ/MaaFramework
      fileName: "MAA-win-${{ matrix.arch }}*"
      tag: ${{ env.MAA_FRAMEWORK_VERSION }}
      out-file-path: "deps"
      extract: true

  # 2. Setup Embedded Python
  - name: Setup Embed Python on Windows
    run: python tools/ci/setup_embed_python.py

  # 3. Download Python dependencies
  - name: Download Python dependencies
    run: ./install/python/python.exe tools/ci/download_deps.py --deps-dir install/deps

  # 4. Execute packaging
  - name: Install
    run: ./install/python/python.exe ./tools/ci/install.py ${{ needs.meta.outputs.tag }} $PLATFORM_TAG

  # 5. Create CLI version
  - name: Create CLI version
    run: ./install/python/python.exe ./tools/ci/install_cli.py ${{ needs.meta.outputs.tag }}

Key Points:

Uses embedded Python (install/python/python.exe)
Dependencies downloaded to install/deps/ directory
Generates two versions: Full (with GUI) and Lite (CLI only)

Linux Platform

steps:
  # 1. Download MaaFramework
  - name: Download MaaFramework for Linux
    uses: robinraju/release-downloader@v1

  # 2. Download Python dependencies
  - name: Download Python dependencies
    run: python tools/ci/download_deps.py --deps-dir install/deps

  # 3. Execute packaging
  - name: Install on Linux
    run: python ./tools/ci/install.py ${{ needs.meta.outputs.tag }} $PLATFORM_TAG

  # 4. Create CLI version
  - name: Create CLI version
    run: python ./tools/ci/install_cli.py ${{ needs.meta.outputs.tag }}

Key Points:

Uses system Python (no embedded Python)
Automatically creates .venv virtual environment at runtime
Dependencies also downloaded to install/deps/ directory

macOS Platform

steps:
  # 1. Download MaaFramework
  - name: Download MaaFramework for macOS
    uses: robinraju/release-downloader@v1

  # 2. Setup Embedded Python
  - name: Setup Embed Python on macOS
    run: python3 tools/ci/setup_embed_python.py

  # 3. Download Python dependencies
  - name: Download Python dependencies
    run: ./install/python/bin/python3 tools/ci/download_deps.py --deps-dir install/deps

  # 4. Execute packaging
  - name: Install on macOS
    run: ./install/python/bin/python3 ./tools/ci/install.py ${{ needs.meta.outputs.tag }} $PLATFORM_TAG

Key Points:

Uses embedded Python (install/python/bin/python3)
Requires setting executable permissions: chmod +x install/python/bin/python3
Packaging process similar to Windows

Packaging Artifacts

Each platform generates two versions:

Full Version (M9A-{platform}-{arch}-{version}-Full)
- Contains complete GUI interface (MFAAvalonia)
- Includes Python Agent and all dependencies
- Includes MaaFramework runtime
- Suitable for regular users
Lite Version (M9A-{platform}-{arch}-{version}-Lite)
- Contains only command-line interface (MaaPiCli)
- Includes Python Agent and all dependencies
- Does not include GUI-related files
- Suitable for server deployment or automation scripts

Agent Usage

Startup Process

The Agent is launched through MaaFramework's AgentServer:

# main.py entry point
if __name__ == "__main__":
    main()

def main():
    # 1. Read version info, determine if in development mode
    current_version = read_interface_version()
    is_dev_mode = current_version == "DEBUG"

    # 2. Start virtual environment on Linux/development mode
    if sys.platform.startswith("linux") or is_dev_mode:
        ensure_venv_and_relaunch_if_needed()

    # 3. Check and install dependencies
    check_and_install_dependencies()

    # 4. Start AgentServer
    agent(is_dev_mode=is_dev_mode)

Command Line Arguments

The Agent needs to receive a socket_id parameter for communication with MaaFramework:

# Get socket_id from command line arguments
socket_id = sys.argv[-1]

# Start AgentServer
AgentServer.start_up(socket_id)
AgentServer.join()
AgentServer.shut_down()

Usage Example:

python agent/main.py <socket_id>

How to Adapt for Other Projects

1. Copy Core Files

Copy the following files from the M9A project to your project:

your-project/
├── .github/workflows/
│   └── install.yml                 # Modify as needed
├── agent/
│   ├── main.py                     # Copy and modify
│   └── utils/                      # Utility modules
│       ├── __init__.py             # Required: Module initialization, modify as needed
│       ├── logger.py               # Required: Logging module
│       ├── exceptions.py           # Optional: Exception definitions
│       ├── version_checker.py      # Optional: Version checking
│       ├── manifest_checker.py     # Optional: Hot update - manifest checking, needs modification
│       └── resource_updater.py     # Optional: Hot update - resource updating, needs modification
├── tools/ci/
│   ├── setup_embed_python.py       # Copy directly
│   ├── download_deps.py            # Copy directly
│   ├── install.py                  # Modify as needed
│   └── install_cli.py              # Modify as needed
└── requirements.txt                # Modify as needed

2. Key Modifications

agent/main.py

Parts that need modification:

Project Root Directory Setup

# Adjust according to your project structure
current_script_dir = os.path.dirname(current_file_path)
project_root_dir = os.path.dirname(current_script_dir)

Business Logic Section

def agent(is_dev_mode=False):
    # Keep common logic like virtual environment, dependency installation, config management
    # Modify the business logic section:

    # Replace with your business module
    from your_module import YourAgent

    # Replace with your initialization logic
    YourAgent.init()
    YourAgent.start()

Hot Update Logic (Optional)
- If hot update is not needed, delete related code
- If needed, implement utils.manifest_checker and utils.resource_updater modules

.github/workflows/install.yml

Parts that need modification:

Environment Variables

env:
  MAA_FRAMEWORK_VERSION: "v5.4.2"  # Change to your required version
  # Add your other dependency versions

Dependency Download Steps

# Modify according to your project dependencies
- name: Download MaaFramework
  uses: robinraju/release-downloader@v1
  with:
    repository: MaaXYZ/MaaFramework  # Change to your dependency repository
    fileName: "MAA-win-${{ matrix.arch }}*"
    tag: ${{ env.MAA_FRAMEWORK_VERSION }}

Packaging Steps
- Keep Python environment setup and dependency download steps
- Modify call parameters for install.py and install_cli.py
- Add or remove other steps as needed (e.g., icon conversion, file copying)

3. Important Notes

Platform Differences

Python Environment
- Windows/macOS: Use embedded Python, requires setup_embed_python.py
- Linux: Use system Python + virtual environment
Path Separators
- Use pathlib.Path to handle paths, automatically adapts to different platforms
- Avoid hardcoding / or \
Executable Permissions
- Linux/macOS require setting executable permissions: chmod +x
- Handle in CI/CD release steps

Dependency Management

Local whl Files Priority
- Download all dependencies to deps/ directory during packaging
- Users prioritize local whl on first run, no internet required
- Automatically fallback to online installation if local installation fails
Mirror Source Configuration
- Provide default mirror sources (Tsinghua, USTC, etc.)
- Allow users to customize mirror sources through configuration files
- Support dual-source strategy with primary + backup sources
Version Locking
- Lock dependency versions in requirements.txt
- Avoid compatibility issues due to dependency updates

Error Handling

Configuration File Read Failure
- Use default configuration, do not interrupt the program
- Log warnings for troubleshooting
Dependency Installation Failure
- Provide multiple installation strategies (local → online primary → online backup)
- Log detailed error information
- Provide user-friendly prompts
Virtual Environment Creation Failure
- Check Python version and venv module availability
- Provide clear error messages and solutions

Summary

M9A's Python Agent packaging solution has the following advantages:

Cross-Platform Support: Unified codebase supports Windows, Linux, and macOS platforms
Offline Availability: Packages all dependencies during packaging, users can run without internet
High Automation: Virtual environment management and dependency installation are fully automated
Flexible Configuration: Manages various settings through JSON configuration files
Strong Fault Tolerance: Multi-level fallback strategies ensure normal operation in various environments