Enable MediaTek NPU runtime via uses-native-library manifest entries#893
Open
jcwtw wants to merge 398 commits into
Open
Enable MediaTek NPU runtime via uses-native-library manifest entries#893jcwtw wants to merge 398 commits into
jcwtw wants to merge 398 commits into
Conversation
PiperOrigin-RevId: 811965499
Corrects the file extension from ".litermlm" to ".litertlm" in the Core Features list.
docs: Fix typo in .litertlm file extension
PiperOrigin-RevId: 815959499
Note: the cancellation exception will be used in an upcoming release of the litertlm API. PiperOrigin-RevId: 817754886
PiperOrigin-RevId: 819451826
PiperOrigin-RevId: 820431462
PiperOrigin-RevId: 822768793
Temporary disable benchmark in Gallery. We will reenable it once a new Maven package is released. PiperOrigin-RevId: 823574437
ExperimentalFlags is newly added to `0.0.0-alpha05`. PiperOrigin-RevId: 823610298
PiperOrigin-RevId: 824676152
PiperOrigin-RevId: 831537495
- Long message cannot be scrolled manually. - Copying message text can only copy partial text. Also, show error messages caught from the engine better. PiperOrigin-RevId: 831690400
…is no longer the default empty model. PiperOrigin-RevId: 832399534
…ages PiperOrigin-RevId: 832463657
…k's model list. PiperOrigin-RevId: 834335411
…owing up on launch (observed on certain vivo phones) PiperOrigin-RevId: 834838597
PiperOrigin-RevId: 836817743
PiperOrigin-RevId: 837565940
PiperOrigin-RevId: 838968861
PiperOrigin-RevId: 839002664
PiperOrigin-RevId: 914549933
PiperOrigin-RevId: 914978962
PiperOrigin-RevId: 914988398
…ption opens a link to GitHub discussions for exploring community-contributed skills. PiperOrigin-RevId: 915028065
…ification skill. PiperOrigin-RevId: 915044738
PiperOrigin-RevId: 915130429
PiperOrigin-RevId: 915136269
…ew try-out chip for "Schedule Reminder". Several skills ("calculate-hash", "kitchen-adventure", "text-spinner", "send-email") are now disabled by default. The corresponding try-out chips for these have been removed.
PiperOrigin-RevId: 915151168
PiperOrigin-RevId: 915507718
Integrates the **Model Context Protocol (MCP)** into the AI Edge Gallery Android app, specifically within the Agent Chat task. It allows the AI assistant to use tools provided by remote MCP servers.
### Key Changes:
* **MCP Integration**:
* Introduces `McpManagerViewModel` to handle server connections, tool discovery, and state management.
* Updates `AgentTools` with a new `runMcpTool` that communicates with MCP servers.
* **AI Agent Updates**:
* Modifies the `AgentChatTask` system prompt to include a routing logic between "Skills" (instruction-based) and "MCP Tools" (API-based).
* Injects enabled MCP tool definitions (name, description, schema) directly into the system prompt.
* **User Interface**:
* **Management UI**: Adds `McpManagerBottomSheet` and `McpToolManagerBottomSheet` for users to add (via URL), toggle, or delete MCP servers and individual tools.
* **Configuration**: Implements `AddMcpServerFromUrlDialog` with support for custom request headers for authorization.
* **Chat Integration**: Updates the chat input area to display an MCP server toggle and enabled skill/mcp count.
* **Persistence**:
* Defines a new `mcp.proto` and uses **Proto DataStore** (`McpServersSerializer`) to persist configured servers and their enablement states.
* Extends `UserData` proto to store MCP authentication data (e.g., API keys in headers).
* **General Improvements**:
* Refactors `LlmChatViewModel` and `ModelManagerViewModel` to use the correct way to update ui state.
* Enhances common UI components like `SmallFilledTonalButton` and `ClickableLink` for better reusability in the new MCP screens.
* Update the Config Dialog to show min/max slider values on labels so sliders can align better.
PiperOrigin-RevId: 915539581
PiperOrigin-RevId: 915693518
…any tool call triggered by the agent prompts the user for approval unless the tool has been explicitly marked as "Always allow." ### Key Changes: * **Permission Dialog:** Adds `McpToolCallPermissionDialog` which displays the tool name and its arguments (formatted as JSON) when a tool is about to be called. It offers three options: "Always allow," "Allow once," and "Don't allow." * **Persistence:** Updates the `mcp.proto` and `McpManagerViewModel` to store the "always allow" preference for each tool. * **Management UI:** Enhances the MCP tool management bottom sheet to show which tools have "always allow" enabled and provides a way for users to revoke this permission. * **Action Handling:** Integrates a new `AskMcpToolCallPermissionAction` into the `AgentTools` execution flow. This suspends tool execution until the user provides a response via the UI. * **Code Cleanup:** Refactors `McpManagerViewModel` to use a centralized `persistServers` helper method, reducing code duplication when updating the DataStore. PiperOrigin-RevId: 915751277
PiperOrigin-RevId: 916210621
PiperOrigin-RevId: 916210820
PiperOrigin-RevId: 916336529
PiperOrigin-RevId: 917305569
PiperOrigin-RevId: 917406504
PiperOrigin-RevId: 917458688
PiperOrigin-RevId: 917488696
…formatted floating point numbers in non-US locales (e.g. numbers like "8186,03") by passing raw value to `StatRow`. PiperOrigin-RevId: 917488806
Add doc for mcp
…Skills. The agent skill task now uses the default system prompt tailored for "skills only" if no MCP tools are loaded. This helps address some skill loading issue. PiperOrigin-RevId: 918659144
PiperOrigin-RevId: 919114845
PiperOrigin-RevId: 919168433
Added model allowlist for version 1.0.15 including multiple models with configurations and capabilities.
…patch-1 Add model allowlists for version 1.0.15
The MediaTek NPU adapter (libneuronusdk_adapter.mtk.so, bundled in
split_mediatek_runtime_v8) runtime-dlopens ~29 sibling MediaTek vendor
libs from /vendor/lib64/ during its static constructor. MediaTek
publishes these as part of the public app-namespace surface via
/system/etc/public.libraries-mtk.txt on supported devices.
Without matching <uses-native-library> entries in the app manifest,
Android's nativeloader refuses the dlopen and the adapter's constructor
crashes with a SIGSEGV (null function pointer) the first time the app
tries to use the MediaTek NPU. Observed on Vivo PD2417 (iQOO Z9 Turbo+,
D9300, firmware 16.2.9.2.W10.V000L1) running the official
Gemma3-1B-IT_q4_ekv1280_mt6989.litertlm bundle:
F libc: Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
in tid 30440 (DefaultDispatch), pid 30400 (ai.edge.gallery)
F DEBUG: Process uptime: 35s
#00 pc 0x0 <unknown>
google-ai-edge#1-03 libneuronusdk_adapter.mtk.so (constructor)
google-ai-edge#4 linker64 __dl__ZN6soinfo17call_constructorsEv+532
google-ai-edge#7 libdl.so dlopen+16
google-ai-edge#8-11 libLiteRtDispatch_MediaTek.so
After adding these manifest entries, Android logs:
D nativeloader: Extending system_exposed_libraries: libapuwareapusys.mtk.so:
libapuwareapusys_v2.mtk.so:...:libmvpuop25_mtk_nn.mtk.so:libmvpu_config.mtk.so
and Gemma 3 1B runs end-to-end on the D9300 MDLA NPU: CreateCompiledModel
returns rc=0, IsFullyAccelerated reports is_full=1, LiteRtRunCompiledModel
executes successfully, and the app generates text from the prompt
"What is the capital of Taiwan?" -> "The capital of Taiwan was Taipei."
All entries marked android:required="false" so the app continues to
install on non-MediaTek SoCs; the linker simply skips libs the firmware
doesn't expose. The library list is the union of (a) names referenced by
libneuronusdk_adapter.mtk.so's runtime dlopen (extracted with `strings`)
and (b) names published in MediaTek's public.libraries-mtk.txt on a
retail D9300 device.
Tested 2026-05-25 on Vivo PD2417 (D9300). Sign-off by uninstalling stock
Gallery 1.0.15, reinstalling the rebuilt + re-signed split APK set, and
verifying Gemma 3 1B NPU model runs without the previous SIGSEGV.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add 29
<uses-native-library>entries toAndroid/src/app/src/main/AndroidManifest.xmlso that the MediaTek NPU adapter —libneuronusdk_adapter.mtk.so(shipped in the existingsplit_mediatek_runtime_v8split APK) — candlopenits transitive vendor library dependencies on devices with a MediaTek SoC. Mirrors the existing entries for Qualcomm (libcdsprpc.so) and Google Tensor (libedgetpu_litert.so). All markedandroid:required="false"so non-MediaTek devices install and run as before.Why
libneuronusdk_adapter.mtk.so's static constructordlopens ~29 sibling MediaTek vendor libs (libmvpu_runtime_25_pub.mtk.so,libapuwarexrp_v2.mtk.so,libneuron_sys_util.mtk.so, …) from/vendor/lib64/. MediaTek publishes the full list in/system/etc/public.libraries-mtk.txton supported devices, but Android'snativeloaderonly routes those into an app's linker namespace if the app declares them via<uses-native-library>.Without these entries, on every device I've tested with the official
Gemma3-1B-IT_q4_ekv1280_mt6989.litertlmbundle the app crashes on first NPU invoke:The null PC is the function pointer the constructor would have populated from a sibling lib that the linker refused to load.
Verification
After applying the patch on Vivo PD2417 (iQOO Z9 Turbo+, MediaTek Dimensity 9300, firmware
PD2417_A_16.2.9.2.W10.V000L1, Android 16 / SDK 36) and re-signing/re-installing the split APK set,logcatshows the namespace extension working:End-to-end smoke test (
Gemma3-1B-IT_q4_ekv1280_mt6989.litertlm):LiteRtCreateCompiledModel(NPU)→rc=0LiteRtCompiledModelIsFullyAccelerated→is_full=1LiteRtRunCompiledModel→rc=0How the library list was derived
Union of:
strings libneuronusdk_adapter.mtk.sothat match^lib.*\.so$and are reachable from the constructor's dlopen chain./system/etc/public.libraries-mtk.txton a retail D9300 device.Libraries that exist in the adapter's lookup table but aren't in any public list (e.g.
libmdla_standalone.so,libedma_custom.so) are deliberately not declared — they appear to be optional probe paths the adapter tolerates failing.Test plan
required="false").Gemma3-1B-IT_q4_ekv1280_mt6989.litertlmloads withoutSIGSEGV.IsFullyAcceleratedreturns true.Scope
Manifest-only change. No code, no resources, no behavior change on non-MediaTek devices.