Add voice-first UI mode and MCP command executor improvements#1940
Merged
Add voice-first UI mode and MCP command executor improvements#1940
Conversation
- Add terminalUI module with EnhancedSpinner, TerminalLayout, InputBox, CompletionMenu classes - EnhancedSpinner supports output-above-spinner pattern and streaming text - InputBox provides bordered input areas with emoji-aware width handling - CompletionMenu adds trigger-based autocompletion (@ for agents, / for commands) - Add ESM module support and string-width dependency for Unicode handling - Include demo script and design documentation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add --testUI flag to CLI interactive command for enhanced terminal UI - Convert debug output to use 'debug' npm package across grammar/cache modules - Restore async grammar generation mode (fire-and-forget) for faster command responses - Add grammar result display when new rules are added to cache - Fix terminal input double-echo and cursor position issues - Fix prompt bracket overlap with emoji icons Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix manifest grammarFile paths to be manifest-relative (for patchPaths resolution) - Add NFA type extraction in AgentGrammar.validateEntityReferences to allow generated rules to reference types like TrackName, ArtistName from base grammar - Remove diagnostic console.log statements, keep debug() logging Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add single console.log showing generated rule or rejection reason - Remove verbose startup logging from setupGrammarGeneration - Keep debug() logging for detailed tracing when needed Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Include the generated rule in rejection messages to help diagnose entity validation failures and other rule addition errors. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Call registerBuiltInEntities() in setupGrammarGeneration to register Ordinal, Cardinal, CalendarDate converters - Make entity registry check case-insensitive (ordinal matches Ordinal) - Include generated rule in rejection messages for debugging Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…itive matching - Revert case-insensitive entity validation (too dangerous for all symbols) - Register lowercase aliases (ordinal, cardinal, calendarDate) alongside PascalCase versions to match paramSpec convention from .pas.json schemas Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Use ClientIO notify for grammar rule notifications instead of console.log - Log grammar notifications to ~/.typeagent/grammar.log in CLI - Suppress startup agent errors (use debug logging instead of error notifications) - Fix entityWildcardPropertyNames: exclude basic wildcard types (wildcard, word, number, string) from being treated as entity wildcards The entity wildcard fix resolves the smoke test failure where $(listName:wildcard) was incorrectly added to entityWildcardPropertyNames, causing "Invalid entity wildcard property name" errors. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The first getList rule was too greedy - "what's on the shopping list?" was matching with wildcard capturing "on the shopping" instead of just "shopping". Added a more specific rule for "what's on (my)? $(listName) (list)?" that takes precedence over the generic pattern. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…rminal rendering - Fix smoke test failure: "what's on the shopping list" was parsing "the shopping" as list name instead of "shopping" - added (the)? as optional article in grammar rules - Add marked and marked-terminal dependencies for CLI markdown rendering - Add convertMarkdownToTerminal function for styled terminal markdown output - Enhance HTML-to-text conversion with better formatting for track lists - Support DisplayType "markdown" in CLI content rendering Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Actions like pause, resume, next, previous, status don't have parameters
in their schema. The evaluators in dfaMatcher.ts and environment.ts were
always adding `parameters: {}` which caused validation to fail with:
"Action schema parameter type mismatch: player.pause"
Now only include parameters if there are any to include.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add grammar rules for: - next -> nextTrack - skip -> nextTrack - skip track -> nextTrack Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Remove custom selectors using invalid format names (orderedList, unorderedList, listItem, dataTable) that caused "format is not a function" errors. Use only valid built-in formats (block, inline, skip). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The rule (what's)? (show)? (me)? (the)? (my)? $(listName:wildcard) list could match with ALL optionals empty, becoming just $(listName:wildcard) list which captured everything before "list" as the list name. Split into specific rules that each require at least one fixed word: - what's (the)? (my)? $(listName:wildcard) list - show (me)? (the)? $(listName:wildcard) list - the $(listName:wildcard) list - my $(listName:wildcard) list This prevents "add bread, milk, flour to the shopping list" from being incorrectly parsed as getList with listName="add bread, milk, flour to the shopping" Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fix issue where "list?" wouldn't match "list" in grammar rules. Also adds local test file for faster grammar testing. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replace regex-based punctuation stripping with a simple loop that runs in O(n) time where n is the number of trailing punctuation chars. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add grammar normalization for passthrough rules before NFA construction Passthrough rules like `@ <S> = <C>` are converted to explicit form `@ <S> = $(_result:<C>) -> $(_result)` during preprocessing - Add popEnvironment flag for exiting nested rules without parent capture When rules like `(<Item>)?` create environments but the parent doesn't capture their result, we now properly pop the environment stack - Track anyRuleCreatedEnvironment to avoid unnecessary pops Rules like `(the)?` that don't create environments don't trigger a pop - Skip CalendarDate test pending grammar imports work Fixes "play the second track" returning trackNumber: undefined instead of 2 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement comprehensive Windows settings control through natural language: - 47 new settings actions across 14 categories (Network, Display, Taskbar, Mouse, Privacy, Power, Accessibility, etc.) - C# handlers in AutoShell_Settings.cs using Win32 APIs, Registry, WMI, and COM - Grammar patterns supporting 300+ natural language variations - Full TypeScript integration with action schema, connector mapping, and grammar compilation - 100% test pass rate (32/32 test cases) This enables users to control Windows settings like "turn on bluetooth", "increase brightness", "center taskbar", "set mouse speed to 12" through TypeAgent. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The TypeScript test file was causing compilation errors: - Missing Jest configuration and type definitions - Invalid import path for 'action-grammar' package - Unused variables flagged by compiler Functional testing is already provided by test-grammar-matching.mjs which successfully tests all 32 grammar patterns with 100% pass rate. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Follow the code agent pattern by organizing Windows settings actions into focused sub-groups for better discoverability and maintainability. **Changes:** - Keep 26 common actions at top level (window management, volume, wifi, bluetooth, brightness) - Organize 44 specialized actions into 7 sub-categories: - desktop-display: Night light, scaling, orientation (5 actions) - desktop-personalization: Transparency, title bars, contrast (3 actions) - desktop-taskbar: Auto-hide, alignment, widgets, clock (7 actions) - desktop-input: Mouse/touchpad settings (8 actions) - desktop-privacy: Camera, microphone, location access (3 actions) - desktop-power: Battery saver, power modes (3 actions) - desktop-system: Accessibility, file explorer, time, focus, etc. (15 actions) **Technical:** - Created `src/windows/` subdirectory with 7 schema files - Updated `manifest.json` with `subActionManifests` section - Created `allActionsSchema.ts` for internal type unions - Main `actionsSchema.ts` reduced from 704 to 274 lines - All actions remain fully functional via connector.ts - Build passes: tsc ✅ asc ✅ agc ✅ Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Keep refactored schema structure with sub-categories.
Include smoke tests, full test scenarios, debugging tips, and success criteria. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Update package.json to compile each sub-schema separately with asc:* scripts - Add compiledSchemaFile to all sub-action manifests (without grammarFile) - Update actionHandler to use AllDesktopActions type - Each sub-schema now generates its own .pas.json file for action resolution Note: Grammar system doesn't yet support schemaName field passthrough Need to implement post-processing or NFA matcher fix for sub-schema routing
Brings in 38+ commits with: - Enhanced terminal UI features - Grammar generation improvements - NFA matching fixes - CLI prompt and completion functionality # Conflicts: # ts/pnpm-lock.yaml
…ovements - NFA completion engine: accepts token arrays, token-over-wildcard preference, minimal (next-token) completions with property completions for checked wildcards - Shell requests completions only at token boundaries (spaces), filters locally between boundaries, backspace re-requests at last boundary - Grammar cache: strip optional LLM-inferred parameters instead of rejecting, add schema optionality info to ParameterValidationInfo - Calendar events open in embedded browser (target="_blank") like email links - Player agent: fix artist property name matching for NFA completions - Completion groups: add kind field (literal/entity), sorted groups support Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
# Conflicts: # ts/packages/cache/src/cache/grammarStore.ts # ts/packages/shell/src/renderer/assets/styles.less # ts/pnpm-lock.yaml
…ams from cache - Sort browser package.json devDependencies and scripts for policy compliance - Strip optional LLM-inferred parameters from grammar cache instead of rejecting - Apply prettier formatting fixes across actionGrammar and shell packages - Open calendar event links in embedded browser instead of system browser - Add backspace recovery for mid-word completion filtering Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ocal manifest paths - Remove console.log [GRAMMAR] diagnostics from grammarStore.ts (debug() calls remain) - Fix Spotify setVolume URL: remove stray ?volume_percent from base URL that caused double query param after getUrlWithParams was rewritten in PR #1923 - Fix playerLocal manifest to use agents/playerLocal/dist/... path convention matching getPackageFilePath expectations Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…adata - Add InlineSearchMenuUI: ghost text completions (like VS Code) as alternative to dropdown menu, with arrow key cycling and Tab accept - Add inlineCompletions setting toggle (ui.inlineCompletions, default true) controllable via @shell set ui.inlineCompletions true/false - Fix completion sort order to preserve backend group ordering (grammar completions like "by" appear before entity completions like song titles) - Fix NFA wildcard self-loop: don't emit property completions from loopState, only from entry state — wildcard is token+ so completions after consumed tokens come from following grammar, not the entity list again - Wire inline mode through SearchMenu, PartialCompletion, ChatView, SettingsView Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Apply token-boundary logic to @ command completions so partial words (e.g. "@config c") don't hit the backend, which fails to resolve the partial token and permanently poisons noCompletion. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…cuts - Show shell window immediately before restoring browser tabs - Parallelize browser tab restoration (21s → 3s) - Skip awaiting CDP setup for non-Google URLs (fire-and-forget) - Add "Restoring N browser tabs..." / "Browser tabs restored." notifications - Fall back to Claude reasoning when translation produces unknown actions - Change global focus shortcut from Ctrl+E to Alt+E - Add window-level Ctrl+E to focus chat input from browser tabs - Remove startup timing instrumentation - Disable agent greeting by default Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The agentGreeting default is now false, so the smoke tests can no longer wait for a greeting agent message to confirm startup. Instead, wait for the chat input element to have contenteditable="true", which signals that the dispatcher is initialized and the shell is ready for input. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…tching Replaces the old confirmation system with a generalized choice mechanism that works identically for in-process and out-of-process agents. Adds multi-token entity lookahead to the NFA interpreter so entity converters can match spans like "from 1-2pm" as a single CalendarTimeRange. Choice system: - ChoiceManager class stores callbacks agent-side (no closures over RPC) - Two choice types: yesNo (boolean) and multiChoice (checkbox selection) - Dispatcher routes responses to agents via handleChoice RPC - Shell renders Yes/No buttons or checkbox panels inline - CLI supports both modes with keyboard input NFA improvements: - Multi-token entity lookahead with skipCount on execution threads - Combined verified-part priority (fixed + checked) in match sorting - Entity-validated matches now rank equal to fixed string matches Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The y/n choice prompt was fire-and-forget, causing the main loop to show "Complete" and the next prompt before the user could respond. Now the main loop awaits a pendingChoicePromise before proceeding. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Build plain text results separately instead of regex-stripping HTML tags, which CodeQL flags as incomplete multi-character sanitization. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…spatcher schema RPC Voice UI: - Add voice mode setting (persisted, follows dark mode pattern) - Add VoiceOrb component with idle/wake-word-waiting/listening/thinking/speaking states - Add voice-mode CSS layer: orb animations, last-result card, mode banner - Wire orb state to speech recognition and message submission - Auto-start/stop wake word continuous listening when voice mode toggled - Change wake word from "hey type agent" to "type agent" with "typeagent" normalization - Add wakeWord to UISettings (default: "type agent") - Add voice mode checkbox to settings panel MCP Command Executor: - Add live agent schema resolution via dispatcher RPC (getAgentSchemas) - Add generateSchemaRegistry.mjs script and pre-built generatedSchemaRegistry.json fallback - Add naturalLanguage param to execute_action to populate TypeAgent NL cache - Fix apostrophe in JSON parameters using \u0027 Unicode escape - Update tool descriptions to steer Claude toward discover+execute for multi-step tasks Dispatcher: - Add ActionInfo, AgentSubSchemaInfo, AgentSchemaInfo types - Add getAgentSchemas(agentName?) RPC method returning live schema data - Wire through dispatcherTypes, dispatcherServer, dispatcherClient Shell agents: - Register code, email, image, photo, video agents in shell manifests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
steveluc
had a problem deploying
to
development-fork
GitHub Actions
Error
steveluc
had a problem deploying
to
development-fork
GitHub Actions
Error
) and fix package.json sort order - grammarGenerator.ts: Update system prompt examples and RuleRHS comments to use bare variable identifiers in action body value positions (e.g., "trackName" not "$(trackName)", "[artist]" not "[$(artist)]") to match the syntax change from PR #1939 - schemaToGrammarGenerator.ts: Update TimeSpec example from "-> $(t)" to "-> t" - packages/agents/code/package.json: Fix sort order per npm-package-sort-metadata policy - packages/commandExecutor/package.json: Fix sort order per npm-package-sort-metadata policy All 17 grammar test suites pass (352 tests). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
steveluc
had a problem deploying
to
development-fork
GitHub Actions
Error
steveluc
had a problem deploying
to
development-fork
GitHub Actions
Error
Without the trailing newline, CI prettier --check fails after npm run build regenerates the file. Adding "\n" ensures the generated JSON matches what prettier expects. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
steveluc
temporarily deployed
to
development-fork
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Voice-first UI mode: New
.voice-modeCSS layer for large-screen/AR/car use cases — animatedVoiceOrbcomponent with idle/wake-word-waiting/listening/thinking/speaking states, last-result glanceable card, mode banner, and settings toggle. Auto-starts/stops continuous wake word detection. Wake word changed from"hey type agent"→"type agent"with normalization for"typeagent".MCP command executor improvements: Live agent schema resolution via new
getAgentSchemas()dispatcher RPC (with static registry fallback). AddednaturalLanguageparam toexecute_actionso TypeAgent's NL cache gets populated. Fixed apostrophe-in-JSON bug using\u0027Unicode escape. Updated tool descriptions to steer Claude towarddiscover_agents+execute_actionfor multi-step orchestration.Dispatcher schema RPC: New
ActionInfo,AgentSubSchemaInfo,AgentSchemaInfotypes;getAgentSchemas(agentName?)wired through dispatcher types, server, and client.Shell agent manifests: Added code, email, image, photo, video agents to shell manifest registrations.
Test plan
pnpm run buildfromts/— no type errors@shell voice on→ body getsvoice-modeclass, orb appears, "Voice Mode" banner visiblethinking, response arrives → back toidlewake-word-waiting→listening→thinking→idle@shell voice off→ desktop UI unchanged, orb hiddenexecute_actionwith apostrophes in params (e.g., song titles) no longer breaks JSON parsingdiscover_agentsreturns live schema data from dispatcherexecute_actionwithnaturalLanguagepopulates TypeAgent NL cache🤖 Generated with Claude Code