From 0e4f1ba823219fff92b4b16ecea8d0932b68cd0c Mon Sep 17 00:00:00 2001 From: MayaTheShy Date: Sun, 9 Nov 2025 16:55:35 -0500 Subject: [PATCH] feat: add future enhancements documentation for Overte to Stardust entity rendering --- docs/FUTURE_ENHANCEMENTS.md | 486 ++++++++++++++++++++++++++++++++++++ 1 file changed, 486 insertions(+) create mode 100644 docs/FUTURE_ENHANCEMENTS.md diff --git a/docs/FUTURE_ENHANCEMENTS.md b/docs/FUTURE_ENHANCEMENTS.md new file mode 100644 index 0000000..5168ccf --- /dev/null +++ b/docs/FUTURE_ENHANCEMENTS.md @@ -0,0 +1,486 @@ +# Future Enhancements - Overte to Stardust Entity Rendering + +## Overview + +This document outlines the features that are **not yet implemented** but would be valuable additions to complete the Overte-to-Stardust integration. All core functionality for basic 3D model rendering is working, but these enhancements would improve visual fidelity and feature parity with native Overte. + +## Priority 1: Visual Fidelity + +### 1.1 Color Tinting for Models + +**Status**: 🟡 Data captured, not applied +**Difficulty**: Medium +**Requires**: Extension to asteroids Model API or server-side material manipulation + +**Current State**: +- Entity color values (RGB + alpha) are parsed from Overte packets ✅ +- Color data is stored in the Rust bridge state ✅ +- Color is logged during rendering ✅ +- Color is NOT applied to model materials ❌ + +**What's Needed**: +1. **Option A: Asteroids API Extension** + ```rust + Model::direct(&path) + .color_tint([r, g, b, a]) // New API needed + .build() + ``` + +2. **Option B: Material Modification in reify()** + - Access model's materials after loading + - Multiply base color by tint color + - Update material in Stardust server + - Requires deeper integration with Bevy's material system + +**Implementation Sketch**: +```rust +// In reify() function +match Model::direct(&model_path) { + Ok(model) => { + // Hypothetical API: + let tinted_model = if node.color != [1.0, 1.0, 1.0, 1.0] { + model.tint_color(node.color) + } else { + model + }; + Some(tinted_model.build()) + } + Err(e) => { /* ... */ } +} +``` + +**Alternative Approach**: +- Modify GLTF files on download to include color multiplier +- Use shader modifications via material extensions +- Server-side post-processing of materials + +**References**: +- Stardust server: `src/nodes/drawable/model.rs` +- Bevy PBR materials: `bevy::pbr::StandardMaterial` +- GLTF color factors: `material.pbrMetallicRoughness.baseColorFactor` + +### 1.2 Texture Application + +**Status**: 🟡 Data captured, not applied +**Difficulty**: High +**Requires**: Texture download system + material texture binding API + +**Current State**: +- Entity texture URLs are parsed from Overte packets ✅ +- Texture URLs are stored in bridge state ✅ +- Texture URLs are logged ✅ +- Textures are NOT downloaded or applied ❌ + +**What's Needed**: +1. **Texture Downloader** (similar to ModelDownloader): + ```rust + pub struct TextureDownloader { + cache_dir: PathBuf, + // Similar to ModelDownloader but for images + } + + // Downloads .png, .jpg, .webp, etc. + fn download_texture(url: &str) -> Option + ``` + +2. **Material Texture Binding**: + - Load texture as Bevy Image asset + - Bind to model's material base color texture + - Handle UV mapping and texture coordinates + +3. **Asteroids API Extension** (hypothetical): + ```rust + Model::direct(&model_path) + .base_color_texture(&texture_path) + .build() + ``` + +**Implementation Challenges**: +- GLTF models may already have embedded textures +- Overte textures should override embedded ones +- Need to handle texture tiling/repeat settings +- UV mapping compatibility between Overte and GLTF + +**Workaround (Current)**: +- Use GLTF/GLB models with embedded textures +- Entity texture URLs are ignored + +### 1.3 Transparency (Alpha) Support + +**Status**: 🟡 Data captured, partially working +**Difficulty**: Low-Medium +**Requires**: Material alpha mode configuration + +**Current State**: +- Entity alpha values parsed ✅ +- Alpha stored with color data ✅ +- Alpha logged but not applied ❌ +- GLTF models can have embedded alpha ✅ + +**What's Needed**: +- Set material alpha mode based on entity.alpha value +- Configure blending for transparent materials +- Handle alpha testing vs alpha blending + +**Implementation**: +```rust +// When alpha < 1.0, configure material for transparency +if node.color[3] < 1.0 { + // Set alpha mode on material + // May require material modification API +} +``` + +## Priority 2: Protocol Support + +### 2.1 ATP Protocol (Overte Asset Server) + +**Status**: 🔴 Not implemented +**Difficulty**: High +**Requires**: AssetClient integration, ATP protocol implementation + +**Current State**: +- HTTP/HTTPS URLs download correctly ✅ +- file:// URLs work ✅ +- atp:// URLs are ignored ❌ + +**What's Needed**: +1. **ATP Client Implementation**: + - Connect to Overte asset server + - Authenticate with asset server credentials + - Request assets by hash + - Handle asset caching + +2. **Integration with ModelCache**: + ```cpp + // In StardustBridge::setNodeModel() + if (modelUrl.starts_with("atp:")) { + // Parse ATP URL: atp://hash.format + // Connect to asset server + // Download asset + // Cache locally + // Pass local path to bridge + } + ``` + +**ATP URL Format**: +``` +atp://. +Example: atp://8f3e9a1b2c4d5e6f.glb +``` + +**References**: +- Overte AssetClient: `libraries/networking/src/AssetClient.cpp` +- ATP protocol documentation in Overte source + +### 2.2 Entity Script Execution + +**Status**: 🔴 Not implemented +**Difficulty**: Very High +**Requires**: JavaScript engine, Overte API compatibility layer + +**Current State**: +- Entity script URLs are not parsed +- Scripts are not executed +- No script API available + +**What's Needed**: +- JavaScript runtime (QuickJS, Deno, or V8) +- Overte entity script API implementation +- Script lifecycle management (load, update, unload) +- Sandboxing for security + +**Out of Scope**: This is a massive undertaking and likely not worth it unless there's significant demand for scripted entities. + +## Priority 3: Performance + +### 3.1 Model Download Optimization + +**Status**: 🟡 Works but could be better +**Difficulty**: Medium + +**Current Issues**: +- First render of HTTP models blocks until download completes +- No visual indication of download progress to user +- No retry logic for failed downloads + +**Improvements Needed**: +1. **Async Rendering Update**: + - Render placeholder while downloading + - Replace with actual model when download completes + - Requires frame update notification to bridge + +2. **Progress Indicators**: + - Show download progress in XR (progress bar, spinner) + - Use Stardust UI elements for visual feedback + +3. **Retry Logic**: + - Exponential backoff for failed downloads + - Max retry count + - Fallback to primitives after retries exhausted + +4. **Parallel Downloads**: + - Download multiple models simultaneously + - Connection pooling + - Rate limiting to avoid overwhelming servers + +### 3.2 Cache Management + +**Status**: 🟡 Works but unbounded +**Difficulty**: Low + +**Current Issues**: +- Cache grows indefinitely +- No LRU eviction +- No cache size limits +- No cache cleanup on app exit + +**Improvements Needed**: +1. **LRU Eviction**: + - Track last access time for each cached model + - Remove least recently used when cache exceeds size limit + - Configurable max cache size (default: 1GB) + +2. **Cache Validation**: + - Check if remote models have been updated (HTTP ETag) + - Re-download if changed + - Configurable refresh policy + +3. **Manual Cache Management**: + - CLI command to clear cache + - UI option in settings + - Selective deletion (by domain, by age, etc.) + +### 3.3 In-Memory Model Caching + +**Status**: 🔴 Not implemented +**Difficulty**: Medium + +**Current State**: +- Primitive models loaded from disk every frame +- No in-memory caching of model data +- Redundant I/O for same models + +**What's Needed**: +```rust +static MODEL_CACHE: OnceLock>>> = OnceLock::new(); + +fn get_cached_model(path: &PathBuf) -> Option> { + MODEL_CACHE.get()?.lock().unwrap().get(path).cloned() +} +``` + +**Benefits**: +- Faster rendering for repeated models +- Less disk I/O +- Better frame times + +## Priority 4: Entity Type Support + +### 4.1 Light Entities + +**Status**: 🔴 Not implemented +**Difficulty**: Medium + +**What's Needed**: +- Parse light type (point, spot, directional) +- Parse light color, intensity, range +- Create Stardust PointLight/SpotLight/DirectionalLight nodes +- Position and orient lights correctly + +**Stardust API** (likely exists): +```rust +use stardust_xr_asteroids::elements::Light; + +Light::point() + .color([r, g, b]) + .intensity(intensity) + .range(range) + .build() +``` + +### 4.2 Text Entities + +**Status**: 🔴 Not implemented +**Difficulty**: Low-Medium + +**What's Needed**: +- Parse text content and font +- Parse text alignment and size +- Create Stardust Text nodes +- Handle line breaks and wrapping + +**Stardust API** (exists): +```rust +use stardust_xr_asteroids::elements::Text; + +Text::new(content) + .character_height(size) + .align_x(XAlign::Center) + .build() +``` + +### 4.3 Zone Entities + +**Status**: 🔴 Not implemented +**Difficulty**: Medium + +**What's Needed**: +- Parse zone dimensions and shape +- Parse zone properties (ambient light, gravity, etc.) +- Create Stardust zone/spatial with properties +- Handle enter/exit events + +**Stardust API**: +```rust +Spatial::default() + .zoneable(true) + .zone_radius(radius) + .build() +``` + +### 4.4 ParticleEffect Entities + +**Status**: 🔴 Not implemented +**Difficulty**: High + +**What's Needed**: +- Parse particle emitter properties +- Create particle system in Stardust +- Handle particle textures and physics +- Synchronize particle state + +**Challenge**: Stardust may not have particle system support yet. + +## Priority 5: Dynamic Updates + +### 5.1 Real-Time Entity Property Updates + +**Status**: 🟡 Transform updates work, other properties don't +**Difficulty**: Medium + +**Current State**: +- Transform (position, rotation) updates work ✅ +- Color changes not reflected ❌ +- Dimension changes not reflected ❌ +- Model URL changes not reflected ❌ + +**What's Needed**: +- Detect property changes in SceneSync +- Call appropriate bridge update functions +- Trigger re-render in Rust bridge when properties change +- Incremental updates instead of full node recreation + +**Implementation**: +```cpp +// In SceneSync.cpp +if (it->second.color != e.color) { + stardust.setNodeColor(nodeId, e.color, e.alpha); +} +if (it->second.dimensions != e.dimensions) { + stardust.setNodeDimensions(nodeId, e.dimensions); +} +// etc. +``` + +### 5.2 Physics Synchronization + +**Status**: 🔴 Not implemented +**Difficulty**: Very High + +**What's Needed**: +- Sync physics state from Overte +- Apply velocities and forces in Stardust +- Handle collisions and rigid body dynamics +- Keep physics in sync across network + +**Challenge**: Stardust physics API is limited. May need custom integration. + +## Priority 6: Advanced Features + +### 6.1 Avatar Rendering + +**Status**: 🔴 Not implemented +**Difficulty**: Very High + +**What's Needed**: +- Parse avatar models and skeletons +- Sync avatar animations +- Handle avatar attachments and wearables +- Render other users' avatars + +**Challenge**: Avatar system is complex and Overte-specific. + +### 6.2 Spatial Audio + +**Status**: 🔴 Not implemented +**Difficulty**: High + +**What's Needed**: +- Connect to Overte audio mixer +- Receive audio streams +- Position audio sources in 3D space +- Mix and play audio in Stardust + +**Challenge**: Requires audio mixer protocol implementation. + +## Implementation Priority + +### Phase 1: Visual Quality (1-2 weeks) +1. Color tinting - Requires asteroids API extension +2. Transparency/alpha - Material configuration +3. Texture support - Download + material binding + +### Phase 2: Protocols (2-3 weeks) +1. ATP protocol support +2. Model download optimization +3. Cache management improvements + +### Phase 3: Entity Types (1-2 weeks) +1. Light entities +2. Text entities +3. Zone entities (basic) + +### Phase 4: Dynamic Features (2-3 weeks) +1. Real-time property updates +2. In-memory caching +3. Progress indicators + +### Phase 5: Advanced (4+ weeks) +1. Physics synchronization +2. Avatar rendering +3. Spatial audio +4. Entity scripts + +## Getting Help + +### To Implement Color Tinting +- Contact Stardust developers about extending asteroids Model API +- Check if material tinting can be done via existing Material API +- Look into shader-based color multiplication + +### To Implement ATP Protocol +- Study Overte's AssetClient implementation +- Document ATP packet format +- Create minimal ATP client in C++ + +### To Implement Physics +- Investigate Stardust's physics capabilities +- Determine if Rapier or other physics engine is available +- Design physics sync protocol + +## Conclusion + +The core rendering pipeline is **complete and functional**. These enhancements would improve visual fidelity and feature parity, but are not required for basic Overte world viewing in StardustXR. + +Prioritize based on user needs: +- **Visual artists**: Color tinting, textures, transparency +- **Content creators**: ATP protocol, advanced entity types +- **Developers**: Physics sync, dynamic updates +- **Users**: Performance, cache management, progress indicators + +--- + +**Document Version**: 1.0 +**Last Updated**: November 9, 2025 +**Status**: Planning Phase