MayaTheShy/Starworld
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Refactor FUTURE_ENHANCEMENTS.md to streamline enhancement priorities and clarify implementation status

Browse Source
This commit is contained in:
MayaTheShy
2025-11-16 21:20:57 -05:00
parent 5a701447c7
commit 6359f92c2d
1 changed files with 24 additions and 451 deletions
Download Patch File Download Diff File Expand all files Collapse all files

475
docs/FUTURE_ENHANCEMENTS.md
View File

@@ -1,486 +1,59 @@
# Future Enhancements - Overte to Stardust Entity Rendering
## Overview
This document outlines the features that are **not yet implemented** but would be valuable additions to complete the Overte-to-Stardust integration. All core functionality for basic 3D model rendering is working, but these enhancements would improve visual fidelity and feature parity with native Overte.
All core functionality for basic 3D model rendering is implemented and stable. The following enhancements would improve visual fidelity and feature parity with native Overte. See [`docs/IMPLEMENTATION_COMPLETE.md`](IMPLEMENTATION_COMPLETE.md) and [`docs/ENTITY_TROUBLESHOOTING.md`](ENTITY_TROUBLESHOOTING.md) for current status and limitations.
## Priority 1: Visual Fidelity
### 1.1 Color Tinting for Models
## Visual Fidelity
**Status**: 🟡 Data captured, not applied
**Difficulty**: Medium
**Requires**: Extension to asteroids Model API or server-side material manipulation
- **Color Tinting for Models**: Data is parsed, stored, and logged, but not visually applied (pending StardustXR asteroids API extension).
- **Texture Application**: Texture URLs are parsed, textures are downloaded and cached, but not visually applied (pending StardustXR material API).
- **Transparency (Alpha) Support**: Alpha values are parsed and stored, but not visually applied (pending material API support).
**Current State**:
- Entity color values (RGB + alpha) are parsed from Overte packets ✅
- Color data is stored in the Rust bridge state ✅
- Color is logged during rendering ✅
- Color is NOT applied to model materials ❌
**What's Needed**:
1. **Option A: Asteroids API Extension**
```rust
Model::direct(&path)
.color_tint([r, g, b, a]) // New API needed
.build()
```
2. **Option B: Material Modification in reify()**
- Access model's materials after loading
- Multiply base color by tint color
- Update material in Stardust server
- Requires deeper integration with Bevy's material system
**Implementation Sketch**:
```rust
// In reify() function
match Model::direct(&model_path) {
Ok(model) => {
// Hypothetical API:
let tinted_model = if node.color != [1.0, 1.0, 1.0, 1.0] {
model.tint_color(node.color)
} else {
model
};
Some(tinted_model.build())
}
Err(e) => { /* ... */ }
}
```
**Alternative Approach**:
- Modify GLTF files on download to include color multiplier
- Use shader modifications via material extensions
- Server-side post-processing of materials
**References**:
- Stardust server: `src/nodes/drawable/model.rs`
- Bevy PBR materials: `bevy::pbr::StandardMaterial`
- GLTF color factors: `material.pbrMetallicRoughness.baseColorFactor`
### 1.2 Texture Application
**Status**: 🟡 Data captured, not applied
**Difficulty**: High
**Requires**: Texture download system + material texture binding API
**Current State**:
- Entity texture URLs are parsed from Overte packets ✅
- Texture URLs are stored in bridge state ✅
- Texture URLs are logged ✅
- Textures are NOT downloaded or applied ❌
**What's Needed**:
1. **Texture Downloader** (similar to ModelDownloader):
```rust
pub struct TextureDownloader {
cache_dir: PathBuf,
// Similar to ModelDownloader but for images
}
// Downloads .png, .jpg, .webp, etc.
fn download_texture(url: &str) -> Option<PathBuf>
```
2. **Material Texture Binding**:
- Load texture as Bevy Image asset
- Bind to model's material base color texture
- Handle UV mapping and texture coordinates
3. **Asteroids API Extension** (hypothetical):
```rust
Model::direct(&model_path)
.base_color_texture(&texture_path)
.build()
```
**Implementation Challenges**:
- GLTF models may already have embedded textures
- Overte textures should override embedded ones
- Need to handle texture tiling/repeat settings
- UV mapping compatibility between Overte and GLTF
**Workaround (Current)**:
- Use GLTF/GLB models with embedded textures
- Entity texture URLs are ignored
### 1.3 Transparency (Alpha) Support
**Status**: 🟡 Data captured, partially working
**Difficulty**: Low-Medium
**Requires**: Material alpha mode configuration
**Current State**:
- Entity alpha values parsed ✅
- Alpha stored with color data ✅
- Alpha logged but not applied ❌
- GLTF models can have embedded alpha ✅
**What's Needed**:
- Set material alpha mode based on entity.alpha value
- Configure blending for transparent materials
- Handle alpha testing vs alpha blending
**Implementation**:
```rust
// When alpha < 1.0, configure material for transparency
if node.color[3] < 1.0 {
// Set alpha mode on material
// May require material modification API
}
```
## Priority 2: Protocol Support
### 2.1 ATP Protocol (Overte Asset Server)
**Status**: 🔴 Not implemented
**Difficulty**: High
**Requires**: AssetClient integration, ATP protocol implementation
**Current State**:
- HTTP/HTTPS URLs download correctly ✅
- file:// URLs work ✅
- atp:// URLs are ignored ❌
**What's Needed**:
1. **ATP Client Implementation**:
- Connect to Overte asset server
- Authenticate with asset server credentials
- Request assets by hash
- Handle asset caching
2. **Integration with ModelCache**:
```cpp
// In StardustBridge::setNodeModel()
if (modelUrl.starts_with("atp:")) {
// Parse ATP URL: atp://hash.format
// Connect to asset server
// Download asset
// Cache locally
// Pass local path to bridge
}
```
**ATP URL Format**:
```
atp://<hash>.<extension>
Example: atp://8f3e9a1b2c4d5e6f.glb
```
**References**:
- Overte AssetClient: `libraries/networking/src/AssetClient.cpp`
- ATP protocol documentation in Overte source
## Protocol Support
### 2.2 Entity Script Execution
- **ATP Protocol (Overte Asset Server)**: Not implemented. Use HTTP URLs for now. atp:// support requires AssetClient integration.
- **Entity Script Execution**: Not implemented. Out of scope unless there is significant demand.
**Status**: 🔴 Not implemented
**Difficulty**: Very High
**Requires**: JavaScript engine, Overte API compatibility layer
**Current State**:
- Entity script URLs are not parsed
- Scripts are not executed
- No script API available
## Performance
**What's Needed**:
- JavaScript runtime (QuickJS, Deno, or V8)
- Overte entity script API implementation
- Script lifecycle management (load, update, unload)
- Sandboxing for security
- **Model Download Optimization**: HTTP/HTTPS download and caching is implemented. Async rendering and progress indicators are not yet implemented.
- **Cache Management**: Cache grows indefinitely; LRU eviction and manual management are not yet implemented.
- **In-Memory Model Caching**: Not implemented; would improve performance for repeated models.
**Out of Scope**: This is a massive undertaking and likely not worth it unless there's significant demand for scripted entities.
## Priority 3: Performance
## Entity Type Support
### 3.1 Model Download Optimization
- **Light, Text, Zone, ParticleEffect Entities**: Not implemented. Only Box, Sphere, Model are currently supported.
**Status**: 🟡 Works but could be better
**Difficulty**: Medium
**Current Issues**:
- First render of HTTP models blocks until download completes
- No visual indication of download progress to user
- No retry logic for failed downloads
## Dynamic Updates
**Improvements Needed**:
1. **Async Rendering Update**:
- Render placeholder while downloading
- Replace with actual model when download completes
- Requires frame update notification to bridge
- **Real-Time Entity Property Updates**: Transform updates work; other property changes (color, dimension, model URL) are not yet reflected in real time.
- **Physics Synchronization**: Not implemented; would require Stardust physics API integration.
2. **Progress Indicators**:
- Show download progress in XR (progress bar, spinner)
- Use Stardust UI elements for visual feedback
3. **Retry Logic**:
- Exponential backoff for failed downloads
- Max retry count
- Fallback to primitives after retries exhausted
## Advanced Features
4. **Parallel Downloads**:
- Download multiple models simultaneously
- Connection pooling
- Rate limiting to avoid overwhelming servers
- **Avatar Rendering**: Not implemented.
- **Spatial Audio**: Not implemented.
### 3.2 Cache Management
**Status**: 🟡 Works but unbounded
**Difficulty**: Low
**Current Issues**:
- Cache grows indefinitely
- No LRU eviction
- No cache size limits
- No cache cleanup on app exit
**Improvements Needed**:
1. **LRU Eviction**:
- Track last access time for each cached model
- Remove least recently used when cache exceeds size limit
- Configurable max cache size (default: 1GB)
2. **Cache Validation**:
- Check if remote models have been updated (HTTP ETag)
- Re-download if changed
- Configurable refresh policy
3. **Manual Cache Management**:
- CLI command to clear cache
- UI option in settings
- Selective deletion (by domain, by age, etc.)
### 3.3 In-Memory Model Caching
**Status**: 🔴 Not implemented
**Difficulty**: Medium
**Current State**:
- Primitive models loaded from disk every frame
- No in-memory caching of model data
- Redundant I/O for same models
**What's Needed**:
```rust
static MODEL_CACHE: OnceLock<Mutex<HashMap<PathBuf, Handle<Scene>>>> = OnceLock::new();
fn get_cached_model(path: &PathBuf) -> Option<Handle<Scene>> {
MODEL_CACHE.get()?.lock().unwrap().get(path).cloned()
}
```
**Benefits**:
- Faster rendering for repeated models
- Less disk I/O
- Better frame times
## Priority 4: Entity Type Support
### 4.1 Light Entities
**Status**: 🔴 Not implemented
**Difficulty**: Medium
**What's Needed**:
- Parse light type (point, spot, directional)
- Parse light color, intensity, range
- Create Stardust PointLight/SpotLight/DirectionalLight nodes
- Position and orient lights correctly
**Stardust API** (likely exists):
```rust
use stardust_xr_asteroids::elements::Light;
Light::point()
.color([r, g, b])
.intensity(intensity)
.range(range)
.build()
```
### 4.2 Text Entities
**Status**: 🔴 Not implemented
**Difficulty**: Low-Medium
**What's Needed**:
- Parse text content and font
- Parse text alignment and size
- Create Stardust Text nodes
- Handle line breaks and wrapping
**Stardust API** (exists):
```rust
use stardust_xr_asteroids::elements::Text;
Text::new(content)
.character_height(size)
.align_x(XAlign::Center)
.build()
```
### 4.3 Zone Entities
**Status**: 🔴 Not implemented
**Difficulty**: Medium
**What's Needed**:
- Parse zone dimensions and shape
- Parse zone properties (ambient light, gravity, etc.)
- Create Stardust zone/spatial with properties
- Handle enter/exit events
**Stardust API**:
```rust
Spatial::default()
.zoneable(true)
.zone_radius(radius)
.build()
```
### 4.4 ParticleEffect Entities
**Status**: 🔴 Not implemented
**Difficulty**: High
**What's Needed**:
- Parse particle emitter properties
- Create particle system in Stardust
- Handle particle textures and physics
- Synchronize particle state
**Challenge**: Stardust may not have particle system support yet.
## Priority 5: Dynamic Updates
### 5.1 Real-Time Entity Property Updates
**Status**: 🟡 Transform updates work, other properties don't
**Difficulty**: Medium
**Current State**:
- Transform (position, rotation) updates work ✅
- Color changes not reflected ❌
- Dimension changes not reflected ❌
- Model URL changes not reflected ❌
**What's Needed**:
- Detect property changes in SceneSync
- Call appropriate bridge update functions
- Trigger re-render in Rust bridge when properties change
- Incremental updates instead of full node recreation
**Implementation**:
```cpp
// In SceneSync.cpp
if (it->second.color != e.color) {
stardust.setNodeColor(nodeId, e.color, e.alpha);
}
if (it->second.dimensions != e.dimensions) {
stardust.setNodeDimensions(nodeId, e.dimensions);
}
// etc.
```
### 5.2 Physics Synchronization
**Status**: 🔴 Not implemented
**Difficulty**: Very High
**What's Needed**:
- Sync physics state from Overte
- Apply velocities and forces in Stardust
- Handle collisions and rigid body dynamics
- Keep physics in sync across network
**Challenge**: Stardust physics API is limited. May need custom integration.
## Priority 6: Advanced Features
### 6.1 Avatar Rendering
**Status**: 🔴 Not implemented
**Difficulty**: Very High
**What's Needed**:
- Parse avatar models and skeletons
- Sync avatar animations
- Handle avatar attachments and wearables
- Render other users' avatars
**Challenge**: Avatar system is complex and Overte-specific.
### 6.2 Spatial Audio
**Status**: 🔴 Not implemented
**Difficulty**: High
**What's Needed**:
- Connect to Overte audio mixer
- Receive audio streams
- Position audio sources in 3D space
- Mix and play audio in Stardust
**Challenge**: Requires audio mixer protocol implementation.
## Implementation Priority
### Phase 1: Visual Quality (1-2 weeks)
1. Color tinting - Requires asteroids API extension
2. Transparency/alpha - Material configuration
3. Texture support - Download + material binding
See [`docs/IMPLEMENTATION_COMPLETE.md`](IMPLEMENTATION_COMPLETE.md) for current status and priorities.
### Phase 2: Protocols (2-3 weeks)
1. ATP protocol support
2. Model download optimization
3. Cache management improvements
### Phase 3: Entity Types (1-2 weeks)
1. Light entities
2. Text entities
3. Zone entities (basic)
### Phase 4: Dynamic Features (2-3 weeks)
1. Real-time property updates
2. In-memory caching
3. Progress indicators
### Phase 5: Advanced (4+ weeks)
1. Physics synchronization
2. Avatar rendering
3. Spatial audio
4. Entity scripts
## Getting Help
### To Implement Color Tinting
- Contact Stardust developers about extending asteroids Model API
- Check if material tinting can be done via existing Material API
- Look into shader-based color multiplication
See [`docs/ENTITY_TROUBLESHOOTING.md`](ENTITY_TROUBLESHOOTING.md) for troubleshooting and [`docs/IMPLEMENTATION_COMPLETE.md`](IMPLEMENTATION_COMPLETE.md) for implementation details. For API extensions, contact StardustXR developers.
### To Implement ATP Protocol
- Study Overte's AssetClient implementation
- Document ATP packet format
- Create minimal ATP client in C++
### To Implement Physics
- Investigate Stardust's physics capabilities
- Determine if Rapier or other physics engine is available
- Design physics sync protocol
## Conclusion
The core rendering pipeline is **complete and functional**. These enhancements would improve visual fidelity and feature parity, but are not required for basic Overte world viewing in StardustXR.
Prioritize based on user needs:
- **Visual artists**: Color tinting, textures, transparency
- **Content creators**: ATP protocol, advanced entity types
- **Developers**: Physics sync, dynamic updates
- **Users**: Performance, cache management, progress indicators
---
**Document Version**: 1.0
**Last Updated**: November 9, 2025
**Status**: Planning Phase
The core rendering pipeline is complete and functional. Enhancements listed here would improve visual fidelity and feature parity, but are not required for basic Overte world viewing in StardustXR. See [`docs/IMPLEMENTATION_COMPLETE.md`](IMPLEMENTATION_COMPLETE.md) for the latest status.