6 Commits

Author	SHA1	Message	Date
geoffsee	9e28e259ad	Add support for listing available models via CLI and HTTP endpoint	2025-08-27 16:35:08 -04:00
geoffsee	432c04d9df	Removed legacy inference engine assets.	2025-08-27 16:19:31 -04:00
geoffsee	8338750beb	Refactor apply_cached_repeat_penalty for optimized caching and reuse, add extensive unit tests, and integrate special handling for gemma-specific models. ... Removed `test_request.sh`, deprecated functionality, and unused imports; introduced a new CLI tool (`cli.ts`) for testing inference engine and adjusted handling of non-streaming/streaming chat completions. - Add CPU fallback support for text generation when primary device is unsupported - Introduce `execute_with_fallback` method to handle device compatibility and shape mismatch errors - Extend unit tests to reproduce tensor shape mismatch errors specific to model configurations - Increase HTTP timeout limits in `curl_chat_stream.sh` script for reliable API testing chat completion endpoint functions with gemma3 (no streaming) Add benchmarking guide with HTML reporting, Leptos chat crate, and middleware for metrics tracking	2025-08-27 16:15:01 -04:00
geoffsee	b8ba994783	Integrate create_inference_router from inference-engine into predict-otron-9000, simplify server routing, and update dependencies to unify versions.	2025-08-16 19:53:33 -04:00
Geoff Seemueller	411ad78026	Remove stale reference in documentation.	2025-08-16 19:29:11 -04:00
geoffsee	2aa6d4cdf8	Introduce predict-otron-9000: Unified server combining embeddings and inference engines. Includes OpenAI-compatible APIs, full documentation, and example scripts.	2025-08-16 19:11:35 -04:00