update stream docs

README.md

@@ -2,14 +2,32 @@
 Remote genaiscript host for integration into conversational AI applications.
 > This project is actively being developed to suit more use-cases, expect breaking changes.
 
+### Disclaimer
+This has not undergone a formal security assessment. You should do your own evaluation before using this.
+
+
+### Features not included in this fork
+- Capabilities API: Reports available agents via HTTP (useful for dynamic intent mapping)
+
+### Planned Features
+- Embed Model Context Protocol for client connectivity
+
+## Documentation
+
+Comprehensive documentation is available in the [docs](./docs) directory:
+
+- [Installation Guide](./docs/installation.md) - How to install and set up the project
+- [Configuration Guide](./docs/configuration.md) - Environment variables and configuration options
+- [API Documentation](./docs/api.md) - API endpoints and usage examples
+- [Authentication](./docs/tokens.md) - Authentication system documentation
+- [Agents Guide](./docs/agents.md) - How to create and use agents
+- [Input Documentation](./docs/input.md) - How input works for agents
+- [Stream Data Format](./docs/streams.md) - How stream data is formatted for clients
+
 
 ### Setup
 See [Installation](./docs/installation.md)
 
-
-### Disclaimer
-This has not undergone a formal security assessment. You should do your own evaluation before using this.
-
 ### How it works
 1. A chat client specifies the URL to this host in their environment.
 2. They send a request with their credentials to create a stream resource 
@@ -90,26 +108,12 @@ let cmd = match resource.as_str() {
 
 If your agent requires specific API keys or configuration, add them to the `ShimBinding` struct in `src/utils/utils.rs`.
 
-## Existing Agents
 
-The project currently includes the following agents:
-
-- **Web Search**: Performs web searches using SearxNG
-- **News Search**: Searches for news articles
-- **Image Generator**: Generates images based on text prompts
-- **Finance Query**: Provides financial information
-- **Web Scrape**: Scrapes content from web pages
-
-## Documentation
-
-Comprehensive documentation is available in the [docs](./docs) directory:
-
-- [Installation Guide](./docs/installation.md) - How to install and set up the project
-- [Configuration Guide](./docs/configuration.md) - Environment variables and configuration options
-- [API Documentation](./docs/api.md) - API endpoints and usage examples
-- [Authentication](./docs/tokens.md) - Authentication system documentation
-- [Agents Guide](./docs/agents.md) - How to create and use agents
-- [Input Documentation](./docs/input.md) - How input works for agents
+### Fast Agent Development Workflow
+1. Create script: create a new genaiscript script in `packages/genaiscript/genaisrc`
+2. Setup a development executor: Map a package script in `package.json` to the script in step 1 following the existing examples
+3. Iterate until agent is functional.
+4. Follow the guide on adding a new agent to integrate it into the rust server.
 
 ## License
 

docs/streams.md

@@ -0,0 +1,73 @@
+# Stream Data Format
+
+This document describes how the stream data is formatted as it comes across the wire to the client.
+
+## Overview
+
+The web-agent-rs uses Server-Sent Events (SSE) to stream data from agents to clients. This allows for real-time updates as the agent processes the request and generates responses.
+
+## Stream Format
+
+When you consume a stream resource by making a GET request to `/webhooks/:stream_id`, the server responds with a stream of data in the SSE format. Each piece of data from the agent is sent as an SSE event with the following format:
+
+```
+data: <content>\n\n
+```
+
+Where `<content>` is a line of output from the agent.
+
+### Stream Completion
+
+When the agent has finished processing and there is no more data to send, the server sends a final event to indicate the stream has completed:
+
+```
+data: [DONE]\n\n
+```
+
+This allows clients to know when the stream has ended and they can stop listening for events.
+
+## HTTP Headers
+
+The server includes the following HTTP headers in the response:
+
+- `Content-Type: text/event-stream` - Indicates that the response is an SSE stream
+- `Cache-Control: no-cache, no-transform` - Prevents caching of the stream
+- `Connection: keep-alive` - Keeps the connection open for the duration of the stream
+- `X-Accel-Buffering: yes` - Controls buffering behavior for certain proxies
+
+## Client-Side Handling
+
+Clients should use an EventSource or similar mechanism to consume the SSE stream. Here's an example of how to consume the stream using JavaScript:
+
+```javascript
+const eventSource = new EventSource('/webhooks/your-stream-id', {
+  headers: {
+    'Authorization': 'Bearer your-session-token'
+  }
+});
+
+eventSource.onmessage = (event) => {
+  if (event.data === '[DONE]') {
+    // Stream is complete, close the connection
+    eventSource.close();
+    return;
+  }
+  
+  // Process the data
+  console.log('Received data:', event.data);
+};
+
+eventSource.onerror = (error) => {
+  console.error('EventSource error:', error);
+  eventSource.close();
+};
+```
+
+## Data Content
+
+The content of each data event depends on the specific agent being used. For example:
+- Web search agents may return search results and snippets
+- News search agents may return article headlines and summaries
+- Image generator agents may return image URLs or base64-encoded images
+
+Refer to the specific agent documentation for details on the format of the data they return.