geoffsee/open-web-agent-rs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

remove legacy tests/docs

Browse Source
This commit is contained in:
geoffsee
2025-06-05 21:50:38 -04:00
parent 9a84675ffa
commit 7f641559ae
10 changed files with 0 additions and 794 deletions
Download Patch File Download Diff File Expand all files Collapse all files

142
docs/agents.md
View File

@@ -1,142 +0,0 @@
# Agents Documentation
## Overview
Agents are the core components of web-agent-rs that perform specific tasks. Each agent is implemented as a GenAIScript file that defines its behavior and a Rust function that wraps the script.
## Available Agents
The following agents are currently available:
| Agent Type | Description | Resource Name |
|------------|-------------|---------------|
| Web Search | Performs web searches using SearxNG | `web-search` |
| News Search | Searches for news articles | `news-search` |
| Image Generator | Generates images based on text prompts | `image-generator` |
| Web Scrape | Scrapes content from web pages | `web-scrape` |
## Creating a New Agent
### 1. Create a GenAIScript File
Create a new `.genai.mts` file in the `packages/genaiscript/genaisrc/` directory. This file will contain the agent's logic.
Example structure of a GenAIScript file:
```typescript
import {SomeClient} from "@agentic/some-package";
import "./tools/some-tool.genai.mjs"
script({
title: "your_agent_name",
maxTokens: 8192,
cache: false,
tools: ["tool-name"],
});
def("USER_INPUT", env.vars.user_input);
$`You are an assistant that performs a specific task.
- Instruction 1
- Instruction 2
- Instruction 3`
```
### 2. Create a Rust Agent Function
Create a new Rust file in the `src/agents/` directory or add a function to an existing file. This function will be a wrapper that calls the GenAIScript file.
Example agent function:
```rust
use tokio::process::Child;
use tracing;
use crate::utils::utils::run_agent;
pub async fn agent(stream_id: &str, input: &str) -> Result<Child, String> {
run_agent(stream_id, input, "./packages/genaiscript/genaisrc/your-agent.genai.mts").await
}
```
### 3. Register the Agent in the Module
Add your agent to the `src/agents/mod.rs` file:
```rust
pub(crate) mod your_module;
```
### 4. Register the Agent in the Webhook Handler
Add your agent to the match statement in the `use_agent` function in `src/handlers/agents.rs`:
```
// In the use_agent function
let cmd = match resource.as_str() {
"web-search" => crate::agents::search::agent(agent_id.as_str(), &*input).await,
"news-search" => crate::agents::news::agent(agent_id.as_str(), &*input).await,
// Add your agent here
"your-resource-name" => crate::agents::your_module::agent(agent_id.as_str(), &*input).await,
_ => {
tracing::error!("Unsupported resource type: {}", resource);
return StatusCode::BAD_REQUEST.into_response();
}
};
```
### 5. Configure Environment Variables
If your agent requires specific API keys or configuration, add them to the `ShimBinding` struct in `src/utils/utils.rs`.
## Agent Tools
Agents can use various tools to perform their tasks. These tools are defined in the `packages/genaiscript/genaisrc/tools/` directory.
To use a tool in your agent, import it in your GenAIScript file:
```typescript
import "./tools/some-tool.genai.mjs"
```
And add it to the `tools` array in the `script` function:
```typescript
script({
title: "your_agent_name",
maxTokens: 8192,
cache: false,
tools: ["tool-name"],
});
```
## Testing Agents
You can test your agent by sending a request to the API:
```bash
curl -X POST https://your-server.com/api/webhooks \
-H "Authorization: Bearer <session_token>" \
-H "Content-Type: application/json" \
-d '{"resource": "your-resource-name", "input": "Your test input"}'
```
Then consume the stream to see the agent's response:
```bash
curl https://your-server.com/webhooks/<stream_id> \
-H "Authorization: Bearer <session_token>"
```
## Best Practices
1. Keep your GenAIScript files focused on a single task
2. Use appropriate tools for the task
3. Handle errors gracefully
4. Provide clear instructions in the agent's prompt
5. Test your agent thoroughly before deploying
## Related Documentation
- [Input Documentation](./input.md) - How input works for agents
- [API Documentation](./api.md) - API endpoints and usage examples

111
docs/api.md
View File

@@ -1,111 +0,0 @@
# API Documentation
## Authentication
All API endpoints require authentication using a session token. The token should be included in the `Authorization` header as a Bearer token:
```
Authorization: Bearer <session_token>
```
For more information on authentication and token generation, see the [Authentication](./tokens.md) documentation.
## Endpoints
### Health Check
```
GET /health
```
Returns a simple "ok" response to indicate that the server is running.
**Response:**
```
200 OK
ok
```
### Create Stream Resource
```
POST /api/webhooks
```
Creates a new stream resource for an agent.
**Request Body:**
```json
{
"resource": "web-search",
"input": "What is the capital of France?"
}
```
| Field | Type | Description |
|-------|------|-------------|
| `resource` | string | The type of agent to use (e.g., "web-search", "news-search", "image-generator", "web-scrape") |
| `input` | string | The input query or prompt for the agent |
**Response:**
```json
{
"stream_id": "abc123"
}
```
| Field | Type | Description |
|-------|------|-------------|
| `stream_id` | string | The ID of the created stream resource |
### Consume Stream Resource
```
GET /webhooks/:stream_id
```
Consumes a stream resource, returning the agent's response as a server-sent event stream.
**Path Parameters:**
- `stream_id`: The ID of the stream resource to consume
**Response:**
Server-sent event stream with the agent's response.
## Available Agents
The following agent types are available for use with the `resource` field in the Create Stream Resource endpoint:
| Agent Type | Description |
|------------|-------------|
| `web-search` | Performs web searches using SearxNG |
| `news-search` | Searches for news articles |
| `image-generator` | Generates images based on text prompts |
| `web-scrape` | Scrapes content from web pages |
## Error Responses
| Status Code | Description |
|-------------|-------------|
| 400 | Bad Request - The request was malformed or missing required fields |
| 401 | Unauthorized - Authentication failed or token is invalid |
| 404 | Not Found - The requested resource was not found |
| 500 | Internal Server Error - An unexpected error occurred |
## Example Usage
### Creating a Stream Resource
```bash
curl -X POST https://your-server.com/api/webhooks \
-H "Authorization: Bearer <session_token>" \
-H "Content-Type: application/json" \
-d '{"resource": "web-search", "input": "What is the capital of France?"}'
```
### Consuming a Stream Resource
```bash
curl https://your-server.com/webhooks/abc123 \
-H "Authorization: Bearer <session_token>"
```

50
docs/configuration.md
View File

@@ -1,50 +0,0 @@
# Configuration Guide
## Environment Variables
open-web-agent-rs uses environment variables for configuration. These can be set in a `.env` file in the root directory or directly in your environment.
### Required Environment Variables
| Variable | Description | Example |
|----------|-------------|---------|
| `OPENAI_API_KEY` | API key for OpenAI services | `sk-...` |
| `BING_SEARCH_API_KEY` | API key for Bing Search | `...` |
| `TAVILY_API_KEY` | API key for Tavily | `tvly-...` |
| `GENAISCRIPT_MODEL_LARGE` | Large language model to use with GenAIScript | `gpt-4-turbo` |
| `GENAISCRIPT_MODEL_SMALL` | Small language model to use with GenAIScript | `gpt-3.5-turbo` |
| `SEARXNG_API_BASE_URL` | Base URL for SearxNG API | `https://searxng.example.com` |
### Optional Environment Variables
You may also want to configure these optional variables:
| Variable | Description | Default | Example |
|----------|-------------|---------|---------|
| `PORT` | Port for the server to listen on | `3006` | `8080` |
| `LOG_LEVEL` | Logging level | `info` | `debug`, `info`, `warn`, `error` |
## Docker Configuration
When running with Docker, you can pass environment variables using the `--env-file` flag or by setting them in the `compose.yml` file:
```yaml
services:
web-agent:
build:
context: .
dockerfile: Local.Dockerfile
ports:
- "3006:3006"
environment:
- OPENAI_API_KEY=${OPENAI_API_KEY}
- BING_SEARCH_API_KEY=${BING_SEARCH_API_KEY}
- TAVILY_API_KEY=${TAVILY_API_KEY}
- GENAISCRIPT_MODEL_LARGE=${GENAISCRIPT_MODEL_LARGE}
- GENAISCRIPT_MODEL_SMALL=${GENAISCRIPT_MODEL_SMALL}
- SEARXNG_API_BASE_URL=${SEARXNG_API_BASE_URL}
```
## Authentication Configuration
The application uses FIPS204 signatures for authentication. See the [Authentication](./tokens.md) documentation for more details on configuring and using the authentication system.

45
docs/index.md
View File

@@ -1,45 +0,0 @@
# open-web-agent-rs Documentation
Welcome to the documentation for open-web-agent-rs, a GenAIScript host for integration into conversational AI applications.
## Table of Contents
- [Installation Guide](./installation.md) - How to install and set up the project
- [Configuration Guide](./configuration.md) - Environment variables and configuration options
- [API Documentation](./api.md) - API endpoints and usage examples
- [Authentication](./tokens.md) - Authentication system documentation
- [Agents Guide](./agents.md) - How to create and use agents
## Overview
open-web-agent-rs is a server that hosts GenAIScript agents for integration into conversational AI applications. It provides a simple API for creating and consuming stream resources that execute various agents to perform tasks like web search, news search, image generation, and web scraping.
## Architecture
The application is built with Rust using the Axum web framework. It uses GenAIScript for defining agent behavior and provides a streaming API for consuming agent responses.
### Key Components
1. **Server** - The main application server that handles HTTP requests and responses
2. **Agents** - GenAIScript files that define agent behavior
3. **Handlers** - Rust functions that process HTTP requests
4. **Authentication** - FIPS204 signature-based authentication system
5. **Configuration** - Environment variables for configuring the application
## Getting Started
To get started with open-web-agent-rs, follow these steps:
1. Read the [Installation Guide](./installation.md) to set up the project
2. Configure the application using the [Configuration Guide](./configuration.md)
3. Learn how to use the API with the [API Documentation](./api.md)
4. Understand the authentication system with the [Authentication](./tokens.md) documentation
5. Create your own agents using the [Agents Guide](./agents.md)
## Security Considerations
Please note that this project has not undergone a formal security assessment. You should do your own evaluation before using it in production environments.
## Contributing
Contributions to open-web-agent-rs are welcome! Please feel free to submit issues and pull requests to improve the project.

139
docs/input.md
View File

@@ -1,139 +0,0 @@
# Agent Input Documentation
## Overview
This document explains how input works for agents in the open-web-agent-rs project. Understanding how input is processed is essential for creating effective agents and integrating them with client applications.
## Input Flow
The input for agents follows this flow:
1. **API Request**: A client sends a request to the API with a resource type and input string.
2. **Webhook Handler**: The webhook handler extracts the input and passes it to the appropriate agent function.
3. **Agent Function**: The agent function passes the input to the `run_agent` utility function.
4. **ShimBinding**: The `run_agent` function creates a `ShimBinding` that passes the input to the GenAIScript shim as a command-line argument.
5. **GenAIScript**: The GenAIScript accesses the input through `env.vars.user_input` and typically assigns it to a variable named "USER_INPUT".
## Input Format
The input is a simple string that can contain any text. There are no specific format requirements, but the input should be relevant to the agent's purpose. For example:
- For a web search agent: "What is the capital of France?"
- For a news search agent: "Latest developments in artificial intelligence"
- For an image generator: "A sunset over a mountain lake with pine trees"
## Accessing Input in GenAIScript
In a GenAIScript file, the input is accessed through the `env.vars.user_input` property. It's common practice to assign this to a variable named "USER_INPUT" using the `def` function:
```typescript
def("USER_INPUT", env.vars.user_input);
```
You can then use this variable in your script's instructions:
```typescript
$`You are an assistant that performs a specific task.
- Use the USER_INPUT to guide your response
- ...other instructions...`
```
## Input Processing Examples
### Web Search Agent
```typescript
// In web-search.genai.mts
def("USER_INPUT", env.vars.user_input);
$`You are an assistant searching for web content using complex queries to pinpoint results.
- tailor search to answer the question in USER_INPUT
- ...other instructions...`
```
### News Search Agent
```typescript
// In news-search.genai.mts
def("USER_INPUT", env.vars.user_input);
$`You are an assistant searching for news using complex queries to pinpoint results.
- tailor search to answer the question in USER_INPUT
- ...other instructions...`
```
## Input Validation
Currently, there is no built-in validation for input. If your agent requires specific input formats or validation, you should implement this in your GenAIScript file. For example:
```typescript
def("USER_INPUT", env.vars.user_input);
// Simple validation example
if (!USER_INPUT || USER_INPUT.trim() === "") {
$`Please provide a valid input query.`;
exit();
}
// More complex validation could be implemented here
```
## Best Practices
1. **Be Clear About Input Requirements**: Document what kind of input your agent expects.
2. **Handle Edge Cases**: Consider how your agent will handle empty, very short, or very long inputs.
3. **Preprocess Input When Necessary**: If your agent needs input in a specific format, consider preprocessing it in the GenAIScript.
4. **Provide Examples**: Include example inputs in your agent's documentation.
## Testing Input
### Using the API
You can test how your agent handles different inputs using the API:
```bash
curl -X POST https://your-server.com/api/webhooks \
-H "Authorization: Bearer <session_token>" \
-H "Content-Type: application/json" \
-d '{"resource": "your-resource-name", "input": "Your test input"}'
```
### Direct Invocation Examples
You can also run agents directly from the command line using the GenAIScript CLI or the shim. Here are examples from the project's package.json:
#### Web Search Agent
```bash
# Using genaiscript CLI
genaiscript run packages/genaiscript/genaisrc/web-search.genai.mts --vars USER_INPUT='who won the 2024 election?'
# Using the shim
./dist/shim.js --file=genaisrc/search.genai.mts USER_INPUT="Who won the 2024 presidential election?"
```
#### News Search Agent
```bash
genaiscript run packages/genaiscript/genaisrc/news-search.genai.mts --vars USER_INPUT='What are the latest updates and developments in the Ukraine war?'
```
#### Web Scrape Agent
```bash
# Read mode
genaiscript run packages/genaiscript/genaisrc/web-scrape.genai.mts --vars USER_INPUT='{"url":"https://geoff.seemueller.io/about","query":"Describe the details of the page.", "action": "read"}'
# Scrape mode
genaiscript run packages/genaiscript/genaisrc/web-scrape.genai.mts --vars USER_INPUT='{"url":"https://www.time4learning.com/homeschool-curriculum/high-school/eleventh-grade/math.html","query":"What is on this page?", "action": "scrape"}'
```
Note the different input formats:
- Simple text queries for search and news agents
- JSON objects for the web scrape agent, specifying URL, query, and action
## Related Documentation
- [Agents Guide](./agents.md) - General information about agents
- [API Documentation](./api.md) - API endpoints and usage examples

71
docs/installation.md
View File

@@ -1,71 +0,0 @@
# Installation Guide
## Prerequisites
Before installing open-open-web-agent-rs, ensure you have the following prerequisites:
- [Rust](https://www.rust-lang.org/tools/install) (latest stable version)
- [Node.js](https://nodejs.org/) (for GenAIScript)
- [Bun](https://bun.sh/) (for package management)
- [Docker](https://www.docker.com/get-started) (optional, for containerized deployment)
## Environment Setup
1. Clone the repository:
```bash
git clone https://github.com/seemueller-io/open-web-agent-rs.git
cd open-web-agent-rs
```
2. Create a `.env` file in the root directory with the following variables:
```
OPENAI_API_KEY=your_openai_api_key
BING_SEARCH_API_KEY=your_bing_search_api_key
TAVILY_API_KEY=your_tavily_api_key
GENAISCRIPT_MODEL_LARGE=gpt-4-turbo
GENAISCRIPT_MODEL_SMALL=gpt-3.5-turbo
SEARXNG_API_BASE_URL=your_searxng_url
```
## Local Development
1. Install Rust dependencies:
```bash
cargo build
```
2. Install JavaScript dependencies:
```bash
bun install
```
3. Run the server:
```bash
cargo run
```
The server will start on `http://localhost:3006`.
## Docker Deployment
You can also run the application using Docker:
1. Build the Docker image:
```bash
docker build -t open-web-agent-rs -f <Local|Remote>.Dockerfile .
```
2. Run the container:
```bash
docker run -p 3006:3006 --env-file .env open-web-agent-rs
```
Alternatively, you can use Docker Compose:
```bash
docker-compose up
```
## Configuration Options
The application can be configured using environment variables. See the [Configuration](./configuration.md) documentation for more details.

73
docs/streams.md
View File

@@ -1,73 +0,0 @@
# Stream Data Format
This document describes how the stream data is formatted as it comes across the wire to the client.
## Overview
The open-web-agent-rs uses Server-Sent Events (SSE) to stream data from agents to clients. This allows for real-time updates as the agent processes the request and generates responses.
## Stream Format
When you consume a stream resource by making a GET request to `/webhooks/:stream_id`, the server responds with a stream of data in the SSE format. Each piece of data from the agent is sent as an SSE event with the following format:
```
data: <content>\n\n
```
Where `<content>` is a line of output from the agent.
### Stream Completion
When the agent has finished processing and there is no more data to send, the server sends a final event to indicate the stream has completed:
```
data: [DONE]\n\n
```
This allows clients to know when the stream has ended and they can stop listening for events.
## HTTP Headers
The server includes the following HTTP headers in the response:
- `Content-Type: text/event-stream` - Indicates that the response is an SSE stream
- `Cache-Control: no-cache, no-transform` - Prevents caching of the stream
- `Connection: keep-alive` - Keeps the connection open for the duration of the stream
- `X-Accel-Buffering: yes` - Controls buffering behavior for certain proxies
## Client-Side Handling
Clients should use an EventSource or similar mechanism to consume the SSE stream. Here's an example of how to consume the stream using JavaScript:
```javascript
const eventSource = new EventSource('/webhooks/your-stream-id', {
headers: {
'Authorization': 'Bearer your-session-token'
}
});
eventSource.onmessage = (event) => {
if (event.data === '[DONE]') {
// Stream is complete, close the connection
eventSource.close();
return;
}
// Process the data
console.log('Received data:', event.data);
};
eventSource.onerror = (error) => {
console.error('EventSource error:', error);
eventSource.close();
};
```
## Data Content
The content of each data event depends on the specific agent being used. For example:
- Web search agents may return search results and snippets
- News search agents may return article headlines and summaries
- Image generator agents may return image URLs or base64-encoded images
Refer to the specific agent documentation for details on the format of the data they return.

77
docs/tokens.md
View File

@@ -1,77 +0,0 @@
# Authentication System Documentation
## Overview
This document outlines the token-based authentication system used in open-web-agent-rs. The system uses FIPS204 signatures to
generate secure session tokens containing user data.
## Core Components
TODO: In the meantime, here's some hamfisted knowledge.
```javascript
class FIPS204KeyPair {
constructor() {
this.publicKey = "FIPS204_PUBLIC_KEY"; // Placeholder
this.privateKey = "FIPS204_PRIVATE_KEY"; // Placeholder
}
sign(data) {
// Placeholder for actual FIPS204 signing logic
return `FIPS204_${data}_SIGNED`;
}
verify(data, signature) {
// Placeholder for actual FIPS204 verification
return true;
}
}
/* NOTES:
- the public key needs to be retrievable, so it can be used to verify payload signature at the time of the request.
- the private key is disposed so it can't be used to create more signatures
- future tokens should use a completely new keypair
- The fips204 authentication scheme was selected for its performance, flexibility, and key-length.
- It would be wise to configure additional protections like ip whitelisting and rate limiting.
*/
// User object representing token payload data
const user = {
sub: "user123",
name: "John Doe",
email: "john@example.com",
roles: ["user"],
iat: Math.floor(Date.now() / 1000),
exp: Math.floor(Date.now() / 1000) + (60 * 60) // 1 hour from now
};
const keyPair = new FIPS204KeyPair();
const signature = keyPair.sign(JSON.stringify(user));
function createToken(payload, signature) {
const encodedPayload = Buffer.from(JSON.stringify(payload)).toString('base64');
const encodedSignature = Buffer.from(signature).toString('base64');
return `${encodedPayload}.${encodedSignature}`;
}
const token = createToken(user, signature);
async function createStreamRequest(eventHost = "https://agent.example.com") {
const requestParams = {
// will automagically include the session token as a cookie, where it will be parsed by the agent server
credentials: "include"
}
const response = await fetch(eventHost, requestParams);
const {streamId} = await response.json();
// This stream id is then supplied as a path parameter to stream, the token is validated to ensure the stream belongs to the user, and the stream is returned.
return streamId;
}
```

43
smoke-tests/test-deep-research.ts
View File

@@ -1,43 +0,0 @@
#!/usr/bin/env deno -A
const API_ROOT = "http://localhost:3006";
const sid = crypto.randomUUID();
// -------------------- 1. Create the agent --------------------
const createAgentBody = {
id: sid,
resource: "deep-research",
parent: sid,
payload: { input: "Research unexplored cures for cancer." },
};
const createRes = await fetch(`${API_ROOT}/api/agents`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(createAgentBody),
});
const raw = await createRes.text();
console.log({raw});
const {stream_url: streamId} = JSON.parse(raw);
console.log("Agent created with streamId:", streamId);
// -------------------- 2. Listen to the SSE stream --------------------
const streamUrl = `${API_ROOT}${streamId}`;
const es = new EventSource(streamUrl);
es.onopen = (e) => {
console.log("connected", e);
};
es.onmessage = (e) => {
console.log("⟶", e.data);
};
es.onerror = (e) => {
console.error("SSE error:", e);
es.close();
};

43
smoke-tests/test-search.ts
View File

@@ -1,43 +0,0 @@
#!/usr/bin/env deno -A
const API_ROOT = "http://localhost:3006";
const sid = crypto.randomUUID();
// -------------------- 1. Create the agent --------------------
const createAgentBody = {
id: sid,
resource: "web-search",
parent: sid,
payload: { input: "What is the capital of France?" },
};
const createRes = await fetch(`${API_ROOT}/api/agents`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(createAgentBody),
});
const raw = await createRes.text();
console.log({raw});
const {stream_url: streamId} = JSON.parse(raw);
console.log("Agent created with streamId:", streamId);
// -------------------- 2. Listen to the SSE stream --------------------
const streamUrl = `${API_ROOT}${streamId}`;
const es = new EventSource(streamUrl);
es.onopen = (e) => {
console.log("connected", e);
};
es.onmessage = (e) => {
console.log("⟶", e.data);
};
es.onerror = (e) => {
console.error("SSE error:", e);
es.close();
};