make package name memorable

This commit is contained in:
2024-12-12 17:24:12 -05:00
parent 63e02153f2
commit 79d16cd02e
5 changed files with 24 additions and 25 deletions

View File

@@ -1,4 +1,4 @@
# This is just for testing to make sure the glob patterns work
# Check is valid when test-for-ignore.css is not included in the program output
**/*.css
**/.code-tokenizer-md-ignore
**/.toak-ignore

View File

@@ -1,23 +1,22 @@
# code-tokenizer-md
for me but you can use it too
# toak
![code-tokenizer-md](./code-tokenizer-md.jpg)
> Push the limits of possible
## Quick Start
```bash
```shell
$ cd your-git-repo
$ npx code-tokenizer-md
$ npx toak
```
![toak](./toak.jpg)
## Overview
`code-tokenizer-md` is a tool that processes git repository files, cleans code, redacts sensitive information, and generates markdown documentation with token counts using the Llama 3 tokenizer.
`toak` is a tool that processes git repository files, cleans code, redacts sensitive information, and generates markdown documentation with token counts using the Llama 3 tokenizer.
---
## Philosophy
> Human-first technologies for a better tomorrow.
---
```mermaid
graph TD
@@ -41,7 +40,7 @@ graph TD
- Removes comments, imports, and unnecessary whitespace
- Redacts sensitive information (API keys, tokens, JWT, hashes)
- Counts tokens using llama3-tokenizer-js
- Supports nested .code-tokenizer-md-ignore files
- Supports nested .toak-ignore files
### Token Cleaning
- Removes single-line and multi-line comments
@@ -65,20 +64,20 @@ graph TD
## Installation
```bash
npm install code-tokenizer-md
npm install toak
```
## Usage
### CLI
```bash
npx code-tokenizer-md
npx toak
```
### Programmatic Usage
```typescript
import { MarkdownGenerator } from 'code-tokenizer-md';
import { MarkdownGenerator } from 'toak';
const generator = new MarkdownGenerator({
dir: './project',
@@ -107,9 +106,9 @@ interface MarkdownGeneratorOptions {
### Ignore File Configuration
Create a `.code-tokenizer-md-ignore` file in any directory to specify exclusions. The tool supports nested ignore files that affect their directory and subdirectories.
Create a `.toak-ignore` file in any directory to specify exclusions. The tool supports nested ignore files that affect their directory and subdirectories.
Example `.code-tokenizer-md-ignore`:
Example `.toak-ignore`:
```plaintext
# Ignore specific files
secrets.json
@@ -154,7 +153,7 @@ This project uses [Bun](https://bun.sh) for development. To contribute:
### Setup
```bash
git clone <repository>
cd code-tokenizer-md
cd toak
bun install
```

View File

@@ -1,11 +1,11 @@
{
"name": "code-tokenizer-md",
"version": "1.2.7",
"name": "toak",
"version": "3.0.0",
"type": "module",
"license": "AGPL-3.0-or-later",
"repository": "https://github.com/geoffsee/code-tokenizer-md.git",
"repository": "https://github.com/seemueller-io/toak.git",
"author": "Geoff Seemueller",
"description": "Open-source CLI/Library for turning a git repository's contents into a single well-formatted markdown file.",
"description": "Open-source CLI/Library for tokenizing a git repository into a markdown file.",
"exports": {
".": {
"import": {
@@ -15,7 +15,7 @@
}
},
"bin": {
"code-tokenizer-md": "./dist/cli.js"
"toak": "./dist/cli.js"
},
"files": [
"dist"

View File

@@ -64,7 +64,7 @@ export class MarkdownGenerator {
}
/**
* Loads and processes .code-tokenizer-md-ignore files recursively from the project directory.
* Loads and processes .toak-ignore files recursively from the project directory.
* These files contain patterns for files to exclude from processing.
* @async
* @returns {Promise<void>}
@@ -76,7 +76,7 @@ export class MarkdownGenerator {
console.log('Loading ignore patterns...');
}
const ignoreFiles = await glob('**/.code-tokenizer-md-ignore', {
const ignoreFiles = await glob('**/.toak-ignore', {
cwd: this.dir,
dot: true,
absolute: true,

View File

Before

Width:  |  Height:  |  Size: 638 KiB

After

Width:  |  Height:  |  Size: 638 KiB