make package name memorable
This commit is contained in:
@@ -1,4 +1,4 @@
|
|||||||
# This is just for testing to make sure the glob patterns work
|
# This is just for testing to make sure the glob patterns work
|
||||||
# Check is valid when test-for-ignore.css is not included in the program output
|
# Check is valid when test-for-ignore.css is not included in the program output
|
||||||
**/*.css
|
**/*.css
|
||||||
**/.code-tokenizer-md-ignore
|
**/.toak-ignore
|
33
README.md
33
README.md
@@ -1,23 +1,22 @@
|
|||||||
# code-tokenizer-md
|
# toak
|
||||||
for me but you can use it too
|
|
||||||
|
|
||||||

|
```shell
|
||||||
|
|
||||||
> Push the limits of possible
|
|
||||||
|
|
||||||
## Quick Start
|
|
||||||
```bash
|
|
||||||
$ cd your-git-repo
|
$ cd your-git-repo
|
||||||
$ npx code-tokenizer-md
|
$ npx toak
|
||||||
```
|
```
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
`code-tokenizer-md` is a tool that processes git repository files, cleans code, redacts sensitive information, and generates markdown documentation with token counts using the Llama 3 tokenizer.
|
`toak` is a tool that processes git repository files, cleans code, redacts sensitive information, and generates markdown documentation with token counts using the Llama 3 tokenizer.
|
||||||
|
|
||||||
|
---
|
||||||
## Philosophy
|
## Philosophy
|
||||||
> Human-first technologies for a better tomorrow.
|
> Human-first technologies for a better tomorrow.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
graph TD
|
graph TD
|
||||||
@@ -41,7 +40,7 @@ graph TD
|
|||||||
- Removes comments, imports, and unnecessary whitespace
|
- Removes comments, imports, and unnecessary whitespace
|
||||||
- Redacts sensitive information (API keys, tokens, JWT, hashes)
|
- Redacts sensitive information (API keys, tokens, JWT, hashes)
|
||||||
- Counts tokens using llama3-tokenizer-js
|
- Counts tokens using llama3-tokenizer-js
|
||||||
- Supports nested .code-tokenizer-md-ignore files
|
- Supports nested .toak-ignore files
|
||||||
|
|
||||||
### Token Cleaning
|
### Token Cleaning
|
||||||
- Removes single-line and multi-line comments
|
- Removes single-line and multi-line comments
|
||||||
@@ -65,20 +64,20 @@ graph TD
|
|||||||
## Installation
|
## Installation
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm install code-tokenizer-md
|
npm install toak
|
||||||
```
|
```
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
### CLI
|
### CLI
|
||||||
```bash
|
```bash
|
||||||
npx code-tokenizer-md
|
npx toak
|
||||||
```
|
```
|
||||||
|
|
||||||
### Programmatic Usage
|
### Programmatic Usage
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
import { MarkdownGenerator } from 'code-tokenizer-md';
|
import { MarkdownGenerator } from 'toak';
|
||||||
|
|
||||||
const generator = new MarkdownGenerator({
|
const generator = new MarkdownGenerator({
|
||||||
dir: './project',
|
dir: './project',
|
||||||
@@ -107,9 +106,9 @@ interface MarkdownGeneratorOptions {
|
|||||||
|
|
||||||
### Ignore File Configuration
|
### Ignore File Configuration
|
||||||
|
|
||||||
Create a `.code-tokenizer-md-ignore` file in any directory to specify exclusions. The tool supports nested ignore files that affect their directory and subdirectories.
|
Create a `.toak-ignore` file in any directory to specify exclusions. The tool supports nested ignore files that affect their directory and subdirectories.
|
||||||
|
|
||||||
Example `.code-tokenizer-md-ignore`:
|
Example `.toak-ignore`:
|
||||||
```plaintext
|
```plaintext
|
||||||
# Ignore specific files
|
# Ignore specific files
|
||||||
secrets.json
|
secrets.json
|
||||||
@@ -154,7 +153,7 @@ This project uses [Bun](https://bun.sh) for development. To contribute:
|
|||||||
### Setup
|
### Setup
|
||||||
```bash
|
```bash
|
||||||
git clone <repository>
|
git clone <repository>
|
||||||
cd code-tokenizer-md
|
cd toak
|
||||||
bun install
|
bun install
|
||||||
```
|
```
|
||||||
|
|
||||||
|
10
package.json
10
package.json
@@ -1,11 +1,11 @@
|
|||||||
{
|
{
|
||||||
"name": "code-tokenizer-md",
|
"name": "toak",
|
||||||
"version": "1.2.7",
|
"version": "3.0.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"license": "AGPL-3.0-or-later",
|
"license": "AGPL-3.0-or-later",
|
||||||
"repository": "https://github.com/geoffsee/code-tokenizer-md.git",
|
"repository": "https://github.com/seemueller-io/toak.git",
|
||||||
"author": "Geoff Seemueller",
|
"author": "Geoff Seemueller",
|
||||||
"description": "Open-source CLI/Library for turning a git repository's contents into a single well-formatted markdown file.",
|
"description": "Open-source CLI/Library for tokenizing a git repository into a markdown file.",
|
||||||
"exports": {
|
"exports": {
|
||||||
".": {
|
".": {
|
||||||
"import": {
|
"import": {
|
||||||
@@ -15,7 +15,7 @@
|
|||||||
}
|
}
|
||||||
},
|
},
|
||||||
"bin": {
|
"bin": {
|
||||||
"code-tokenizer-md": "./dist/cli.js"
|
"toak": "./dist/cli.js"
|
||||||
},
|
},
|
||||||
"files": [
|
"files": [
|
||||||
"dist"
|
"dist"
|
||||||
|
@@ -64,7 +64,7 @@ export class MarkdownGenerator {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Loads and processes .code-tokenizer-md-ignore files recursively from the project directory.
|
* Loads and processes .toak-ignore files recursively from the project directory.
|
||||||
* These files contain patterns for files to exclude from processing.
|
* These files contain patterns for files to exclude from processing.
|
||||||
* @async
|
* @async
|
||||||
* @returns {Promise<void>}
|
* @returns {Promise<void>}
|
||||||
@@ -76,7 +76,7 @@ export class MarkdownGenerator {
|
|||||||
console.log('Loading ignore patterns...');
|
console.log('Loading ignore patterns...');
|
||||||
}
|
}
|
||||||
|
|
||||||
const ignoreFiles = await glob('**/.code-tokenizer-md-ignore', {
|
const ignoreFiles = await glob('**/.toak-ignore', {
|
||||||
cwd: this.dir,
|
cwd: this.dir,
|
||||||
dot: true,
|
dot: true,
|
||||||
absolute: true,
|
absolute: true,
|
||||||
|
Before Width: | Height: | Size: 638 KiB After Width: | Height: | Size: 638 KiB |
Reference in New Issue
Block a user