chunker

package

v0.0.0-beta.18 Latest Latest Go to latest Published: Mar 20, 2026 License: MIT Imports: 2 Imported by: 0

Details

This section is empty.

This section is empty.

This section is empty.

type Chunk struct {
	Index   int
	Content string
	Tokens  int
}

Chunk is a piece of text from a document.

type Chunker struct {
	ChunkSize    int
	ChunkOverlap int
	// contains filtered or unexported fields
}

Chunker splits text into overlapping chunks.

func New(chunkSize, chunkOverlap int) *Chunker

New creates a recursive character text splitter.

func (c *Chunker) Split(text string) []Chunk

Split splits text into chunks, returning them with indices.