slimjson 🎯

Go Version Go Reference License CI/CD Go Report Card GitHub Release Docker Image GitHub Pages

slimjson is a Go tool and library designed to “slim down” JSON data before sending it to AI models or other bandwidth-constrained systems. It removes unnecessary fields, truncates deep nesting, shortens lists and strings, and strips empty values to create an optimized “diet” version of your JSON.

✨ Perfect for reducing token usage when sending large JSON payloads to LLMs and AI APIs.

Quick Links 📚

Features ⚡

Installation 📦

Go Package

Install as a Go library:

go get github.com/tradik/slimjson@latest

Or install a specific version:

go get github.com/tradik/slimjson@v1.0.0

Pre-built Binaries

Download the latest pre-built binaries for Linux, macOS, and FreeBSD from the Releases page.

Installation Guide

Linux (amd64/arm64) 🐧

  1. Download the binary: 
    # For amd64
wget https://github.com/tradik/slimjson/releases/latest/download/slimjson-linux-amd64 -O slimjson
   
# For arm64
wget https://github.com/tradik/slimjson/releases/latest/download/slimjson-linux-arm64 -O slimjson
  2. Make it executable: 
    chmod +x slimjson
  3. Move to path: 
    sudo mv slimjson /usr/local/bin/

macOS (Intel/Apple Silicon) 🍎

  1. Download the binary: 
    # For Apple Silicon (M1/M2/etc)
curl -L https://github.com/tradik/slimjson/releases/latest/download/slimjson-darwin-arm64 -o slimjson
   
# For Intel
curl -L https://github.com/tradik/slimjson/releases/latest/download/slimjson-darwin-amd64 -o slimjson
  2. Make it executable: 
    chmod +x slimjson
  3. Move to path: 
    sudo mv slimjson /usr/local/bin/

    Note: You might need to allow the application in System Settings > Privacy & Security if macOS blocks it.

FreeBSD 👹

  1. Download the binary: 
    # For amd64
fetch -o slimjson https://github.com/tradik/slimjson/releases/latest/download/slimjson-freebsd-amd64
   
# For arm64
fetch -o slimjson https://github.com/tradik/slimjson/releases/latest/download/slimjson-freebsd-arm64
  2. Make it executable: 
    chmod +x slimjson
  3. Move to path: 
    sudo mv slimjson /usr/local/bin/

Build from Source 🔨

If you have Go 1.25+ installed:

go install github.com/tradik/slimjson/cmd/slimjson@latest

Or clone and build using Makefile:

git clone https://github.com/tradik/slimjson.git
cd slimjson
make build
sudo mv bin/slimjson /usr/local/bin/

Usage 🚀

Configuration File

SlimJSON supports a .slimjson configuration file for defining custom profiles. The file is searched in:

  1. Current directory (./.slimjson)
  2. User home directory (~/.slimjson)

Format:

# Comments start with # or //
[profile-name]
parameter=value

[another-profile]
parameter=value

Example .slimjson:

# Custom profile for API responses
[api-response]
depth=5
list-len=20
strip-empty=true
decimal-places=2
deduplicate=true
block=metadata,debug,trace

# Custom profile for LLM context
[llm-context]
depth=4
list-len=15
strip-empty=true
string-pooling=true
type-inference=true
bool-compression=true
block=avatar_url,url,html_url

# Maximum compression
[maximum]
depth=3
list-len=5
strip-empty=true
decimal-places=2
deduplicate=true
sample-strategy=first_last
sample-size=10
null-compression=true
type-inference=true
bool-compression=true
timestamp-compression=true
string-pooling=true
number-delta=true
enum-detection=true

Using custom profiles:

slimjson -profile api-response data.json
slimjson -profile llm-context data.json

Note: Custom profiles take precedence over built-in profiles. If a parameter is not specified, it defaults to the zero value (disabled).

See .slimjson.example for a complete configuration file with all available parameters.

CLI

The slimjson CLI reads JSON from stdin or a file and outputs the slimmed JSON to stdout. It can also run as an HTTP daemon for processing JSON via REST API.

Quick Start

# Show help
slimjson

# Process file
slimjson data.json

# Process stdin
cat data.json | slimjson -profile medium

# Run as daemon
slimjson -d -port 8080

# Light compression - preserve most data
slimjson -profile light input.json > output.json

# Medium compression - balanced reduction
slimjson -profile medium input.json > output.json

# Aggressive compression - maximum reduction
slimjson -profile aggressive input.json > output.json

# AI-Optimized - optimized for LLM token reduction
slimjson -profile ai-optimized input.json > output.json

Custom Parameters

# Read from file
slimjson -depth 3 -list-len 5 input.json > output.json

# Read from stdin
cat input.json | slimjson -strip-empty=true -block "password,secret"

# Round numbers and remove duplicates
slimjson -decimal-places 2 -deduplicate data.json

# Sample large arrays - keep first 5 and last 5
slimjson -sample-strategy first_last -sample-size 10 data.json

# Representative sampling - evenly distributed
slimjson -sample-strategy representative -sample-size 20 data.json

# Combine with profile
slimjson -profile medium -decimal-places 2 -deduplicate data.json

# Advanced compression - all optimizations
slimjson -string-pooling -enum-detection -timestamp-compression data.json

# Maximum compression (use all features)
slimjson -profile ai-optimized \
  -decimal-places 2 \
  -deduplicate \
  -sample-strategy representative \
  -sample-size 50 \
  -null-compression \
  -type-inference \
  -bool-compression \
  -timestamp-compression \
  -string-pooling \
  -number-delta \
  -enum-detection \
  data.json

Flags:

Basic Options:

Optimization Options:

Advanced Compression:

Profile Details:

Profile MaxDepth MaxListLength BlockList Strategy
Light 10 20 none Preserve data, limit depth/lists
Medium 5 10 none Balanced reduction
Aggressive 3 5 description, summary, comment, notes, bio, readme Remove verbose text fields
AI-Optimized 4 8 avatar_url, gravatar_id, url, html_url, *_url Remove URLs and metadata

Default Values Summary:

Parameter Default Description
-depth 5 Maximum nesting depth
-list-len 10 Maximum array length
-string-len 0 No string truncation (unlimited)
-strip-empty true Remove empty values
-decimal-places -1 No rounding
-sample-strategy none No sampling
-sample-size 0 Use list-len value
-string-pool-min 2 Min occurrences for pooling
-number-delta-threshold 5 Min array size for delta
-enum-max-values 10 Max unique values for enum

Note: Profiles do NOT truncate strings to preserve data integrity. Use -string-len manually if needed, but be aware this may lose information.

Daemon Mode (HTTP Server)

Run SlimJSON as an HTTP daemon to process JSON via REST API:

# Start daemon on default port 8080
slimjson -d

# Start on custom port
slimjson -d -port 3000

# Use custom config file
slimjson -d -c /path/to/.slimjson

API Endpoints:

# Health check
curl http://localhost:8080/health
# Response: {"status":"ok","version":"1.0"}

# List available profiles
curl http://localhost:8080/profiles
# Response: {"builtin":["light","medium","aggressive","ai-optimized"],"custom":["my-profile"]}

# Compress JSON with default settings
curl -X POST http://localhost:8080/slim \
  -H "Content-Type: application/json" \
  -d '{"users":[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"}]}'

# Compress JSON with specific profile
curl -X POST 'http://localhost:8080/slim?profile=medium' \
  -H "Content-Type: application/json" \
  -d '{"users":[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"}]}'

# Use custom profile from config file
curl -X POST 'http://localhost:8080/slim?profile=my-custom-profile' \
  -H "Content-Type: application/json" \
  -d @data.json

Daemon Features:

Use Cases:

Custom Config File Priority

# Priority 1: Specified config file (highest priority)
slimjson -c /path/to/custom.slimjson -profile my-profile data.json

# Priority 2: .slimjson in current directory
slimjson -profile my-profile data.json

# Priority 3: .slimjson in home directory
slimjson -profile my-profile data.json

# Priority 4: Built-in profiles
slimjson -profile medium data.json

📚 See EXAMPLES.md for CLI examples and LIBRARY_EXAMPLES.md for complete library usage guide.

Library (Go Package)

SlimJSON can be used as a Go library in your applications.

Installation

go get github.com/tradik/slimjson

Basic Usage

package main

import (
	"encoding/json"
	"fmt"
	"github.com/tradik/slimjson"
)

func main() {
	data := map[string]interface{}{
		"name": "Test",
		"details": map[string]interface{}{
			"deep": map[string]interface{}{
				"too_deep": "value",
			},
		},
		"list": []interface{}{1, 2, 3, 4, 5},
		"empty": "",
	}

	// Create configuration
	cfg := slimjson.Config{
		MaxDepth:      2,
		MaxListLength: 3,
		StripEmpty:    true,
	}

	// Create slimmer and process data
	slimmer := slimjson.New(cfg)
	result := slimmer.Slim(data)

	// Output result
	out, _ := json.MarshalIndent(result, "", "  ")
	fmt.Println(string(out))
}

Using Built-in Profiles

package main

import (
	"encoding/json"
	"fmt"
	"github.com/tradik/slimjson"
)

func main() {
	// Get built-in profile
	profiles := slimjson.GetBuiltinProfiles()
	cfg := profiles["medium"]
	
	// Or use a specific profile directly
	cfg := slimjson.Config{
		MaxDepth:      5,
		MaxListLength: 10,
		StripEmpty:    true,
	}

	slimmer := slimjson.New(cfg)
	result := slimmer.Slim(yourData)
	
	// ... use result
}

Advanced Configuration

package main

import (
	"github.com/tradik/slimjson"
)

func main() {
	// Full configuration with all options
	cfg := slimjson.Config{
		// Basic options
		MaxDepth:        5,
		MaxListLength:   10,
		MaxStringLength: 0,  // 0 = unlimited
		StripEmpty:      true,
		BlockList:       []string{"password", "secret", "token"},
		
		// Optimization options
		DecimalPlaces:     2,
		DeduplicateArrays: true,
		SampleStrategy:    "first_last",
		SampleSize:        20,
		
		// Advanced compression
		NullCompression:          true,
		TypeInference:            true,
		BoolCompression:          true,
		TimestampCompression:     true,
		StringPooling:            true,
		StringPoolMinOccurrences: 2,
		NumberDeltaEncoding:      true,
		NumberDeltaThreshold:     5,
		EnumDetection:            true,
		EnumMaxValues:            10,
	}

	slimmer := slimjson.New(cfg)
	result := slimmer.Slim(data)
	
	// Result may contain metadata fields:
	// - _strings: String pool (if StringPooling enabled)
	// - _enums: Enum mappings (if EnumDetection enabled)
	// - _nulls: Tracked null fields (if NullCompression enabled)
}

Loading Custom Profiles from File

package main

import (
	"fmt"
	"github.com/tradik/slimjson"
)

func main() {
	// Load profiles from .slimjson file
	customProfiles, err := slimjson.LoadConfigFile()
	if err != nil {
		fmt.Printf("Warning: %v\n", err)
		customProfiles = make(map[string]slimjson.Config)
	}
	
	// Use custom profile
	if cfg, ok := customProfiles["my-custom-profile"]; ok {
		slimmer := slimjson.New(cfg)
		result := slimmer.Slim(data)
		// ... use result
	}
	
	// Or combine built-in and custom profiles
	allProfiles := slimjson.GetBuiltinProfiles()
	for name, cfg := range customProfiles {
		allProfiles[name] = cfg
	}
	
	// Use any profile
	cfg := allProfiles["api-response"]
	slimmer := slimjson.New(cfg)
	result := slimmer.Slim(data)
}

Parsing Config File Manually

package main

import (
	"github.com/tradik/slimjson"
)

func main() {
	// Parse specific config file
	profiles, err := slimjson.ParseConfigFile("/path/to/.slimjson")
	if err != nil {
		panic(err)
	}
	
	cfg := profiles["my-profile"]
	slimmer := slimjson.New(cfg)
	result := slimmer.Slim(data)
}

Config Structure Reference

type Config struct {
	// Basic options
	MaxDepth        int      // Maximum nesting depth (0 = unlimited)
	MaxListLength   int      // Maximum array length (0 = unlimited)
	MaxStringLength int      // Maximum string length (0 = unlimited)
	StripEmpty      bool     // Remove nulls, empty strings, empty arrays/objects
	BlockList       []string // List of field names to remove (case-insensitive)
	
	// Optimization options
	DecimalPlaces     int    // Round floats to N decimal places (-1 = no rounding)
	DeduplicateArrays bool   // Remove duplicate values from arrays
	SampleStrategy    string // Array sampling: "none", "first_last", "random", "representative"
	SampleSize        int    // Number of items when sampling (0 = use MaxListLength)
	
	// Advanced compression
	NullCompression          bool   // Track removed null fields in _nulls array
	TypeInference            bool   // Convert uniform arrays to schema+data format
	BoolCompression          bool   // Convert booleans to bit flags
	TimestampCompression     bool   // Convert ISO timestamps to unix timestamps
	StringPooling            bool   // Deduplicate repeated strings using string pool
	StringPoolMinOccurrences int    // Minimum occurrences for string pooling (default: 2)
	NumberDeltaEncoding      bool   // Use delta encoding for sequential numbers
	NumberDeltaThreshold     int    // Minimum array size for delta encoding (default: 5)
	EnumDetection            bool   // Convert repeated categorical values to enums
	EnumMaxValues            int    // Maximum unique values to consider as enum (default: 10)
}

Example: API Response Compression

package main

import (
	"encoding/json"
	"net/http"
	"github.com/tradik/slimjson"
)

func apiHandler(w http.ResponseWriter, r *http.Request) {
	// Get data from database
	data := fetchDataFromDB()
	
	// Configure compression for API response
	cfg := slimjson.Config{
		MaxDepth:          5,
		MaxListLength:     20,
		StripEmpty:        true,
		DecimalPlaces:     2,
		DeduplicateArrays: true,
		BlockList:         []string{"internal_id", "metadata", "debug"},
	}
	
	// Compress data
	slimmer := slimjson.New(cfg)
	compressed := slimmer.Slim(data)
	
	// Send response
	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(compressed)
}

Example: LLM Context Optimization

package main

import (
	"encoding/json"
	"github.com/tradik/slimjson"
)

func prepareLLMContext(data interface{}) ([]byte, error) {
	// Maximum compression for LLM context
	cfg := slimjson.Config{
		MaxDepth:             4,
		MaxListLength:        15,
		StripEmpty:           true,
		DecimalPlaces:        2,
		DeduplicateArrays:    true,
		StringPooling:        true,
		TypeInference:        true,
		BoolCompression:      true,
		TimestampCompression: true,
		BlockList:            []string{"avatar_url", "url", "html_url"},
	}
	
	slimmer := slimjson.New(cfg)
	compressed := slimmer.Slim(data)
	
	// Convert to JSON for LLM
	return json.Marshal(compressed)
}

Example: Processing Large Datasets

package main

import (
	"encoding/json"
	"github.com/tradik/slimjson"
)

func processLargeDataset(records []map[string]interface{}) []byte {
	// Wrap in container
	data := map[string]interface{}{
		"records": records,
	}
	
	// Aggressive compression with sampling
	cfg := slimjson.Config{
		MaxDepth:          3,
		MaxListLength:     100,
		StripEmpty:        true,
		DecimalPlaces:     2,
		SampleStrategy:    "representative",
		SampleSize:        50,
		TypeInference:     true,
		NumberDeltaEncoding: true,
	}
	
	slimmer := slimjson.New(cfg)
	compressed := slimmer.Slim(data)
	
	result, _ := json.Marshal(compressed)
	return result
}

Docker / Podman 🐳

Run slimjson as a containerized service using Docker or Podman.

Pull from GitHub Container Registry

Pre-built multi-arch images are available:

# Pull latest version
docker pull ghcr.io/tradik/slimjson:latest

# Pull specific version
docker pull ghcr.io/tradik/slimjson:v0.1.X

# Using Podman
podman pull ghcr.io/tradik/slimjson:latest

Build the image locally

# Using Docker
make docker-build
# or
docker build -t slimjson:latest .

# Using Podman
make podman-build
# or
podman build -t slimjson:latest .

Run the container

# Using Docker (from ghcr.io)
cat input.json | docker run -i --rm ghcr.io/tradik/slimjson:latest -depth 3 -list-len 5

# Using local image
cat input.json | docker run -i --rm slimjson:latest -depth 3 -list-len 5

# Using Podman
cat input.json | podman run -i --rm ghcr.io/tradik/slimjson:latest -depth 3 -list-len 5

# Using docker-compose
docker-compose up

Example with file mounting

# Docker
docker run -i --rm -v $(pwd)/data:/data ghcr.io/tradik/slimjson:latest -depth 5 < /data/input.json > /data/output.json

# Podman
podman run -i --rm -v $(pwd)/data:/data:z ghcr.io/tradik/slimjson:latest -depth 5 < /data/input.json > /data/output.json

Compression Results 📊

Real-world compression tests on various JSON files:

File Original Size Config Compressed Size Reduction Reduction % Original Tokens Compressed Tokens Token Reduction %
resume.json 28.2 KB Light 21.4 KB 6.8 KB 24.2% 7230 5478 24.2%
resume.json 28.2 KB Medium 18.9 KB 9.3 KB 33.1% 7230 4841 33.0%
resume.json 28.2 KB Aggressive 530 B 27.7 KB 98.2% 7230 133 98.2%
resume.json 28.2 KB AI-Optimized 11.4 KB 16.8 KB 59.5% 7230 2928 59.5%
schema-resume.json 24.8 KB Light 17.9 KB 7.0 KB 28.0% 6359 4579 28.0%
schema-resume.json 24.8 KB Medium 15.3 KB 9.5 KB 38.3% 6359 3922 38.3%
schema-resume.json 24.8 KB Aggressive 530 B 24.3 KB 97.9% 6359 133 97.9%
schema-resume.json 24.8 KB AI-Optimized 9.5 KB 15.4 KB 61.8% 6359 2427 61.8%
users.json 5.5 KB Light 4.0 KB 1.5 KB 27.5% 1412 1024 27.5%
users.json 5.5 KB Medium 4.0 KB 1.5 KB 27.5% 1412 1024 27.5%
users.json 5.5 KB Aggressive 691 B 4.8 KB 87.8% 1412 173 87.7%
users.json 5.5 KB AI-Optimized 2.9 KB 2.6 KB 47.9% 1412 736 47.9%

Token Estimation: Tokens are estimated using ~4 characters per token, approximating GPT-style tokenization for JSON/English text.

Statistical Analysis: Each test is run 10 times. Processing times show mean ± standard deviation (n=10) for statistical reliability. See testing/METHODOLOGY.md for detailed methodology.

Configuration Profiles

Performance Benchmarks

Benchmarks run on Apple M2 (arm64):

BenchmarkSlim_Small-8              196236    16327 ns/op    19384 B/op    442 allocs/op
BenchmarkSlim_Medium-8              90555    38951 ns/op    42472 B/op    955 allocs/op
BenchmarkSlim_Large-8               77553    46666 ns/op    50032 B/op   1143 allocs/op
BenchmarkSlim_Aggressive-8         181198    20002 ns/op    12720 B/op    598 allocs/op
BenchmarkSlim_Parallel-8           177736    20209 ns/op    42473 B/op    955 allocs/op

Key Performance Metrics:

Run your own compression tests:

cd testing
./run_tests.sh

Run performance benchmarks:

go test -bench=. -benchmem

Development 🛠️

Requirements

Testing

go test ./...

Benchmarking

go test -bench=. -benchmem -benchtime=3s

Compression Testing

cd testing
go run compression_benchmark.go

Linting

golangci-lint run

Documentation 📚

Publishing & Releases 🚀

Creating a Release

SlimJSON uses automated GitHub Actions for releases. To publish a new version:

# 1. Update CHANGELOG.md with release notes
# 2. Commit changes
git add CHANGELOG.md
git commit -m "chore: prepare release v1.0.0"
git push origin main

# 3. Create and push tag
git tag -a v1.0.0 -m "Release v1.0.0"
git push origin v1.0.0

GitHub Actions will automatically:

See RELEASING.md for detailed release process documentation.

Installation from Release

Go Package:

go get github.com/tradik/slimjson@v1.0.0

Binary:

wget https://github.com/tradik/slimjson/releases/download/v1.0.0/slimjson-linux-amd64
chmod +x slimjson-linux-amd64
sudo mv slimjson-linux-amd64 /usr/local/bin/slimjson

Docker:

docker pull ghcr.io/tradik/slimjson:v1.0.0
docker run --rm ghcr.io/tradik/slimjson:v1.0.0 --help

Contributing 🤝

Contributions are welcome! Please feel free to submit a Pull Request.

License 📄

BSD-3-Clause License - see LICENSE file for details.