shipitandpray/TurboQuant — Quantize LLM
Compress any LLM up to 6x in one step. Quantize to GGUF, GPTQ, or AWQ and optionally push to HuggingFace Hub.
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Stale
- License
- None
Inputs
| name | description | required | default |
|---|---|---|---|
| model | HuggingFace model ID or path to local model directory | yes | — |
| format | Output format: gguf, gptq, awq, or all | no | gguf |
| bits | Quantization bits: 2, 3, 4, 5, or 8 | no | 4 |
| target | Target platform: ollama, vllm, llamacpp, lmstudio (overrides format) | no | "" |
| push-to-hub | HuggingFace repo to upload quantized model (e.g. user/model-GGUF) | no | "" |
| eval | Run quality evaluation after quantization | no | false |
| output | Output directory for quantized model | no | ./turboquant-output |
| hf-token | HuggingFace API token (required for --push-to-hub and gated models) | no | "" |
Outputs
| name | description |
|---|---|
| output-dir | Directory containing quantized model files |
| report | Path to the JSON compression report |