思维导图备注
vLLM v0.5.5 Documentation
首页 AI助手 白天 夜间 BookChat 小程序 小程序 阅读
  • 书签 我的书签
  • 添加书签 添加书签 移除书签 移除书签

Quantization

 Sponsor  来源:vLLM 浏览 190 扫码 2025-02-09 14:47:14
  • Supported Hardware for Quantization Kernels
  • AutoAWQ
  • BitsAndBytes
  • INT8 W8A8
  • FP8 W8A8
  • FP8 E5M2 KV Cache
  • FP8 E4M3 KV Cache
当前内容版权归 vLLM 或其关联方所有，如需对内容或内容相关联开源项目进行关注与资助，请访问 vLLM .
上一篇:
下一篇:
  • 书签
  • 添加书签 移除书签
  • vLLM Recipes
  • vLLM v0.7.2 Documentation
  • vLLM v0.7.1 Documentation
  • vLLM v0.7.0 Documentation
  • vLLM v0.6.6 Documentation
  • vLLM v0.6.5 Documentation
  • vLLM v0.6.4 Documentation
  • vLLM v0.6.3 Documentation
  • vLLM v0.6.2 Documentation
  • vLLM v0.6.1 Documentation
  • vLLM v0.6.0 Documentation
  • vLLM v0.5.5 Documentation
  • vLLM v0.5.4 Documentation
  • vLLM v0.5.3 Documentation
  • vLLM v0.5.2 Documentation
  • vLLM v0.5.1 Documentation
  • vLLM v0.5.0 Documentation
  • vLLM v0.4.3 Documentation
  • vLLM v0.4.2 Documentation
  • vLLM v0.4.1 Documentation
  • Getting Started
    • Installation
      • Installation with ROCm
        • Installation with OpenVINO
          • Installation with CPU
            • Installation with Neuron
              • Installation with TPU
                • Installation with XPU
                  • Quickstart
                    • Debugging Tips
                      • Examples
                        • API Client
                          • Aqlm Example
                            • Cpu Offload
                              • Gguf Inference
                                • Gradio OpenAI Chatbot Webserver
                                  • Gradio Webserver
                                    • LLM Engine Example
                                      • Lora With Quantization Inference
                                        • MultiLoRA Inference
                                          • Offline Inference
                                            • Offline Inference Arctic
                                              • Offline Inference Audio Language
                                                • Offline Inference Chat
                                                  • Offline Inference Distributed
                                                    • Offline Inference Embedding
                                                      • Offline Inference Encoder Decoder
                                                        • Offline Inference Mlpspeculator
                                                          • Offline Inference Neuron
                                                            • Offline Inference Tpu
                                                              • Offline Inference Vision Language
                                                                • Offline Inference With Prefix
                                                                  • OpenAI Audio API Client
                                                                    • OpenAI Chat Completion Client
                                                                      • OpenAI Completion Client
                                                                        • OpenAI Embedding Client
                                                                          • OpenAI Vision API Client
                                                                            • Save Sharded State
                                                                              • Tensorize vLLM Model
                                                                            • Serving
                                                                              • OpenAI Compatible Server
                                                                                • Deploying with Docker
                                                                                  • Distributed Inference and Serving
                                                                                    • Production Metrics
                                                                                      • Environment Variables
                                                                                        • Usage Stats Collection
                                                                                          • Integrations
                                                                                            • Deploying and scaling up with SkyPilot
                                                                                              • Deploying with KServe
                                                                                                • Deploying with NVIDIA Triton
                                                                                                  • Deploying with BentoML
                                                                                                    • Deploying with Cerebrium
                                                                                                      • Deploying with LWS
                                                                                                        • Deploying with dstack
                                                                                                          • Serving with Langchain
                                                                                                            • Serving with llama_index
                                                                                                            • Loading Models with CoreWeave’s Tensorizer
                                                                                                              • Frequently Asked Questions
                                                                                                              • Models
                                                                                                                • Supported Models
                                                                                                                  • Adding a New Model
                                                                                                                    • Enabling Multimodal Inputs
                                                                                                                      • Engine Arguments
                                                                                                                        • Using LoRA adapters
                                                                                                                          • Using VLMs
                                                                                                                            • Speculative decoding in vLLM
                                                                                                                              • Performance and Tuning
                                                                                                                              • Quantization
                                                                                                                                • Supported Hardware for Quantization Kernels
                                                                                                                                  • AutoAWQ
                                                                                                                                    • BitsAndBytes
                                                                                                                                      • INT8 W8A8
                                                                                                                                        • FP8 W8A8
                                                                                                                                          • FP8 E5M2 KV Cache
                                                                                                                                            • FP8 E4M3 KV Cache
                                                                                                                                            • Automatic Prefix Caching
                                                                                                                                              • Introduction
                                                                                                                                                • Implementation
                                                                                                                                                • Performance benchmarks
                                                                                                                                                  • Benchmark suites of vLLM
                                                                                                                                                  • Developer Documentation
                                                                                                                                                    • Sampling Parameters
                                                                                                                                                      • Offline Inference
                                                                                                                                                        • LLM Class
                                                                                                                                                          • LLM Inputs
                                                                                                                                                          • vLLM Engine
                                                                                                                                                            • LLMEngine
                                                                                                                                                              • AsyncLLMEngine
                                                                                                                                                              • vLLM Paged Attention
                                                                                                                                                                • Input Processing
                                                                                                                                                                  • Input Processing Pipeline
                                                                                                                                                                  • Multi-Modality
                                                                                                                                                                    • Adding a Multimodal Plugin
                                                                                                                                                                    • Dockerfile
                                                                                                                                                                      • Profiling vLLM
                                                                                                                                                                      • Community
                                                                                                                                                                        • vLLM Meetups
                                                                                                                                                                          • Sponsors
                                                                                                                                                                          暂无相关搜索结果！

                                                                                                                                                                            本文档使用 BookStack 构建

                                                                                                                                                                            文章二维码

                                                                                                                                                                            手机扫一扫，轻松掌上读

                                                                                                                                                                            文档下载

                                                                                                                                                                            • 普通下载
                                                                                                                                                                            • 下载码下载(免登录无限下载)
                                                                                                                                                                            你与大神的距离，只差一个APP
                                                                                                                                                                            APP下载
                                                                                                                                                                            请下载您需要的格式的文档，随时随地，享受汲取知识的乐趣！
                                                                                                                                                                            PDF文档 EPUB文档 MOBI文档
                                                                                                                                                                            温馨提示 每天每在网站阅读学习一分钟时长可下载一本电子书，每天连续签到可增加阅读时长
                                                                                                                                                                            下载码方式下载：免费、免登录、无限制。 免费获取下载码

                                                                                                                                                                            微信小程序阅读

                                                                                                                                                                            BookChat 微信小程序阅读
                                                                                                                                                                            您与他人的薪资差距，只差一个随时随地学习的小程序

                                                                                                                                                                            书签列表

                                                                                                                                                                              阅读记录

                                                                                                                                                                              阅读进度: 0.00% ( 0/0 ) 重置阅读进度
                                                                                                                                                                                欢迎使用AI助手 AI助手
                                                                                                                                                                                全屏 缩小 隐藏 清空