For example, vllm is an inference engine, and according to their roadmap they will incororate vllm into several LLM API engines such as openllm, rayserve, and nvidia triton.

  • What are examples of other inference engines?

  • You can run the same model (e.g. llama2) under several inference engines. What is the specification/doc/etc that the inference engine authors would use to implement this?

  • What specifications/documentation/etc are used when inference engine authors write an engine to support these models?



You must log in to answer this question.

Browse other questions tagged .