The ACSeL method for selecting LLM inference configuration on resource-constrained devices
Researchers have proposed the ACSeL method, which searches for an LLM configuration (model size, batch, quantization) for operation on edge devices with limited memory and latency. Tests were conducted on an NVIDIA RTX A6000 and Jetson Thor.
Deploying large language models (LLMs) on devices with limited memory and latency, including edge devices, requires jointly choosing model size, batch size, quantization algorithm, and mixed-precision recipes. According to the authors, searching the entire configuration space is practically infeasible. Unlike datacenter operation, which typically optimizes throughput, resource-constrained environments, according to the article, first set memory and latency limits as fixed conditions and only then search among the compliant configurations for the one with the highest generation quality.
The authors therefore present the ACSeL framework, which identifies configurations that satisfy deployment constraints while having high predicted generation quality. A key component is the RMQD (Relative Model Quality Degradation) metric — a proxy based on the divergence between the next-token distributions of the candidate and reference models on a small calibration corpus, which, according to the authors, strongly correlates with downstream benchmark results without the need to actually evaluate the benchmark. Other components are Few-shot Median Calibration (FMC), which calibrates latency estimates from a small number of end-to-end profiling measurements, and the BandPareto search algorithm for searching layerwise mixed-precision configurations.
The experiments were conducted on an NVIDIA RTX A6000 workstation and the NVIDIA Jetson Thor edge platform. According to the authors, FMC significantly increases deployment feasibility (i.e., the proportion of found configurations meeting the specified limits), and the combination of BandPareto + FMC achieves the highest feasibility and the smallest RMQD differences compared to the best measured configurations in the tested set. This is a publication on arXiv (2611.07570); details on the methodology and numerical results can be found in the source article.
Why it matters
The method addresses a specific engineering problem: how to quickly find a configuration with the best possible quality when deploying an LLM on a device with limited memory and latency (mobile devices, robotics, edge compute), without having to exhaustively test combinations of model size, batch size, and quantization against full benchmarks. This shortens the time needed to tune deployment and reduces the computational resources consumed by the configuration search itself.
Two audiences, two different impacts
What this means
For individuals
For developers and ML engineers working on deploying LLMs on edge devices, this is a specific technique (RMQD, FMC, BandPareto) that can be studied and potentially tried out when tuning model configuration, instead of manually testing combinations.
For a business
Companies developing products with LLMs on devices with limited memory and latency (edge computing, robotics, embedded systems) can use this method to reduce the time and computational costs associated with finding a suitable model configuration while maintaining the highest possible generation quality.
DevelopmentCheck the original
Event sources
only one source so far · 1 publisher, 1 independent. We count feeds from the same owner only once.