Least Outstanding Request routing using Lua
Aug 15th ‘24 Update: If you are using k8s for deployments of your service, there’s an option to use ISTIO-ENVOY setting with LEAST_REQUESTS
option in the charts to enable LOR. More on this in another blog.
It’s a well known technique of predictive load balancing to route requests to hosts/servers that have consumed least requests. In this blog post, I try to do the same with nginx and lua script stored as nginx.conf file.
http {
lua_shared_dict outstanding_requests 10m;
server {
listen 80;
location / {
content_by_lua_block {
local backends = {
{ "127.0.0.1", 8081 },
{ "127.0.0.1", 8082 },
{ "127.0.0.1", 8083 }
}
local min_requests = math.huge
local selected_backend = nil
for _, backend in ipairs(backends) do
local ip, port = unpack(backend)
local key = ip .. ":" .. port
local outstanding_requests = ngx.shared.outstanding_requests:get(key) or 0
if outstanding_requests < min_requests then
min_requests = outstanding_requests
selected_backend = backend
end
end
local ip, port = unpack(selected_backend)
local key = ip .. ":" .. port
-- Increment outstanding requests count
ngx.shared.outstanding_requests:incr(key, 1, 0)
local res = ngx.location.capture("/proxy_pass", {
method = ngx.HTTP_GET,
ctx = { backend_ip = ip, backend_port = port }
})
-- Decrement outstanding requests count
ngx.shared.outstanding_requests:incr(key, -1)
ngx.status = res.status
ngx.say(res.body)
}
location /proxy_pass {
internal;
proxy_pass http://$ctx.backend_ip:$ctx.backend_port;
}
}
}
}
To use the script, we need to run following steps on the linux host on which we want to do the LOR splitting:
Step 1: Install nginx-extras.
sudo apt-get install nginx-extras
Step 2 Install Lua for Nginx: If it's not already installed with nginx-extras
, you can install LuaJIT and the Lua module for Nginx:
sudo apt-get install lua5.3
Step 3 Setup Nginx Configuration with Lua:
sudo nano /etc/nginx/nginx.conf
After setup, copy the lua script (assuming the hosts are locally on the same machine as nginx. If not, you can use something like redis pubsub to communicate the outstanding requests between hosts. More on this in another blog post).
Start your nginx and bam!
sudo service nginx start