Least Outstanding Request routing using Lua

Omkar Kulkarni
2 min readAug 15, 2024

--

Aug 15th ‘24 Update: If you are using k8s for deployments of your service, there’s an option to use ISTIO-ENVOY setting with LEAST_REQUESTS option in the charts to enable LOR. More on this in another blog.

Happy to use abstract art made for my by Dall-E!

It’s a well known technique of predictive load balancing to route requests to hosts/servers that have consumed least requests. In this blog post, I try to do the same with nginx and lua script stored as nginx.conf file.

http {
lua_shared_dict outstanding_requests 10m;

server {
listen 80;

location / {
content_by_lua_block {
local backends = {
{ "127.0.0.1", 8081 },
{ "127.0.0.1", 8082 },
{ "127.0.0.1", 8083 }
}

local min_requests = math.huge
local selected_backend = nil

for _, backend in ipairs(backends) do
local ip, port = unpack(backend)
local key = ip .. ":" .. port
local outstanding_requests = ngx.shared.outstanding_requests:get(key) or 0

if outstanding_requests < min_requests then
min_requests = outstanding_requests
selected_backend = backend
end
end

local ip, port = unpack(selected_backend)
local key = ip .. ":" .. port

-- Increment outstanding requests count
ngx.shared.outstanding_requests:incr(key, 1, 0)

local res = ngx.location.capture("/proxy_pass", {
method = ngx.HTTP_GET,
ctx = { backend_ip = ip, backend_port = port }
})

-- Decrement outstanding requests count
ngx.shared.outstanding_requests:incr(key, -1)

ngx.status = res.status
ngx.say(res.body)
}

location /proxy_pass {
internal;
proxy_pass http://$ctx.backend_ip:$ctx.backend_port;
}
}
}
}

To use the script, we need to run following steps on the linux host on which we want to do the LOR splitting:

Step 1: Install nginx-extras.

sudo apt-get install nginx-extras

Step 2 Install Lua for Nginx: If it's not already installed with nginx-extras, you can install LuaJIT and the Lua module for Nginx:

sudo apt-get install lua5.3

Step 3 Setup Nginx Configuration with Lua:

sudo nano /etc/nginx/nginx.conf

After setup, copy the lua script (assuming the hosts are locally on the same machine as nginx. If not, you can use something like redis pubsub to communicate the outstanding requests between hosts. More on this in another blog post).

Start your nginx and bam!

sudo service nginx start

--

--

No responses yet