The following tuning did the job (thanks to the video sharing script)

/etc/lighttpd/lighttpd.conf

server.max-keep-alive-requests = 4
server.max-keep-alive-idle = 4
server.event-handler = "linux-sysepoll"
server.network-backend = "linux-sendfile"
server.max-fds = 8192
server.stat-cache-engine = "fam"

/etc/sysctl.conf

# These ensure that TIME_WAIT ports either get reused or closed fast.
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_tw_recycle = 1
# TCP memory
net.core.rmem_max = 16777216
net.core.rmem_default = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
# For Large File Hosting Servers
net.core.wmem_max = 1048576
net.ipv4.tcp_wmem = 4096 87380 524288

in the apache virtual host we just use proxy pass to forward our static urls to the lighttpd server running on port 8080

ProxyRequests Off
ProxyPreserveHost On
ProxyPass /images http://127.0.0.1:8080/images
ProxyPass /js http://127.0.0.1:8080/js
ProxyPass /css http://127.0.0.1:8080/css
ProxyPassReverse / http://127.0.0.1:8080/