Too many header key-value pairs

dc42

~~@chrishamm do you know why middleware would add key/value pairs to GET requests? Isn't there a danger of a clash with key/value pairs already used? Or perhaps there are some reserved key names?~~

I guess a Wireshark trace would show us what is being added, but I don't know whether it is possible to get a Wireshark trace to display packets that don't involve the PC that Wireshark is running on.

Masterjuggler

@chrishamm It sounds like this is something inherent to the way DWC is set up? If that's the case though, why would it be fine in firefox and only throw this error in chrome? I've been using this middleware with about 15 other services I'm exposing and haven't had any issues so far.

alankilian

@Masterjuggler Ar those other services running on an embedded processor with like 128 kilobytes of RAM?

The Duet is a pretty specialized machine and "normal" (read wasteful. profligate bloatware) software development practices do not necessarily apply.

I worked with a vendor once that wanted to send me a 4 gigabyte JSON file to grab some values out of and when I mentioned I had a TOTAL ram of 96 kilobytes, he called me a liar on a Zoom call. Well...... We eventually chose a different data vendor and they lost out on a million dollar a year contract.

So, this is a special machine that has very strict limits and "It works with other PC-based applications" isn't really going to apply here.

Can you grab the GET packet and post it so we can see all the headers your application is adding to the stream please? That would be incredibly helpful and you might be able to help the Duet team make an even better product.

chrishamm

@Masterjuggler That may be but RRF running on an embedded processor has limited RAM. Version 3.5 of DWC adds a single HTTP header to HTTP requests, older versions don't add anything at all. Everything else originates from your browser+middleware. When I send HTTP requests to my printer the number of HTTP headers is 13, which is still pretty far away from the limit of 30 headers. Perhaps some kind of browser extension adds more headers, I don't know - try to connect to your Duet via middleware from a private/incongnito tab and see if that makes a difference.

I'm afraid I cannot say what else is added without a dump of a GET request or a Wireshark trace between Duet and middleware.

@dc42 I think some reverse proxies add X-Forward-* headers but I can't say much more about that.

dc42

@chrishamm @Masterjuggler if you can find out what headers your middleware is adding (perhaps by using Wireshark, or by asking the supplier of the middleware) then we may be able to add a filter to ignore the ones that the Duet doesn't care about, to avoid filling up the table.

Masterjuggler

@dc42 I'll have to figure out a proper method of tracking down exactly what headers are being added. I'm a little bit in information overload right now lol, learning this stuff as I go.

After some playing around with the configuration I did find that removing authelia's default authResponseHeaders of Remote-User, Remote-Groups, Remote-Name, and Remote-Email allows DWC to load on chrome. That's only 4 headers and still means that a lot are being added somewhere along the chain, so I'll still try to figure out what's going on when I have time, but at least right now it's not a big problem. Unless I need those headers for some reason, I don't fully understand what they do tbh.

dc42

@Masterjuggler I've created this issue https://github.com/Duet3D/RepRapFirmware/issues/870 to track this. Please tell us which Duet, firmware version and DWC version you are using.

dc42 created this issue in Duet3D/RepRapFirmware

open "too many header key-value pairs" error when authelia middleware is used #870

alankilian

If anyone is interested, Authelia is open-source, and there are many ways to contact them listed in this github page.

Maybe jumping on a discord server or Matrix Space (whatever that is) and asking this question could provide an answer and maybe even a solution.

https://github.com/authelia/authelia

Masterjuggler

@dc42 I'm using a Duet 2 WiFi with firmware & DWC version 3.4.5. No expansion boards or RPI. To expand on my setup a bit, I'm running an ubuntu headless server with docker containers for traefik 2.10.1 and authelia. I'm then exposing via cloudflare tunnel (cloudflared) to avoid having to forward a port from my router. It's possible headers are being added at multiple stages through this chain, I was focusing on authelia because that was the straw that broke the camel's back so to speak.

As mentioned, removing authelia's four default authResponseHeaders (taken from here) reduced the number of headers such that DWC now loads in chrome with authelia in place, though I don't understand the system well enough to know if that is secure or if it causes any other problems.

chrishamm

@Masterjuggler I don't think removing those extra headers has any effect - they're ignored by RRF anyway. As previously said, a Wireshark dump taken between Duet and middleware may help us further figure out what and how many headers need to be handled in your config scenario.

benisfroms

This post is deleted!