CONSTANT AJAX disconnect errors

MiR

I today tested latest firmware 1.20beta4+WifiServer1.20beta2, unfortunately the test ended with an Ajax error when I started printing a real file.

But, to start at the beginning, I first simply let autocalibration run for quite a while (30++ minutes) and all was fine.
Then I edited a gcode file stored on the printer, saved it and started printing the file.
Shortly after that I saw the Ajax Disconnect Error message.

RSSI Values and Noise Floor are very good, Powersave is disabled (see flags, no PS in flags) and the WiFi Card is still associated and answering fine to sta_info but the TCP-Stack is dead, cannot ping the Duet

in network 2897 seconds
flags 0x1e03a: WME N_CAP AMPDU AMSDU
per antenna average rssi of rx data frames: -39 -43 -45 0
per antenna noise floor: -95 -96 -95 0

What I see in sta_info is that after the error happend I have rx decrypt errors and one tx error, before the error decrypt failures were 0:

tx failures: 1
rx decrypt succeeds: 30153
rx decrypt failures: 144

Not sure if this means anything, at least it looks suspicious.

I have now rebooted the board and have directly started the print job, right now it runs for 30 minutes, decrypt errors are still 0 and tx failures are also 0.

Michael

MiR

@David: Could it help when I provide remote debugging for you? I could set up a pc with windows on it that is also connected to duet's usb

dc42

KeeganB, stepper motor wires should ideally be twin twisted pair, with one pair used per phase, to minimise EMI. But like most people, I just use ordinary 4 core cable, in which the cores slowly twist around each other along the length of the cable.

Jarery

I have two printers, different builds, both exhibit the same disconnects.
One is a metal delta from a kickstarter where I added a duet and it is mounted on the back upright, the other is based of dc42's large kosel build except i mounted my motors and duet on top and power supply underneath.

here is the Kossel with duet and motors on the top of printer

And here is second printer, with duet mounted off the back, no motor wiring anywhere near the wifi module

dc42

A major difference with my delta is that I have the wifi antenna poking outside the machine through the lower extrusions, well away from stepper motors cables, the PSU and other sources of interference, and partially shielded from stepper motors cables etc. by the extrusions. I've never had it any other way, so I don't know how significant this is.

Jarery

I can unscrew my board and mount it standing up on end, or at farthest from the motors this weekend. The end panel of my housing is also removable for access to the usb and sd card, i'll remove it also

MiR

I let the printer run over night, no disconnect and the print job ran through without an issue.

What I saw this morning is that the wires of one stepper motor were running close to the wifi antenna, I will do a test tonight with stepper motor wires disconnected. tx failures are still 0 and also no decrypt errors reported by the router.

KeeganB

Well, last night went horribly for me.

The first night I put the replacement board in was Monday night. I didn't have much free time, so I took an hour and swapped out the boards. Two minutes after getting the firmwares updated and the duet connected to my network, it disconnected. In frustration I turned it off and walked away for the day.

Last night I went back to it to work on more troubleshooting. The disconnects immediately came back. I wanted to test the voltage at the main power inputs to make sure that when the board was under load, the PSU was supplying enough power. With the board powered on and the steppers OFF, I was getting 12.1 volts. Next, it measured 11.92V with the bed and hot end preheating and the stepper motors engaged. The third test I wanted to try was to check the voltage while the steppers were moving. Running an autocal was the easiest way to do this. This was the first time running an autocal or anything that would move the steppers since the "new" board was installed. It probed the first 2 points and then halfway to the third the stepper(s) started making a horrible grinding sound and skipping. I killed the power immediately. Upon powering back up it homed fine. G32 again, and the same thing, in the same area. again, killed power, rebooted and it homed fine. Further testing shows that after a reboot, it can move around X amount of mm, and then it will have issues. For example, I am able to drop it straight down 100mm and it is nice and smooth. When i click home from there, it makes it about 50mm back up before the grinding starts. Once the grinding starts, it grinds no matter how small the movement.

I THINK that it is specifically the Z stepper that is experience the issue.
I checked my stepper wires and connections to the board, they are were good and tight.
I went back to 16x micro stepping instead of 32x.
I stared longingly into the middle distance for a while.
I verified that the endstops are all connected well and functioning properly.

David, is it possible that this board you had sent to me is bad? It didn't look like a brand new board when I received it, so I am guessing it was a warranty repair or something?

So now, on top of having to solve the disconnects, I have to solve this mystery stepper nonsense too.

MiR

I today disconnected the stepper motors and removed G28/G32 from the gcode file I wanted to print.
Shortly after I started the print I got a disconnect, funny thing is that the disconnect was nearly in the same moment when I changed the printer temperature. After that I was unable to reconnect and I saw:

tx failures: 3
and
rx decrypt failures: 242 in sta_info command on the router.

Is it possible that problems on the duet cause it to crash and wifi is not properly recovered in the process? One thing I read was that decrypt failures can come from a wpa2 key changing unexpectedly.

MiR

Not sure if this is helpful or something totally different:

I can now create the Ajax Errors whenever I want. Turns out that it is good enough to remove G32 and G28 from my GCode file (I reconnected the steppers), then I only need to change temperature when the printer is heating up for the 2nd layer and I get the disconnect (most likely because Duet crashes, the whole gcode behaves a little strange without the homing command)

Do you have a watchdog process running that restarts duet? In that case I am perhaps onto something, if not then what I am seeing is simply a crash that kills duet plus wifi, no more, no less.
Just in case I have uploaded the gcode here:

http://temp.michael-ring.org/D_Delta2LowerPulleys.gcode

Michael

KeeganB

Michael,

In my situation, I get the disconnects even when the heaters are off. I haven't been able to figure out any pattern with mine lately. It happens when the heaters are off, or on, the steppers are off, idle, or moving, when the printer is printing or not, etc…

Where is you duet mounted in proximity to your PSU? after I figure out my stepper issues I posted above, my next step is to move the duet as far away as possible from the rest of the printer to test for disconnects that way.

MiR

I also have random disconnects, sometimes a few seconds after startup, sometimes shortly after starting a print. The only pattern I see is the pattern on what I see with station Info provided by my router. Besides that everything feels random, crashes after a few seconds, no problems to connect after several hours of idle after a print, I have it all

dc42

Once again you have the WiFi antenna inside a metal reflector along with several sources of interference. We designed the Duet WiFi with the antenna right at the edge of the board and no vertical connectors etc. in the last 20mm of the board along that edge, specifically so that you can mount the Duet with the antenna outside the printer metalwork.

MiR

David, can you please check my comments a little higher than this last comment, I‘d like your idea if what I saw is something worth following up or just a crash. At least the sympthons look exactly the same as a random crash, at keast from what I see in the logs of the router

Jarery

David

As to your suggestion that maybe its the way you have yours mounted vs those of us with issues, i tried the following.
On my delta, with only board and motors in the top triangle, i moved my board so its standing on end instead of flat, in the center of the triangle such that the wifi module is near no motor or motor wires and is as far away from all 3 motors as possible, without moving it completely out of the printer.
Disconnected after about an hour of idling and using autocorrect a few times.

Jarery

Looking back I think everyone with issues reported using routers with Merlins firmware? Perhaps anyone with Ajax disconnect issues who is running stock firmware on their router could confirm/deny.

Researching Asus routers that drop wireless clients, one thread had success turning off beamforming. In Merlins firmware there are two beamforming settings under "Professional" tab, Explicit and Universal.
Another thread reported wireless client issues dropping with both Merlins and stock firmware but problem cleared when they used DD-WRT firmware.

Last night I turned off both beamforming options. My system has stayed connected overnight. I wont be able to run it through any prints till tomorrow, but perhaps some others with the issue capable of printing sooner could also test and let us know prior to that.

If this route fails I'll start messing with other router firmware. If anyone with issues is using dd-wrt firmware please let me know so i don't go down that path unnecessarily.

fma

I'm running an OpenWRT firmware on an APU2 board, and I have these errors. On the other hand, my signal is really weak: -67dB

KeeganB

I was running the stock firmware on my Apple Airport Extreme router, and now I am running the stock firmware on my Tenda AC18 router.

I am about 99% sure the Apple router didn't have beam forming.

The Tenda does have beam forming, so I will see if I can turn it off.

Having had the same issues over two different routers and two different network setups I don't think it is a router issue in my case.

I've had a hell of a time with it over the past few days, and my frustration level reached a max last night. I have some new stuff to type up, but I am just not in the frame of mind to do it.

KeeganB

If anyone is interested in joining a Slack channel where we can have more instant (and mobile) conversations about this I have created one here:

https://join.slack.com/t/duetwifi/shared_invite/enQtMjY3MDQwOTAwMzc1LWUxMWYwYTg2OWI5NWU3NzI3YWI0ZmRhNGVkMmJjNjU4MTEwZDdjNWQwMTI4Yjk5MGViNWRmNjRiYTQ1NjFmYTY

If you have never used Slack, it is a FANTASTIC messaging system that works on nearly every platform and is loaded with wonderful features for file sharing, images, pics, chat, etc…

EDIT: If this slack channel is well received, we should probably still post to this post to keep everyone that may be following along updated on progress and findings.

Jarery

@KeeganB - According to google the apple airport extreme has beamforming. Ive never used their router, perhaps its not able to be turned on/off by the user. No mention if its explicit only or if they implicit also. In my test I've currently turned off both. I also share your pain with the frustration part

@fma, Thanks that helps to know.