MPG Pendant for Duet 2 using PanelDue input.

CJD

I see a few discussions about building an MPG Pendant for use on CNC machines but with no obvious conclusion.

I had the same idea and have now built two prototype pendants which have been tested on two different CNC machines fitted with Duet 2 boards.

The pendant uses an Arduino Pro Mini, 12 push switches and a rotary encoder.
Connection is using the PanelDue port set to 57600 baud.

8 of the switches send M95 commands to run 8 different macros. This works fine.

3 switches are used to select the X,Y or Z axis for the rotary encoder Jog Wheel.

Turning the rotary encoder initially sends G91 followed by multiple G0 X0.1 to incrementally move by 0.1mm per click.
G90 is sent after a short period of inactivity to return to Absolute mode.

The rotary encoder normally works well but occasionally the Duet Web interface looses connection and does not reconnect unless the Duet 2 is power cycled. The jog wheel and buttons continue to work and the machine continues to function but the web interface does not reconnect. The machine can still be pinged in this condition so the wifi connection is good but for some reason the web interface does not respond.

I am not actively using the web interface at the time, it is just open on the workshop PC.

This seems to happen more often when turning the jog wheel rapidly, this is sending lots of G0 X0.1 commands in succession so it may be that this is causing some sort of buffer overflow in the RepRap code and somehow crashing the web interface.

I am trying to identify the cause of the problem and have a few questions:-

Is there any flow control implemented on the PanelDue input?
Assuming there is an input buffer on the PanelDue input what happens if it overflows?
Is there any way large amounts of commands on the PanelDue input could be clashing with the Web interface?

Colin

Phaedrux

Just for information what firmware and dwc versions are you using?

CJD

I am not at the machine at the moment but I downloaded the latest of both 3 days ago.

dc42

@CJD said in MPG Pendant for Duet 2 using PanelDue input.:

Is there any flow control implemented on the PanelDue input?

No.

Assuming there is an input buffer on the PanelDue input what happens if it overflows?

The AUX input buffer is 511 bytes long. if it overflows, then additional characters will be lost. PanelDue runs with checksumming enabled, so in most cases this would lead to the resulting command being recognised as invalid.

Is there any way large amounts of commands on the PanelDue input could be clashing with the Web interface?

I can see two possible mechanisms for that:

If the aux input buffer overflows, and due to character loss a command gets interpreted as a command to disable networking or disable HTTP protocol. Are you running with checksumming, or have you disabled it?
If for some reason a large amount of data gets sent to the aux port, and the output buffer fills up, then the main processing loop will wait while the data is sent. This will slow most things down. I would not expect it to cause a network disconnection.

Bear in mind that the aux buffer will also fill up of you send it GCode commands too fast for the machine to execute.

Here are some suggestions:

Keep the GCode commands short, by not including the F parameter in G1 commands unless you want to change the feed rate.
Limit the rate at which you send commands. Don't always send a command on every click of the wheel, instead if you get several clicks in succession then send one movement command to cover all of them.

If necessary I can improve the response to input buffer overrun.

CJD

Thanks for the response.

It is strange that the web interface stops completely and only reconnects if the power is cycled. Even a GCode to reboot doesn’t restart it. I will try debugging using the USB port to see if I can work out what’s happening.

I am not currently using checksum for the commands, I have considered that and will rewrite the sketch and give it a try.
I am already keeping the commands as short as possible. I only send G0 X-0.1 for each click.

I will also try enforcing a maximum rate for the commands so as to leave a short gap between them.

Jogging is something that is quite important for CNC use when lining up on an existing component. It probably doesn’t get as much use in 3D printing. Ideally a method of commanding an axis to move continuously at a set speed until told to stop would be useful here. Then a pendant could be made with jog buttons for each axis. An idea rotary MPG in the traditional sense gives a direct connection from the rotary encoder to the stepper motor. So 10 pulses from the rotary gives 10 pulses of the stepper with no added acceleration or deceleration filtering. However that is probably not something that could be easily done using GCode.

Colin.

dc42

@CJD said in MPG Pendant for Duet 2 using PanelDue input.:

It is strange that the web interface stops completely and only reconnects if the power is cycled. Even a GCode to reboot doesn’t restart it. I will try debugging using the USB port to see if I can work out what’s happening.

Yes, when the problem occurs send M122 from USB and check the output buffer count. Also send M552 to check that the network is still enabled, and M586 to check that HTTP is still enabled.

CJD

One of my previous statements was wrong. I initially though that a software reboot following an M112 Emergency Stop did not restore the web interface but in fact it does. It just takes longer than I was expecting. So it does look like for some reason the HTTP protocol is being disabled.

I am just adding checksums to the commands to see if that makes any difference.

Colin.

CJD

Adding checksums to the commands made no difference. So I restricted the rate of commands to no more than 10 per second, increasing the movement size as necessary. This seems to have improved the situation. I have not seen any failures yet. Unfortunately the jogging is now a bit jerky as variable size movements coupled with acceleration and deceleration don't produce smooth movement.
It seems to be a usable work-around but still not the smooth motion I would really like.

Colin.

dc42

Maybe 10 commands/second is lower than it needs to be?

CJD

Yes, it is still not clear what is happening. This is only a temporary work around. For my own peace of mind I would like to find the root cause. More investigation to come when I get the time.

CJD

This is a long post but I think I have made some progress.....

After a lot of testing today I think I can see what is going wrong.
Even after reducing the rate of commands sent through the paneldue port I could still reproduce the web interface hangup.

I then enabled debugging of the http and gcodes on the USB port using M111.

When it was working I saw the following typical debug output.

HTTP connection accepted
HTTP req, command words { GET /rr_status HTTP/1.1 }, parameters { type=1 }
Sending JSON reply, length 547
aux: G0 X0.50
aux: G0 X0.50
aux: G0 X1.00

By repeatedly sending commands over the PanelDue port I recorded the following when the web interface crashed.

HTTP connection accepted
HTTP req, command words { GET /rr_status HTTP/1.1 }, parameters { type=2 }
aux: G0 X1.00
aux: G0 X0.50
aux: G0 X1.00
aux: G0 X1.00
aux: G0 X1.00
aux: G0 X1.50
aux: G0 X1.50
aux: G0 X1.50
aux: G0 X1.00
aux: G0 X0.50
Ran out of output buffers at ../src/Networking/HttpResponder.cpp(980)

when I closed the Web browser I saw

HTTP connection accepted
HTTP req, command words { GET /rr_disconnect HTTP/1.1 }, parameters { }
Sending JSON reply, length 9
Ran out of output buffers at ../src/Networking/HttpResponder.cpp(980)
Can't send anymore

So it is definitely an output buffer problem.

I could then see the Web interface trying to reconnect but it was failing because the "Ran out of output buffers" message kept appearing.

It looked like the output buffers were never being reset.

Sending an M122 command from the USB port gave the following.

serial: m122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 2.05 running on Duet WiFi 1.02 or later
Board ID: 08DGM-9T6BU-FG3SJ-6J1FJ-3SS6K-98Z3H
Used output buffers: 22 of 24 (24 max)
=== RTOS ===
Static ram: 25712
Dynamic ram: 92844 of which 0 recycled
Exception stack ram used: 456
Never used ram: 12060
Tasks: NETWORK(ready,404) HEAT(blocked,1232) MAIN(running,4080) IDLE(ready,160)
Owned mutexes:
=== Platform ===
Last reset 00:11:30 ago, cause: software
Last software reset time unknown, reason: Heat task stuck, spinning module GCodes, available RAM 11428 bytes (slot 2)
Software reset code 0x00a3 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f80f BFAR 0xe000ed38 SP 0x200049fc Task 0x4e49414d
Stack: 0042bf17 0042bfda 01000000 00000000 00000000 00000000

The used output buffers line would tend to say that 22 buffers were in use even though the machine was idle.

Following up on this clue I reset and repeated the exercise whilst regularly doing M122 commands.
I could see the used buffer count gradually increasing. So it appears that something in the firmware is consuming buffers and not releasing them. I believe that this is the root cause of the problem. (And maybe also similar problems others have had with output buffers).

By accident I found that if I used my G0 commands to move the machine to its maximum limit the stepper motor stopped moving as expected. If I then continued to send G0 commands the used buffer count increased quite rapidly until the web interface crashed.
The only reason I can see for this is that the machine would have been sending 'Movement out of range' error messages after each command and I was probably sending my G0 commands at the same time.

I have a theory which might explain this behavior:-

If a new G command is received while the firmware is trying to send a response to the previous command I suspect the response could be being abandoned without freeing up the associated output buffer.

I tried to rewrite my Pendant code to wait for an "ok" response after each command to try to avoid this condition but it appears that the PanelDue port does not send responses to anything except an M408 Json status request.

Is there any way to get the PanelDue port to output the 'ok' after each command?

Colin.

dc42

You must periodically read the responses using M408, otherwises responses will pile up in the output buffer. But I'll implement a timeout too in the next RRF3 build.

CJD

Thanks for that. I hadn’t realised that was necessary, I assumed if I did not need the responses I did no have to read them.

Not much time to devote to this at the moment, hopefully I can get time to look at it soon. When it is working reliably I will publish the design for the pendant on github.

Colin.

CJD

Thanks, that fixed my problem.

For anyone else trying to use the PanelDue port for their own hardware the summary is this....

The PanelDue port does not send responses to commands in the same way that the USB port does. Responses are queued up in the output buffer and must be regularly retrieved using the M408 command with a sequence number.

You must issue the M408 command regularly even if you are not using the responses or the firmware will run out of output buffers and begin behaving strangely.

If you are not interested in the responses then just sending
M408 S0 R1 every few seconds and ignoring the response will flush out the buffers.

CJD

if anyone is interested in making one the Arduino sketch and schematic for this pendent can be found at
https://github.com/g4eml/JogBox

Colin.

wesc

@CJD are you using a COTS pendant or one of your own design?

CJD

@wesc

I am using my own design. However the Arduino code could probably be adapted to use any pendant with switches and rotary encoder.

Colin.