Expansion board 1 stopped sending status

droftarts

@chris94 said in Expansion board 1 stopped sending status:

The M122 reports this as the reason for the crash, on the 3HC:

reason: HardFault imprec

The main reason we see these is due to static discharge causing memory corruption. This is generally caused by insufficient grounding of hot end and extruder parts, and stepper motors. See https://docs.duet3d.com/en/User_manual/Connecting_hardware/Power_wiring#grounding

Ian

dc42

@chris94 from the console output it is clear that an Emergency Stop was commanded. The "Failed to switch off remote heater" and "Expansion board stopped sending status" messages came slightly later and were consequences of the Emergency Stop.

The Hard Fault that @droftarts refers to occurred over a month ago so it not related to tis event.

So the question is: what commanded the Emergency Stop? Here are some possible reasons:

M112 was encountered in the GCode file, or was received through an input channel e.g. USB or Telnet (if enabled)
STOP button pressed on an attached PanelDue
The wires connecting PanelDue are picking up interference (e.g. from nearby stepper motors wires) and you have not enabled CRC protection in the corresponding M575 command in config.g
Emergency Stop button pressed in DWC

HTH David

droftarts

@dc42 said in Expansion board 1 stopped sending status:

The Hard Fault that @droftarts refers to occurred over a month ago

Good point!

Ian

chris94

Thanks for the quick answers.

@droftarts I am pretty sure that everything is grounded but I am not a professional.

@dc42 I don't think the M112 comes from a G-code or macro as it also occurs when the printer is in idle. I am sitting 2m away from my machine and I was alone in this room so I would say that no EMO button was pressed.
The PanelDue uses a ~1m long unshielded cable so I guess I could start there. Is there a way to check my system for interference?
I do not really understand the difference between the CRC modes but I am currently running my M575 with P1 S1 B57600

(Great learning experience for me)
Chris

droftarts

@chris94 What version of the PanelDue firmware is it running? Check on the 'Setup' screen. S1 sets either checksum or CRC, depending on the PanelDue firmware, CRC is set on PD firmware from v3.4.1 and later.

Do you have an emergency stop button? If so, how is that wired? If it's NO (normally open) as most of them are, that could also be picking up interference.

Ian

chris94

@droftarts I have to admit (full of shame) I have never checked my PD firmware and it is still on 3.2.11 but I will update it as soon as possible.
I do have an emergency stop button and it is normally open.

droftarts

@chris94 Try disabling the emergency stop button (comment out the trigger in config.g), and/or updating the PanelDue (see https://docs.duet3d.com/User_manual/RepRapFirmware/Updating_PanelDue).

If that seems to work stably, re-enable the emergency stop, but check the wiring, move it as far as possibly from the stepper motor and heater wires.

Ian

chris94

@droftarts Thanks for the help.
I am already working on the update and I will disable the EMO after that. It will probably take a few days before I can reliably say whether this solves the problem or not. Sometimes everything runs without problems for days, but I will let you know as soon as I know more.

droftarts

@chris94 It's also possible that any housing you have on the PanelDue is moving just enough (usually with temperature changes) to put pressure on the PandelDue Emergency stop. Users have had that problem in the past, too.

Ian

dc42

@chris94 when you've updated your PanelDue firmware, also change the M575 P1 command in config.g from S1 to S4. This ensures that RRF will only recognise a valid CRC from PaneDue. With S1 it recognises either a valid CRC (which is good protection against interference) or a valid checksum (which is not).

chris94

Thanks again for the help

@droftarts I have a clearance gap of 0.5mm between the display and the case. However, I should be able to hear the display beep when something triggers the display and I did not hear anything.

@dc42 PD update is done and I’ve changed my config as you suggested.

The EMO trigger is still active, but I changed the trigger G-code to just display a message and play a sound. I'll let the printer idle for the rest of the day to see if anything happens. If nothing happens today, I will run a test print tomorrow if I have time. Rerouting the EMO cable is on my to-do list this evening.
I guess it's time to wait and see what happens.

chris94

OK, the two testing days are over and the title of this topic is probably wrong. I thought that the emergency stop was caused by the loss of connection but now I have learned that it was the other way around.
It looks like all of my NO switches are affected by this problem. I can't say why, but I suspect that my printer is not directly the cause of the switch activations because I noticed that the problems almost always occur around the same time.
If I understand this correctly and I replace all NO switches with NC switches, the problem should be solved right? The only problem that would remain is that my Z-Probe cannot be replaced by an NC switch because I am using a CS067A-HT1 from Metrol, which is very expensive. So, I guess I also have to find the cause for this.
The two possible causes I can imagine are either interference from other devices or main power problems. Unfortunately, I don't know how to find out where it comes from. Can I even figure this out by myself and if so, how?

droftarts

@chris94 said in Expansion board 1 stopped sending status:

It looks like all of my NO switches are affected by this problem.

Generally, endstops are only checked while homing, and the rest of the time they are ignored. Interference on these shouldn't be causing emergency stops or glitches, because they are not checked, unless you have them set to trigger for 'over-travel' events.

@chris94 said in Expansion board 1 stopped sending status:

I can't say why, but I suspect that my printer is not directly the cause of the switch activations because I noticed that the problems almost always occur around the same time.

If it is at a specific time of day, I'd guess it's an electrically-noisy heating or air conditioning pump turning on, something like that. You may get around this by putting a surge protector on the mains input, as the interference seems to be external to the machine. What time of day does it cause the switch activations? See this post, which had an issue with a motor causing electrical interference (though the noisy motor was actually part of the machine): https://forum.duet3d.com/post/346470

@chris94 said in Expansion board 1 stopped sending status:

The only problem that would remain is that my Z-Probe cannot be replaced by an NC switch because I am using a CS067A-HT1 from Metrol, which is very expensive.

I doubt you're using the probe during the print, and it seemed that it was the emergency stop that was causing the issue because it is specifically monitored all of the time. So I wouldn't worry too much about this. If you think it is causing issues, you could remove the configuration for it from config.g, and only configure it within macros that need to use it, and un-configure it at the end of the macro.

Ian

chris94

@droftarts said in Expansion board 1 stopped sending status:

Generally, endstops are only checked while homing, and the rest of the time they are ignored. Interference on these shouldn't be causing emergency stops or glitches, because they are not checked, unless you have them set to trigger for 'over-travel' events.

Yes I do have one as an over-travel safety endstop because I crashed the axis few time and that’s not really healthy on a double spindle driven axis. If I had been smart I would have installed it as an NC

I doubt you're using the probe during the print, and it seemed that it was the emergency stop that was causing the issue because it is specifically monitored all of the time. So I wouldn't worry too much about this. If you think it is causing issues, you could remove the configuration for it from config.g, and only configure it within macros that need to use it, and un-configure it at the end of the macro.

Thanks for the Idea. I will take a look at it.

If it is at a specific time of day, I'd guess it's an electrically-noisy heating or air conditioning pump turning on, something like that. You may get around this by putting a surge protector on the mains input, as the interference seems to be external to the machine. What time of day does it cause the switch activations?

Mostly it occurs at 6 o'clock in the evening but there have also been days when it has happened 3 times in a row in the afternoon. Unfortunately, my neighbours aren't very helpful when it comes to topics like this. For example, one of my neighbours doors has been squeaking for 2 years and they don't care (so I'm on my own). The surge protector is already on my list, but I still need to find a place to put it.

I think that the issue is solved for now and if the problem occurs again after I have implemented everything that was discussed here, I will reach out again. Thanks again for the help and for the ideas from both of you.