PanelDueFirmware 3.2-RC3 released

docbobo

Just noticed the following in my DSF output:

[error] Failed to merge JSON: {"key":"","flags":"d99fn","result":{"boards":[{"mcuTemp":{},"vIn":{}}],"fans":[{"actualValue":0,"requestedValue":0,"rpm":-1},{"actualValue":1.00,"requestedValue":1.00,"rpm":-1}],"heat":{"heaters":[{"active":0,"current":61.0,"standby":0,"state":"off"},{"active":0,"current":84.5,"standby":0,"state":"off"}]},"inputs":[{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":205,"state":"idle"},{"feedRate":50.0,"lineNumber":151,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"}],"job":{"duration":null,"filePosition":0,"layerTime":null,"timesLeft":{"filament":null,"file":null,"layer":null}},"move":{"axes":[{"homed":false,"machinePosition":0,"userPosition":0},{"homed":false,"machinePosition":0,"userPosition":0},{"homed":false,"machinePosition":0,"userPosition":0}],"currentMove":{"acceleration":0,"deceleration":0,"laserPwm":null,"requestedSpeed":0,"topSpeed
   System.Text.Json.JsonReaderException: Expected end of string, but instead reached end of data. LineNumber: 0 | BytePositionInLine: 1280.
   at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
   at System.Text.Json.Utf8JsonReader.ConsumeString()
   at System.Text.Json.Utf8JsonReader.ConsumePropertyName()
   at System.Text.Json.Utf8JsonReader.ConsumeNextToken(Byte marker)
   at System.Text.Json.Utf8JsonReader.ConsumeNextTokenOrRollback(Byte marker)
   at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
   at System.Text.Json.Utf8JsonReader.Read()
   at System.Text.Json.JsonDocument.Parse(ReadOnlySpan`1 utf8JsonSpan, Utf8JsonReader reader, MetadataDb& database, StackRowStack& stack)
   at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory`1 utf8Json, JsonReaderOptions readerOptions, Byte[] extraRentedBytes)
   at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory`1 utf8Json, JsonDocumentOptions options)
   at DuetControlServer.Model.Updater.Run() in /home/christian/Duet3D/DuetSoftwareFramework/src/DuetControlServer/Model/Updater.cs:line 100

I also have that on startup for some other keys, e.g. "inputs" and "move". But the one above is reoccurring. It stop when the PanelDue's screensaver is on.

Some more system info:

M115
FIRMWARE_NAME: RepRapFirmware for LPC176x based Boards FIRMWARE_VERSION: 3.2-beta1_2 ELECTRONICS: LPC176x FIRMWARE_DATE: 2020-09-15b1

Board: biquskr_1.4 (LPCSBC)
DSF Version: 3.2.0-beta1+1

wilriker

@docbobo When the screensaver is active it reduces update poll rate from 1Hz to 0.25Hz. So this rather looks to me as if the LPC build has issues with available RAM or better: cleanly handling the lack of available output buffers.

gloomyandy

@wilriker I look after the LPC port and have a couple of questions if you can spare some time...

Is the panelDue causing any extra json requests to be sent to DSF, or is it just likely that having both the panelDue and DSF making requests is putting extra strain on the available buffer space?
What are the assumptions made about available output buffers (if any)?
It seems odd that a partial response is being sent to DSF. Is that what you would expect if the system runs out of output buffers? I would have thought that it would make more sense to assemble all of a json response before sending it?

The LPC port uses the same mechanism for handling a lack of buffers as does the Duet ports, so it seems likely that if the LPC has problems then in other circumstances (under heavy load or whatever) the Duet versions will hit the same problem. We do have a smaller number of buffers defined (16), but that is the same as number used by some other Duet hardware (that using SAM3XA mcus), but perhaps those Duets will never be used with an SBC?

T3P3Tony

@gloomyandy said in PanelDueFirmware 3.2-RC3 released:

Duet hardware (that using SAM3XA mcus), but perhaps those Duets will never be used with an SBC?

I cant answer the other questions, but the old SAM3X MCUs wont be supported connected to an SBC. Its close to the resource limit on Duet 2s using SAM4s

gloomyandy

@T3P3Tony said in PanelDueFirmware 3.2-RC3 released:

I cant answer the other questions, but the old SAM3X MCUs wont be supported connected to an SBC. Its close to the resource limit on Duet 2s using SAM4s

Thanks for the information Tony. I suspected that might well be the case!

dc42

@gloomyandy said in PanelDueFirmware 3.2-RC3 released:

Is the panelDue causing any extra json requests to be sent to DSF, or is it just likely that having both the panelDue and DSF making requests is putting extra strain on the available buffer space?

PanelDueFirmware 3.2 requests the frequently-changing parts of the object model using M409; whereas earlier firmware uses M408 to get the status response. The data returned by M409 is almost certainly larger than the data that was returned by M408, so it's more likely to cause a shortage of output buffers.

The code is supposed to detect when output buffer overflow occurs, and when that happens return {err:-1} instead of an incomplete response, to avoid JSON parse errors. Maybe that change hasn't reached your fork yet; or maybe there are some conditions under which that overflow detection doesn't work properly.

BTW we are looking to release RRF 3.2beta2 on Monday or Tuesday.

HTH David

gloomyandy

@dc42 Thanks for the update. The current 3.2 LPC fork is based on the Duet3D 3.2-beta2 (as released) version not sure if the checks you mention are present in that or not. I see there have been a lot of changes in the SBC interface for beta2. I will be integrating/testing those as soon as beta2 is released and will investigate this further once I have the beta2 changes.

wilriker

@gloomyandy said in PanelDueFirmware 3.2-RC3 released:

@wilriker I look after the LPC port and have a couple of questions if you can spare some time...

Is the panelDue causing any extra json requests to be sent to DSF, or is it just likely that having both the panelDue and DSF making requests is putting extra strain on the available buffer space?

There are a very few extra requests going through to DCS (macro and file listings) but they are sent very rarely. It's your second thought that I'd true though, PanelDue with the latest firmware does make an ObjectModel request once per second and that puts extra strain on the available output buffers such that e.g. Duet 2 with no SBC and 24 buffers regularly responds with out of buffer messages if PanelDue and DWC are connected (I use this example since SBC support needs even more memory)

What are the assumptions made about available output buffers (if any)?

Sorry, I don't understand this question.

It seems odd that a partial response is being sent to DSF. Is that what you would expect if the system runs out of output buffers? I would have thought that it would make more sense to assemble all of a json response before sending it?

It really seems odd to me that the binary communication between RRF and DCS sees partial responses, though I am not totally familiar how these are assembled. For JSON responses, AFAIK, it does actually try to allocate the required output buffers dynamically and errors out if along the way there are not enough available. @dc42 and/or @chrishamm should be able to shed some more light on this part.

There has been a fix added to RRF 3.2-beta1 such that in this case it at least returns a clean error instead of a partial response. Since I don't know how close your fork actually is to "mainline" it was an assumption that this might not have been included yet.
Did not want to offend you or your work. Sorry, if that came across like "not my fault, go blame someone else".

The LPC port uses the same mechanism for handling a lack of buffers as does the Duet ports, so it seems likely that if the LPC has problems then in other circumstances (under heavy load or whatever) the Duet versions will hit the same problem. We do have a smaller number of buffers defined (16), but that is the same as number used by some other Duet hardware (that using SAM3XA mcus), but perhaps those Duets will never be used with an SBC?

As an addition to what @T3P3Tony already mentioned the old SAM3 based Duet boards never even got to RRF 2 - due to the lack of RAM. So they will never support SBC (officially).

gloomyandy

@wilriker Thanks for the reply, my question about "assumptions" was just speculation I wasn't sure if it was assumed/required that there was sufficient buffer space for say a certain type of json response. I can confirm that that when running with the panelDue/SBC all 16 buffers (on the LPC version) are indeed used so I think your diagnosis of a lack of buffers is correct.

It looks like the SBC interface has changed a fair bit between beta1 and beta2 so I'll take a closer look at the partial json problem once I've merged the beta2 changes. Will let you know if I think there may be a general problem there.

@wilriker @dc42 @T3P3Tony Thanks again for you help.

wilriker

@docbobo said in PanelDueFirmware 3.2-RC3 released:

@wilriker Let me know if there's anything I can do.

So, I tried but unfortunately (or luckily) I cannot reproduce your issue with M300 not playing on PanelDue. I tried playing various tunes from various sources (USB, HTTP, Panel itself) in in all cases sounds were played as expected.

Can you provide me with an example? Also I am running RRF 3.2-beta2 now. Please check if that makes a difference.

docbobo

@wilriker said in PanelDueFirmware 3.2-RC3 released:

So, I tried but unfortunately (or luckily) I cannot reproduce your issue with M300 not playing on PanelDue. I tried playing various tunes from various sources (USB, HTTP, Panel itself) in in all cases sounds were played as expected.

Can you provide me with an example? Also I am running RRF 3.2-beta2 now. Please check if that makes a difference.

Will do. However, won’t be able to do this before the end of week.

dc42

@gloomyandy said in PanelDueFirmware 3.2-RC3 released:

It looks like the SBC interface has changed a fair bit between beta1 and beta2

Yes, it uses a separate task now.

dc42

I have just checked, and the code to avoid sending malformed JSON responses when there is output buffer overflow is present only for M409 responses. I've made a note to fix this in the next beta.

wilriker

@oozeBot said in PanelDueFirmware 3.2-RC3 released:

@wilriker we are using Duet 3 / RPi combos running 3.2b1 and the PanelDue is on 3.2-RC3+1. I just had it happen again on a machine that had been turned off overnight. What I didn't catch was the state reported in the top right corner of the PanelDue, but I believe it was "Printing". I feel sure this will happen again today, so I'll report back.

I will point out that I didn't notice this nearly as often (or at all?) on the earlier betas.. it feels new to 3.2-RC3+1. Thanks

I tracked this down to be an issue (or intended? limitation - waiting on feedback from @chrishamm) in DSF. M291 in macros will currently not update state.messageBox in the ObjectModel and that's the source for these dialogs showing up on PanelDue. It works if M291 is sent directly.
As mentioned before I notified @chrishamm about this.

wilriker

@docbobo said in PanelDueFirmware 3.2-RC3 released:

@wilriker no, it’s not waking up the panel. It was mostly black, just the numbers for x, y, and z were appearing.

Unfortunately I cannot reproduce this one either. Can you tell me exact steps to reproduce this?

wilriker

I just now released 3.2.0-RC4. Closing this thread now.