Notifications
Clear all

Danger: Extruder thermistor failsafe didn't work  

  RSS
Geoff
(@geoff)
Active Member
Danger: Extruder thermistor failsafe didn't work

Prusa Mk2.5

My Mk2.5 recently started to produce prints with large chunks of missing extrusion. I found that this was caused by an intermittent break in the wire from the heat-block thermistor (about half way along the cable bundle).
I have replaced the lead and now all is well. However, I am concerned that the firmware (3.5.0) did not detect the failure, because when present the indicated temperature dropped to 27C and the head heater tried to ramp up the power to compensate. I caught it at one point (when connection momentarily restored) up to 270C. Luckily the break was intermittent and dependent on head position, so at intervals the correct reading would be measured and the head would cool again for a few minutes.
The symptoms on my printer when the wire was o/c were: the indicated temperature dropped to 27C, the heatsink cooling fan stopped, the extruder stopped, the printer X,Y and Z actions continued, the heater power increased to try and restore the erroneous temperature.
Now I thought that the Prusa printers were smart enough to detect a faulty thermistor in order to prevent thermal runaway and possible meltdown or fire. This did not happen on my printer - does anyone know why?

Publié : 19/01/2019 11:24 am
RetireeJay
(@retireejay)
Reputable Member
Re: Danger: Extruder thermistor failsafe didn't work

Safety is always a priority! Thanks for bringing this up; it may indeed be a problem.

Normally when a thermistor wire breaks completely, the indicated temperature drops to zero. Technically, a thermistor of the type we use is 100k ohms at room temperature, and the resistance goes down significantly as the thermistor gets hotter, or up if the thermistor gets colder. So technically, an infinite resistance corresponds to an undefined temperature well below zero, but the table of values in Marlin firmware stops at zero (Prusa firmware is based on Marlin).

Normally, if the firmware detects a temperature of zero, it will declare a fault and stop the print.

You could try unplugging the thermistor from the board and see if the indicated temperature drops to zero (and/or causes the board to report a fault). If it does not, then there may be a problem in the firmware configuration because open circuit is definitely a fault. If your indicated temperature when unplugged does drop to zero, that probably means that in the earlier case your broken cable was not truly and absolutely broken but was maintaining a weak connection of about 100k ohms.

I can't run the same experiment for you because my MK3 uses a different board and therefore also slightly different firmware.

Publié : 19/01/2019 2:33 pm
Geoff
(@geoff)
Active Member
Topic starter answered:
Re: Danger: Extruder thermistor failsafe didn't work

Yes I will try that when I can. I have updated firmware now to 3.5.1, but I don't think there have been any changes that might affect the situation above.

Certainly the break could well have gone through a high-resistance phase as it made & broke connection so might not have given an instantaneous step change in resistance.
The firmware must have been detecting the temperature change because the Noctua fan stopped and started as the open circuit came and went - presumably because when the apparent temperature was low the firmware decided there was no need to cool the heatsink!

I dissected the wire afterwards and found a clean break in all the strands of one conductor at the same point about halfway along the cable.

Just an addition: when the connection was broken (during the fault) the firmware didn't indicate 0 it indicated 27 degrees.

Publié : 19/01/2019 2:50 pm
RetireeJay
(@retireejay)
Reputable Member
Re: Danger: Extruder thermistor failsafe didn't work

I've looked at ALL the thermistor tables in "thermistortables.h" and ALL of them carry the temperature down to zero (some of them even below zero). So if your readout says that the temperature is greater than zero when the thermistor is unplugged (or definitively open-circuit) then something is not working the way it should. The logic inside the temperature code (temperature.cpp) is rather large and complex and seems to involve the possibility of an "ambient" thermistor as well as the extruder, bed, and PINDA thermistors, although I am not aware of an ambient thermistor in Prusa printers.

I have had my printer declare a temperature fault when the print fan cooled the extruder temperature more than five degrees below setpoint for more than a few seconds, so I know that generally speaking there are multiple fail-safes built into the firmware.

It's crucial for us to figure out if your printer keeps running happily with the thermistor unplugged. That would be a serious safety issue and needs to be thoroughly understood and fixed. It could be a defect on the board that generates 100k of resistance across the terminals, or it could be a newly-introduced bug in the V3.5 firmware.

Publié : 19/01/2019 5:22 pm
Geoff
(@geoff)
Active Member
Topic starter answered:
Re: Danger: Extruder thermistor failsafe didn't work

Right. I've just unplugged the thermistor. The temperature reads 20C. Turned on PREHEAT with the thermistor still disconnected, the printhead and bed both started heating but of course the indicated temperature didn't rise.
Blowing warm air into the electronics box causes the indicated temperature to rise, even with the thermistor disconnected. The temperature indicated seems to be the ambient in the box - as if there was a temperature sensor contained in the electronics.

Publié : 19/01/2019 5:44 pm
RetireeJay
(@retireejay)
Reputable Member
Re: Danger: Extruder thermistor failsafe didn't work

You said that this is a recent problem. Can you correlate it with any particular event besides the cable breaking? Did the printer work fine before you re-flashed the firmware to 3.5? Can you try re-flashing it with 3.4 or earlier to see if the problem goes away? Did you build this from a kit or buy it pre-assembled? What kind of board is used for the controller?

Obviously you can't use your printer until you get this figured out. If the indicated temperature is greater than zero when the thermistor is unplugged, then you can't trust it to be correct when the thermistor is plugged in.

Publié : 19/01/2019 8:14 pm
Geoff
(@geoff)
Active Member
Topic starter answered:
Re: Danger: Extruder thermistor failsafe didn't work

I have never had the thermistor unplugged or disconnected before, so I wouldn't have noticed anything unusual until the lead failure.

The printer was a Mk2s kit, bought end of September 2017, and upgraded to Mk2.5 in April 2018. The board is whatever was being supplied with the Mk2s in September 2017.

The printer has performed faultlessly since I had it both before and after the upgrade, until the recent faulty thermistor lead and I have had no reason to think that the temperatures were outside normal tolerances. All materials seem to print normally.

I have just checked the heat-block with a thermocouple thermometer and it measures about 209 degrees when set for 215 degrees; given that the probe has to be fixed on the side of the block that seems to me to be reasonable.

I haven't yet tried re-flashing with 3.4 but 3.5.0 and 3.5.1 both behave the same. I'll try 3.4 tomorrow.

Publié : 19/01/2019 10:52 pm
RetireeJay
(@retireejay)
Reputable Member
Re: Danger: Extruder thermistor failsafe didn't work

Based on the Prusa website documentation for the MK2S, your board is a "Mini-Rambo". I've looked at the schematic ( ) and I don't see anything that would normally prevent the voltage at the input of the Atmel A/D converter from going to full scale when the thermistor is unplugged. Which should correspond to an indicated temperature of 0C. However, I note that all the thermistors have a common "reference" voltage, and maybe if that's not where it should be it could cause the erroneous 20C reading. If that is the case, then if you unplug the bed thermistor the bed temperature reading should also go to 20C instead of zero (or at least the extruder and bed temperatures should track each other if both thermistors are unplugged). Your board has three thermistor sockets; are all three in use?

Publié : 19/01/2019 11:22 pm
Geoff
(@geoff)
Active Member
Topic starter answered:
Re: Danger: Extruder thermistor failsafe didn't work

I think I have tracked this problem down: the thermistor D12 which provides the reference voltage is very temperature sensitive - a brief shot of freezer spray sends the temperature reading down rapidly, enough that with the thermistor disconnected I get a min temp error, which is as it should be.

I'm debating the difficulty of replacing this diode or whether a new board is going to be needed. seems a shame to replace the whole board for the sake of a component costing a few pence!

Notwithstanding the above, there would seem to be room for improvement in the failure detection. Consider: the printer is printing happily away when the measured heat block temperature suddenly drops from 210 degrees to 27 degrees. The Noctua fan controller says "Aha! the heat block is cold, I'll stop the fan". The extruder controller says "Aha! the heat block is cold, I'll stop trying to extrude". Meanwhile the X, Y and Z drivers carry on merrily as if nothing has happened.

Publié : 20/01/2019 10:54 am
Geoff
(@geoff)
Active Member
Topic starter answered:
Re: Danger: Extruder thermistor failsafe didn't work

Thinking about it, either way I'm going to have to take the board out so I'll try replacing the diode first and if I don't have any luck I'll order a new board.

Publié : 20/01/2019 11:15 am
Geoff
(@geoff)
Active Member
Topic starter answered:
Re: Danger: Extruder thermistor failsafe didn't work

I have to admit I'm slightly stumped here. I've replaced the zener diode with no real improvement. The ref voltage it provides is basically a clamp voltage for the thermistor inputs to the CPU which should limit them to one diode drop + the zener voltage. The zener voltage is (or should be) 4.7V but I'm only measuring 4.47V even with a new diode. This is causing the voltage on the thermistor inputs when open circuit to limit at 4.95V which is presumably too low, I think it should go to Vcc (Vcc measured at 5.0V).
The curious thing is that I can't see why I'm not getting the correct voltage on the zener: it's fed through a 10k resistor (measures 9.9k) from the 12V supply (measures 12.0V). This should give a current of about 0.7mA through the zener which is well above the current it needs to stabilise, which is 0.1mA according to the data sheet. There is no decoupling capacitor which might be leaky and if one of the diodes D1, D14 and D11 was leaky it can't draw current from the zener regulator because with the thermistors o/c the voltage on the anodes of the diodes is > than the zener voltage anyway.

I hate it when something that can't be wrong is wrong.

Can you think of anyone who might be able to shed some light on this? I hate to be beaten by such a simple circuit!

Publié : 20/01/2019 3:06 pm
RetireeJay
(@retireejay)
Reputable Member
Re: Danger: Extruder thermistor failsafe didn't work

Oh wow. The datasheet https://www.mouser.com/datasheet/2/308/MM3Z2V4T1-D-269162.pdf says the nominal 4.7V zener can be as low as 4.4V and still be within spec. So this is a design error: a diode at the bottom end of the allowable specification will cause the behavior you are seeing, i.e. the open-circuit voltage never goes high enough to trigger the "zero degrees = fault" condition in the printer.

I can see why they have the zener - in order to prevent the input voltage from ever going above VCC. It seems like they should use a zener with a tighter tolerance.

Or maybe just connect the clamping diode to VCC. The spec sheet for the ATMega2560 http://ww1.microchip.com/downloads/en/devicedoc/atmel-2549-8-bit-avr-microcontroller-atmega640-1280-1281-2560-2561_datasheet.pdf does say that the maximum voltage on an analog input is VREF (on page 365). But on page 274, it says "Single ended channels that exceed VREF will result in codes close to 0x3FF" which implies that it's safe to exceed VREF. I couldn't find an "Absolute Max" specification.

The only purpose of the clamping diode is protection from ESD or other circuit faults. When the wiring is normal, it's impossible for the input voltage to ever exceed VCC. So if I were you I'd remove the zener and jumper the plus lead (OVREF) to VCC. :geek:

Publié : 20/01/2019 8:05 pm
Geoff
(@geoff)
Active Member
Topic starter answered:
Re: Danger: Extruder thermistor failsafe didn't work

Thanks for confirming my suspicions!

Interestingly I found this alternative design of the the thermistor circuitry which would avoid the problem (and is effectively the same as your suggestion) but I notice that it uses schottky diodes instead of 1N4148s, presumably to limit the voltage range as close as possible to 0V-Vcc.

Do you think this is a problem that Prusa should be made aware of? I wonder how many board are affected.

Sorry, can't seem to work out how to just paste an image into this post, but here is a link to the circuit diagram I am talking about:

https://github.com/ultimachine/Einsy-Rambo/blob/1.1a/board/Project%20Outputs/Schematic%20Prints_Einsy%20Rambo_1.1a.PDF

[Update - it attached my image after all - twice!]

Publié : 21/01/2019 9:44 am
RetireeJay
(@retireejay)
Reputable Member
Re: Danger: Extruder thermistor failsafe didn't work

The Einsy board is used in the MK3. So hopefully that means that the correct solution is now the current and best supported design.

But Prusa is still selling the older machines.

I agree with you that this is a serious problem. If the heater wires break (really, when the heater wires break) it's extremely important for safety that the machine recognize this and shut down. Clearly not all mini-Rambo boards will be affected because the nominal zener voltage should be OK, but some boards will be affected by the problem due to manufacturing tolerances of the zeners - and unless Prusa does screening before shipping the boards, most users will never know if they are at risk of a fire or not.

I think you and I should at least raise this as an "Issue" on the github page for the mini-Rambo, as well as tell Prusa directly.

Publié : 21/01/2019 2:31 pm
Geoff
(@geoff)
Active Member
Topic starter answered:
Re: Danger: Extruder thermistor failsafe didn't work

Yes agreed.

By the way, the latest regarding my own machine: I found that the tolerance of my replacement zeners was actually quite close to the target voltage, BUT only if the current was increased slightly to put the diode well over the knee voltage (which is not particularly sharp on these low voltage zeners).

I Increased the current from around 0.7mA (with the original 10k dropper) to just over 3mA by putting a 2k7 resistor in parallel with the 10k.

The voltage on the zener is now pretty much spot on 4.7V and open-circuit thermistors are now detected properly - so I'm back in business.

Thanks for your input!

Publié : 21/01/2019 2:44 pm
RetireeJay
(@retireejay)
Reputable Member
Re: Danger: Extruder thermistor failsafe didn't work

Very interesting! I'm glad you're back in business!

Publié : 21/01/2019 4:29 pm
RetireeJay
(@retireejay)
Reputable Member
Re: Danger: Extruder thermistor failsafe didn't work

I posted the following on the Mini-Rambo GitHub site ( https://github.com/johnnyr/Mini-Rambo/issues ):
The clamping circuit for thermistor inputs uses a zener diode to theoretically limit the positive voltage excursion at the ATMega input to just a shade over 5.0V. However, there are two problems with the circuit: (1) the current through the 4.7V zener, 0.73mA, is not sufficient to put the zener fully into its rated clamping voltage. In practice, one user has seen 4.4V. This was solved by increasing the current to about 3mA, well above the "knee" for his particular zener.
(2) Even with sufficient current, some zeners with this part number will be as low as 4.4V due to manufacturing tolerances.
The problem is this: In either of the above scenarios, the analog voltage at the ATMega input is constrained to slightly less than 5V. (Note that the clamping diode is operating at a very low current, so its voltage drop added to the zener voltage is less than the canonical "0.6V". At 0.01mA, the 1N4148 has a forward drop of about 0.4V). If the input voltage at the ATMega is not allowed to go all the way up to VCC of 5V, then the A/D conversion will never return a full-scale reading. But if (when) the thermistor leads break, the firmware requires a full-scale reading in order to detect the break and shut down the associated heater. At room temperature of 20C the input voltage is 4.819V, and increasing slowly; 0C yields 4.930. The open-circuit voltage should always, unconditionally, be allowed to go above 4.930 regardless of zener tolerance and other part tolerances.

This concern is not theoretical. A user of a Prusa MK2S has actually experienced a thermistor break that did not trigger a firmware shutdown and did result in an extremely hot extruder.

Failure to trigger the firmware shutdown is a fire hazard.

However, that GitHub site seems to have extremely little traffic, so I don't know if the issue will be noticed.

Publié : 21/01/2019 9:06 pm
Partager :