Thank you for the reply.
I think I have figured it out, the driver is fine (I think) as I was able to change the direction finally.
This is what happened:
I have moved the extruder from E0 to E1 motor output.
I have set the E0 drive to use 4(E1) Motor driver
The issue is that configurator.reprapfirmware.org generated config for E0 instead of E1, I do not know if that is a bug or a feature, but setting manually M569 P4 S0 did the trick.
Thank you everyone for you support.