@stellator Normally, the 'X' stepper is back right on a CoreXY, and the 'Y' stepper is back left. I think you have these the wrong way around.
I think it should be:
Motor XY - 'Y' motor - 1 goes to Duet2Wifi E0-Stepper (driver 3)
Motor XY - 'Y' motor - 2 goes to Duet2Wifi Y-Stepper (driver 1)
Motor XY 3 - 'X' motor - goes to Duet2Wifi E1-Stepper (driver 4)
Motor XY 4 - 'X' motor - goes to Duet2Wifi X-Stepper (driver 0)
So your M584 should be
M584 X0.0:0.4 Y0.1:0.3 Z0.5:0.6:0.7 ; set axis mapping
Then check the direction of movement for the motors, using https://docs.duet3d.com/User_manual/Machine_configuration/Configuration_coreXY#testing-motor-movement, reversing the pairs of motors' direction as necessary.
Edit: I don't think it's 4WD problem, so long as the pairs of motors are working in the same direction as each other. Though it may be easier to get it working in 2WD first!
Ian