@Bustable-0 on my delta, the rare layer shifts I get are caused by blobs forming on the print after a while, which the nozzle collides with. To counter this I've done a few things:
Use Z hop for travel moves (I use firmware retraction and include Z hop in the M207 settings)
Reduce extrusion factor to 95% when printing with filaments that are more prone to cause blobbing
Increase motor current
Reduce speed when using these problem filaments