Bit late to the party here but here are a couple other factors that can affect the printed part sizing.
Backlash in axis. Backlash due to loose belts, excessively compliant mechanics, etc can present as oval holes if one axis suffers from significant backlash and the other does not. This is pretty easy to check. If you have a dial indicator you can use that to take two measurements at the same axis position, approaching the measurement point from opposite directions for each reading. If you don't have a dial you can print a large and small part to see if the error is an absolute value or scales with part size. Absolute error would indicate something mechanical like backlash rather than a material setting / steps/mm issue.
Another easy backlash check is to look at the first layer of the part. IF the diagonal fill lines appear to be in sets of two (two lines close together, a small gap, then another two lines close together) this indicates excessive backlash in an axis.
Material shrink. Plastics will always shrink a little bit while cooling. For a PLA I print with a lot I see shrink rates around 0.5%, around 0.1mm over 20mm (for my printer and material). Extrusion factor can't correct for this error and it's also not really a positioning error as my axes move very accurately. It's possible to scale x/y steps but I prefer to keep the positioning steps correct and scale up my part in the slicer in XY by my shrink rate. Z tends to be pretty stable and not need scaling for shrink in my experience.