I think I might have figured it out and I'm surprised I didn't notice it before. My cooling duct fan wasn't running at all and when I looked at how S3D was commanding it, it wasn't sending the fan index so it was defaulting to the extruder fan (based on the notes for M106). I fixed that to explicitly call P0 on the M106 command it the blower works now. Since then I haven't seen any imperfections.
I'm guessing between the extruder calibration and the fan that was probably my problems. Thank you @Phaedrux for the help.
