I would suggest a 1/4" palm router like a DeWalt DWP 611 or Bosch Colt. That is about the bare minimum. They are basically turbo charged laminate trimmers.
Speed does matter. You want to cut fast enough to maintain sufficient chip load, for the depth, and RPM. This cools the bit, and keeps it from burning up. You will also create more chips, less dust. The bit should be cool enough to touch (with the router unplugged) when your done cutting. Typical handheld wood routers have universal brushed motors so our minimum spindle RPM is limited due to poor speed control/torque at the low end of the speed range. So chip load is going to be controlled by feed rate.
So with a 1/4" end mill in wood you would probably want to be able to cut at at least 1.5 IPS or 40mm/s. More ideal would be 3-5 IPS, or 80-120MM/S, while climb cutting at 1/4"-3/8" depth. I own a machine that does that but it also cost $15,000 and has a 3.5" hp router motor, so obviously thats out of the question here.
Just an example of how CNC routers quickly get exponentially heavier and more expensive.