Four good sized self threading screws should be sufficient to fix the block. Then the swivel is also fixed with four self threaders. Particularly in hardwood it is important to drill holes with diameters equal to the core diameter of the self threaders. I would use marine grade ply rather then wood for ply cannot split. You are after all, putting eight fair sized screws in a tiny block. Before drilling, make sure the swivel screws will not be obstructed by the bottom screws. (Don’t ask 😳 ).
Using penny washers will help to spread the load over a larger area and are a good idea indeed.
Some dip the screws in varnish or epoxy before final fitting. It locks the screws in place so they won’t wobble out on a country road. It also seals the screw holes and toughens the (wooden) threads in the holes.