NH 644 Top Idler Loose

Finally got a chance to work on the 644 from my neighbor, and found that the top sledge idler is loose.  The bearing seems to be ok, but the whole idler assembly is loose in the sledge frame.  I followed mike10's guide and removed the idler hub, and can see where the recess in the sledge frame sidewall is now worn as is the shim washer between the idler and the sledge frame.  I'm not going to use this baler much at all and don't really want to remove the sledge, so I'm hoping for a simpler fix.  It is hard for me to tell what bottoms out on what, so I'm wondering if I should try and place a shim between the idler support and the outside of the sledge frame, or if I can just clean everything up well and by tightening the bolt it will drawn everything tight again?

NH 644 Top Idler Loose