Firstly my sympathies for your situation. I can't possibly guide you on an arrangement, but I will put my two cents in on behalf of your youngest. I'm from a "broken" home as they call it, with older siblings, and younger half-siblings.
While I agree that keeping a 3 year old on the sidelines of Saturday sporting events for older siblings is not ideal, its still important that they are treated the same at all times. Separating them for any period of time might be harmful to your three year old - it will certainly be confusing. They are not used to being on their own at that age; they were brought home to a busy household.
My advice is to find a playground near to where the sports activities are - if the 3 year old is restless or bored one week, pop to the playground for 45 minutes the next. It allows them one on one time with their dad, while still having fun. The older siblings will be playing so won't really notice you've slipped away, and if they do, they are old enough to understand you have to entertain the youngest.
If that's something you're not comfortable doing (I'm not sure of the set up surrounding these sport activities - is it safe for you not to be present, etc.), you'll need to get creative on how to keep the 3 year old entertained while on the sidelines. Books, games, etc - trial and error is the only way about it. I remember lots of occasions of being on the sidelines for my siblings, and my parents would come up with inventive ways to make the time pass. We used to have a competition to see who could hold on to the top of a goal post the longest while my dad tickled us - not to far a drop and he would catch us every time
- funny what you remember but its easily one of my best memories.
The only thing I can be sure of for you is that time with their father is essential, so its good to see you pushing for it. In time I hope things become easier for you all.