Electric issues on Yamaha 200hp

by Jerry
(Charleston sc)

It is a Yamaha saltwater series , 200 hp it is on a Grady white 1995" 209 escape , the problem is as follows.

I was fishing and turned off the engine,, when I went to restart it , there was no power at all in the boat, I fiddled around with the battery connector , for a while and "poof" it started . So going on that information I replaced the battery connector switch.

I was on the water today and a similar thing happened , except this time there was power to the boat , but the motor would not turn over and it was not seeming to be connected to any electrical source.
Finally after fiddleing around with it it started back up.

I am wondering if it's the starter or solonoid, I can't figure out what would cause it randomly to not have power to the starter. When I turn the key , nothing happens not a sound , and yet I have electricity, I can raise or lower the moter, or use the radio and gps etc
Any ideas?