Depending on crew experience (yours did not seem very keen) sailing in force 5 with full sails (main and genoa) should be alright. Sailing towards windward may be slow in any event, due to having to allow the sails to spill wind in order to stay upright (a heavy crew leaning out will always help). As you were racing, then using the jib from the outset would be the better option in your situation.
Do not sail to windward with a reefed main with genoa, as this is an unbalanced sail configuration, which may account for your difficulties. If you need to sail with a reefed mainsail, then use the jib or furling jib / genoa.