Infighting and rifts herald start of Your Party conference