Tom, there were a number of problems with the old SA, for instance the constant pressure from the SWP in particular to minimise ongoing organisation which sought to put down roots in workplaces and communities in favour of occasional election-dashes; and its sectarian attitude towards the struggle in the Labour Party (though this was nowhere near as serious as in the SSP). Not to mention its bad politics on eg imperialism and the Middle East.
But such problems paled in comparison to the fact that, yes, the two largest groups did behave badly.
1. The SP left because the SWP were (at least at that time) bigger than them and they calculated they could do better outside.
2. The SWP deliberately downgraded the Alliance to promote themselves in the Stop the War movement and then destroyed the it in order to pursue the Respect project.
We should not let this history be rewritten! Not by the SP and SWP, and not by 'left-wing' critics who argue the project was doomed from the start either (not saying you necessarily fall into this category).