The LAP (Linear Assignment Problem) sometimes can make very poor choices because the marginals have been already flattened (because of the epsilon relaxation) and it's impossible to extract valuable information. We need either a better algorithm or a better source of information other than the flattened marginals.
It might be possible to use directly the score matrix or run again the NAP problem only on a subset of the nodes.