These are known as Suckers.
It's best to remove them if you don't need them in your design, because they can siphon energy away from the main tree.
That said, Elms are very vigorous trees so this is not that big of an issue on these species.
Looking at your tree, you may actually want more branching lower down, as there is a long section without any.
You could consider letting these suckers grow in order to thread graft them in the winter.