Well, those leaves aren't going to do you any good. You can take them off if you want. It would probably be easier to clean them up all at once.
How cold did it get? How dry did it get? I've heard of them taking a light frost in the ground, but I always bring mine inside at 50F with my other tropicals. I've also seen mine drop all of its leaves once under stress. It took far longer than I would have expected to grow them back. These may look like brazilian rain trees, but they grow at 1/4 speed.