I believe that trees grow far better outside where they evolved. I'm a long way from California but Chinese elm is pretty hardy. I think it should be fine outside and will probably do far better than indoors. Way back when I moved some of my subtropicals to different location for winter and summer I thought I saw delayed growth after each move. Now I keep all trees in the same location all year and they seem to do better overall. I know some of you must move trees for winter. That can't b helped if they are to survive but in warmer climates many trees can overwinter outside and will be better off for it. Check with other locals for actual needs.
Soil mix and/or overwatering could be a cause. Trees can grow in almost any mix IF you adjust watering to suit but most of us do not understand water needs properly so trees can be too wet or too dry. Modern, open bonsai mix allows much greater margin for error and minimises risk of root rot.
I can't see or feel the tree so cannot make informed judgement. If you believe the mix is a problem I would repot. Bad soil is likely to cause death. Repotting may possibly cause death. Being outside in sun and wind should reduce root problems as well. Maybe you are sentimental about this tree but it is really only starter material rather than valuable bonsai. Nice to keep it alive but if not, no real loss.