olives are common trees of the Mediterranean climate.
In theses area the weather is pretty warm to very hot most of the year however it could go down to freezing T in Winter with sometimes snow (but not for very long)
According to the USA climate zone you are in a 9A area, it should be fine for your olives outside with a minimum Winter protection.