Fortunately it was not poisonous. It was a jujube. I figured it was either that or an olive and the tree didn't look rough enough to be an olive. So I ate it and lo, it tasted like a dry, not so sweet, apple.
(for reference here is the jujube I ate and a picture of an olive. the pits look similar too.)
One of the local nurseries said that they grow very well in the area even during a drought. It won't be until a few more weeks until they are ripe though. At that time I will go back and get some more and see if I can't make jam out of it. They are supposed to be really good for the stomach.