Is it appropriate to use the term "Semite" for Muslims?

I am an American Jew who has been reaching out to Muslim friends and neighbors. To stress the connection I feel, I sometimes describe myself as a fellow (or sister) Semite or say that I consider anti-Semitism to be just as unacceptable whether directed against Jews or Muslims. Is this terminology appropriate, or does it show me to be ignorant or insensitive? I know that not all Muslims (or Jews) are of Middle Eastern descent, although both Islam and Judaism are considered Semitic religions.