According to Bible History. com, the phrase "one jot or tittle" in Matthew 5:18 refers to a tiny letter and an even tinier "decorative mark":
The "Jot" is the Hebrew word "Yodh" which is the 10th letter of the Hebrew alphabet. It is also the smallest letter. Its European or English equivalent is the letter "Y" as in the English term Yahweh or in Hebrew YHVH since there were no vowels used in the ancient script.
The word "jot" itself is an English transliteration of "iota" which is the 9th letter of the Greek alphabet. "Iota," in turn, is the nearest Greek equivalent for the Hebrew yodh.
The "tittle" is the small decorative spur or point on the upper edge of the yodh. If you can imagine a tiny letter with a slightly visible decorative mark. Tittle is used by Greek grammarians of the accents and diacritical points.
