probably, defragmentation is best performed on a routine basis but not after a specific action. Unless, as stated, you're using a billion really small files. Have you considered getting an SSD? It may be faster and more reliable for such purposes, i.e. that type of data storage, mesh optimization, etc.
Leaving things "fragmented" isn't particularly bad, it's just cluttered. If there are frequently-accessed files that end up split far across the drive platter, then it may cause more read-time than necessary, but ultimately it's not going to do a ton. It basically depends on the type of files, i.e. quantity vs. size, in the filesystem as well. Other people here could probably explain it more thoroughly or accurately, I rarely defrag as I think ext4 does something like that automatically. And I never remember to on Windows.