Spam transactions are transactions which create undesirable extra load on the network due to not following Bitcoin best practices, either maliciously or out of ignorance. Even the very first versions of the Bitcoin software had some spam filtering rules, and these have been improved over time, but it is probably impossible to automatically filter all spam.
Transaction spam is always more costly to the spammer than not spamming. Someone who is intentionally spamming the network will have to burn BTC constantly to do so, and so will eventually be forced to stop.
The most common types of spam are:
Sending many without sendmany
If you want to send payments to many people at around the same time, then you should not do tons of individual transactions one after the other. Doing so is grossly wasteful of network resources because each transactions produces a extraneous change output. Instead, all of the outgoing transactions should be bundled into one transaction. The bitcoind JSON-RPC command for doing this is sendmany.
Signalling and data transmission
Bitcoin is for storing & transferring value between people, and for limited hash-based timestamping. Transactions should never be used as a signal (such as a win/loss signal in a gambling game) or to send data directly.
Attackers will sometimes waste network resources by sending BTC between their own addresses uselessly.
A simplistic attack that people often try is to send tens of thousands of zero- or very-low-fee transactions at once. This makes the "mempool" number shown by some websites go up massively, which sometimes causes people to panic, but in reality it is only a slight drain on the network's bandwidth resources, and hardly a problem at all. Transactions with very low fees will never be confirmed, and will typically be expired from nodes' mempools after a few days.