Not sure if I did something wrong...or guys are just like this sometimes [21F]

I've gone on a few dates with a guy I met on OkCupid. On the fourth date we hooked up, and we agreed on doing the whole casual thing, which I pretty much assume is the nice way of saying friends with benefits. We were both pretty clear that we didn't want anything serious.

We made solid plans for Saturday afternoon, and he just never texted me back. Initially I texted him telling him to just let me know when he was on his way (about 15 minutes before we said he'd come over, 2pm), but after a couple hours of not hearing I texted him again asking if his interview (where he was coming from) was running late. After STILL not hearing back, I texted him saying that if he didn't want to hang out that that was fine, but I would've appreciated him letting me know.

I don't have any real feelings for him, but it really irks me when people stand me up. Was I wrong to get annoyed with him since it's supposed to be casual? I still really want to hook up with him because I enjoy it, but I'm afraid I scared him off by texting him too much.