Should I delete myetherwallet private key from my computer and only store it on USB's?

As the title says - Should I delete myetherwallet private key from my computer and only store it on USB's

Or is there nothing to worry about and have it on my computer?