Check upper/lower bounds on various RTS flags (#3633)

Check upper/lower bounds on various RTS flags (#3633)

Also, make K mean 1024 rather than 1000, in RTS flags (similarly for M
and G).  The main reason I want to change it is that otherwise this
might be confusing:

exp3_8: error in RTS option -H4k: size outside allowed range (4096 - 18446744073709551615)

And I think the original reason for using 1000 instead of 1024,
worries about direct-mapped caches, is not an issue in this context
(even if you can find a direct-mapped cache these days).