Your correct. We have several abandoned railroads in North Central Ok. Most have been returned to the landowners on either side, but some are still restricted.I may be incorrect in this, but I think that railroad rights-of-way are owned by the railroad. I know of a rail line up in the Panhandle that was abandoned decades ago, but most of that right-of-way is still not accessible or usable by the landowners across which land the R-O-W crosses.
The reason I make this statement is because I've heard of photographers that wanted to get onto railroad rights-of-way and photograph passing trains. They've been run off when the railroad reported them as trespassers.