Yeah. Unless the coral tosses you back there's no way to get back. But if it helps, time doesn't pass back in your world. If you do go back you'll get right back to where you came from.