Ouch! I can imagine that’s an absolute pain.
I have an old spare mobile which I’ve recently started throwing in the bottom of my backpack when I travel, because I realised that (in addition to basically everything I need to survive in a foreign country being on my phone - boarding passes, maps, train/bus times, tickets, emergency contacts, etc) all of my online accounts are almost too secure.
I use a password manager with randomised passwords and have two-factor authentication set up for my Gmail, Facebook etc. so if my phone was lost , even if I could get to an internet cafe, I wouldn’t know the password and wouldn’t be able to authenticate to log into my accounts! Old phone is now set up with key apps/data preinstalled and authorised - Gmail, Facebook, Google Authenticator, contacts. Another good tip is to print off your 2FA recovery codes or give them to a trusted friend/family member, so that if you do need to bypass authentication in an emergency, you can.
It’s still not perfect because, for example, WhatsApp can only be installed on one phone at a time (and I wouldn’t be able to move it to the backup phone without the lost/stolen SIM card) and as you say many banking apps would hit the same roadblock - but I at least feel I have a much better level of preparedness should I lose my phone, especially if I’m travelling in a developing country where it can be more difficult to find help.