1. Do native speakers say “open up (an application” as in “just open up YouTube on your phone”? 2. Does “open up YouTube back” sound natural where “back” means “a return to an earlier or normal condition”?

"Open YouTube back up" sounds more natural, in my (native) opinion.

You would not say "open up YouTube back", but you could say "go back \[over\] to YouTube", or "open YouTube back up"

Go on YouTube again.

yes we’d say the first one, with “up” being optional. i wouldn’t say “open up youtube back,” but might say “reopen youtube” “go back to youtube” or “open youtube again” if they opened youtube before, closed it, and now should go back.

In 1) no up. There's nothing wrong with it, I just don't say it. In 2) "open YouTube back up" is what you want.

Also an option: “go back to YouTube”

Open YouTube or Open up YouTube both work I would say "Open back up Youtube" for 2