There is a way to make Powershell act like a normal, readline-like terminal! ♥‿♥

Set-PSReadLineOption -EditMode Emacs

To make this change permanent, add the above line to your $profile which basically seems to be an analogue to ~/.bash_profile or .zshrc:

notepad $profile

Just like with any customization I will miss this dearly on all remote systems which I haven’t added this change to…oh well ¯\(ツ)/¯