I tried a few out-dated methods using xmodmap{: .language-} and getting ~/.xinitrc{: .language-} to run on startup but even if .xinitrc{: .language-} is executed, the key is still not mapped.
At the end I stumbled upon this answer when I was about to give up, and it worked.
I needed to change /usr/share/X11/xkb/symbols/pc{: .language-} with as root to modify the behavior of Insert key to Delete. Saved the file, reboot and it worked like a charm.
- key <INS> { [ Insert ] };
+ key <INS> { [ Delete ] };