This joy of an issue starts of with these exact words: Error 789: The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer. Do not mistake it with VPN Error 807 which is a completely different pair of shoes.
In the simplest of terms, the VPN Error 789 happens when your own machine can’t establish a L2TP connection with another. Though the error dialogue might sound confusing with talks of security layers and processing errors, the fixes are actually pretty straightforward and resemble what you might do if you stumbled upon or any of the most common nuisances.
Fix #1: Try some resets to do away with VPN Error 789
We use Ivacy VPN in our office, and though the software is as solid as they come, plenty of co-workers will get the Error 789 and believe it to be a tough-to-solve technical issue. In truth, just like with Error 807 or any of the most common issues, the Error 789 can often be solved by a few unplugs and replugs, be it on the software or hardware side.
Since hardware is the easiest, we suggest you start there. Unplug both your modem and router from your own machine and then unplug both devices from their power source. Wait around 30 seconds and then plug them back in. If the devices are separate, it can be a good idea to add a 30 seconds between plugging each of them back in for optimal results. This will set the network back to default and oftentimes solve Error 789.
If it persists, navigate to the Device Manager in your Windows machine, get to your network adapter, uninstall it with the right click menu and then restart the computer. This will reinstall the network adapter and, more likely than not, fix Error 789.
Fix #2: Verify the certificates and enable IPSec
If it persists, don’t panic: it doesn’t necessarily mean you have to reach out to tech support yet. First, make sure that both the device you are connecting from and the server you are trying to reach out to have the same security certificates. A different security certificate on both machines can essentially act as a firewall stopping the connection.
Another thing to do is to open the start menu, navigate to Run, type services.msc and press Enter. From here, find the IPSec section by scrolling to the IKE and AuthIP IPSec Keying Modules. An offline or malfunctioning IPSec Policy Agent can often cause Error 789, so either enable it if it’s disabled or restart it with the right click if it’s enabled. Set it to automatic start-up through the same menu and either restart just your VPN or the machine as well.
3#: Ensure open connectivity on both ends
By now, you’ll have run out of most easy solutions for Error 789, but let’s try a few things more first. The server your VPN is connecting to should be highlighted as green or alternatively have a full bar, which translates to the best connectivity. If it isn’t, use the drop-down menu to find a closer one.
Likewise, try to connect to another network to see if the ports on your machine are open at all. This can be another computer in your Home or Office network or anything else that you find secure. If the Error 789 persists, you are probably dealing with closed ports on your machine that need to be opened or a firewall that’s blocking most or all traffic.
The last point also signals the end of easy solutions for fixing this error. By now, you’ll most likely need assistance from a tech specialist, be it to open up the machine without causing security issues or to fin whichever deeper-rooted issue is causing this.