Setting up an IPsec VPN between a Fortigate firewall and a Mikrotik router can seem daunting, but with a clear guide, it becomes a manageable task. This article walks you through the process, ensuring a secure and stable connection between your networks. So, let's dive into the configurations needed on both ends!

Understanding IPsec VPN

Before we jump into the configurations, let's briefly understand what an IPsec VPN is. IPsec (Internet Protocol Security) is a suite of protocols used to secure internet protocol (IP) communications by authenticating and encrypting each IP packet of a communication session. IPsec includes protocols for establishing mutual authentication between agents at the beginning of the session and negotiation of cryptographic keys to use during the session.

The primary goal of IPsec is to provide confidentiality, integrity, and authentication. It operates in two main modes: Tunnel mode and Transport mode. In Tunnel mode, the entire IP packet is encrypted and encapsulated within a new IP packet, providing a secure tunnel between networks, which is typically used for VPNs. In Transport mode, only the payload of the IP packet is encrypted, which is more suitable for host-to-host communication where the endpoints need to know each other's IP addresses.

When configuring an IPsec VPN, there are two phases: Phase 1 and Phase 2. Phase 1 establishes a secure channel for further communication and involves negotiating security parameters like encryption and hashing algorithms. Phase 2 then uses this secure channel to negotiate the specific parameters for data transfer. Getting these phases configured correctly on both the Fortigate and Mikrotik devices is crucial for a successful VPN connection. Proper configuration ensures that data transmitted between the two networks remains confidential and secure from potential eavesdropping or tampering.

Configuring IPsec involves several key steps, starting with defining the IPsec policies, configuring the cryptographic settings, and setting up the authentication methods. Each device must be configured to understand and trust the other, which requires careful attention to detail. The common pitfalls in setting up IPsec VPNs often involve mismatched encryption settings, incorrect IP address configurations, or improperly configured firewall rules. Addressing these common issues through methodical configuration and testing ensures a stable and reliable VPN connection. Understanding the underlying concepts and correctly implementing the configuration steps is essential for creating a secure and efficient network connection.

Fortigate Configuration

First, let's configure the Fortigate side. We'll start by setting up the IPsec Phase 1 settings, followed by the Phase 2 settings, and finally, the firewall policies to allow traffic through the VPN tunnel.

Phase 1 Configuration

To configure Phase 1 on your Fortigate, navigate to VPN > IPsec Tunnels and create a new custom tunnel. Here's what you need to configure:

• Name: Give your VPN tunnel a descriptive name (e.g., "Mikrotik-VPN").
• Interface: Choose the external interface that will be used for the VPN.
• Remote Gateway: Select "Static IP Address" and enter the public IP address of your Mikrotik router.
• Authentication Method: Choose "Pre-shared Key" and enter a strong, complex key. Make sure to use the exact same key on the Mikrotik device.
• IKE Version: Select IKEv2.
• Encryption: Choose AES256-SHA256 (or another strong combination).
• DH Group: Select Group 14 (or another strong group).
• Key Lifetime: Set a reasonable key lifetime (e.g., 28800 seconds).
• NAT Traversal: Enable this option to handle NAT scenarios.
• Dead Peer Detection (DPD): Enable DPD to detect and handle inactive peers. Configure the DPD retry interval and threshold appropriately.

Phase 1 is crucial because it establishes the initial secure connection between the Fortigate and Mikrotik devices. The settings you choose here define the security parameters for the control channel, which protects all subsequent communication. Using strong encryption and hashing algorithms ensures that this initial connection is resistant to eavesdropping and tampering. One common mistake is using weak or default encryption settings, which can expose the VPN to security vulnerabilities. Always opt for the strongest algorithms supported by both devices to maximize security. Moreover, the pre-shared key must be complex and kept secret. A weak pre-shared key can be easily cracked, compromising the entire VPN. Regularly changing the pre-shared key can also enhance security.

Also, ensure that the IKE version is compatible with the Mikrotik. IKEv2 is generally preferred for its improved security and performance. The DH Group determines the strength of the key exchange, so selecting a strong group is essential. Group 14 is a good choice, but higher groups provide even greater security at the cost of increased computational overhead. Configuring NAT traversal is particularly important when either the Fortigate or Mikrotik device is behind a NAT gateway. Enabling NAT traversal ensures that the VPN connection can be established and maintained even when NAT is present. Finally, Dead Peer Detection (DPD) is vital for maintaining the VPN's reliability. DPD periodically checks the availability of the remote peer and can automatically re-establish the connection if the peer becomes unreachable. This feature helps prevent prolonged outages and ensures that the VPN remains active.

Phase 2 Configuration

Next, configure Phase 2. In the same IPsec Tunnel configuration page, navigate to Phase 2 Selectors. Here’s what you need:

• Name: Give this phase a descriptive name (e.g., "Mikrotik-Phase2").
• Protocol: Choose ESP.
• Encryption: Choose AES256-SHA256 (must match Phase 1).
• Perfect Forward Secrecy (PFS): Enable and select Group 14 (must match Phase 1).
• Auto-negotiate: Enable this option.
• Source Address: Define the local network behind the Fortigate (e.g., 192.168.1.0/24).
• Destination Address: Define the remote network behind the Mikrotik (e.g., 192.168.2.0/24).

Phase 2 is where you define the security parameters for the data channel, which is used to transmit actual network traffic. Matching the encryption settings between Phase 1 and Phase 2 is absolutely critical. If the encryption algorithms, hashing algorithms, or DH groups don't match, the VPN will fail to establish. AES256-SHA256 is a strong and commonly used combination, but you can choose other options as long as they are supported by both devices. Perfect Forward Secrecy (PFS) adds an extra layer of security by ensuring that each session key is unique and not derived from previous keys. This prevents an attacker who compromises a session key from decrypting past sessions.

Enabling auto-negotiate allows the Fortigate to automatically negotiate the security parameters with the Mikrotik, simplifying the configuration process. However, it's still important to ensure that the settings are compatible on both sides. Defining the source and destination addresses correctly is essential for routing traffic through the VPN tunnel. The source address should be the local network behind the Fortigate, and the destination address should be the remote network behind the Mikrotik. These addresses tell the Fortigate which traffic should be encrypted and sent through the VPN. Incorrectly configured source and destination addresses are a common cause of VPN connectivity issues. It's also important to consider overlapping IP address ranges. If both networks use the same IP address range, you'll need to implement NAT or change the IP addressing scheme to avoid conflicts.

Firewall Policies

Finally, create firewall policies to allow traffic to pass through the VPN tunnel. You'll need two policies:

• Policy 1: From the internal network to the VPN tunnel, allowing traffic to the Mikrotik network.
• Policy 2: From the VPN tunnel to the internal network, allowing traffic from the Mikrotik network.

Make sure to enable logging on these policies to monitor traffic and troubleshoot any issues. Creating the firewall policies is the final step in configuring the Fortigate. These policies define which traffic is allowed to pass through the VPN tunnel. Without these policies, traffic will be blocked, and the VPN will be ineffective. The first policy should allow traffic from the internal network behind the Fortigate to the VPN tunnel, with the destination being the network behind the Mikrotik. The second policy should allow traffic from the VPN tunnel to the internal network behind the Fortigate, with the source being the network behind the Mikrotik.

When creating these policies, it's important to specify the correct source and destination interfaces. The source interface for the first policy should be the internal interface, and the destination interface should be the VPN tunnel interface. Conversely, the source interface for the second policy should be the VPN tunnel interface, and the destination interface should be the internal interface. Enable logging on these policies to help troubleshoot any connectivity issues. Logging provides valuable information about the traffic passing through the VPN, including source and destination IP addresses, ports, and protocols. This information can be invaluable for diagnosing problems such as blocked traffic, incorrect routing, or authentication failures. Regularly reviewing the logs can also help identify potential security threats.

Mikrotik Configuration

Now, let's move on to the Mikrotik configuration. Similar to the Fortigate, we'll configure the IPsec policy, proposal, and identity settings.

IPsec Policy

On your Mikrotik router, open Winbox and navigate to IP > IPsec > Policies. Add a new policy with the following settings:

• Enabled: Check this box to enable the policy.
• Comment: Add a descriptive comment (e.g., "Fortigate-VPN").
• Src. Address: The local network behind the Mikrotik (e.g., 192.168.2.0/24).
• Dst. Address: The remote network behind the Fortigate (e.g., 192.168.1.0/24).
• IPsec Protocol: ESP.
• Tunnel: Check this box.
• Level: Require.
• Auth. Algorithms: sha256.
• Encr. Algorithms: aes-256.
• PFS Group: modp1024 (or the equivalent of DH Group 14).
• Lifetime: 8h (or the equivalent of 28800 seconds).
• Secret: Enter the same pre-shared key you used on the Fortigate.

The IPsec policy on the Mikrotik defines the parameters for securing the VPN connection. Enabling the policy is the first step, and adding a descriptive comment helps you identify it later. The source and destination addresses must match the networks you want to connect through the VPN. The source address should be the local network behind the Mikrotik, and the destination address should be the remote network behind the Fortigate. These addresses tell the Mikrotik which traffic should be encrypted and sent through the VPN tunnel. Setting the IPsec protocol to ESP (Encapsulating Security Payload) specifies that ESP will be used to provide confidentiality, integrity, and authentication for the data packets.

Checking the tunnel box indicates that this policy is for a tunnel-mode VPN, where the entire IP packet is encapsulated and encrypted. Setting the level to require ensures that IPsec is mandatory for traffic matching this policy. The authentication and encryption algorithms must match the settings on the Fortigate. Using sha256 for authentication and aes-256 for encryption provides strong security. The PFS (Perfect Forward Secrecy) group should also match the DH Group configured on the Fortigate. Modp1024 is equivalent to DH Group 14. The lifetime setting specifies how long the IPsec security association is valid before it needs to be renegotiated. Setting the lifetime to 8 hours (28800 seconds) is a common practice. Finally, the secret is the pre-shared key that must match the key configured on the Fortigate. This key is used to authenticate the VPN connection. A mismatch in the pre-shared key is one of the most common reasons for VPN connection failures.

IPsec Proposal

Next, go to IP > IPsec > Proposals and create a new proposal with these settings:

• Name: Give it a descriptive name (e.g., "Fortigate-Proposal").
• Auth. Algorithms: sha256.
• Encr. Algorithms: aes-256.
• Lifetime: 8h.

The IPsec proposal defines the cryptographic algorithms and lifetime settings for the VPN connection. Giving the proposal a descriptive name helps you identify it later. The authentication algorithm should match the setting in the IPsec policy, which is sha256 in this case. The encryption algorithm should also match the setting in the IPsec policy, which is aes-256. The lifetime setting specifies how long the security association is valid before it needs to be renegotiated. Setting the lifetime to 8 hours is consistent with the policy settings. It is crucial to ensure that the settings in the proposal match the settings in the policy and the Fortigate configuration. Mismatched settings are a common cause of VPN connection failures.

IPsec Identity

Finally, configure the IPsec identity under IP > IPsec > Identities:

• Policy: Select the policy you created earlier (e.g., "Fortigate-VPN").
• Auth. Method: pre-shared-key.
• Secret: Enter the same pre-shared key used on the Fortigate.
• My ID: The public IP address of your Mikrotik router.
• Remote ID: The public IP address of your Fortigate.

The IPsec identity defines how the Mikrotik router identifies itself to the Fortigate and authenticates the VPN connection. Selecting the correct policy associates this identity with the VPN policy you created earlier. Setting the authentication method to pre-shared-key specifies that a pre-shared key will be used for authentication. The secret is the pre-shared key, which must match the key configured on both the Fortigate and in the IPsec policy. The My ID is the public IP address of the Mikrotik router, and the Remote ID is the public IP address of the Fortigate. These IDs are used to identify the endpoints of the VPN connection. Incorrectly configured IDs can prevent the VPN from establishing correctly. It's also important to ensure that the Mikrotik router can resolve the public IP address of the Fortigate. If DNS resolution is required, make sure that the Mikrotik router is configured with a valid DNS server.

Testing the VPN Connection

After configuring both the Fortigate and Mikrotik devices, it's time to test the VPN connection. Here’s how:

1. Check IPsec Status: On both devices, check the IPsec status to see if the tunnel is established. Look for active security associations (SAs).
2. Ping Test: Ping a device on the remote network from a device on the local network. For example, ping a server on the Mikrotik network from a computer on the Fortigate network.
3. Traceroute: Use traceroute to verify that traffic is indeed passing through the VPN tunnel.
4. Firewall Logs: Check the firewall logs on both devices to ensure that traffic is being allowed through the VPN policies.

Testing the VPN connection is crucial to ensure that the configuration is working correctly and that traffic is flowing as expected. Checking the IPsec status on both the Fortigate and Mikrotik devices is the first step. Look for active security associations (SAs), which indicate that the tunnel is established and that encryption keys have been negotiated. If the SAs are not active, there may be a problem with the configuration, such as mismatched settings or authentication failures.

Pinging a device on the remote network from a device on the local network is a simple way to test basic connectivity. If the ping is successful, it indicates that traffic is able to pass through the VPN tunnel. If the ping fails, there may be a problem with routing, firewall policies, or IP address configuration. Using traceroute can help you verify that traffic is indeed passing through the VPN tunnel. Traceroute shows the path that traffic takes from source to destination, allowing you to confirm that the traffic is being routed through the VPN tunnel interface. Checking the firewall logs on both devices is essential for ensuring that traffic is being allowed through the VPN policies. The logs can provide valuable information about blocked traffic, incorrect routing, or authentication failures.

Troubleshooting Tips

If you encounter issues, here are some troubleshooting tips:

• Mismatched Settings: Double-check that all settings (encryption, hashing, DH group, pre-shared key) match exactly on both devices.
• Firewall Rules: Ensure that your firewall rules allow traffic to pass through the VPN tunnel.
• NAT Issues: If either device is behind NAT, make sure NAT traversal is enabled and configured correctly.
• Logs: Check the logs on both devices for error messages or clues about what might be wrong.
• MTU Issues: Adjust the MTU size if you experience packet fragmentation issues.

Troubleshooting VPN connections can be challenging, but following a systematic approach can help you identify and resolve the issues. One of the most common causes of VPN connection failures is mismatched settings. Double-check that all the settings, including encryption algorithms, hashing algorithms, DH groups, and the pre-shared key, match exactly on both the Fortigate and Mikrotik devices. Even a small difference in these settings can prevent the VPN from establishing correctly.

Ensure that your firewall rules allow traffic to pass through the VPN tunnel. The firewall rules should allow traffic from the internal network to the VPN tunnel and from the VPN tunnel to the internal network. If either device is behind NAT, make sure that NAT traversal is enabled and configured correctly. NAT can interfere with VPN connections by changing the source IP addresses of the packets. Enabling NAT traversal allows the VPN to work correctly even when NAT is present. Checking the logs on both devices is essential for troubleshooting VPN issues. The logs can provide valuable information about error messages, authentication failures, and blocked traffic.

If you experience packet fragmentation issues, try adjusting the MTU (Maximum Transmission Unit) size. Packet fragmentation can occur when packets are too large to be transmitted over the network. Reducing the MTU size can help prevent fragmentation and improve the reliability of the VPN connection. Start by reducing the MTU size on the Fortigate and Mikrotik devices and then test the VPN connection to see if the issue is resolved.

Conclusion

Setting up an IPsec VPN between a Fortigate and a Mikrotik device requires careful configuration, but following this guide should make the process smoother. Ensure that all settings match on both sides, and don't forget to create the necessary firewall policies. With a properly configured VPN, you can securely connect your networks and protect your data.

By following this comprehensive guide, you can successfully establish a secure and reliable IPsec VPN connection between your Fortigate firewall and Mikrotik router. Remember to double-check all settings, pay close attention to firewall policies, and use the troubleshooting tips to resolve any issues that may arise. With a properly configured VPN, you can securely connect your networks and protect your data, ensuring that your business operations remain secure and efficient.