How to Design a Secure Architecture for Distributed System

Securing distributed systems is a complex challenge due to the large-scale and diverse components involved. With multiple services interacting over potentially insecure networks, the risk of unauthorized access and data leakage increases significantly. This article explores a practical approach to securing distributed systems using an open source project that demonstrates how to integrate several security mechanisms and technologies to address common security challenges, such as authentication, authorization, and secure communication.

Understanding security challenges in distributed systems

Distributed systems involve multiple services or microservices that must communicate securely over the network. The main security challenges in this type of :

1. Secure communication : Ensure that data transmitted between services is encrypted , secure and reliable to prevent eavesdropping or tampering .

2. Authentication: Verify the identity of users and services to prevent unauthorized access .

3. Authorization : Control the actions that users and services are allowed to perform based on the roles and permissions of authenticated users and services .

4. Policy enforcement : Implement fine-grained access controls and policies that govern service-to-service and user interactions .

5. Certificate management : Manage digital certificates used to encrypt data and establish trust between services .

This open source project uses several integrated technologies and solutions to overcome these challenges.

Project Setup and Configuration

The project first sets up a secure environment using shell scripts and Docker . The setup requires providing digital certificates and starting the necessary services to ensure that all components are ready for secure communication.

Steps to set up the environment

1. Provide certificates

The project uses a shell script ( provisioning.sh ) to simulate a Certificate Authority ( CA ) and generate the necessary certificates for the service.

./provisioning.sh

2. Start the service

Docker Compose is used to start all the services defined in the project and ensure that they are correctly configured for secure operation.

docker-compose up

3. Test service-to-service communication

In order to authenticate service-to-service communication using certificates and JWT tokens, a test_services.sh script is provided. This script demonstrates how different services can interact securely using the certificates assigned to them .

Addressing security challenges in distributed systems

The project integrates several key technologies to address the major security challenges mentioned earlier. Here is how each challenge is addressed :

1. Secure communication using mutual TLS ( mTLS )

challenge

In a distributed system, services must communicate securely to prevent unauthorized access and data leakage.

Solution

The project uses mutual TLS ( mTLS ) to secure communications between services. mTLS ensures that both the client and server authenticate each other using their respective certificates. This mutual authentication prevents unauthorized services from communicating with legitimate services.

Implementation

Nginx is configured as a reverse proxy to handle mTLS. It requires both client and server certificates to establish a secure connection, ensuring that data transmitted between services remains confidential and tamper-proof.

2. Authentication using Keycloak

challenge

Properly authenticating users and services is critical to

Solution

The project leverages the open source identity and access management solution Keycloak to manage authentication. Keycloak supports multiple authentication methods, including OpenID Connect and client credentials , suitable for both user authentication and service authentication.

• User Authentication:

Use OpenID Connect to authenticate users. Keycloak is configured with a client ( appTest-login-client ) that handles the user authentication flow, including login, token issuance , and callback handling.

• Service Authentication:

For service-to-service authentication, the project uses a Keycloak client ( client_credentials-test ) configured for the client credentials grant type . This approach is ideal for authenticating to services without user intervention.

Authentication flow example

• User navigates to the login page.
• After a successful login, Keycloak redirects the user to the callback page with the authorization code.
• The authorization code is then exchanged for a JWT token for subsequent requests. The authn.js file in the nginx/njs directory provides a detailed implementation of this flow .

Service authentication example using client credentials

curl -X POST "http://localhost:9000/realms/tenantA/protocol/openid-connect/token" \
 -H "Content-Type: application/x-www-form-urlencoded" \
 -d "grant_type=client_credentials" \
 -d "client_id=client_credentials-test" \
 -d "client_secret=your-client-secret-here"

3. User Authorization with Open Policy Agent ( OPA ) and JWT

challenge

Implement fine-grained access controls to ensure that authenticated users and services can access only authorized resources .

Solution

This project uses Open Policy Agent ( OPA ) in conjunction with JWT tokens to enforce authorization policies. This project demonstrates three different JWT validation strategies to ensure solid security :

• Get Certificate from Keycloak : Dynamically get a certificate from Keycloak to validate the token.
• Use x5t ( thumbprint ): Use the thumbprint embedded in the token to retrieve the public key from the local trust store.
• Embedded Certificate Validation : Use an embedded certificate validation token to ensure that certificates are validated against a trusted Certificate Authority ( CA ) .

For a detailed implementation of these policies , see the nginx/njs/token.js file : https://github.com/apssouza22/security-architecture/blob/main/nginx/njs/token.js.

4. Policy Enforcement Using Open Policy Agent ( OPA )

challenge

Implement dynamic and flexible access control policies for services and users .

Solution

OPA is used to enforce fine-grained access control policies. Policies are written in a declarative language ( Rego ) and stored in the opa/ directory. These policies dictate the conditions under which services can communicate and users can access resources, ensuring that access controls are applied consistently across the system .

5. Certificate Management

challenge

Manage digital certificates for services to establish trust and secure communications .

Solution :

The project includes a robust certificate management system. A shell script ( provisioning.sh ) is used to emulate a certificate authority ( CA ) and generate certificates for each service. This approach simplifies certificate management and ensures that all services have the credentials they need to communicate securely.

We also added an endpoint to renew the serving certificate without restarting nginx.

curl --insecure https://localhost/certs --cert certificates/gen/serviceA/client.crt --key certificates/gen/serviceA/client.key -F cert=@certificates/gen/serviceA/client.crt -F key=@certificates/gen/serviceA/client.key

In Conclusion

Building secure distributed systems requires careful consideration of various security aspects, including secure communication, authentication, authorization, policy enforcement , and certificate management. This open source project provides a comprehensive example of how to integrate multiple security mechanisms to effectively address these challenges.

If you follow the setup and configuration demonstrated in this project, developers can build a robust security architecture using mutual TLS, Keycloak, Open Policy Agent, and Nginx . Together, these technologies provide a solid foundation for protecting distributed systems from a variety of threats, ensuring data protection and secure access control.

Original title: Designing a Secure Architecture for Distributed Systems , author: Alexsandro Souza