Skip to content

API Endpoints Refactoring and Keycloak Integration Enhancement

Date: 2025-04-24 Participants: Rakesh Gangwar, Cascade AI

Context

As part of the ongoing development of the Meta Agent Platform, we identified two key areas for improvement:

  1. The API endpoints structure needed refactoring as all endpoints were defined directly in the main.py file, making it difficult to maintain as the application grew.

  2. The custom Keycloak implementation needed to be replaced with a more robust solution using the python-keycloak library to improve authentication and user management capabilities.

This work aimed to improve code organization, maintainability, and enhance the Keycloak integration for better user management.

Discussion

Current Structure Issues

The original implementation had several issues:

  1. Code Organization: All API endpoints were defined in the main.py file, making it increasingly difficult to navigate and maintain.
  2. Separation of Concerns: The main application file was handling too many responsibilities, from application setup to endpoint definitions.
  3. Scalability: Adding new endpoints would further bloat the main.py file, making it harder to manage over time.
  4. Maintainability: Related endpoints were not grouped together, making it difficult to understand the API structure.
  5. Custom Keycloak Implementation: The custom implementation lacked robustness and had limitations in user management capabilities.

Proposed Solution

We decided to implement improvements in two main areas:

API Structure Refactoring

  1. Creating Dedicated Router Files: Moving endpoints to dedicated files based on their functionality.
  2. Implementing a Router Structure: Using FastAPI's APIRouter to organize endpoints.
  3. Centralizing Dependencies: Creating reusable dependencies in a dedicated file.
  4. Simplifying the Main Application: Keeping only essential setup code in the main.py file.

Keycloak Integration Enhancement

  1. Migrate to python-keycloak: Replace the custom implementation with the more robust python-keycloak library.
  2. Improve Token Handling: Enhance token validation and decoding using the jose library.
  3. Enhance User Management: Implement better user management capabilities, including user creation and synchronization.
  4. Configure Admin Client: Set up a separate admin client for administrative operations in Keycloak.

Implementation

API Router Structure

We created a structured API router system with the following components:

  1. Main API Router: Defined in /app/api/v1/__init__.py, which includes all sub-routers.
  2. User Endpoints: Moved to /app/api/v1/users.py.
  3. Authentication Endpoints: Moved to /app/api/v1/auth.py.
  4. General Endpoints: Moved to /app/api/v1/general.py.

API Dependencies

We created a dedicated file for API dependencies at /app/api/dependencies.py, which includes:

  1. User Service Dependency: For accessing user-related services.
  2. Authentication Dependencies: For getting the current user with different role requirements.

Endpoint Organization

We organized the endpoints into logical groups:

  1. User Endpoints:
  2. /users/me - Get current user info
  3. /users/{user_id} - Get specific user (admin only)
  4. /users/ - List all users (admin only)
  5. /users/sync - Sync users from Keycloak (admin only)

  6. /users/create - Create new user (admin only)

  7. Authentication Endpoints:

  8. /login - Redirect to Keycloak login
  9. /callback - Handle login callback
  10. /logout - Logout from Keycloak

  11. /token - Get client credentials token

  12. General Endpoints:

  13. / - Root endpoint
  14. /protected - Protected endpoint requiring authentication

Keycloak Integration Enhancement

We significantly improved the Keycloak integration by:

  1. Implementing python-keycloak:
  2. Replaced the custom Keycloak implementation in app/core/auth.py with the python-keycloak library
  3. Configured both a regular client for authentication and an admin client for administrative operations

  4. Improved token validation and user information retrieval

  5. Enhancing User Management:

  6. Updated the UserService in app/services/user.py to leverage the KeycloakAdmin API
  7. Implemented functions to create, update, and delete users in Keycloak

  8. Added synchronization between Keycloak and the database with sync_user_from_keycloak

  9. Configuring Admin Client:

  10. Added separate admin client configuration in app/core/config.py
  11. Set up admin credentials for administrative operations

  12. Implemented proper initialization of the KeycloakAdmin client

  13. Improving Token Handling:

  14. Enhanced token validation using the jose library
  15. Implemented a simplified token validation approach for development
  16. Added better error handling for authentication issues

Main Application Update

We simplified the main.py file to:

  1. Include the API Router: Added the main API router with the appropriate prefix.
  2. Include Root-Level Routes: Added auth and general routers at the root level for backward compatibility.
  3. Keep Essential Setup: Retained only the necessary application setup code.
  4. Update CORS Settings: Explicitly allowed the frontend origin for better security.

Decision

  1. Modular API Structure: We adopted a modular API structure with dedicated files for different endpoint groups.
  2. Router-Based Organization: We used FastAPI's APIRouter for organizing endpoints.
  3. Centralized Dependencies: We created a central location for API dependencies.
  4. Backward Compatibility: We maintained backward compatibility by including root-level routes.
  5. python-keycloak Adoption: We replaced the custom Keycloak implementation with the more robust python-keycloak library.
  6. Separate Admin Client: We configured a separate admin client for administrative operations in Keycloak.

Benefits

  1. Improved Maintainability: The codebase is now more organized and easier to maintain.
  2. Better Scalability: Adding new endpoints is simpler as they can be added to the appropriate router file.
  3. Enhanced Readability: The code is more readable with related endpoints grouped together.
  4. Easier Debugging: Issues with specific endpoints are easier to locate and fix.
  5. Simplified Main Application: The main.py file is now focused on application setup rather than endpoint definitions.
  6. Robust Authentication: The python-keycloak library provides a more robust and maintainable authentication solution.
  7. Enhanced User Management: The KeycloakAdmin API enables better user management capabilities.
  8. Improved Security: Better token validation and handling improves the security of the application.

Next Steps

  1. API Documentation: Enhance API documentation with more detailed descriptions and examples.
  2. Endpoint Testing: Implement comprehensive tests for all API endpoints.
  3. Versioning Strategy: Develop a strategy for API versioning as the application evolves.
  4. Authentication Enhancements: Further improve the authentication system with more granular role-based access control.
  5. Swagger UI Improvements: Enhance the Swagger UI for better API exploration and testing.
  6. Security Hardening: Enable JWT signature verification for production and move Keycloak secrets to environment variables.
  7. Role-Based Access Control: Implement more extensive role-based access control throughout the application.
  8. User Interface Improvements: Enhance the user management interface with better error handling and user feedback.

References