Skip to main content

​
Secrets Management

TextLayer Core provides flexible options for managing secrets and environment variables across different environments. This guide covers the various secrets management tools supported by TextLayer Core.

​
Supported Providers

TextLayer Core supports several secrets management solutions:

​
Doppler Setup

​
Setting Up Doppler

​
Prerequisites

Before getting started with Doppler for TextLayer Core, you’ll need:
  1. A Doppler account
  2. The Doppler CLI installed on your machine
  3. Access to your TextLayer Core project in Doppler (ask your administrator for access)

​
Installation

  • macOS
  • Linux
  • Windows
# Using Homebrew
brew install dopplerhq/cli/doppler

# Verify installation
doppler --version

​
Configuration

Once you have the Doppler CLI installed, you need to authenticate and set up your project:
  1. Login to Doppler:
doppler login
  1. Set up your project:
# Navigate to your TextLayer Core project directory
cd ~/repos/textlayer-core

# Initialize Doppler with your project
doppler setup
During setup, you’ll be prompted to select:
  • Your Doppler project (select the TextLayer Core project)
  • The config (environment) you want to use (e.g., dev, staging, prod)
TextLayer Core provides a convenient Make command for setting up Doppler: 
# Run the Doppler setup helper
make doppler-setup

​
Using Doppler with TextLayer Core

​
Local Development

There are two primary ways to use Doppler with TextLayer Core for local development:

​
Managing Secrets

Doppler provides a web dashboard and CLI for managing your secrets:
  1. Viewing secrets:
# List all secrets in your current config
doppler secrets

# View a specific secret
doppler secrets get API_KEY
  1. Setting secrets:
# Set a single secret
doppler secrets set API_KEY=your-api-key

# Set multiple secrets
doppler secrets set --config dev \
  API_KEY=your-api-key \
  DATABASE_URL=your-database-url
  1. Removing secrets:
# Remove a secret
doppler secrets delete API_KEY

​
Environment-Specific Configurations

Doppler allows you to manage different configurations for different environments (development, staging, production):
  1. Switching environments:
# Switch to the development environment
doppler setup --config dev

# Switch to the production environment
doppler setup --config prod
  1. Running with a specific config:
# Run with a specific config without changing your default
doppler run --config prod -- flask run

​
CI/CD Integration

While TextLayer Core uses GitHub Actions secrets directly for CI/CD pipelines, you can integrate Doppler with your CI/CD system for consistent secrets management across all environments.

​
GitHub Actions

To use Doppler with GitHub Actions:
  1. Add Doppler Service Token:
# Generate a service token in Doppler dashboard for your CI environment
doppler configs tokens create ci-token --access readonly --config prod

# Add the token to GitHub repository secrets as DOPPLER_TOKEN
  1. Use in GitHub Actions workflow:
name: Build and Deploy

on:
  push:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v2
      
      - name: Install Doppler CLI
        uses: dopplerhq/cli-action@v1
        
      - name: Set up environment
        run: doppler run --token=${{ secrets.DOPPLER_TOKEN }} -- echo "Doppler configured"
        
      - name: Build and test
        run: doppler run --token=${{ secrets.DOPPLER_TOKEN }} -- make test
        
      - name: Deploy
        run: doppler run --token=${{ secrets.DOPPLER_TOKEN }} -- make deploy

​
Best Practices

​
Secret Management Workflow

  1. Never store secrets in code or version control
    • Use Doppler for all secrets and environment variables
    • Add .env files to .gitignore
  2. Use descriptive names for secrets
    • Follow a consistent naming pattern (e.g., SERVICE_CREDENTIAL_TYPE)
    • Use uppercase for environment variables
  3. Limit access to production secrets
    • Use Doppler’s access controls to restrict who can view or modify production secrets
    • Use readonly service tokens for CI/CD systems
  4. Rotate secrets regularly
    • Set up a schedule for rotating important credentials
    • Doppler maintains a history of changes for auditability
  5. Use different configs for different environments
    • Create separate configs for development, staging, and production
    • Use config syncing for promoting changes between environments

​
Secure Local Development

For local development with TextLayer Core:
  1. Always use doppler run instead of .env files
    • This prevents secrets from being written to disk
    • Example: doppler run -- flask run
  2. Verify your active configuration
    • Check which Doppler config you’re using: doppler configure
    • Ensure you’re not accidentally using production secrets in development
  3. Set up a personal development config
    • For team environments, consider having personal configs (e.g., dev-john)
    • This prevents conflicts when multiple developers need different values

​
Troubleshooting

​
Common Issues

  1. Authentication problems: 
    # Re-authenticate with Doppler
doppler login
  2. Wrong environment variables: 
    # Verify your active config
doppler configure

# List all secrets in your current config
doppler secrets
  3. Doppler CLI not found: 
    # Verify installation
which doppler

# Reinstall if needed
brew install dopplerhq/cli/doppler  # For macOS
For more information, consult the Doppler Documentation.

​
Keeper Setup

Keeper Secrets Manager provides zero-knowledge secure storage for secrets. This section covers how to set up and use Keeper with TextLayer Core.

​
Installation

  • Python SDK
# Install the Keeper SDK
pip install keeper-secrets-manager-core

​
Configuration

To use Keeper with TextLayer Core:
  1. Create a Keeper Secrets Manager account at Keeper Security
  2. Create a secrets manager application in the Keeper Admin Console
  3. Generate a One-Time Token for initializing the SDK
  4. Initialize the SDK with your token: 
# Add to your initialization code
import keeper_secrets_manager_core as ksm

# Initialize with your One-Time Token
config = ksm.core.SecretsManager(
    token="your-one-time-token"
)

# Get a secret
secret = config.get_secret("your/secret/notation")

# Access fields
api_key = secret["apiKey"]
  1. Add the Keeper integration to your TextLayer Core setup:
Create a keeper_secrets.py file in your project: 
import os
import keeper_secrets_manager_core as ksm
from dotenv import load_dotenv

def load_keeper_secrets():
    # Check if we need to load secrets from Keeper
    if os.environ.get("USE_KEEPER_SECRETS") == "1":
        try:
            # Initialize Keeper client
            config = ksm.core.SecretsManager(
                config=os.environ.get("KEEPER_CONFIG_FILE"),
                token=os.environ.get("KEEPER_TOKEN")
            )
            
            # Get secrets record
            record = config.get_secret(os.environ.get("KEEPER_SECRET_NOTATION"))
            
            # Set environment variables from record
            for key, value in record.items():
                os.environ[key] = value
                
            print("Loaded secrets from Keeper Secrets Manager")
        except Exception as e:
            print(f"Error loading Keeper secrets: {e}")
            # Fall back to .env file
            load_dotenv()
    else:
        # Load from .env file
        load_dotenv()
Then import and call this function in your application initialization.

​
Infisical Setup

Infisical is an open-source secrets management platform. This section covers how to set up and use Infisical with TextLayer Core.

​
Installation

  • CLI
  • Python SDK
# Using npm
npm install -g infisical

# Verify installation
infisical --version

​
Configuration

To use Infisical with TextLayer Core:
  1. Create an Infisical account at Infisical
  2. Create a project in the Infisical dashboard
  3. Initialize Infisical in your project: 
# Navigate to your TextLayer Core project directory
cd ~/repos/textlayer-core

# Initialize Infisical
infisical init
  1. Create a service token in the Infisical dashboard
  2. Add the Infisical integration to your TextLayer Core setup:
Create an infisical_secrets.py file in your project: 
import os
from infisical import InfisicalClient
from dotenv import load_dotenv

def load_infisical_secrets():
    # Check if we need to load secrets from Infisical
    if os.environ.get("USE_INFISICAL") == "1":
        try:
            # Initialize Infisical client
            client = InfisicalClient(token=os.environ.get("INFISICAL_TOKEN"))
            
            # Get all secrets for the environment
            secrets = client.get_all_secrets(
                environment=os.environ.get("INFISICAL_ENVIRONMENT", "dev")
            )
            
            # Set environment variables from secrets
            for secret in secrets:
                os.environ[secret.key] = secret.value
                
            print("Loaded secrets from Infisical")
        except Exception as e:
            print(f"Error loading Infisical secrets: {e}")
            # Fall back to .env file
            load_dotenv()
    else:
        # Load from .env file
        load_dotenv()
Then import and call this function in your application initialization.

​
HashiCorp Vault Setup

HashiCorp Vault is an advanced secrets management and data protection platform. This section covers how to set up and use Vault with TextLayer Core.

​
Installation

  • CLI
  • Python SDK
# Download Vault
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install vault

# Verify installation
vault --version

​
Configuration

To use HashiCorp Vault with TextLayer Core:
  1. Set up a Vault server or use HashiCorp Cloud Platform (HCP) Vault
  2. Create a policy for TextLayer Core access: 
# textlayer-policy.hcl
path "secret/data/textlayer/*" {
  capabilities = ["read"]
}
  1. Create a token with the policy:
vault token create -policy=textlayer-policy
  1. Add the Vault integration to your TextLayer Core setup:
Create a vault_secrets.py file in your project: 
import os
import hvac
from dotenv import load_dotenv

def load_vault_secrets():
    # Check if we need to load secrets from Vault
    if os.environ.get("USE_VAULT") == "1":
        try:
            # Initialize Vault client
            client = hvac.Client(
                url=os.environ.get("VAULT_ADDR"),
                token=os.environ.get("VAULT_TOKEN")
            )
            
            # Verify authentication
            if not client.is_authenticated():
                raise Exception("Vault authentication failed")
            
            # Get secrets from Vault
            secret_path = os.environ.get("VAULT_SECRET_PATH", "textlayer/config")
            secret = client.secrets.kv.v2.read_secret_version(
                path=secret_path
            )
            
            # Set environment variables from secret data
            for key, value in secret["data"]["data"].items():
                os.environ[key] = value
                
            print("Loaded secrets from HashiCorp Vault")
        except Exception as e:
            print(f"Error loading Vault secrets: {e}")
            # Fall back to .env file
            load_dotenv()
    else:
        # Load from .env file
        load_dotenv()
Then import and call this function in your application initialization.

​
Using .env Files

For local development or simple deployments, TextLayer Core supports using .env files for environment variables.

​
Creating a .env File

Create a .env file in the root of your TextLayer Core project: 
# TextLayer Core .env file

# Flask Configuration
FLASK_APP=application.py
FLASK_DEBUG=1
FLASK_RUN_PORT=5000
FLASK_RUN_HOST=0.0.0.0
LOG_LEVEL=INFO

# OpenSearch Configuration
ELASTICSEARCH_URL=http://localhost:9200
ELASTICSEARCH_USER=admin
ELASTICSEARCH_PASSWORD=admin
KNN_EMBEDDING_DIMENSION=1536

# LLM Configuration
CHAT_MODEL=gpt-4
EMBEDDING_MODEL=text-embedding-ada-002

# API Keys
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-...

# Langfuse Configuration
LANGFUSE_PUBLIC_KEY=pk-...
LANGFUSE_SECRET_KEY=sk-...
LANGFUSE_HOST=https://cloud.langfuse.com

​
Loading Environment Variables

TextLayer Core automatically loads environment variables from the .env file using the python-dotenv package: 
# This is already configured in TextLayer Core
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()

​
Best Practices for .env Files

While .env files are convenient for local development, they have security limitations:
  1. Never commit .env files to version control
    • Always add .env to your .gitignore file
    • Consider using .env.example with placeholder values as a template
  2. Limit access to .env files
    • Restrict file permissions: chmod 600 .env
    • Share secrets securely with team members
  3. Consider using a secrets manager for production
    • .env files are best for local development only
    • Use Doppler or another secrets manager for production environments
  4. Validate required variables
    • Add validation in your application startup to ensure all required variables are present

​
Choosing the Right Solution

When selecting a secrets management solution for TextLayer Core, consider:
  1. Team Size and Collaboration
    • Doppler and Infisical excel at team collaboration
    • Keeper provides strong access controls for larger teams
    • Vault offers advanced policy-based access control
  2. Security Requirements
    • All solutions provide encryption at rest and in transit
    • Keeper and Vault offer advanced security features
    • Consider compliance requirements for your organization
  3. Integration Needs
    • Doppler has the tightest integration with TextLayer Core
    • All solutions provide CLI and SDK options
    • Consider existing infrastructure and tooling
  4. Deployment Model
    • Doppler and Keeper are primarily cloud-based
    • Infisical and Vault can be self-hosted
    • Consider your organization’s cloud vs. on-premises requirements
For most TextLayer Core users, we recommend Doppler for its ease of use, strong security, and seamless integration.

​
Troubleshooting

For troubleshooting secrets management issues, see our FAQ or contact support.
⌘I