Create a project

Projects are the foundation of your AI evaluation workflow in Evaligo. They organize all your datasets, experiments, evaluations, and production traces around a specific AI application or use case, providing structure and collaboration capabilities for your team.

A well-configured project sets the stage for effective AI development and quality assurance. It establishes permissions, integrations, and organizational patterns that will serve your team throughout the development lifecycle from initial experimentation to production monitoring.

This guide walks you through creating and configuring projects that scale with your needs, support team collaboration, and integrate seamlessly with your existing development workflows and infrastructure.

Whether you're working on a chatbot, content generation system, code assistant, or any other AI application, proper project setup ensures your evaluation efforts are organized, discoverable, and maintainable over time.

Project Planning and Architecture

Before creating your first project, consider how you want to organize your AI initiatives. Projects should align with your team structure, product boundaries, and deployment environments to maximize clarity and collaboration.

Most organizations start with one project per AI application or major feature. As your AI initiatives grow, you can create additional projects for different environments (development, staging, production) or different teams working on related applications.

Step-by-Step Project Creation

Creating a project in Evaligo involves several configuration steps that establish the foundation for all your future evaluation work. Take time to set these up thoughtfully from the beginning.

1. 1

   Basic Information Name your project descriptively and add a clear description of its purpose and scope.

2. 2

   Team and Permissions Invite collaborators and set up appropriate access levels for different team members.

3. 3

   Environment Configuration Configure API keys, model providers, and environment-specific settings.

4. 4

   Integrations Set up notifications, webhooks, and connections to your existing tools.

Project Naming and Description

Choose project names that clearly indicate the AI application, team, and optionally the environment or version. Good names make projects discoverable and help team members understand scope at a glance.

Include a comprehensive description that explains the project's purpose, the AI application being evaluated, key stakeholders, and any special considerations or constraints. This context helps new team members get oriented quickly.

// Recommended project naming patterns:

// Application-focused names
"Customer-Support-Chatbot"
"Code-Review-Assistant" 
"Content-Generation-Marketing"
"SQL-Query-Assistant"

// Team and environment specific
"Support-Bot-CustomerSuccess-Prod"
"CodeGen-DevTools-Staging"
"Chatbot-Sales-Development"

// Version or time-based
"Support-Bot-v2-2024"
"Content-Gen-Q1-Campaign"
"Assistant-Winter-Release"

Team Collaboration and Permissions

Effective AI evaluation requires collaboration between different roles: product managers define requirements, engineers implement solutions, data scientists build evaluators, and stakeholders review results. Project permissions support this collaboration.

Evaligo provides role-based access control that balances collaboration with security. Assign appropriate permissions based on each team member's responsibilities and need for access to sensitive data or configuration.

Permission Levels

Understanding permission levels helps you assign appropriate access without over-permissioning team members or creating bottlenecks for common operations.

Owner

Full access to all project features including deletion, billing, and permission management. Typically assigned to project leads or senior engineers.

Admin

Can modify project settings, manage team members, configure integrations, and access all evaluation data. Good for technical leads and senior team members.

Member

Can create and run experiments, view results, and contribute to datasets. The standard level for most team members actively working on the project.

Viewer

Read-only access to experiments, results, and dashboards. Appropriate for stakeholders who need visibility but don't actively contribute to evaluation work.

Environment Configuration

Configure your project to work with your AI infrastructure by setting up API keys, model providers, and environment-specific settings. This enables Evaligo to run experiments and evaluations against your chosen models and services.

Evaligo supports all major LLM providers including OpenAI, Anthropic, AWS Bedrock, Google Vertex AI, and Azure OpenAI. You can configure multiple providers within a single project to compare across different models and services.

// Example environment configuration
{
  "providers": {
    "openai": {
      "api_key": "${OPENAI_API_KEY}",
      "models": ["gpt-4", "gpt-4-turbo", "gpt-3.5-turbo"]
    },
    "anthropic": {
      "api_key": "${ANTHROPIC_API_KEY}",
      "models": ["claude-3-opus", "claude-3-sonnet"]
    }
  },
  "environment": {
    "name": "production",
    "cost_limits": {
      "daily_max": 100.00,
      "experiment_max": 25.00
    }
  }
}

Integrations and Notifications

Connect your project to existing tools and workflows to maximize its value and ensure important results reach the right people at the right time. Integrations reduce context switching and automate common workflows.

Set up Slack notifications for experiment completions, webhook integrations for CI/CD pipelines, and dashboard embeds for stakeholder reporting. These integrations make evaluation results more accessible and actionable.

First Steps After Creation

Once your project is created and configured, establish your evaluation baseline by importing your first dataset and running initial experiments. This creates the foundation for all future evaluation work.

Video

Complete project setup walkthrough and first experiment

1. 1

   Import Core Dataset Start with 20-50 high-quality examples that represent your most important use cases.

2. 2

   Run Baseline Experiment Test a simple, direct prompt to establish initial performance metrics.

3. 3

   Configure Key Evaluators Set up the 2-3 most important quality metrics for your application.

4. 4

   Share Results Review initial results with your team to validate approach and priorities.

Next Steps

With your project properly configured, you're ready to begin building your evaluation workflow. Start by importing datasets and running your first experiments to establish baseline performance.