Align Evals: LangSmith’s New Playground for Human‑Aligned LLM Evaluation

When LangChain announced Align Evals on July 29, 2025, it answered a pain point that has dogged almost every LLM team: evaluator scores that don’t line up with human judgment. The new feature—now live for all LangSmith Cloud users—lets builders calibrate their “LLM‑as‑a‑judge” prompts until automated scores track closely with what real reviewers would say. 

Why alignment matters in evaluation

Even the best prompt tweaks or model upgrades lose value if your test harness misfires. LangChain notes that teams waste time “chasing false signals” when evaluators over‑ or under‑score outputs versus human reviewers. Align Evals gives immediate feedback on that gap, quantifying it as an alignment score you can iterate against. 

A feature set built for rapid iteration

Align Evals drops a playground‑style interface into LangSmith with three marquee capabilities:

1. Real‑time alignment score for each evaluator prompt revision.

2. Side‑by‑side comparison of human‑graded “golden set” examples and LLM‑generated scores, sortable to surface the worst mismatches.

3. Baseline snapshots so you can track whether your latest prompt improved or regressed alignment. 

The alignment flow in four steps

LangChain distills evaluator creation into a structured loop:

1. Select evaluation criteria that reflect app priorities (e.g., correctness and conciseness for chatbots).

2. Curate representative data—good and bad outputs alike—to form a realistic test bed.

3. Assign human scores to create a gold standard.

4. Draft an evaluator prompt, run it against the set, and refine until its judgments mirror the human baseline. The UI highlights over‑scored or under‑scored cases so you know exactly what to fix next. 

Availability and roadmap

Align Evals is already shipping in LangSmith Cloud; a self‑hosted release drops later this week. Looking ahead, LangChain teases analytics for long‑term tracking and even automatic prompt optimization that will generate alternative evaluator prompts for you. 

Why AI builders should care

Evaluations are the backbone of continuous improvement—whether you’re evaluating a single prompt, a RAG pipeline, or a multi‑agent workflow. Yet teams often discover that a “99 % accurate” evaluator still lets bad outputs slip through. Align Evals lowers that friction, turning evaluator design into a measurable, repeatable process.

For AI enthusiasts and practitioners, the message is clear: before you chase bigger models or flashier agents, make sure your evaluators speak the same language as your users. With Align Evals, LangChain just handed the community a calibrated mic—and the feedback loop we’ve been missing.

Building a Real-Time AI Assistant with Jina Search, LangChain, and Gemini 2.0 Flash

 In the evolving landscape of artificial intelligence, creating responsive and intelligent assistants capable of real-time information retrieval is becoming increasingly feasible. A recent tutorial by MarkTechPost demonstrates how to build such an AI assistant by integrating three powerful tools: Jina Search, LangChain, and Gemini 2.0 Flash. 

Integrating Jina Search for Semantic Retrieval

Jina Search serves as the backbone for semantic search capabilities within the assistant. By leveraging vector search technology, it enables the system to understand and retrieve contextually relevant information from vast datasets, ensuring that user queries are met with precise and meaningful responses.

Utilizing LangChain for Modular AI Workflows

LangChain provides a framework for constructing modular and scalable AI workflows. In this implementation, it facilitates the orchestration of various components, allowing for seamless integration between the retrieval mechanisms of Jina Search and the generative capabilities of Gemini 2.0 Flash.

Employing Gemini 2.0 Flash for Generative Responses

Gemini 2.0 Flash, a lightweight and efficient language model, is utilized to generate coherent and contextually appropriate responses based on the information retrieved. Its integration ensures that the assistant can provide users with articulate and relevant answers in real-time.

Constructing the Retrieval-Augmented Generation (RAG) Pipeline

The assistant's architecture follows a Retrieval-Augmented Generation (RAG) approach. This involves:

1. Query Processing: User inputs are processed and transformed into vector representations.

2. Information Retrieval: Jina Search retrieves relevant documents or data segments based on the vectorized query.

3. Response Generation: LangChain coordinates the flow of retrieved information to Gemini 2.0 Flash, which then generates a coherent response.

Benefits and Applications

This integrated approach offers several advantages:

• Real-Time Responses: The assistant can provide immediate answers to user queries by accessing and processing information on-the-fly.

• Contextual Understanding: Semantic search ensures that responses are not just keyword matches but are contextually relevant.

• Scalability: The modular design allows for easy expansion and adaptation to various domains or datasets.

Conclusion

By combining Jina Search, LangChain, and Gemini 2.0 Flash, developers can construct intelligent AI assistants capable of real-time, context-aware interactions. This tutorial serves as a valuable resource for those looking to explore the integration of retrieval and generation mechanisms in AI systems.

Mistral AI Launches Agents API to Simplify AI Agent Creation for Developers

 Mistral AI has unveiled its Agents API, a developer-centric platform designed to simplify the creation of autonomous AI agents. This launch represents a significant advancement in agentic AI, offering developers a structured and modular approach to building agents that can interact with external tools, data sources, and APIs.

Key Features of the Agents API

1. Built-in Connectors:
   The Agents API provides out-of-the-box connectors, including:

   • Web Search: Enables agents to access up-to-date information from the web, enhancing their responses with current data.

   • Document Library: Allows agents to retrieve and utilize information from user-uploaded documents, supporting retrieval-augmented generation (RAG) tasks.

   • Code Execution: Facilitates the execution of code snippets, enabling agents to perform computations or run scripts as part of their workflow.

   • Image Generation: Empowers agents to create images based on textual prompts, expanding their multimodal capabilities.

2. Model Context Protocol (MCP) Integration:
   The API supports MCP, an open standard that allows agents to seamlessly interact with external systems such as APIs, databases, and user data. This integration ensures that agents can access and process real-world context effectively.

3. Persistent State Management:
   Agents built with the API can maintain state across multiple interactions, enabling more coherent and context-aware conversations.

4. Agent Handoff Capability:
   The platform allows for the delegation of tasks between agents, facilitating complex workflows where different agents handle specific subtasks.

5. Support for Multiple Models:
   Developers can leverage various Mistral models, including Mistral Medium and Mistral Large, to power their agents, depending on the complexity and requirements of the tasks.

Performance and Benchmarking

In evaluations using the SimpleQA benchmark, agents utilizing the web search connector demonstrated significant improvements in accuracy. For instance, Mistral Large achieved a score of 75% with web search enabled, compared to 23% without it. Similarly, Mistral Medium scored 82.32% with web search, up from 22.08% without. (Source)

Developer Resources and Accessibility

Mistral provides comprehensive documentation and SDKs to assist developers in building and deploying agents. The platform includes cookbooks and examples for various use cases, such as GitHub integration, financial analysis, and customer support. (Docs)

The Agents API is currently available to developers, with Mistral encouraging feedback to further refine and enhance the platform.

Implications for AI Development

The introduction of the Agents API by Mistral AI signifies a move toward more accessible and modular AI development. By providing a platform that simplifies the integration of AI agents into various applications, Mistral empowers developers to create sophisticated, context-aware agents without extensive overhead. This democratization of agentic AI has the potential to accelerate innovation across industries, from customer service to data analysis.