We recommend new projects start with resources from the AWS provider.
published on Wednesday, Nov 18, 2026 by Pulumi
We recommend new projects start with resources from the AWS provider.
published on Wednesday, Nov 18, 2026 by Pulumi
Resource Type definition for AWS::BedrockAgentCore::Evaluator - Creates a custom evaluator for agent quality assessment using LLM-as-a-Judge, code-based, or derived evaluator configurations. Derived evaluators run an existing built-in or third-party evaluator’s logic on a model that you supply.
Create Evaluator Resource
Resources are created with functions called constructors. To learn more about declaring and configuring resources, see Resources.
Constructor syntax
new Evaluator(name: string, args: EvaluatorArgs, opts?: CustomResourceOptions);@overload
def Evaluator(resource_name: str,
args: EvaluatorArgs,
opts: Optional[ResourceOptions] = None)
@overload
def Evaluator(resource_name: str,
opts: Optional[ResourceOptions] = None,
evaluator_config: Optional[EvaluatorConfigArgs] = None,
level: Optional[EvaluatorLevel] = None,
description: Optional[str] = None,
evaluator_name: Optional[str] = None,
kms_key_arn: Optional[str] = None,
tags: Optional[Sequence[_root_inputs.TagArgs]] = None)func NewEvaluator(ctx *Context, name string, args EvaluatorArgs, opts ...ResourceOption) (*Evaluator, error)public Evaluator(string name, EvaluatorArgs args, CustomResourceOptions? opts = null)
public Evaluator(String name, EvaluatorArgs args)
public Evaluator(String name, EvaluatorArgs args, CustomResourceOptions options)
type: aws-native:bedrockagentcore:Evaluator
properties: # The arguments to resource properties.
options: # Bag of options to control resource's behavior.
resource "aws-native_bedrockagentcore_evaluator" "name" {
# resource properties
}Parameters
- name string
- The unique name of the resource.
- args EvaluatorArgs
- The arguments to resource properties.
- opts CustomResourceOptions
- Bag of options to control resource's behavior.
- resource_name str
- The unique name of the resource.
- args EvaluatorArgs
- The arguments to resource properties.
- opts ResourceOptions
- Bag of options to control resource's behavior.
- ctx Context
- Context object for the current deployment.
- name string
- The unique name of the resource.
- args EvaluatorArgs
- The arguments to resource properties.
- opts ResourceOption
- Bag of options to control resource's behavior.
- name string
- The unique name of the resource.
- args EvaluatorArgs
- The arguments to resource properties.
- opts CustomResourceOptions
- Bag of options to control resource's behavior.
- name String
- The unique name of the resource.
- args EvaluatorArgs
- The arguments to resource properties.
- options CustomResourceOptions
- Bag of options to control resource's behavior.
Evaluator Resource Properties
To learn more about resource properties and how to use them, see Inputs and Outputs in the Architecture and Concepts docs.
Inputs
In Python, inputs that are objects can be passed either as argument classes or as dictionary literals.
The Evaluator resource accepts the following input properties:
- Evaluator
Config Pulumi.Aws Native. Bedrock Agent Core. Inputs. Evaluator Config - The configuration for the evaluator, including LLM-as-a-Judge, code-based, or derived settings.
- Level
Pulumi.
Aws Native. Bedrock Agent Core. Evaluator Level - The evaluation level that determines the scope of evaluation. Required for all evaluator configuration types. For Derived evaluators it must match the level of the base evaluator identified by BaseEvaluatorId.
- Description string
- The description of the evaluator.
- Evaluator
Name string - The name of the evaluator. Must be unique within your account.
- Kms
Key stringArn - The ARN of the KMS key used to encrypt evaluator data.
-
List<Pulumi.
Aws Native. Inputs. Tag> - A list of tags to assign to the evaluator.
- Evaluator
Config EvaluatorConfig Args - The configuration for the evaluator, including LLM-as-a-Judge, code-based, or derived settings.
- Level
Evaluator
Level - The evaluation level that determines the scope of evaluation. Required for all evaluator configuration types. For Derived evaluators it must match the level of the base evaluator identified by BaseEvaluatorId.
- Description string
- The description of the evaluator.
- Evaluator
Name string - The name of the evaluator. Must be unique within your account.
- Kms
Key stringArn - The ARN of the KMS key used to encrypt evaluator data.
-
Tag
Args - A list of tags to assign to the evaluator.
- evaluator_
config object - The configuration for the evaluator, including LLM-as-a-Judge, code-based, or derived settings.
- level "TOOL_CALL" | "TRACE" | "SESSION"
- The evaluation level that determines the scope of evaluation. Required for all evaluator configuration types. For Derived evaluators it must match the level of the base evaluator identified by BaseEvaluatorId.
- description string
- The description of the evaluator.
- evaluator_
name string - The name of the evaluator. Must be unique within your account.
- kms_
key_ stringarn - The ARN of the KMS key used to encrypt evaluator data.
- list(object)
- A list of tags to assign to the evaluator.
- evaluator
Config EvaluatorConfig - The configuration for the evaluator, including LLM-as-a-Judge, code-based, or derived settings.
- level
Evaluator
Level - The evaluation level that determines the scope of evaluation. Required for all evaluator configuration types. For Derived evaluators it must match the level of the base evaluator identified by BaseEvaluatorId.
- description String
- The description of the evaluator.
- evaluator
Name String - The name of the evaluator. Must be unique within your account.
- kms
Key StringArn - The ARN of the KMS key used to encrypt evaluator data.
- List<Tag>
- A list of tags to assign to the evaluator.
- evaluator
Config EvaluatorConfig - The configuration for the evaluator, including LLM-as-a-Judge, code-based, or derived settings.
- level
Evaluator
Level - The evaluation level that determines the scope of evaluation. Required for all evaluator configuration types. For Derived evaluators it must match the level of the base evaluator identified by BaseEvaluatorId.
- description string
- The description of the evaluator.
- evaluator
Name string - The name of the evaluator. Must be unique within your account.
- kms
Key stringArn - The ARN of the KMS key used to encrypt evaluator data.
- Tag[]
- A list of tags to assign to the evaluator.
- evaluator_
config EvaluatorConfig Args - The configuration for the evaluator, including LLM-as-a-Judge, code-based, or derived settings.
- level
Evaluator
Level - The evaluation level that determines the scope of evaluation. Required for all evaluator configuration types. For Derived evaluators it must match the level of the base evaluator identified by BaseEvaluatorId.
- description str
- The description of the evaluator.
- evaluator_
name str - The name of the evaluator. Must be unique within your account.
- kms_
key_ strarn - The ARN of the KMS key used to encrypt evaluator data.
-
Sequence[Tag
Args] - A list of tags to assign to the evaluator.
- evaluator
Config Property Map - The configuration for the evaluator, including LLM-as-a-Judge, code-based, or derived settings.
- level "TOOL_CALL" | "TRACE" | "SESSION"
- The evaluation level that determines the scope of evaluation. Required for all evaluator configuration types. For Derived evaluators it must match the level of the base evaluator identified by BaseEvaluatorId.
- description String
- The description of the evaluator.
- evaluator
Name String - The name of the evaluator. Must be unique within your account.
- kms
Key StringArn - The ARN of the KMS key used to encrypt evaluator data.
- List<Property Map>
- A list of tags to assign to the evaluator.
Outputs
All input properties are implicitly available as output properties. Additionally, the Evaluator resource produces the following output properties:
- Created
At string - The timestamp when the evaluator was created.
- Evaluator
Arn string - The Amazon Resource Name (ARN) of the evaluator.
- Evaluator
Id string - The unique identifier of the evaluator.
- Evaluator
Type Pulumi.Aws Native. Bedrock Agent Core. Evaluator Type - The kind of evaluator resource. Builtin and ThirdParty are AWS-managed global evaluators that you reference but don't create; Custom, CustomCode, and CustomDerived are evaluators that you create.
- Id string
- The provider-assigned unique ID for this managed resource.
- Provider
Pulumi.
Aws Native. Bedrock Agent Core. Evaluator Provider - Where the evaluator's logic comes from: AWS, a third-party library (DeepEval or AutoEval), or you. Derived evaluators inherit the provider of their base evaluator.
- Status
Pulumi.
Aws Native. Bedrock Agent Core. Evaluator Status - The current status of the evaluator.
- Updated
At string - The timestamp when the evaluator was last updated.
- Created
At string - The timestamp when the evaluator was created.
- Evaluator
Arn string - The Amazon Resource Name (ARN) of the evaluator.
- Evaluator
Id string - The unique identifier of the evaluator.
- Evaluator
Type EvaluatorType - The kind of evaluator resource. Builtin and ThirdParty are AWS-managed global evaluators that you reference but don't create; Custom, CustomCode, and CustomDerived are evaluators that you create.
- Id string
- The provider-assigned unique ID for this managed resource.
- Provider
Evaluator
Provider - Where the evaluator's logic comes from: AWS, a third-party library (DeepEval or AutoEval), or you. Derived evaluators inherit the provider of their base evaluator.
- Status
Evaluator
Status - The current status of the evaluator.
- Updated
At string - The timestamp when the evaluator was last updated.
- created_
at string - The timestamp when the evaluator was created.
- evaluator_
arn string - The Amazon Resource Name (ARN) of the evaluator.
- evaluator_
id string - The unique identifier of the evaluator.
- evaluator_
type "Builtin" | "ThirdParty" | "Custom" | "Custom Code" | "Custom Derived" - The kind of evaluator resource. Builtin and ThirdParty are AWS-managed global evaluators that you reference but don't create; Custom, CustomCode, and CustomDerived are evaluators that you create.
- id string
- The provider-assigned unique ID for this managed resource.
- provider
"AWS" | "Deep
Eval" | "Auto Eval" | "Custom" - Where the evaluator's logic comes from: AWS, a third-party library (DeepEval or AutoEval), or you. Derived evaluators inherit the provider of their base evaluator.
- status "ACTIVE" | "CREATING" | "CREATE_FAILED" | "UPDATING" | "UPDATE_FAILED" | "DELETING"
- The current status of the evaluator.
- updated_
at string - The timestamp when the evaluator was last updated.
- created
At String - The timestamp when the evaluator was created.
- evaluator
Arn String - The Amazon Resource Name (ARN) of the evaluator.
- evaluator
Id String - The unique identifier of the evaluator.
- evaluator
Type EvaluatorType - The kind of evaluator resource. Builtin and ThirdParty are AWS-managed global evaluators that you reference but don't create; Custom, CustomCode, and CustomDerived are evaluators that you create.
- id String
- The provider-assigned unique ID for this managed resource.
- provider
Evaluator
Provider - Where the evaluator's logic comes from: AWS, a third-party library (DeepEval or AutoEval), or you. Derived evaluators inherit the provider of their base evaluator.
- status
Evaluator
Status - The current status of the evaluator.
- updated
At String - The timestamp when the evaluator was last updated.
- created
At string - The timestamp when the evaluator was created.
- evaluator
Arn string - The Amazon Resource Name (ARN) of the evaluator.
- evaluator
Id string - The unique identifier of the evaluator.
- evaluator
Type EvaluatorType - The kind of evaluator resource. Builtin and ThirdParty are AWS-managed global evaluators that you reference but don't create; Custom, CustomCode, and CustomDerived are evaluators that you create.
- id string
- The provider-assigned unique ID for this managed resource.
- provider
Evaluator
Provider - Where the evaluator's logic comes from: AWS, a third-party library (DeepEval or AutoEval), or you. Derived evaluators inherit the provider of their base evaluator.
- status
Evaluator
Status - The current status of the evaluator.
- updated
At string - The timestamp when the evaluator was last updated.
- created_
at str - The timestamp when the evaluator was created.
- evaluator_
arn str - The Amazon Resource Name (ARN) of the evaluator.
- evaluator_
id str - The unique identifier of the evaluator.
- evaluator_
type EvaluatorType - The kind of evaluator resource. Builtin and ThirdParty are AWS-managed global evaluators that you reference but don't create; Custom, CustomCode, and CustomDerived are evaluators that you create.
- id str
- The provider-assigned unique ID for this managed resource.
- provider
Evaluator
Provider - Where the evaluator's logic comes from: AWS, a third-party library (DeepEval or AutoEval), or you. Derived evaluators inherit the provider of their base evaluator.
- status
Evaluator
Status - The current status of the evaluator.
- updated_
at str - The timestamp when the evaluator was last updated.
- created
At String - The timestamp when the evaluator was created.
- evaluator
Arn String - The Amazon Resource Name (ARN) of the evaluator.
- evaluator
Id String - The unique identifier of the evaluator.
- evaluator
Type "Builtin" | "ThirdParty" | "Custom" | "Custom Code" | "Custom Derived" - The kind of evaluator resource. Builtin and ThirdParty are AWS-managed global evaluators that you reference but don't create; Custom, CustomCode, and CustomDerived are evaluators that you create.
- id String
- The provider-assigned unique ID for this managed resource.
- provider
"AWS" | "Deep
Eval" | "Auto Eval" | "Custom" - Where the evaluator's logic comes from: AWS, a third-party library (DeepEval or AutoEval), or you. Derived evaluators inherit the provider of their base evaluator.
- status "ACTIVE" | "CREATING" | "CREATE_FAILED" | "UPDATING" | "UPDATE_FAILED" | "DELETING"
- The current status of the evaluator.
- updated
At String - The timestamp when the evaluator was last updated.
Look up Existing Evaluator Resource
Get an existing Evaluator resource’s state with the given name, ID, and optional extra properties used to qualify the lookup.
public static get(name: string, id: Input<ID>, opts?: CustomResourceOptions): Evaluator@staticmethod
def get(resource_name: str,
id: str,
opts: Optional[ResourceOptions] = None) -> Evaluatorfunc GetEvaluator(ctx *Context, name string, id IDInput, state *EvaluatorState, opts ...ResourceOption) (*Evaluator, error)public static Evaluator Get(string name, Input<string> id, CustomResourceOptions? opts = null)public static Evaluator get(String name, Output<String> id, CustomResourceOptions options)resources: existingEvaluator: type: aws-native:bedrockagentcore:Evaluator get: id: ${id}import {
to = aws-native_bedrockagentcore_evaluator.example
id = "${id}"
}
- name string
- The unique name of the resulting resource.
- id Input<ID>
- The unique provider ID of the resource to look up.
- opts CustomResourceOptions
- Bag of options to control resource's behavior.
- resource_name str
- The unique name of the resulting resource.
- id str
- The unique provider ID of the resource to look up.
- opts ResourceOptions
- Bag of options to control resource's behavior.
- ctx Context
- Context object for the current deployment.
- name string
- The unique name of the resulting resource.
- id IDInput
- The unique provider ID of the resource to look up.
- state EvaluatorState
- Pass nil. This resource has no state arguments.
- opts ResourceOption
- Bag of options to control resource's behavior.
- name string
- The unique name of the resulting resource.
- id Input<string>
- The unique provider ID of the resource to look up.
- opts CustomResourceOptions
- Bag of options to control resource's behavior.
- name String
- The unique name of the resulting resource.
- id Output<String>
- The unique provider ID of the resource to look up.
- options CustomResourceOptions
- Bag of options to control resource's behavior.
Supporting Types
EvaluatorBedrockEvaluatorModelConfig, EvaluatorBedrockEvaluatorModelConfigArgs
The configuration for using Amazon Bedrock models in evaluator assessments.
- Model
Id string - The identifier of the Amazon Bedrock model to use for evaluation.
- Additional
Model objectRequest Fields - Additional model-specific request fields.
- Inference
Config Pulumi.Aws Native. Bedrock Agent Core. Inputs. Evaluator Inference Configuration
- Model
Id string - The identifier of the Amazon Bedrock model to use for evaluation.
- Additional
Model interface{}Request Fields - Additional model-specific request fields.
- Inference
Config EvaluatorInference Configuration
- model_
id string - The identifier of the Amazon Bedrock model to use for evaluation.
- additional_
model_ anyrequest_ fields - Additional model-specific request fields.
- inference_
config object
- model
Id String - The identifier of the Amazon Bedrock model to use for evaluation.
- additional
Model ObjectRequest Fields - Additional model-specific request fields.
- inference
Config EvaluatorInference Configuration
- model
Id string - The identifier of the Amazon Bedrock model to use for evaluation.
- additional
Model anyRequest Fields - Additional model-specific request fields.
- inference
Config EvaluatorInference Configuration
- model_
id str - The identifier of the Amazon Bedrock model to use for evaluation.
- additional_
model_ Anyrequest_ fields - Additional model-specific request fields.
- inference_
config EvaluatorInference Configuration
- model
Id String - The identifier of the Amazon Bedrock model to use for evaluation.
- additional
Model AnyRequest Fields - Additional model-specific request fields.
- inference
Config Property Map
EvaluatorCategoricalScaleDefinition, EvaluatorCategoricalScaleDefinitionArgs
A categorical rating scale option.
- Definition string
- The description that explains what this categorical rating represents.
- Label string
- The label of this categorical rating option.
- Definition string
- The description that explains what this categorical rating represents.
- Label string
- The label of this categorical rating option.
- definition string
- The description that explains what this categorical rating represents.
- label string
- The label of this categorical rating option.
- definition String
- The description that explains what this categorical rating represents.
- label String
- The label of this categorical rating option.
- definition string
- The description that explains what this categorical rating represents.
- label string
- The label of this categorical rating option.
- definition str
- The description that explains what this categorical rating represents.
- label str
- The label of this categorical rating option.
- definition String
- The description that explains what this categorical rating represents.
- label String
- The label of this categorical rating option.
EvaluatorCodeBasedEvaluatorConfig, EvaluatorCodeBasedEvaluatorConfigArgs
The configuration for code-based evaluation using a Lambda function.
EvaluatorConfig, EvaluatorConfigArgs
The configuration that defines how an evaluator assesses agent performance.
EvaluatorDerivedEvaluatorConfig, EvaluatorDerivedEvaluatorConfigArgs
The configuration for an evaluator derived from an existing base evaluator (a built-in or third-party evaluator), run on your own model. The base evaluator supplies the prompt and scoring logic.
- Base
Evaluator stringId - The identifier of the base evaluator whose logic to run (a Builtin.* or ThirdParty.* evaluator).
- Model
Config Pulumi.Aws Native. Bedrock Agent Core. Inputs. Evaluator Model Config
- Base
Evaluator stringId - The identifier of the base evaluator whose logic to run (a Builtin.* or ThirdParty.* evaluator).
- Model
Config EvaluatorModel Config
- base_
evaluator_ stringid - The identifier of the base evaluator whose logic to run (a Builtin.* or ThirdParty.* evaluator).
- model_
config object
- base
Evaluator StringId - The identifier of the base evaluator whose logic to run (a Builtin.* or ThirdParty.* evaluator).
- model
Config EvaluatorModel Config
- base
Evaluator stringId - The identifier of the base evaluator whose logic to run (a Builtin.* or ThirdParty.* evaluator).
- model
Config EvaluatorModel Config
- base_
evaluator_ strid - The identifier of the base evaluator whose logic to run (a Builtin.* or ThirdParty.* evaluator).
- model_
config EvaluatorModel Config
- base
Evaluator StringId - The identifier of the base evaluator whose logic to run (a Builtin.* or ThirdParty.* evaluator).
- model
Config Property Map
EvaluatorDimensionRatingScale, EvaluatorDimensionRatingScaleArgs
The rating scale for one dimension of an LLM-as-a-Judge evaluator.
- Categorical
List<Pulumi.
Aws Native. Bedrock Agent Core. Inputs. Evaluator Categorical Scale Definition> - The categorical rating scale for this dimension.
- Numerical
List<Pulumi.
Aws Native. Bedrock Agent Core. Inputs. Evaluator Numerical Scale Definition> - The numerical rating scale for this dimension.
- Categorical
[]Evaluator
Categorical Scale Definition - The categorical rating scale for this dimension.
- Numerical
[]Evaluator
Numerical Scale Definition - The numerical rating scale for this dimension.
- categorical list(object)
- The categorical rating scale for this dimension.
- numerical list(object)
- The numerical rating scale for this dimension.
- categorical
List<Evaluator
Categorical Scale Definition> - The categorical rating scale for this dimension.
- numerical
List<Evaluator
Numerical Scale Definition> - The numerical rating scale for this dimension.
- categorical
Evaluator
Categorical Scale Definition[] - The categorical rating scale for this dimension.
- numerical
Evaluator
Numerical Scale Definition[] - The numerical rating scale for this dimension.
- categorical
Sequence[Evaluator
Categorical Scale Definition] - The categorical rating scale for this dimension.
- numerical
Sequence[Evaluator
Numerical Scale Definition] - The numerical rating scale for this dimension.
- categorical List<Property Map>
- The categorical rating scale for this dimension.
- numerical List<Property Map>
- The numerical rating scale for this dimension.
EvaluatorEvaluationDimension, EvaluatorEvaluationDimensionArgs
One dimension of an LLM-as-a-Judge evaluator.
- Dimension
Instruction string - The instruction to the evaluator model describing what to judge for this dimension.
- Dimension
Name string - The name of the dimension. Must be unique within the evaluator.
- Dimension
Rating Pulumi.Scale Aws Native. Bedrock Agent Core. Inputs. Evaluator Dimension Rating Scale
- Dimension
Instruction string - The instruction to the evaluator model describing what to judge for this dimension.
- Dimension
Name string - The name of the dimension. Must be unique within the evaluator.
- Dimension
Rating EvaluatorScale Dimension Rating Scale
- dimension_
instruction string - The instruction to the evaluator model describing what to judge for this dimension.
- dimension_
name string - The name of the dimension. Must be unique within the evaluator.
- dimension_
rating_ objectscale
- dimension
Instruction String - The instruction to the evaluator model describing what to judge for this dimension.
- dimension
Name String - The name of the dimension. Must be unique within the evaluator.
- dimension
Rating EvaluatorScale Dimension Rating Scale
- dimension
Instruction string - The instruction to the evaluator model describing what to judge for this dimension.
- dimension
Name string - The name of the dimension. Must be unique within the evaluator.
- dimension
Rating EvaluatorScale Dimension Rating Scale
- dimension_
instruction str - The instruction to the evaluator model describing what to judge for this dimension.
- dimension_
name str - The name of the dimension. Must be unique within the evaluator.
- dimension_
rating_ Evaluatorscale Dimension Rating Scale
- dimension
Instruction String - The instruction to the evaluator model describing what to judge for this dimension.
- dimension
Name String - The name of the dimension. Must be unique within the evaluator.
- dimension
Rating Property MapScale
EvaluatorInferenceConfiguration, EvaluatorInferenceConfigurationArgs
The inference configuration parameters that control model behavior during evaluation.
- Max
Tokens int - The maximum number of tokens to generate in the model response.
- Stop
Sequences List<string> - The sequences that make the model stop generating further tokens.
- Temperature double
- The temperature value that controls randomness in the model's responses.
- Top
P double - The top-p sampling parameter that controls the diversity of the model's responses.
- Max
Tokens int - The maximum number of tokens to generate in the model response.
- Stop
Sequences []string - The sequences that make the model stop generating further tokens.
- Temperature float64
- The temperature value that controls randomness in the model's responses.
- Top
P float64 - The top-p sampling parameter that controls the diversity of the model's responses.
- max_
tokens number - The maximum number of tokens to generate in the model response.
- stop_
sequences list(string) - The sequences that make the model stop generating further tokens.
- temperature number
- The temperature value that controls randomness in the model's responses.
- top_
p number - The top-p sampling parameter that controls the diversity of the model's responses.
- max
Tokens Integer - The maximum number of tokens to generate in the model response.
- stop
Sequences List<String> - The sequences that make the model stop generating further tokens.
- temperature Double
- The temperature value that controls randomness in the model's responses.
- top
P Double - The top-p sampling parameter that controls the diversity of the model's responses.
- max
Tokens number - The maximum number of tokens to generate in the model response.
- stop
Sequences string[] - The sequences that make the model stop generating further tokens.
- temperature number
- The temperature value that controls randomness in the model's responses.
- top
P number - The top-p sampling parameter that controls the diversity of the model's responses.
- max_
tokens int - The maximum number of tokens to generate in the model response.
- stop_
sequences Sequence[str] - The sequences that make the model stop generating further tokens.
- temperature float
- The temperature value that controls randomness in the model's responses.
- top_
p float - The top-p sampling parameter that controls the diversity of the model's responses.
- max
Tokens Number - The maximum number of tokens to generate in the model response.
- stop
Sequences List<String> - The sequences that make the model stop generating further tokens.
- temperature Number
- The temperature value that controls randomness in the model's responses.
- top
P Number - The top-p sampling parameter that controls the diversity of the model's responses.
EvaluatorLambdaEvaluatorConfig, EvaluatorLambdaEvaluatorConfigArgs
The Lambda function configuration for code-based evaluation.
- Lambda
Arn string - The ARN of the Lambda function used for evaluation.
- Lambda
Timeout intIn Seconds - The timeout in seconds for the Lambda function invocation.
- Lambda
Arn string - The ARN of the Lambda function used for evaluation.
- Lambda
Timeout intIn Seconds - The timeout in seconds for the Lambda function invocation.
- lambda_
arn string - The ARN of the Lambda function used for evaluation.
- lambda_
timeout_ numberin_ seconds - The timeout in seconds for the Lambda function invocation.
- lambda
Arn String - The ARN of the Lambda function used for evaluation.
- lambda
Timeout IntegerIn Seconds - The timeout in seconds for the Lambda function invocation.
- lambda
Arn string - The ARN of the Lambda function used for evaluation.
- lambda
Timeout numberIn Seconds - The timeout in seconds for the Lambda function invocation.
- lambda_
arn str - The ARN of the Lambda function used for evaluation.
- lambda_
timeout_ intin_ seconds - The timeout in seconds for the Lambda function invocation.
- lambda
Arn String - The ARN of the Lambda function used for evaluation.
- lambda
Timeout NumberIn Seconds - The timeout in seconds for the Lambda function invocation.
EvaluatorLevel
| Name | Value |
|---|---|
| Tool | TOOL_CALL |
| Trace | TRACE |
| Session | SESSION |
| Name | Value |
|---|---|
| Evaluator | TOOL_CALL |
| Evaluator | TRACE |
| Evaluator | SESSION |
| Value |
|---|
TOOL_CALL |
TRACE |
SESSION |
| Name | Value |
|---|---|
| Tool | TOOL_CALL |
| Trace | TRACE |
| Session | SESSION |
| Name | Value |
|---|---|
| Tool | TOOL_CALL |
| Trace | TRACE |
| Session | SESSION |
| Name | Value |
|---|---|
| TOOL_CALL | TOOL_CALL |
| TRACE | TRACE |
| SESSION | SESSION |
| Value |
|---|
TOOL_CALL |
TRACE |
SESSION |
EvaluatorLlmAsAJudgeEvaluatorConfig, EvaluatorLlmAsAJudgeEvaluatorConfigArgs
The configuration for LLM-as-a-Judge evaluation.
- Instructions string
- The evaluation instructions that guide the language model in assessing agent performance.
- Model
Config Pulumi.Aws Native. Bedrock Agent Core. Inputs. Evaluator Model Config - Rating
Scale Pulumi.Aws Native. Bedrock Agent Core. Inputs. Evaluator Rating Scale
- Instructions string
- The evaluation instructions that guide the language model in assessing agent performance.
- Model
Config EvaluatorModel Config - Rating
Scale EvaluatorRating Scale
- instructions string
- The evaluation instructions that guide the language model in assessing agent performance.
- model_
config object - rating_
scale object
- instructions String
- The evaluation instructions that guide the language model in assessing agent performance.
- model
Config EvaluatorModel Config - rating
Scale EvaluatorRating Scale
- instructions string
- The evaluation instructions that guide the language model in assessing agent performance.
- model
Config EvaluatorModel Config - rating
Scale EvaluatorRating Scale
- instructions str
- The evaluation instructions that guide the language model in assessing agent performance.
- model_
config EvaluatorModel Config - rating_
scale EvaluatorRating Scale
- instructions String
- The evaluation instructions that guide the language model in assessing agent performance.
- model
Config Property Map - rating
Scale Property Map
EvaluatorModelConfig, EvaluatorModelConfigArgs
The model configuration that specifies which foundation model to use for evaluation.
EvaluatorNumericalScaleDefinition, EvaluatorNumericalScaleDefinitionArgs
A numerical rating scale option.
- Definition string
- The description that explains what this numerical rating represents.
- Label string
- The label that describes this numerical rating option.
- Value double
- The numerical value for this rating scale option.
- Definition string
- The description that explains what this numerical rating represents.
- Label string
- The label that describes this numerical rating option.
- Value float64
- The numerical value for this rating scale option.
- definition string
- The description that explains what this numerical rating represents.
- label string
- The label that describes this numerical rating option.
- value number
- The numerical value for this rating scale option.
- definition String
- The description that explains what this numerical rating represents.
- label String
- The label that describes this numerical rating option.
- value Double
- The numerical value for this rating scale option.
- definition string
- The description that explains what this numerical rating represents.
- label string
- The label that describes this numerical rating option.
- value number
- The numerical value for this rating scale option.
- definition str
- The description that explains what this numerical rating represents.
- label str
- The label that describes this numerical rating option.
- value float
- The numerical value for this rating scale option.
- definition String
- The description that explains what this numerical rating represents.
- label String
- The label that describes this numerical rating option.
- value Number
- The numerical value for this rating scale option.
EvaluatorOpenResponsesEvaluatorModelConfig, EvaluatorOpenResponsesEvaluatorModelConfigArgs
The configuration for using OpenResponses-compatible models in evaluator assessments.
- Model
Id string - The identifier of the model to use for evaluation.
- Max
Output intTokens - The maximum number of output tokens to generate, including visible output and reasoning tokens.
- Reasoning
Pulumi.
Aws Native. Bedrock Agent Core. Inputs. Evaluator Reasoning Configuration - Temperature double
- The sampling temperature between 0 and 2.
- Top
P double - The nucleus sampling probability mass between 0 and 1.
- Model
Id string - The identifier of the model to use for evaluation.
- Max
Output intTokens - The maximum number of output tokens to generate, including visible output and reasoning tokens.
- Reasoning
Evaluator
Reasoning Configuration - Temperature float64
- The sampling temperature between 0 and 2.
- Top
P float64 - The nucleus sampling probability mass between 0 and 1.
- model_
id string - The identifier of the model to use for evaluation.
- max_
output_ numbertokens - The maximum number of output tokens to generate, including visible output and reasoning tokens.
- reasoning object
- temperature number
- The sampling temperature between 0 and 2.
- top_
p number - The nucleus sampling probability mass between 0 and 1.
- model
Id String - The identifier of the model to use for evaluation.
- max
Output IntegerTokens - The maximum number of output tokens to generate, including visible output and reasoning tokens.
- reasoning
Evaluator
Reasoning Configuration - temperature Double
- The sampling temperature between 0 and 2.
- top
P Double - The nucleus sampling probability mass between 0 and 1.
- model
Id string - The identifier of the model to use for evaluation.
- max
Output numberTokens - The maximum number of output tokens to generate, including visible output and reasoning tokens.
- reasoning
Evaluator
Reasoning Configuration - temperature number
- The sampling temperature between 0 and 2.
- top
P number - The nucleus sampling probability mass between 0 and 1.
- model_
id str - The identifier of the model to use for evaluation.
- max_
output_ inttokens - The maximum number of output tokens to generate, including visible output and reasoning tokens.
- reasoning
Evaluator
Reasoning Configuration - temperature float
- The sampling temperature between 0 and 2.
- top_
p float - The nucleus sampling probability mass between 0 and 1.
- model
Id String - The identifier of the model to use for evaluation.
- max
Output NumberTokens - The maximum number of output tokens to generate, including visible output and reasoning tokens.
- reasoning Property Map
- temperature Number
- The sampling temperature between 0 and 2.
- top
P Number - The nucleus sampling probability mass between 0 and 1.
EvaluatorProvider
Where the evaluator's logic comes from: AWS for AWS-authored evaluators, DeepEval or AutoEval for the corresponding third-party libraries, or Custom for an evaluator that you authored.
| Name | Value |
|---|---|
| Aws | AWS |
| Deep | DeepEval |
| Auto | AutoEval |
| Custom | Custom |
| Name | Value |
|---|---|
| Evaluator | AWS |
| Evaluator | DeepEval |
| Evaluator | AutoEval |
| Evaluator | Custom |
| Value |
|---|
AWS |
DeepEval |
AutoEval |
Custom |
| Name | Value |
|---|---|
| Aws | AWS |
| Deep | DeepEval |
| Auto | AutoEval |
| Custom | Custom |
| Name | Value |
|---|---|
| Aws | AWS |
| Deep | DeepEval |
| Auto | AutoEval |
| Custom | Custom |
| Name | Value |
|---|---|
| AWS | AWS |
| DEEP_EVAL | DeepEval |
| AUTO_EVAL | AutoEval |
| CUSTOM | Custom |
| Value |
|---|
AWS |
DeepEval |
AutoEval |
Custom |
EvaluatorRatingScale, EvaluatorRatingScaleArgs
The rating scale that defines how evaluators should score agent performance.
- Categorical
List<Pulumi.
Aws Native. Bedrock Agent Core. Inputs. Evaluator Categorical Scale Definition> - Multi
Dimensional List<Pulumi.Aws Native. Bedrock Agent Core. Inputs. Evaluator Evaluation Dimension> - The dimensions an LLM-as-a-Judge evaluator scores in a single judge call. Each dimension has its own instruction and rating scale.
- Numerical
List<Pulumi.
Aws Native. Bedrock Agent Core. Inputs. Evaluator Numerical Scale Definition>
- Categorical
[]Evaluator
Categorical Scale Definition - Multi
Dimensional []EvaluatorEvaluation Dimension - The dimensions an LLM-as-a-Judge evaluator scores in a single judge call. Each dimension has its own instruction and rating scale.
- Numerical
[]Evaluator
Numerical Scale Definition
- categorical list(object)
- multi_
dimensional list(object) - The dimensions an LLM-as-a-Judge evaluator scores in a single judge call. Each dimension has its own instruction and rating scale.
- numerical list(object)
- categorical
List<Evaluator
Categorical Scale Definition> - multi
Dimensional List<EvaluatorEvaluation Dimension> - The dimensions an LLM-as-a-Judge evaluator scores in a single judge call. Each dimension has its own instruction and rating scale.
- numerical
List<Evaluator
Numerical Scale Definition>
- categorical
Evaluator
Categorical Scale Definition[] - multi
Dimensional EvaluatorEvaluation Dimension[] - The dimensions an LLM-as-a-Judge evaluator scores in a single judge call. Each dimension has its own instruction and rating scale.
- numerical
Evaluator
Numerical Scale Definition[]
- categorical
Sequence[Evaluator
Categorical Scale Definition] - multi_
dimensional Sequence[EvaluatorEvaluation Dimension] - The dimensions an LLM-as-a-Judge evaluator scores in a single judge call. Each dimension has its own instruction and rating scale.
- numerical
Sequence[Evaluator
Numerical Scale Definition]
- categorical List<Property Map>
- multi
Dimensional List<Property Map> - The dimensions an LLM-as-a-Judge evaluator scores in a single judge call. Each dimension has its own instruction and rating scale.
- numerical List<Property Map>
EvaluatorReasoningConfiguration, EvaluatorReasoningConfigurationArgs
The reasoning configuration for reasoning models.
- Effort string
- The level of reasoning effort the model applies.
- Effort string
- The level of reasoning effort the model applies.
- effort string
- The level of reasoning effort the model applies.
- effort String
- The level of reasoning effort the model applies.
- effort string
- The level of reasoning effort the model applies.
- effort str
- The level of reasoning effort the model applies.
- effort String
- The level of reasoning effort the model applies.
EvaluatorStatus
| Name | Value |
|---|---|
| Active | ACTIVE |
| Creating | CREATING |
| Create | CREATE_FAILED |
| Updating | UPDATING |
| Update | UPDATE_FAILED |
| Deleting | DELETING |
| Name | Value |
|---|---|
| Evaluator | ACTIVE |
| Evaluator | CREATING |
| Evaluator | CREATE_FAILED |
| Evaluator | UPDATING |
| Evaluator | UPDATE_FAILED |
| Evaluator | DELETING |
| Value |
|---|
ACTIVE |
CREATING |
CREATE_FAILED |
UPDATING |
UPDATE_FAILED |
DELETING |
| Name | Value |
|---|---|
| Active | ACTIVE |
| Creating | CREATING |
| Create | CREATE_FAILED |
| Updating | UPDATING |
| Update | UPDATE_FAILED |
| Deleting | DELETING |
| Name | Value |
|---|---|
| Active | ACTIVE |
| Creating | CREATING |
| Create | CREATE_FAILED |
| Updating | UPDATING |
| Update | UPDATE_FAILED |
| Deleting | DELETING |
| Name | Value |
|---|---|
| ACTIVE | ACTIVE |
| CREATING | CREATING |
| CREATE_FAILED | CREATE_FAILED |
| UPDATING | UPDATING |
| UPDATE_FAILED | UPDATE_FAILED |
| DELETING | DELETING |
| Value |
|---|
ACTIVE |
CREATING |
CREATE_FAILED |
UPDATING |
UPDATE_FAILED |
DELETING |
EvaluatorType
The kind of evaluator resource. Builtin and ThirdParty are AWS-managed global evaluators that you reference but don't create; Custom, CustomCode, and CustomDerived are evaluators that you create.
| Name | Value |
|---|---|
| Builtin | Builtin |
| Third | ThirdParty |
| Custom | Custom |
| Custom | CustomCode |
| Custom | CustomDerived |
| Name | Value |
|---|---|
| Evaluator | Builtin |
| Evaluator | ThirdParty |
| Evaluator | Custom |
| Evaluator | CustomCode |
| Evaluator | CustomDerived |
| Value |
|---|
Builtin |
ThirdParty |
Custom |
CustomCode |
CustomDerived |
| Name | Value |
|---|---|
| Builtin | Builtin |
| Third | ThirdParty |
| Custom | Custom |
| Custom | CustomCode |
| Custom | CustomDerived |
| Name | Value |
|---|---|
| Builtin | Builtin |
| Third | ThirdParty |
| Custom | Custom |
| Custom | CustomCode |
| Custom | CustomDerived |
| Name | Value |
|---|---|
| BUILTIN | Builtin |
| THIRD_PARTY | ThirdParty |
| CUSTOM | Custom |
| CUSTOM_CODE | CustomCode |
| CUSTOM_DERIVED | CustomDerived |
| Value |
|---|
Builtin |
ThirdParty |
Custom |
CustomCode |
CustomDerived |
Tag, TagArgs
A set of tags to apply to the resource.
Package Details
- Repository
- AWS Native pulumi/pulumi-aws-native
- License
- Apache-2.0
We recommend new projects start with resources from the AWS provider.
published on Wednesday, Nov 18, 2026 by Pulumi