Evaluation

Creates evaluation of a conversation model. Note - this resource’s API doesn’t support deletion. When deleted, the resource will persist on Google Cloud even though it will be deleted from Pulumi state.

Create Evaluation Resource

new Evaluation(name: string, args: EvaluationArgs, opts?: CustomResourceOptions);
@overload
def Evaluation(resource_name: str,
               opts: Optional[ResourceOptions] = None,
               conversation_model_id: Optional[str] = None,
               display_name: Optional[str] = None,
               evaluation_config: Optional[GoogleCloudDialogflowV2EvaluationConfigArgs] = None,
               location: Optional[str] = None,
               name: Optional[str] = None,
               project: Optional[str] = None)
@overload
def Evaluation(resource_name: str,
               args: EvaluationArgs,
               opts: Optional[ResourceOptions] = None)
func NewEvaluation(ctx *Context, name string, args EvaluationArgs, opts ...ResourceOption) (*Evaluation, error)
public Evaluation(string name, EvaluationArgs args, CustomResourceOptions? opts = null)
public Evaluation(String name, EvaluationArgs args)
public Evaluation(String name, EvaluationArgs args, CustomResourceOptions options)
type: google-native:dialogflow/v2:Evaluation
properties: # The arguments to resource properties.
options: # Bag of options to control resource's behavior.

name string
The unique name of the resource.
args EvaluationArgs
The arguments to resource properties.
opts CustomResourceOptions
Bag of options to control resource's behavior.
resource_name str
The unique name of the resource.
args EvaluationArgs
The arguments to resource properties.
opts ResourceOptions
Bag of options to control resource's behavior.
ctx Context
Context object for the current deployment.
name string
The unique name of the resource.
args EvaluationArgs
The arguments to resource properties.
opts ResourceOption
Bag of options to control resource's behavior.
name string
The unique name of the resource.
args EvaluationArgs
The arguments to resource properties.
opts CustomResourceOptions
Bag of options to control resource's behavior.
name String
The unique name of the resource.
args EvaluationArgs
The arguments to resource properties.
options CustomResourceOptions
Bag of options to control resource's behavior.

Evaluation Resource Properties

To learn more about resource properties and how to use them, see Inputs and Outputs in the Architecture and Concepts docs.

Inputs

The Evaluation resource accepts the following input properties:

ConversationModelId string
DisplayName string

Optional. The display name of the model evaluation. At most 64 bytes long.

EvaluationConfig Pulumi.GoogleNative.Dialogflow.V2.Inputs.GoogleCloudDialogflowV2EvaluationConfigArgs

Optional. The configuration of the evaluation task.

Location string
Name string

The resource name of the evaluation. Format: projects//conversationModels//evaluations/

Project string
ConversationModelId string
DisplayName string

Optional. The display name of the model evaluation. At most 64 bytes long.

EvaluationConfig GoogleCloudDialogflowV2EvaluationConfigArgs

Optional. The configuration of the evaluation task.

Location string
Name string

The resource name of the evaluation. Format: projects//conversationModels//evaluations/

Project string
conversationModelId String
displayName String

Optional. The display name of the model evaluation. At most 64 bytes long.

evaluationConfig GoogleCloudDialogflowV2EvaluationConfigArgs

Optional. The configuration of the evaluation task.

location String
name String

The resource name of the evaluation. Format: projects//conversationModels//evaluations/

project String
conversationModelId string
displayName string

Optional. The display name of the model evaluation. At most 64 bytes long.

evaluationConfig GoogleCloudDialogflowV2EvaluationConfigArgs

Optional. The configuration of the evaluation task.

location string
name string

The resource name of the evaluation. Format: projects//conversationModels//evaluations/

project string
conversation_model_id str
display_name str

Optional. The display name of the model evaluation. At most 64 bytes long.

evaluation_config GoogleCloudDialogflowV2EvaluationConfigArgs

Optional. The configuration of the evaluation task.

location str
name str

The resource name of the evaluation. Format: projects//conversationModels//evaluations/

project str
conversationModelId String
displayName String

Optional. The display name of the model evaluation. At most 64 bytes long.

evaluationConfig Property Map

Optional. The configuration of the evaluation task.

location String
name String

The resource name of the evaluation. Format: projects//conversationModels//evaluations/

project String

Outputs

All input properties are implicitly available as output properties. Additionally, the Evaluation resource produces the following output properties:

CreateTime string

Creation time of this model.

Id string

The provider-assigned unique ID for this managed resource.

SmartReplyMetrics Pulumi.GoogleNative.Dialogflow.V2.Outputs.GoogleCloudDialogflowV2SmartReplyMetricsResponse

Only available when model is for smart reply.

CreateTime string

Creation time of this model.

Id string

The provider-assigned unique ID for this managed resource.

SmartReplyMetrics GoogleCloudDialogflowV2SmartReplyMetricsResponse

Only available when model is for smart reply.

createTime String

Creation time of this model.

id String

The provider-assigned unique ID for this managed resource.

smartReplyMetrics GoogleCloudDialogflowV2SmartReplyMetricsResponse

Only available when model is for smart reply.

createTime string

Creation time of this model.

id string

The provider-assigned unique ID for this managed resource.

smartReplyMetrics GoogleCloudDialogflowV2SmartReplyMetricsResponse

Only available when model is for smart reply.

create_time str

Creation time of this model.

id str

The provider-assigned unique ID for this managed resource.

smart_reply_metrics GoogleCloudDialogflowV2SmartReplyMetricsResponse

Only available when model is for smart reply.

createTime String

Creation time of this model.

id String

The provider-assigned unique ID for this managed resource.

smartReplyMetrics Property Map

Only available when model is for smart reply.

Supporting Types

GoogleCloudDialogflowV2EvaluationConfig

datasets List<Property Map>

Datasets used for evaluation.

smartComposeConfig Property Map

Configuration for smart compose model evalution.

smartReplyConfig Property Map

Configuration for smart reply model evalution.

GoogleCloudDialogflowV2EvaluationConfigResponse

datasets List<Property Map>

Datasets used for evaluation.

smartComposeConfig Property Map

Configuration for smart compose model evalution.

smartReplyConfig Property Map

Configuration for smart reply model evalution.

GoogleCloudDialogflowV2EvaluationConfigSmartComposeConfig

MaxResultCount int

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

AllowlistDocument string

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.

MaxResultCount int

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

AllowlistDocument string

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.

maxResultCount Integer

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument String

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.

maxResultCount number

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument string

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.

max_result_count int

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlist_document str

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.

maxResultCount Number

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument String

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.

GoogleCloudDialogflowV2EvaluationConfigSmartComposeConfigResponse

AllowlistDocument string

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.

MaxResultCount int

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

AllowlistDocument string

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.

MaxResultCount int

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument String

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.

maxResultCount Integer

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument string

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.

maxResultCount number

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlist_document str

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.

max_result_count int

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument String

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.

maxResultCount Number

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

GoogleCloudDialogflowV2EvaluationConfigSmartReplyConfig

MaxResultCount int

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

AllowlistDocument string

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.

MaxResultCount int

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

AllowlistDocument string

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.

maxResultCount Integer

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument String

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.

maxResultCount number

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument string

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.

max_result_count int

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlist_document str

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.

maxResultCount Number

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument String

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.

GoogleCloudDialogflowV2EvaluationConfigSmartReplyConfigResponse

AllowlistDocument string

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.

MaxResultCount int

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

AllowlistDocument string

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.

MaxResultCount int

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument String

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.

maxResultCount Integer

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument string

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.

maxResultCount number

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlist_document str

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.

max_result_count int

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument String

The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.

maxResultCount Number

The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

GoogleCloudDialogflowV2InputDataset

Dataset string

ConversationDataset resource name. Format: projects//locations//conversationDatasets/

Dataset string

ConversationDataset resource name. Format: projects//locations//conversationDatasets/

dataset String

ConversationDataset resource name. Format: projects//locations//conversationDatasets/

dataset string

ConversationDataset resource name. Format: projects//locations//conversationDatasets/

dataset str

ConversationDataset resource name. Format: projects//locations//conversationDatasets/

dataset String

ConversationDataset resource name. Format: projects//locations//conversationDatasets/

GoogleCloudDialogflowV2InputDatasetResponse

Dataset string

ConversationDataset resource name. Format: projects//locations//conversationDatasets/

Dataset string

ConversationDataset resource name. Format: projects//locations//conversationDatasets/

dataset String

ConversationDataset resource name. Format: projects//locations//conversationDatasets/

dataset string

ConversationDataset resource name. Format: projects//locations//conversationDatasets/

dataset str

ConversationDataset resource name. Format: projects//locations//conversationDatasets/

dataset String

ConversationDataset resource name. Format: projects//locations//conversationDatasets/

GoogleCloudDialogflowV2SmartReplyMetricsResponse

AllowlistCoverage double

Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].

ConversationCount string

Total number of conversations used to generate this metric.

TopNMetrics List<Pulumi.GoogleNative.Dialogflow.V2.Inputs.GoogleCloudDialogflowV2SmartReplyMetricsTopNMetricsResponse>

Metrics of top n smart replies, sorted by TopNMetric.n.

AllowlistCoverage float64

Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].

ConversationCount string

Total number of conversations used to generate this metric.

TopNMetrics []GoogleCloudDialogflowV2SmartReplyMetricsTopNMetricsResponse

Metrics of top n smart replies, sorted by TopNMetric.n.

allowlistCoverage Double

Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].

conversationCount String

Total number of conversations used to generate this metric.

topNMetrics List<GoogleCloudDialogflowV2SmartReplyMetricsTopNMetricsResponse>

Metrics of top n smart replies, sorted by TopNMetric.n.

allowlistCoverage number

Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].

conversationCount string

Total number of conversations used to generate this metric.

topNMetrics GoogleCloudDialogflowV2SmartReplyMetricsTopNMetricsResponse[]

Metrics of top n smart replies, sorted by TopNMetric.n.

allowlist_coverage float

Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].

conversation_count str

Total number of conversations used to generate this metric.

top_n_metrics Sequence[GoogleCloudDialogflowV2SmartReplyMetricsTopNMetricsResponse]

Metrics of top n smart replies, sorted by TopNMetric.n.

allowlistCoverage Number

Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].

conversationCount String

Total number of conversations used to generate this metric.

topNMetrics List<Property Map>

Metrics of top n smart replies, sorted by TopNMetric.n.

GoogleCloudDialogflowV2SmartReplyMetricsTopNMetricsResponse

N int

Number of retrieved smart replies. For example, when n is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model.

Recall double

Defined as number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply divided by number of queries with at least one smart reply. Value ranges from 0.0 to 1.0 inclusive.

N int

Number of retrieved smart replies. For example, when n is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model.

Recall float64

Defined as number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply divided by number of queries with at least one smart reply. Value ranges from 0.0 to 1.0 inclusive.

n Integer

Number of retrieved smart replies. For example, when n is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model.

recall Double

Defined as number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply divided by number of queries with at least one smart reply. Value ranges from 0.0 to 1.0 inclusive.

n number

Number of retrieved smart replies. For example, when n is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model.

recall number

Defined as number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply divided by number of queries with at least one smart reply. Value ranges from 0.0 to 1.0 inclusive.

n int

Number of retrieved smart replies. For example, when n is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model.

recall float

Defined as number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply divided by number of queries with at least one smart reply. Value ranges from 0.0 to 1.0 inclusive.

n Number

Number of retrieved smart replies. For example, when n is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model.

recall Number

Defined as number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply divided by number of queries with at least one smart reply. Value ranges from 0.0 to 1.0 inclusive.

Package Details

Repository
https://github.com/pulumi/pulumi-google-native
License
Apache-2.0