EC2 Instance State Forwarding Configuration

Even after an AWS EC2 instance is terminated, the server can remain registered in the server project. When you enable OpenAgent's EC2 instance state forwarding, the instance states retrieved by EC2 State Exporter are sent to server projects, and the collection server automatically unregisters terminated instances. This guide is for administrators who run OpenAgent together with server monitoring in environments where EC2 instances are frequently created and terminated, such as with auto scaling.

How it works

AWS EC2 state query
  → EC2 State Exporter: /metrics
  → OpenAgent: forwards state to each registered server project
  → Collection server: identifies and unregisters servers in the project

EC2 State Exporter is a separate program from CloudWatch Exporter, which collects CPU and network metrics. If you installed only CloudWatch Exporter, you must also install EC2 State Exporter.
A single exporter queries the state of all EC2 instances in one region of the authenticated AWS account. You do not need to install it on every EC2 instance. It does not automatically discover multiple accounts or regions.
OpenAgent forwards the observed states to each registered server project. You do not need to separately map project codes or instance IDs to agents.
If an instance state is shutting-down (32) or terminated (48), all servers with the same instance-id in the project are unregistered. Even if an agent was reinstalled on the same EC2 instance and both the old and new servers remain, all of them are unregistered.
This feature cleans up only WhaTap server registrations. It does not stop or delete EC2 instances in AWS.

Prerequisites

The following versions are required.

Component	Version
OpenAgent	1.8.11 or later
EC2 State Exporter	0.1.0 or later
Collection server	3.1.10 or later

Caution

With OpenAgent 1.8.10 or earlier, the settings in this document alone do not work. Keep ec2_state_forwarding_enabled set to false until you confirm that the target collection server supports this feature. Older collection servers ignore the forwarded states, so the absence of errors in OpenAgent does not indicate support.

Add the following read permission to the IAM role of the EC2 instance where EC2 State Exporter runs. Skip this if it is already allowed.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ec2:DescribeInstances",
      "Resource": "*"
    }
  ]
}

The exporter uses the AWS SDK default credential chain. In environments that use an EC2 role, you do not enter an Access Key and Secret Key separately, and you do not put AWS keys in whatap.conf. The region is taken from SDK environment variables and profile settings first, and if not found, from EC2 metadata (IMDS).

Running EC2 State Exporter

Download and run the binary with the following commands. For arm64 hosts, replace amd64 in the path with arm64.

mkdir -p bin
curl -fLo bin/ec2_state_exporter https://repo.whatap.io/exporter/ec2_state_exporter/0.1.0/amd64/ec2_state_exporter
chmod +x bin/ec2_state_exporter
./bin/ec2_state_exporter --version
./bin/ec2_state_exporter

The default listen address is http://127.0.0.1:9107/metrics. If the region cannot be determined automatically or you want to query a different region, specify the --region argument.

./bin/ec2_state_exporter --region=ap-northeast-2

The commands above do not register a service or configure automatic start on reboot. In production, keep the process running with your service management method.

In another terminal, check the output with the following command.

curl --fail --show-error --max-time 15 http://127.0.0.1:9107/metrics

The output format is as follows. The instance ID is an example.

# HELP aws_ec2_instance_state EC2 lifecycle state code reported by DescribeInstances.
# TYPE aws_ec2_instance_state gauge
aws_ec2_instance_state{instance_id="i-0123456789abcdef0",region="ap-northeast-2",state_name="running"} 16

The metric value is the AWS instance state code.

Code	State
`0`	`pending`
`16`	`running`
`32`	`shutting-down`
`48`	`terminated`
`64`	`stopping`
`80`	`stopped`

Registering the scrape target

Add an ec2-state entry to the existing features.openAgent.targets list in scrape_config.yaml. Do not overwrite the existing file with the example below or create duplicate features or openAgent keys; merge the entry into the existing YAML structure.

scrape_config.yaml
features:
  openAgent:
    enabled: true
    targets:
      - targetName: ec2-state
        type: StaticEndpoints
        enabled: true
        endpoints:
          - address: "127.0.0.1:9107"
            path: "/metrics"
            scheme: "http"
            interval: "60s"
            timeout: "15s"

The recommended interval is 60s.
The exporter limits a full AWS query to 10 seconds, so set timeout to 15s, which is longer.
Keep existing CloudWatch Exporter scrape targets as they are.
If OpenAgent and the exporter run on different hosts or containers, run the exporter on a private address that OpenAgent can reach using the --web.listen-address argument, and change address to that address. The exporter has no TLS or HTTP authentication, so allow access only from OpenAgent with a firewall or security group.

Registering managed licenses

Add the following two options to whatap.conf. Register them with the feature disabled at first.

whatap.conf
ec2_state_forwarding_enabled=false
ec2_state_licenses=<SERVER_LICENSE_A>,<SERVER_LICENSE_B>

In ec2_state_licenses, enter the licenses of the server projects whose unregistration you want to manage. This value does not replace OpenAgent's own project license (license).

For multiple projects, enter them on one line separated by commas. Do not add a trailing comma or empty entries.
Project codes are resolved automatically from the licenses, so do not enter project codes separately.
The registered projects must be able to send data to the collection server that OpenAgent is connected to. You cannot specify a different collection server address for each project.

After confirming collection server support, change ec2_state_forwarding_enabled to true. This setting leads to actual server unregistration, so first make sure the managed projects are correct.

whatap.conf
ec2_state_forwarding_enabled=true
ec2_state_licenses=<SERVER_LICENSE_A>,<SERVER_LICENSE_B>

OpenAgent running on a regular host detects changes to whatap.conf, reloads it, and uses the new settings from the next scrape.

Configuration options

If you connect EC2 State Exporter with default values, you do not need to set anything other than ec2_state_forwarding_enabled and ec2_state_licenses.

Option	Type	Default	Description
`ec2_state_forwarding_enabled`	`boolean`	`false`	Whether to enable EC2 instance state forwarding
`ec2_state_licenses`	`string`	-	Licenses of the server projects whose unregistration is managed. Comma-separated, up to 64
`ec2_state_metric_name`	`string`	`aws_ec2_instance_state`	Instance state metric name
`ec2_state_instance_id_label`	`string`	`instance_id`	Instance ID label name
`ec2_state_state_name_label`	`string`	`state_name`	State name label name
`ec2_state_state_code_source`	`string`	`value`	Where to read the state code. `value` uses the metric value as the state code
`ec2_state_state_code_label`	`string`	`stateCode`	State code label name. Used only with the `label` method
`ec2_state_exporter_urls`	`string`	-	Allowlist of scrape target URLs whose states are processed. If not set, state metrics are processed from all scraped sources
`ec2_state_param_id`	`integer`	`1002`	Command ID sent to the collection server
`ec2_state_resend_interval`	`string`	`5m`	Interval for resending the same state. Duration format with a unit, such as `5m` or `30s`. Values without a unit such as `300`, `0`, negative values, or unparsable values cause a forwarding error

If you change ec2_state_metric_name to a non-default name, the label defaults change to instanceId, stateName, and stateCode, and the code source default changes to label.

ec2_state_* options can also be set as environment variables with the same names. If whatap.conf has a non-empty value, the file value takes precedence, and environment variable changes take effect only after the process restarts.
ec2_state_exporter_urls is a list of URLs, among already registered scrape targets, that are allowed to be processed. Setting this option alone does not create a new scrape job, and the URLs must exactly match the scheme, address, port, and path of the actual scrape URLs.

Note

Starting with OpenAgent 1.8.11, the default value of ec2_state_param_id changed from 56 to 1002. If your existing settings contain ec2_state_param_id=56, delete that line or change it to 1002. Explicitly set values are not corrected automatically or replaced with the previous ID when forwarding. If metric name or label settings from older examples remain, automatic defaults cannot be used, so remove unnecessary ec2_state_* settings unless you intentionally run a different exporter.

Kubernetes Operator environment configuration

If you installed OpenAgent with the WhaTap Operator, do not directly modify whatap.conf inside the Pod or the ConfigMap and Deployment created by the Operator. The Operator reverts manual changes. Configure settings in spec.features.openAgent of the WhatapAgent CR.

Item	CR field
EC2 State Exporter scrape target	`targets`
Feature activation, managed server project licenses	`envs`
OpenAgent image version	`imageVersion`. Specify only when the current version is earlier than 1.8.11

Deploy EC2 State Exporter to the cluster. Run it with the --web.listen-address=0.0.0.0:9107 argument so that OpenAgent in other Pods can reach it, and specify the region to query with --region, because EC2 metadata queries may be blocked in Pods. The exporter has no authentication, so allow access only from the OpenAgent Pod with a NetworkPolicy or similar. No official container image is provided for the exporter, so you must prepare an image containing the binary yourself.

In Pods where IMDS access is blocked, credentials for the node IAM role cannot be retrieved either. In EKS, grant the ec2:DescribeInstances permission with IRSA or EKS Pod Identity.

Back up the CR with the following commands, then edit it.

umask 077
kubectl get whatapagent whatap -o yaml > "whatapagent-backup-$(date +%Y%m%d%H%M%S).yaml"
kubectl edit whatapagent whatap

Add the following entries to spec.features.openAgent. Do not delete existing envs or targets entries; append to the lists. kubectl patch --type merge replaces the entire list, so do not use it if existing entries are present.
```
spec:
  features:
    openAgent:
      enabled: true
      envs:
        - name: ec2_state_forwarding_enabled
          value: "false"
        - name: ec2_state_licenses
          value: "<SERVER_LICENSE_A>,<SERVER_LICENSE_B>"
      targets:
        - targetName: ec2-state
          type: StaticEndpoints
          enabled: true
          endpoints:
            - address: "ec2-state-exporter.whatap-monitoring.svc:9107"
              path: "/metrics"
              scheme: "http"
              interval: "60s"
```
- Environment variable values are strings, so enclose them in quotes, such as "false" and "true".
- To avoid keeping licenses in the CR as plain values, create a Secret and reference it with valueFrom.secretKeyRef instead of value.
- After confirming collection server support, change ec2_state_forwarding_enabled to "true"; the Operator updates the Deployment and restarts the Pod. If you change only the Secret value, the Pod does not restart automatically, so delete the OpenAgent Pod to recreate it.

Check the result with the following commands.

kubectl -n whatap-monitoring rollout status deploy/whatap-open-agent
kubectl -n whatap-monitoring logs deploy/whatap-open-agent --since=10m | grep -E 'EC2_STATE_'

The namespace whatap-monitoring and CR name whatap in the example are Operator defaults. Change them to match your installation.

Verifying operation

Check that /metrics of EC2 State Exporter returns HTTP 200 and aws_ec2_instance_state samples. If there are no instances in the account and region, there may be no samples even with HTTP 200.
Check that the OpenAgent log has no EC2_STATE_CONFIG, EC2_STATE_INPUT, or EC2_STATE_SEND errors.
When states are put into the send queue, the INFO log EC2_STATE_FORWARD ... locally queued=<count> from <scrape address> is written. This count is the number accepted by OpenAgent's local queue, not the number received by the collection server or successfully unregistered.
Check whether servers were actually unregistered by viewing the agent list of the target server project again.

OpenAgent forwards states at the following times.

Situation	Behavior
First observed state	Forwarded in that scrape
State code or name changed	Forwarded in the scrape that detects the change
Same state repeated	Not forwarded for `ec2_state_resend_interval` (default 5 minutes) after the last forwarding
Resend interval elapsed with the same state	Forwarded again in the next scrape
Query failed, removed from the list	Previous state is not resent and is not treated as `terminated`

State changes are not received immediately as AWS events; they are forwarded when the next scrape detects the change. Terminated instances that have already disappeared from AWS query results cannot be forwarded.

Troubleshooting

Symptom	Check
Connection refused on port 9107	Check the exporter process, listen address and port, and container network
`EC2_STATE_EXPORTER: AWS region is unavailable`	Check the region in the AWS environment or profile, or IMDS access. Specify `--region` if needed
HTTP 503, `EC2_STATE_EXPORTER: collection failed`	Check AWS credentials, the `ec2:DescribeInstances` permission, and the region and network
HTTP 503, `EC2_STATE_EXPORTER: scrape already in progress`	Check for duplicate scrape targets or concurrent scrapes on the same exporter
`EC2_STATE_CONFIG` log	Check for missing licenses, typos, empty entries, and invalid optional settings
`EC2_STATE_INPUT` log	Check the metric format, required labels, and whether the code matches the state name
`EC2_STATE_SEND` log	Rejected by the local send queue. Check the collection server connection and queue status. Retried in the next scrape
Collected but not forwarded	Check the OpenAgent version (1.8.11 or later), the whatap.conf actually read, whether the feature is enabled, and whether old settings remain
`locally queued` log exists but the server remains	Check the collection server version, the target project license, and whether the server's `instance-id` matches. `running` and `stopped` states are not unregistered

Disabling the feature

Change whatap.conf as follows. In an Operator environment, change the value in the CR envs to "false".

whatap.conf
ec2_state_forwarding_enabled=false

After the settings are reloaded, state forwarding stops from the next scrape. Regular metric collection continues, and data already in the send queue is not canceled. To also stop AWS queries, disable the ec2-state scrape target in scrape_config.yaml and stop the exporter process.

How it works​

Prerequisites​

Running EC2 State Exporter​

Registering the scrape target​

Registering managed licenses​

Configuration options​

Kubernetes Operator environment configuration​

Verifying operation​

Troubleshooting​

Disabling the feature​