EEM
Posted inEnterprise Network Network automation

Cisco Embedded Event Manager (EEM)

The Cisco Embedded Event Manager or Cisco EEM is a software component of Cisco IOS, IOS-XR, and NX-OS that provides real-time network event detection and onboard automation.

EEM allows you to automate tasks, perform minor enhancements and create workarounds and can makes life easier for network operators by tracking and classifying events that take place on a network device and providing actions options for those events.

Event detectors and actions

EEM uses event detectors and actions to provide notifications of those events.

EEM event detectors

IOS Embedded Event Manager supports more than 20 event detectors that are highly integrated with different Cisco IOS Software components to trigger actions in response to network events. They can be:

  • SNMP – Monitoring SNMP objects, events and notifications (SNMP traps) of the device.
  • Syslog – Responds to various syslog messages, allowing for matching on regular expressions of the content of the message.
  • Counter(s) – Monitoring and responding to interface counters when cross threshold settings.
  • CLI – Screening CLI input for a regular expression match or when a special parser character is entered, like Tab or “?” (question mark).
  • Timers – Countdown, watchdog and CRON.
  • IP SLA and NetFlows events.
  • Routing – When a route entry changes in the Routing Information Base (RIB).
  • RPC – Provides the ability to invoke EEM policies from outside the router over an encrypted connection using SSH. The RPC event detector uses Simple Object Access Protocol (SOAP) data encoding for exchanging XML-based messages. This event detector can be used to run EEM policies and then receive output in a SOAP XML-formatted reply.
  • OIR (online insertion and removal) – When a card is removed od inserted.
  • Resource – When the Embedded Resource Manager (ERM) reports an event.
  • Enhanced Object Tracking Event Detector (EOT) – When the status of a tracked object changes.
  • Another EEM policy – Another EEM policy may publishes an event.
  • None – This event detector is to test the EEM script/applet using “event manager run” command.

EEM Actions

EEM Actions can be:

  • Executing a Cisco IOS command-line interface (CLI) command.
  • Generating a CNS event for upstream processing by Cisco CNS devices.
  • Setting or modifying a named counter.
  • Switching to a secondary processor in redundant hardware configuration.
  • Requesting system information when an event occurs.
  • Sending a short e-mail message.
  • Manually running an EEM policy.
  • Publishing an application-specific event.
  • Reloading the Cisco IOS software.
  • Generating an SNMP trap.
  • Generating prioritized syslog messages.
  • Reading the state of a tracked object.
  • Setting the state of a tracked object.

EEM applets vs EEM Scripts (TCL) vs Shell Policies (IOS.sh)

There are currently three native policy engines within Cisco EEM:

  1. EEM Applets – Supported since EEM version 1.0, these policies are specified and defined in the configuration of the device and were designed to allow simple interface into Cisco’s EEM feature.
    Only one event configuration command is allowed within an applet configuration, or up to six events with correlation since EEM 2.4, and multiple action are allowed.
    Use the command: show event manager policy registered to display a list of registered applets.
  2. TCL Scripts – Supported since EEM version 2.0, these policies are defined in separate files stored locally on the device and specified (registered) by adding a single configuration command.
    Tcl allows for more complex policies and some EEM features like timer subscribers are (currently) only supported in EEM Tcl.
  3. IOS.sh Shell Policies – Supported since EEM version 3.2, these policies are defined in separate files stored locally on the device and specified (registered) by adding a single configuration command.  Shell policies utilize the IOS shell feature.  Support for shell policies is limited to some switching platforms.

In this article, I will only talk about EEM Applets. TCL Scripts and IOS.sh need both another dedicated post.

EEM Versions

The EEM feature is constantly evolving.  EEM 1.0 was first introduced in 12.3(4)T, but did not offer much in the way of power.  Now, with EEM 3.x and 4.0 in recent versions of IOS, policies can interact with the CLI, send messages between devices, process complex show command output, intercept SNMP messages, manipulate files, and much more.

EEM 2.4 to today

To know what features of EEM your device has, you must first know what version it supports.  If you are not running one of the main IOS version trains, this information may not be obvious.  In fact, it may not be that easy to determine the version of EEM.  Prior to EEM 2.4, there was no command to determine the version of EEM.  As of EEM 2.4, a new command: show event manager version was introduced which shows the version of EEM as well as all the associated Event Detectors:

test-router#show event manager version 
Embedded Event Manager Version 4.00
Component Versions:
eem: (dev10)1.1.5
eem-gold: (rel1)1.0.2
eem-call-home: (rel2)1.0.5
(...)

If your device does not support this command, you do not have EEM 2.4.

EEM 1.0

To find out if your device supports at least EEM 1.0, check for the presence of the show event manager ? command.  That is, if you run show event manager, then type a ‘?‘, you should see command syntax.  If so, then your device supports EEM 1.0 at the very least.

EEM 2.0

EEM 2.0 was only ported to the 12.2(26)S and only existed in the 12.2S train.  As of 12.2(30)S and higher, those platforms support at least version 2.1.

EEM 2.1

To determine if your device supports EEM 2.1, you should check to make sure it supports the show event manager environment command. If this command is supported, then the device supports EEM 2.1 at the very least.

EEM 2.2

The primary feature added to EEM 2.2 was Enhanced Object Tracking (EOT) support.  However, even for devices which would otherwise support EEM 2.2, they may not support EOT.  For example, Catalyst 6500s running 12.2(33)SXH code support EEM 2.3, but do not support EOT/EEM integration.  Catalyst 3K switches (e.g. 3560, 3750, etc.) as of 12.2(40)SE support EEM 2.4, but do not support EOT/EEM integration.  However, for other devices, verify that the event track command is supported when configuring an EEM applet:

test-router# config t
Enter configuration commands, one per line. End with CNTL/Z.
test-router(config)#event manager applet test-version
test-router(config-applet)#event track ?
 <1-1000> Tracking object number

EEM 2.3

The primary user-facing feature of EEM 2.3 was the support for running interactive (e.g. clear counters, crypto key zeroize rsa, etc.).  Therefore, to determine if a device supports EEM 2.3, check to make sure the pattern keyword is supported for the CLI action in applet configuration mode:

test-router(config)#event manager applet test-version
test-router(config-applet)#action test cli ?
 command CLI command
 pattern response pattern

These techniques work for all IOS and IOS-XE devices.

EEM Built-In Environment Variables Used in EEM Applets

EEM built-in environment variables are a subset of the Cisco-defined environment variables and the built-in variables are available to EEM applets only. The built-in variables can be read-only or read-write and may apply to one specific event detector or to all event detectors.

You can find the list of the Cisco built-in environment variables in the Cisco IOS Network Management Configuration Guide online. Here are some examples:

_event_id – Unique number that indicates the ID for this published event.
_event_type – Type of event.
_cli_msg – The fully expanded message that triggered the CLI event.
_track_number – The number of the tracked object.
_track_state – The state of the tracked object; down or up.
_interface_name – The name of the interface to be monitored.
_interface_parameter – The name of the interface counter to be monitored.
_interface_value – A value with which the current interface counter value is compared.
_snmp_oid – The SNMP object ID that caused the event to be published.
_syslog_msg – The syslog message that caused the event to be published.
_email_server – A SMTP mail server used to send e-mail.
_email_to – The address to which e-mail is sent.
_email_from – The address from which e-mail is sent.
_email_cc – The address to which the e-mail is copied.
Etc…

How to Write EEM Policies Using the Cisco IOS CLI

The process to create an EEM policy is quite simple:

  1. Create the EEM policy
  2. Define the triggered event(s)
  3. Define the action(s)
  4. End

Example of simple EEM policies

Syslog Event detector

Imagine that I have a router with an interface that should never be shutdown:

test(config)#event manager applet INT_NO_SHUT
test(config-applet)#event syslog pattern "Interface GigabitEthernet0/0/1, changed state to administratively down"
test(config-applet)#action 1.0 cli command "enable"
test(config-applet)#action 2.0 cli command "config t"
test(config-applet)#action 3.0 cli command "interface gi0/0/1"
test(config-applet)#action 4.0 cli command "no shutdown"
test(config-applet)#action 5.0 cli command "end"
test(config-applet)#action 6.0 cli command "who"
test(config-applet)#action 7.0 mail server "10.10.10.10" to ".noc@my-company.com." from ".EEM@cisco.com." subject ".Interface_gi0/0/1_SHUT." body "Current users $_cli_result"

So here, the triggered event is the syslog message and the action is first we do a “no shut” and then we send an email to the NOC team with the name of the logged-in user at this moment.

First, I can verify my applet with the command:

test#show event manager policy registered 
No. Class Type Event Type Trap Time Registered Name
1 applet user syslog Off Thu Jan 25 10:40:12 2018 INT_NO_SHUT
 pattern {Interface GigabitEthernet0/0/1, changed state to administratively down}
 maxrun 20.000
 action 1.0 cli command "enable"
 action 2.0 cli command "config t"
 action 3.0 cli command "interface gi0/0/1"
 action 4.0 cli command "no shutdown"
 action 5.0 cli command "end"
 action 6.0 cli command "who"
 action 7.0 mail server "10.10.10.10" to ".noc@my-company.com." from ".EEM@cisco.com." subject ".Interface_gi0/0/1_SHUT." body "Current users $_cli_result"

Then, to see the applet in action, let’s enable the debugging of EEM cli and shutdown the interface (I removed unnecessary debug messages):

test#debug event manager action cli 
Debug EEM action cli debugging is on
test#config t
Enter configuration commands, one per line. End with CNTL/Z.
test(config)#int gi0/0/1
test(config-if)#shut
test(config-if)#
%LINK-5-CHANGED: Interface GigabitEthernet0/0/1, changed state to administratively down
DEBUG(cli_lib) : : CTL : cli_open called.
DEBUG(cli_lib) : : IN : test>enable
DEBUG(cli_lib) : : OUT : test#
DEBUG(cli_lib) : : IN : test#config t
DEBUG(cli_lib) : : OUT : Enter configuration commands, one per line. End with CNTL/Z.
DEBUG(cli_lib) : : OUT : test(config)#
DEBUG(cli_lib) : : IN : test(config)#interface gi0/0/1
DEBUG(cli_lib) : : OUT : test(config-if)#
DEBUG(cli_lib) : : IN : test(config-if)#no shutdown
DEBUG(cli_lib) : : OUT : test(config-if)#
DEBUG(cli_lib) : : IN : test(config-if)#end
%SYS-5-CONFIG_I: Configured from console by on vty1 (EEM:INT_NO_SHUT)
DEBUG(cli_lib) : : OUT : test#
DEBUG(cli_lib) : : IN : test#who
DEBUG(cli_lib) : : OUT : Line User Host(s) Idle Location
DEBUG(cli_lib) : : OUT : 866 vty 0 admin idle 00:00:02 10.254.223.168
DEBUG(cli_lib) : : OUT : *867 vty 1 idle 00:00:00 EEM:INT_NO_SHUT
DEBUG(cli_lib) : : OUT : 
DEBUG(cli_lib) : : OUT : Interface User Mode Idle Peer Address
DEBUG(cli_lib) : : OUT : 
DEBUG(cli_lib) : : OUT : test#

CLI event detector

Another example, to disable the use of the reload command:

test#config t
Enter configuration commands, one per line. End with CNTL/Z.
test(config)#event manager applet NO-RELOAD
test(config-applet)#event cli pattern "reload" sync no skip yes occurs 1
test(config-applet)#action 1.0 syslog msg "$_cli_msg has been disabled."
test(config-applet)#exit
test(config)#exit
test#
test#reload
test#
Jan 25 11:36:26.030 MET: %HA_EM-6-LOG: NO-RELOAD: reload has been disabled.
test#

About the event cli pattern

The event cli pattern defines the event criteria to initialize the EEM applet. The arguments are:

  • The pattern itself – Between “”, we can use normal text and also Reg Exp.
  • sync <yes/no> – Specifies if the policy should be executed synchronously before the CLI commands executes or not.
  • skip <yes/no> – Indicates if the CLI commands should be executed or not.
  • occurs <1-32> – Indicates the number of occurrences before the EEM applet is triggers. One is the default.

Multi-Event Correlation

This is supported since EEM 2.4. We can correlate up to 6 events with “and”, “or” and “andnot” operations. For example:

test(config)#event manager applet TEST
test(config-applet)#event tag e1 syslog pattern ".*UPDOWN.*Ethernet1/0.*"
test(config-applet)#event tag e2 syslog pattern ".*UPDOWN.*Ethernet1/1.*"
test(config-applet)#event tag e3 syslog pattern ".*UPDOWN.*FastEthernet0/0.*"
test(config-applet)#trigger occurs 1
test(config-applet-trigger)#correlate event e1 or event e2 or event e3
test(config-applet-trigger)#attribute tag e1 occurs 1
test(config-applet-trigger)#attribute tag e2 occurs 1
test(config-applet-trigger)#attribute tag e3 occurs 1
test(config-applet-trigger)#exit
test(config-applet)#action 1.0 reload
test(config-applet)#end

Cron and environment variables

You can schedule the execution of an applet, like with cron in Linux. And you can define environment variables. Here is an example to backup the current configuration to a TFTP server, every day at midnight:

test(config)#event manager environment filename current_config
test(config)#event manager environment tftploc tftp://10.10.10.10/
test(config)#
test(config)#event manager applet DAILY_BACKUP
test(config-applet)#event timer cron cron-entry "0 0 * * *"
test(config-applet)#action 1.0 cli command “enable”
test(config-applet)#action 2.0 cli command “conf t”
test(config-applet)#action 3.0 cli command “file prompt quiet”
test(config-applet)#action 4.0 cli command “end”
test(config-applet)#action 5.0 cli command “copy start $tftploc$filename”
test(config-applet)#action 6.0 cli command “conf t”
test(config-applet)#action 7.0 cli command “no file prompt quiet”
test(config-applet)#action 8.0 cli command “end”

The file prompt quiet command is to disable the confirmation questions from the IOS.

Now you can see the real firepower of EEM!

 Maxrun

By default, all EEM policies should run to completion within 20 seconds.  This 20 second timer is known as the maxrun timer.  It is not always possible to accomplish everything your policy needs to do in 20 seconds, however.  For policies requiring a longer run time, the maxrun timer can be increased when configuring your event registration line.  The maxrun time is specified in seconds[.milliseconds].  The following examples will set the maxrun timer to 60 seconds:

test(config-applet)#event syslog pattern "Configured from console" maxrun 60

In addition to the maxrun timer, certain event detectors have a default timer.  The default timer is the number of seconds (and milliseconds) before the default action will be taken.  The default timer exists for CLI, RPC, SNMP Object, and None event detectors.  In the case of the CLI event detector, the default action is to execute the intercepted command.  For RPC, SNMP Object, and None, the default timer specifies how long to wait before the event is published up the stack (i.e. to the next consumer).  If not specified at policy registration time, the default timer is 30 seconds.  It is configured just like the maxrun timer using the default keyword.

References

Embedded Event Manager (EEM): http://www.cisco.com/go/eem

Cisco EEM Best Practices: https://supportforums.cisco.com/t5/network-infrastructure-documents/cisco-eem-best-practices/ta-p/3127596

The Tweeting router: https://supportforums.cisco.com/t5/network-infrastructure-documents/tweet-from-ios/ta-p/3122172

 

 

Did you like this article? Please share it…
Tags:

1 Comment

  1. benslimane abderrahim
    Reply
    February 13, 2022,

    Thank you very much, interesting explanation.

Leave a Reply

Your email address will not be published. Required fields are marked *