How Do I Replay Failed Events into the Event Hose?

This page may help you if:

  • You need to upgrade/downgrade RabbitMQ, and you do not want to lose any events.
  • RabbitMQ failed at some point, and you need to replay the events from when it was down.

The Event Hose is a part of Momentum 4.x responsible for taking events from the Platform, and passing them onto the Analytics and Webhooks components. See Chapter 1 "Components" in the Momentum 4 Manual . The Event Hose also underlies the Webhook functionality.

The Event Hose is implemented in several components in the Platform:

  • The event_hydrant module, which generates events.
  • The event_hose module, which picks up events from event_hyrdrant, and injects them into RabbitMQ.
  • RabbitMQ, a freely available message queue.

In the event that RabbitMQ is down or does not accept messages, the event_hose module will store event data in /var/log/ecelerity/event_hose_failed.rt on Platform nodes. The file event_hose_failed.rt, is a JLog file.

The failed events are not automatically picked up. The replay_event_hose_failed script must be manually run, to pick up the event_hose_failed.rt file.

The replay_event_hose_failed script can also be run on a flat file, with the format of one JSON blob per line. Blank lines are ignored. A flat file could be generated by configuring the event_hose module to write events to the log aggregation node using the clustering module.

This script only works with Momentum 4.1+. It has been tested with:

Momentum (4.1 HF4)

  • Do not run this script from cron, or multiple times concurrently. It has no internal locking, and may duplicate data if you run it multiple times.
  • Do not run this script as root. Run it as ecuser

Step-by-step guide (JLog)

  1. On each Platform node, check whether /var/log/ecelerity/event_hose_failed.rt exists.

  2. On each Platform node that has /var/log/ecelerity/event_hose_failed.rt, invoke the script.

    # Note: This may take some time. It limits itself to 500 events/second.
    # Recommendation: Run under screen or tmux.
    sudo -u ecuser /opt/msys/ecelerity/bin/replay_event_hose_failed
    # The JLog filename is optional when running replay_event_hose_failed.
    # You can specify an alternative filename, e.g.:
    sudo -u ecuser /opt/msys/ecelerity/bin/replay_event_hose_failed -l /some/other/path/event_hose_failed.rt
  3. The script will log its progress periodically to STDERR. When it's complete, it will log a summary of records inserted, and records read to STDERR.

  4. The script will checkpoint the JLog after each record. If it is aborted half-way through, it will pick up where it left off when you re-run it on the JLog.

Step-by-step guide (Flat file)

  1. On each Platform node with a flat file invoke the script:

    # Recommendation: Run under screen or tmux,
    # and capture log lines.
    # Filename required:
    sudo -u ecuser /opt/msys/ecelerity/bin/replay_event_hose_failed -l /path/to/flat/file
  2. Progress is reported in the same way as for JLogs.