Dynamic Workflow Management and Monitoring Using DDS

Scientific workflows executing over long periods on unreliable computing infrastructure need both fault tolerance and recovery capabilities with minimal manual intervention. This work innovates by integrating workflow management with distributed monitoring infrastructure using publish-subscribe middleware to decouple workflow components. The hierarchical management structure enables both local rapid response to failures and global coordination of recovery actions across the distributed system.

This paper extends previous workflow management frameworks with dynamic workflow management and monitoring using Data Distribution Services middleware. The work presents a hierarchical framework for managing scientific workflows through workflow managers coordinating job execution and participant managers tracking individual task execution. It integrates monitoring of infrastructure resources and workflow status to enable both fault detection and workflow recovery through stopping and restarting failed workflow portions.

The framework successfully manages scientific workflows with distributed monitoring that tracks both infrastructure status and workflow execution. Results demonstrate effective fault detection and workflow recovery through stopping failed portions and restarting from known checkpoints. The integration with Data Distribution Services enables scalable monitoring across large computing clusters without centralized bottlenecks.