Today's Updates: Workflow Improvements & Monitoring Enhancements
What developments have taken place today?
This commit refactors the location of a workflow directory, moving it from /var/www/www.remotedaemon.com/workflow to /var/www/remotedaemon.com/workflow. This change simplifies the path and reduces potential ambiguity, improving maintainability.
This commit introduces a WorkflowDirectory configuration option for several functions and structs within the application. By passing the workflow directory path as an argument to initHandler, Checker, and NewGeneratorChecker, these components gain explicit access to the location of workflow definitions, enhancing their flexibility and enabling consistent handling across different deployments.
This commit integrates a comprehensive workflow state management system into the lighthouse checks. The LightHouseConsumer now utilizes WorkflowStateManagerInterface and WorkflowLoaderInterface for loading workflows from YAML files, determining steps, dependencies, and execution conditions. The loadWorkflowState() function handles workflow loading and validation, returning errors if issues are encountered. This provides better control and reliability in the execution of lighthouse checks.
Specifically:
* A sample lighthouse.yaml file has been added to define workflow steps for lighthouse checks.
* The LighthouseGenerator now accepts a workflowDirectory parameter to specify the location of workflow definition files.
* The ProcessData() method in the consumer calls loadWorkflowState() to retrieve workflow state before performing lighthouse checks and marks the step as completed.
* The LighthouseGeneratorData struct includes a workflowRunId field, passed to the workflow state manager.
This commit updates project dependencies to include OpenTelemetry tracing and metrics instrumentation for Node.js applications. This addition incorporates packages for OpenTelemetry SDKs, exporters, and instrumentations, facilitating distributed tracing and metric collection. A placeholder message reminds developers to update when a mobile application is available.
This commit upgrades the URL downloader component, integrating it seamlessly into the registry initialization process. The Config struct now includes a script_ping_checker field to store the path to the ping checker script. A new instance of helper_url_downloader.OSCommandRunner is created in the InitRegistry function using this configuration setting. Comprehensive tests, including success, error handling (invalid JSON and script errors), and mock testing, have been added to ensure the downloader’s functionality and reliability.
This commit refines the workflow for checking website certificate expiration dates. It introduces a new workflow that includes generating a message and consuming it to verify the certificate’s expiration date. The description of this workflow step has been updated from “Fetch data from an external API” to “Download data from URL”.
Finally, this commit incorporates WorkflowStateManager and WorkflowLoader into various application components via dependency injection. These changes are reflected in api.NewApiHandler, the main function, CertificateConsumer, CronFunctions.Certificate, WebsiteController.checkCertificate, and GeneratorCertificate.GenerateOne. The loadWorkflowState() function is now used to check workflow status before generating messages or performing other operations. Additionally, the WorkflowRunRepository.DeleteOld function has been modified to delete old workflow runs based on age limits (5 hours for ping checkers, 50 hours for all others).
This commit improves the helper_url_downloader package by enhancing its ability to handle redirects during URL downloads. The code now uses curl’s -L option to automatically follow redirects and extracts detailed metrics about each step in the redirect chain using curl’s -w flag. This includes HTTP status codes, response times, DNS lookup times, connection times, and more. The information is outputted as a JSON string, facilitating programmatic processing of download results. Robust error handling has been implemented to address common issues such as DNS failures, connection problems, and TLS errors.
In summary, these changes represent significant improvements in workflow management, monitoring capabilities, tracing instrumentation, and URL downloading reliability, leading to a more robust and maintainable application.
workflow monitoring url downloader open telemetry certificate checking