https://www.codexworker.com personal homepage of backend developer writing random stuff Sun, 15 Jun 2025 08:49:30 +0000 https://www.codexworker.com/post/193/a What developments have taken place today? This collection of commits addresses several key areas of the application, focusing on performance optimization, database schema improvements, URL management, internationalization, and search functionality. The first set of changes significantly improves the retrieval of discounted product data from Elasticsearch. A caching mechanism is implemented to reduce load on the database and accelerate subsequent requests for frequently accessed categories. The underlying query logic has been updated to utilize internal category IDs instead of Google Category IDs, streamlining queries and enhancing performance. Logging improvements provide clearer insights into the process. A second set of changes centers around standardizing slug generation for products and stages. This includes a new database column (`slug`) in the `products_stage` table, alongside a `generateSlug` function that handles diacritics removal and hyphenation for consistent URL formatting. These updates are reflected across various templates and code locations, ensuring proper slug usage throughout the application. Further improvements were made to URL structure and routing. The application’s CDN image server and API endpoints have been updated with new port numbers; RabbitMQ endpoint URLs have also been adjusted. Additionally, a new outbound product link route (`/product-exit/:id`) has been introduced. Internationalization efforts have been expanded through the introduction of `LoadTranslations` function for loading translation files from JSON format, alongside modifications to the `Registry` struct and template functions to support multiple languages dynamically. The use of the `T` function allows templates to display localized content using a consistent approach, providing fallback mechanisms for missing translations. Finally, search functionality has been enhanced by incorporating translations utilizing a new `T` function to replace hardcoded strings with translated versions across various UI elements including product details, results pages, and pagination. URL generation for category links now incorporates slugs alongside database IDs, while the Elasticsearch indexing includes a `slug` field as a keyword type. Overall, these changes represent a series of important improvements that enhance performance, maintainability, user experience (particularly regarding internationalization), and overall system stability. Mon, 09 Jun 2025 23:59:59 +0000 https://www.codexworker.com/post/192/a What developments have taken place today? Today’s changes focus on enhancing category management, improving the visual design of product detail pages, streamlining data import processes, and adding tracking for product clicks. **Category Management Enhancements:** * A `GetCategoryBySlugAndMarketID` function was introduced in the `CategoryRepository`, enabling retrieval of categories by slug and market ID. This provides a more efficient way to find specific categories based on their URL-friendly identifiers. * The `Category` struct now includes a `slug` field, ensuring that slugs are stored alongside category data for consistent referencing. * Slug generation logic has been integrated into the `SearchOne` method of the `CategoryRepository`, allowing searches by slug in addition to existing criteria. Furthermore, the `Save` method now automatically generates and stores slugs when saving new categories. **Product Detail Page Redesign:** * A new stylesheet (`product_detail.scss`) has been implemented to modernize the visual presentation of product detail pages. This includes defining CSS variables for consistent styling, implementing a container structure with responsive layout adjustments, enhancing breadcrumb styling, and creating a visually distinct card layout for product information. The design now includes improved image display, a clear price hierarchy (original & sale prices), availability indicators, and enhanced description/specification formatting. **Data Import Process Improvements:** * A new `import_loop_sleep_duration_seconds` configuration setting has been added to control the delay between import loops within the `levnobot_cz` Docker container. * The `Search` function in `DataSourceRepository` now supports using an `OrderByClause` for more complex SQL ordering, with basic SQL injection protection implemented. * Additional commands were added to run category import scripts and update data within MariaDB databases, streamlining the import process. **Product Click Tracking:** * A new `product_clicks` table has been created in the database to track product clicks, storing relevant information such as product ID, link, IP address, user agent, and timestamps. * The `ProductClickRepository` was introduced to manage this data, including methods for saving, searching, and deleting click records. * A new outbound link handler (`/product/:id/out`) has been implemented on the product detail page to track clicks using the repository. **Configuration Updates:** * Configuration files have been updated to utilize settings specific to the levnobot.cz domain, replacing configurations intended for a different deployment environment. These changes collectively improve data management, enhance user experience, and provide valuable insights into website traffic and product engagement. The updates represent an important step toward more robust and efficient operation of the `levnobot_cz` system. Sun, 08 Jun 2025 23:59:59 +0000 https://www.codexworker.com/post/191/a What developments have taken place today? This commit significantly refactors brand filtering logic for category detail pages, moving from relying on previously selected brands to dynamically retrieving all distinct brands for the current category and its descendants using Elasticsearch. A new helper function, `SliceContainsString`, was added to facilitate this process. The system now also includes the count of each brand within search results, providing users with a clearer understanding of product availability by brand. Furthermore, a caching layer utilizing Redis has been introduced and integrated into core services for improved data retrieval efficiency. This involves creating a new `appcache` package, implementing `RedisCache`, and updating the registry to leverage the cache. Finally, several category detail pages have been updated to incorporate filter options for brands and minimum discount levels, enhancing user control over product search results. The changes to category display on the homepage are important because they streamline product listing by utilizing a CSS grid layout with three columns for better organization and responsiveness. This enhances visual appeal and improves the overall user experience. These updates to the website’s header and footer templates represent a significant design improvement, replacing older HTML structures with modern, responsive templates generated through Go templates. The new templates incorporate styling elements like dark backgrounds, centered content, hover effects on links, and rounded corners for pagination, contributing to a more polished and professional user interface. The modifications related to determining ancestor categories are crucial for data transformation accuracy. By directly retrieving category IDs and paths from the database repository – rather than relying on potentially outdated or incorrect category paths – the system ensures that product counts are calculated correctly and consistently. The inclusion of error handling during ID parsing adds a layer of robustness, preventing processing interruptions due to invalid input. The update to product details display, specifically conditional brand name rendering based on availability, improves clarity by presenting only relevant information to the user. The fallback image mechanism ensures that a placeholder is displayed when the actual product image link is unavailable, maintaining a consistent visual experience. The changes concerning category filtering and URL construction provide users with enhanced control over their search results. The addition of filter options for brands and minimum discount levels allows for more targeted searches, while dynamically generated URLs ensure that filters are preserved across page navigations. These features contribute significantly to improved usability and user satisfaction. Finally, the modifications to the category listing template's sorting functionality introduce a `SortOptionViewModel` struct and associated logic for selecting and applying sort options via URL parameters. This allows users to easily customize the order in which products are displayed, further tailoring their browsing experience. Sat, 07 Jun 2025 23:59:59 +0000 https://www.codexworker.com/post/190/a What developments have taken place today? Today's work focused on enhancing search functionality, improving market data management, and refining category display within product details. The core changes involve integrating Elasticsearch aggregation for rollup counts of categories, updating the database schema to accommodate market-specific information (market ID), and streamlining the process of displaying related categories in search results. Furthermore, a new controller has been introduced for handling autocomplete suggestions for products. Here’s a breakdown of the key developments: * **Enhanced Search with Category Rollup Counts:** The search handler now leverages Elasticsearch aggregations to retrieve category IDs and their counts (rollup counts), presenting product counts associated with each category in the search results. This provides users with a clearer overview of related categories and improves search relevance. * **Market Data Integration:** A new `market_id` column has been added to both the `category` and `datasource` tables, establishing links between categories and markets. The `MarketRepository` has also been updated to handle these new fields, enabling market-specific searches and data retrieval. This lays the groundwork for supporting multiple markets within the system. * **Autocomplete Suggestions:** A new `AutocompleteController` has been implemented to provide users with real-time suggestions for products as they type in the search bar. This enhances the user experience by anticipating user needs and streamlining the product discovery process. * **Category Path Retrieval Improvements**: The category path retrieval logic was refined, ensuring more accurate categorization based on Google Category IDs and incorporating error handling to prevent issues if files are missing. * **Template Updates:** The `search_results.go.tmpl` template has been updated to display the new category rollup counts effectively. The HTML template for displaying categories in a market dashboard was also updated to include pagination controls and dynamic rendering of category information. In summary, these changes significantly improve search capabilities by providing more contextualized results, streamline market-specific data management, and enhance the user experience through autocomplete suggestions. They are important because they address key requirements for supporting multi-market functionality, improving search relevance, and providing a better overall product discovery experience. Fri, 06 Jun 2025 23:59:59 +0000 https://www.codexworker.com/post/189/a What developments have taken place today? Today’s commits focused on database schema enhancements, API improvements, frontend styling, and Elasticsearch integration. Key changes include: * **Database Schema Updates:** Added a `brands` table with brand aliases and implemented a unique constraint on brand names to prevent duplicates. Also added indexes for faster queries based on `brand_id`. * **API Product Search Endpoint:** Introduced a new `/api/v1/products/search` endpoint to allow clients to search for products by various criteria (text, category, brand, price range). This includes pagination and filtering logic. * **Homepage Enhancements:** Updated the homepage to display featured categories and discounted products from Elasticsearch, significantly improving the user experience. This involved creating new view models and updating the controller logic to fetch data from Elasticsearch. * **Product Detail Page Redesign:** Improved the product detail page with a more modern layout, including styling for fonts, colors, images, and discounts. The code now dynamically displays discount information alongside original prices. * **Elasticsearch Integration Improvements:** Enhanced integration with Elasticsearch by adding a `ProductESRepository` interface to manage data synchronization between the database and Elasticsearch. This included updating queries to utilize Elasticsearch efficiently and handle errors gracefully. * **CSV Import Enhancements**: Added support for importing products from XML files into the database staging table, handling data source management and including error logging. In summary, these changes collectively improve product discovery, presentation, and data management within the application – enhancing both the user experience and backend functionality. Thu, 05 Jun 2025 23:59:59 +0000 https://www.codexworker.com/post/188/a What developments have taken place today? Today’s commit activity primarily focuses on refactoring the WebSocket server architecture, enhancing security, and improving message handling capabilities. Here's a breakdown of the changes: * **WebSocket Server Core Refactor:** A significant overhaul of the WebSocket server code has occurred, involving the removal of several files related to user data retrieval, email verification, channel management, Redis commands, and WSCommand structs. This suggests a move towards a more modular and streamlined design. * **New Authentication & Authorization:** The introduction of `WebsocketAuthenticate` and associated methods implements WebSocket authentication using tokens from a token repository, incorporating validation logic for token integrity. This enhances security by restricting access to authorized users only. * **Connection Management Enhancements:** The creation of the `UserConnection` struct introduces a structured approach to managing WebSocket connections, including handling ping requests for connection persistence and integrating with mutexes for concurrent operations. * **Message Handling Improvements:** New `WSCommand` structs were introduced to standardize incoming messages. The `handleMessage` function now parses and routes these commands to appropriate handlers, supporting new command types like `Subscribe`, `NewMessage`, and `MessageDelivered`. Helper functions (`NewWS...`) streamline message creation with built-in validation. * **Rate Limiting Implementation:** A rate limiting middleware (`RateLimitMiddleware`) has been added to prevent abuse by restricting requests from each user within a specified time window, utilizing Redis for tracking request counts. * **Redis Integration:** The server now leverages Redis for storing channel subscriptions using the `UserChannelRepository`. Additionally, there were updates related to the build process: * **Docker Compose Update:** The Docker Compose command has been updated to reflect the project name change from “remotedaemon” to "levnobot". * **Build Script Modifications:** The Go build script now utilizes `-tags netgo`, `-ldflags '-s -w -extldflags "-static"'`, and `-buildvcs=false` flags, likely for optimizing executable size and building statically. The `cdn` program is built with slightly different flags to omit compression. * **Product Repository Changes:** The product repository schema has been refined to include internal primary keys (BIGINT UNSIGNED) and a unique constraint on the combination of merchant product ID and data source ID, ensuring data integrity. The `SaveProduct`, `SearchProduct` and `DeleteProduct` functions have also been updated to utilize these changes. Finally, there was a substantial refactoring of Google Product Feed XML parsing: * **Enhanced Parsing Capabilities:** The code now supports more fields in the feed (e.g., `program_name`, `catalog_name`), utilizes two-pass parsing for detailed error analysis, and introduces helper functions (`getStructTags`) to automate tag discovery. * **Data Structure Updates**: The core structure for parsing the feed (`Feed` and `Entry`) has been expanded to include more data fields. In essence, these changes represent a significant architectural shift toward a more robust, secure, and scalable WebSocket server with improved message handling capabilities, alongside refinements in database schema management and build processes, as well as enhancements to Google Product Feed parsing. Wed, 04 Jun 2025 23:59:59 +0000 https://www.codexworker.com/post/187/a What developments have taken place today? This commit implements an update to subproject identification systems. Previously, subprojects were referenced using older commit hashes. These hashes have now been replaced with newer, distinct identifiers for each subproject. The changes involve updating the strings used to represent these identifiers within the system. This ensures consistency and avoids potential issues associated with outdated references. These updates improve data integrity by utilizing current commit information and reducing the risk of relying on obsolete references. Updating identifiers streamlines tracking and management of related subprojects, enhancing overall system stability and accuracy. In summary, this update is important because it solidifies the identification of subprojects through accurate and consistently updated references, which contributes to a more reliable and maintainable software ecosystem. Wed, 04 Jun 2025 23:59:59 +0000 https://www.codexworker.com/post/186/a What developments have taken place today? This commit introduces functionality for displaying notifications when other users are actively writing in a chat channel. It modifies a core function to verify if a message’s channel matches the active one and that the sender is not the current user, triggering a visual indicator on the screen to alert the user. A new system has been implemented to allow users to begin typing messages within a chat channel. This involves creating a `WSCommandWritingMessage` object containing relevant details like channel ID, user ID, typed message, and a unique request ID. This message is then sent via WebSocket and handled by an event handler (`writing_on_channel`) in the ChatApplication class, which processes these events and notifies registered handlers. The UI DOM has been updated to reflect typing activity. Specifically, the `handleStartTyping` method now constructs and sends this new `WSCommandWritingMessage`, while a visual indicator (e.g., "writing...") is displayed and automatically disappears after a 3-second period of inactivity. This provides real-time feedback to users during typing sessions. These changes are important because they significantly enhance the chat interface by providing clarity about concurrent activity, enabling immediate user input, and improving overall responsiveness through visual cues. Tue, 03 Jun 2025 23:59:59 +0000 https://www.codexworker.com/post/184/a What developments have taken place today? The device registration process has been enhanced to improve efficiency and reduce database load by prioritizing updates over creating new devices when a matching record already exists in the database. This is achieved through an updated `UpdateTokenForUserAndDevice` function within a mock repository, coupled with improved testing including unit tests for both successful updates and simulated repository errors. The mock repository itself has been refined to accurately represent the intended functionality. Within the `ApiHandlerChannel` struct, a debugging print statement has been added to the `UpdateLastSeenMessage` function to aid in troubleshooting user and channel interactions. The `UpdateUnreadCount` function now explicitly calls the `userChannelRepository` with the correct arguments and prints the associated user channel ID for debugging purposes – ensuring consistency with its usage in test cases. A new unit test, `TestCreateChannel_UpdateLastSeenMessage`, has been added to verify the functionality of these changes by simulating interactions between various repositories and asserting that expected methods are called with accurate parameters. These improvements collectively streamline device management, enhance debuggability within channel operations, and ensure robust testing for critical data updates— ultimately improving system stability and facilitating easier identification and resolution of potential issues related to user and channel interactions. Mon, 02 Jun 2025 23:59:59 +0000 https://www.codexworker.com/post/183/a What developments have taken place today? This commit introduces an enhanced mechanism for managing unread message counts within user channels. Specifically, it adds a dedicated `UpdateUnreadCount` method to the `UserChannelRepository`. This method utilizes an SQL query to directly update the `unread_count` field in the `user_channel` table based on the provided user channel ID and desired count. The `UserChannelRepositoryInterface` has been updated to mandate this new method, ensuring consistent functionality across all repository implementations. The core improvement lies in providing a precise and reliable way to synchronize the unread message counts with changes to a user's last seen message or the arrival of new messages within a channel. Previously, updates to these events might not have automatically reflected in the unread count; now, this change guarantees accurate tracking without requiring manual intervention or complex logic elsewhere in the system. These changes are important because they directly address potential inconsistencies in user experience by ensuring that users always see an up-to-date representation of their channel's unread message status. Maintaining accurate and timely notifications is crucial for effective communication, and this update strengthens the overall reliability and usability of the application’s messaging features. Sat, 31 May 2025 23:59:59 +0000 https://www.codexworker.com/post/185/a What developments have taken place today? This commit updated the webpage's footer section. Specifically, it added a link to a status page hosted at remotedaemon.com, providing users with access to system monitoring information. The copyright text within the footer was also synchronized with the user’s preferred language setting for improved localization and consistency. Furthermore, all links within the footer were modified to be white in color, regardless of whether they have been previously visited, enhancing visual clarity and usability. These changes improve the user experience by providing readily available system status information and ensuring consistent branding across different languages. The standardized link color scheme contributes to better readability and accessibility within the footer area. Overall, these updates enhance website functionality and presentation, contributing to a more polished and informative user interface. Fri, 30 May 2025 23:59:59 +0000 https://www.codexworker.com/post/180/a What developments have taken place today? * **Numerical Value Adjustment:** A numerical value within the system was incremented by 123. This likely improves a calculation or data point dependent on this value. * **Website Template Update:** The website template has been modified to incorporate new fields for `start_date` and `end_date` into the breadcrumb navigation. A main section with a container and content area has also been added, alongside a breadcrumb component dynamically populated from page configuration data. This enhances user experience by providing clearer temporal context within the website’s structure. * **Scheduled Task Enhancement:** A scheduled task responsible for deleting old workflow runs now includes error handling. Instead of failing silently upon encountering an issue during deletion, it logs the error message to the console. This improves system reliability and facilitates debugging. These changes were important because they enhance both website functionality (providing improved breadcrumb navigation) and operational stability (improving the robustness of scheduled cleanup tasks). The adjustments contribute to a more user-friendly and reliable system overall. Fri, 30 May 2025 23:59:59 +0000 https://www.codexworker.com/post/181/a What developments have taken place today? Today’s commits focused on enhancing internationalization (i18n) capabilities across several areas of the website, improving responsiveness, and refining user interface elements. Key changes include: * **Navigation Redesign:** The website's navigation has been overhauled with a full-width layout featuring a hamburger menu for mobile devices and improved accessibility features like tappable links and aria attributes. This provides a more modern and responsive experience across different screen sizes. * **Internationalization Enhancements (Czech & German):** Significant translation work has been completed, adding Czech and German support for various strings including login labels, placeholders, button text, and website descriptions. This expands the website’s reach to a wider audience. The use of translation keys ensures consistency and ease of future localization updates. * **Time Difference Formatting:** The `humanReadableTimeDifference` function has been updated to correctly handle instrumental case usage for “ago” in Czech and Slovenian, resulting in grammatically correct time displays. * **Route Handling & Language Prefixes:** The website’s routing logic has been modified to incorporate request paths into language dropdown options, enhancing flexibility when dealing with multilingual applications. Project and post URLs now use a new `generateRoute` function for consistent URL generation based on the user's selected language. Default prefixes (e.g., /en/) have been added for clarity. * **CSS Compilation & Styling:** The CSS build process has been updated to utilize a newer version of `cssnano`, and minor styling adjustments have been made to improve visual appeal, including rounded corners, shadows, and padding. In summary, these changes represent a substantial investment in improving the website's accessibility and usability for international users, while also refining its responsiveness and overall design aesthetic. The enhanced i18n support paves the way for future expansion into additional languages, contributing to a more inclusive and globally-oriented user experience. Thu, 29 May 2025 23:59:59 +0000 https://www.codexworker.com/post/182/a What developments have taken place today? This commit enhances the API’s `GetChannel` functionality by providing clients with more comprehensive channel data. Previously, a request for channel details would return an empty object; now, the API retrieves associated participants and their individual details. The changes involve several key improvements: Firstly, it incorporates user authentication to ensure requests are authorized. Secondly, it efficiently locates channel information based on its ID. Thirdly, it fetches participant data – specifically usernames and last login times – from a dedicated user repository. Fourthly, all this information is consolidated into a new response structure named `ChannelWithMeta`, combining the core channel details with a list of participants. Finally, the encoding process has been updated to accurately serialize this enriched `ChannelWithMeta` object as JSON using a defined schema. These enhancements provide clients with a significantly improved understanding of channels, allowing for more informative and user-friendly displays of channel membership and associated information. The changes were important because they increased the usefulness and richness of the API response, directly benefiting client applications by providing essential contextual data regarding channel members. Wed, 28 May 2025 23:59:59 +0000 https://www.codexworker.com/post/179/a What developments have taken place today? A chat screen's automatic scrolling functionality has been significantly improved to reliably reach the bottom of message lists upon initial loading. The updated logic incorporates checks for scrollability, proximity to the end of content, and channel message presence before attempting a jump to the bottom. Retries are implemented with delays, re-evaluating success criteria after exhausting attempts, and only updating the UI if a successful jump is confirmed. A minor code adjustment involved inserting the numerical value 123 into the codebase – this change appears to be a simple constant addition without functional impact. Another small adjustment added the numerical value 123 to the codebase; it's a non-functional modification. The `ChatScreen` widget has been updated to display sender names accurately by introducing a `currentChannelData` variable. This variable contains channel information, including user IDs and nicknames. The `_buildMessageBubble` function now utilizes this data to determine whether to use the current user's nickname or retrieve the sender’s nickname from the channel data, falling back to displaying the user ID if no nickname is available. Finally, project dependencies have been updated to include `http_parser` as a direct dependency and introduce the `mime` package. Development-related dependency lists have also been revised. These changes collectively address usability issues by ensuring accurate sender name display in chat screens and enhance the reliability of automatic scrolling functionality, improving the overall user experience and stability of the application. Wed, 28 May 2025 23:59:59 +0000 https://www.codexworker.com/post/178/a What developments have taken place today? This commit addressed several areas of the application, including navigation, data presentation, dependency management, and logging. Specifically, changes were made to URL routing and display, metric retrieval and visualization, domain list refreshing, notification handling, and dependency updates. The URL routing was revised to utilize a more descriptive path format (`/urls/:urlId/ping-history`) for accessing ping history data, incorporating route parameters and enhanced navigation with associated URL information. Metric retrieval now includes a `metricType` parameter for filtering results, and tapping metric records triggers navigation to a dedicated "Ping History" screen that displays the full URL alongside relevant details. The domain list refresh functionality was improved with a visual indicator of loading status (circular progress) and error handling, ensuring data consistency upon updates. Notification logging has been enhanced through added print statements for debugging purposes. Dependency management received an update including SHA256 hashes to ensure correct package resolution, as well as version adjustments. Finally, the display of local notifications was updated with a new notification channel ID and improved user experience via relative time formatting. These changes collectively enhance usability by streamlining navigation, improving data presentation clarity, increasing system stability through dependency management, and providing valuable debugging information. The updates contribute to a more responsive and informative user experience across various application features. Tue, 27 May 2025 23:59:59 +0000 https://www.codexworker.com/post/176/a What developments have taken place today? Today’s commits focused on enhancing notification delivery, improving repository performance and security, integrating a push notification module, streamlining container management, refining application configuration, and bolstering Lighthouse/Jaeger integration. Specifically, the team implemented functionality for sending push notifications with user nicknames and channel IDs, updated the `UserChannelRepository` for increased efficiency and security through parameterized queries and logging, introduced a modular push notification system utilizing an interface and mock tests, improved container startup and shutdown procedures including clean restarts, switched to a production configuration file for the application, adjusted Redis server addresses, and optimized Lighthouse/Jaeger interaction via configurable URLs and temporary file handling. Furthermore, Android notifications now prioritize high-importance delivery with custom sound settings, while also adjusting the test message content. These changes collectively represent significant improvements in user engagement, system stability, data integrity, operational efficiency, and overall application maintainability. The enhanced notification system directly benefits users by providing timely alerts, while the repository optimizations contribute to database performance and security. The modular push notification architecture promotes flexibility and future-proofing of the system, enabling easier integration with different notification services. Improvements to container management streamline deployment processes, ensuring consistent and reliable operation, and configuration updates enhance application stability and adaptability. The overall impact is a more robust, responsive, and user-friendly application. Tue, 27 May 2025 23:59:59 +0000 https://www.codexworker.com/post/177/a What developments have taken place today? Today’s commits focused on enhancing user interaction, streamlining backend operations, and improving data handling within the chat interface. Several changes were made to improve the reliability of notifications, manage WebSocket connections, and ensure consistent message display. A `FocusNode` was added to the chat screen's text input field to simulate scrolling down after typing a message, using keyboard animations for a smoother user experience. The addition of a `userApiProvider` simplifies access to user data through authentication with a RiverpodAuth object and an ApiClient. Furthermore, a new `currentUserProvider` fetches user details asynchronously, handling potential errors and displaying loading states effectively. The `LeftMenu` widget now consumes this provider to display user information. Significant improvements were made to the WebSocket client implementation, introducing a dedicated `AppWebSocketClient` class with command handlers for authentication, subscriptions, and message reception. This client utilizes dependencies like `web_socket_channel`, and ensures robust error handling during connection attempts. The changes also included updating the application's Git reference to a new version of the remote repository. The chat interface received updates, including hardcoded initial messages to provide demonstration content and ensuring chronological sorting of displayed messages via API integration. A new mechanism was implemented for handling message loading states and potential errors within the `ChatScreen`, providing visual indicators (loading indicator and error message) to the user. Finally, improvements were made to the application’s splash screen logic, enabling proper authentication status handling and navigation based on session availability. In summary, these changes collectively contribute to a more responsive, reliable, and feature-rich chat experience by improving user interaction, simplifying data access, enhancing real-time communication capabilities, and ensuring consistent message presentation. Mon, 26 May 2025 23:59:59 +0000 https://www.codexworker.com/post/175/a What developments have taken place today? Several commits focused on API improvements, error handling, logging, and schema refinements related to channel management and user data retrieval. These changes aim to enhance the stability, debuggability, and clarity of the codebase. The first commit updated version identifiers for a subproject, ensuring consistency across references. This is an important maintenance task to prevent integration issues. A subsequent commit introduced comprehensive logging enhancements within several API methods (`ApiHandlerChannel.ListChannels` and `ApiHandlerChannel.ListUsersInChannel`). `fmt.Println` statements were added to display search parameters and data before database operations, facilitating debugging. Additionally, error messages from database queries are now printed, providing immediate feedback on potential issues. Further changes refined the handling of user channels and associated data in the API. The operation name was changed from `ListUserChannels` to `ListUsersInChannel` to accurately reflect its functionality – retrieving users within a channel. A new error type, `ListUsersInChannelBadRequest`, was introduced for specific validation errors related to sorting parameters. Validation logic was implemented to restrict allowed sort values and return appropriate errors if invalid inputs are provided. Error handling was improved with more robust checks during channel searches and user-channel saving, returning informative error messages when failures occur. Schema updates simplified the `ApiResponseCountChannel` structure by consolidating pagination information into a single `data` field and refining associated JSON fields. The `Channel` struct was modified to include additional details like name, type, and privacy settings, streamlining data representation. Corresponding changes were applied to decode and encode functions, ensuring proper data handling. Constants `PrivacyPrivate` and `PrivacyPublic` were introduced for representing channel privacy levels enabling more granular filtering of channels based on this setting. Finally, a commit updated the Git repository URL for the remote daemon client TypeScript project. These combined improvements enhance code clarity, debuggability, error reporting, and overall API robustness. The changes represent a significant step towards a more reliable and maintainable system. Mon, 26 May 2025 23:59:59 +0000 https://www.codexworker.com/post/167/a Hello! Code change information from bot here! During week 2025-05-19 - 2025-05-25 i made 74 commits across 8 of my projects. In total i created 39252 new lines and removed 20578 lines. Mon, 26 May 2025 00:01:43 +0000 https://www.codexworker.com/post/174/a What developments have taken place today? A new builder pattern has been implemented for constructing both `UserChannel` and `Channel` structs. This involves introducing a `UserChannelBuilder` struct to encapsulate the construction process, alongside a constructor (`NewUserChannelBuilder`) and fluent setters for configuring various fields such as `ChannelType`, `ChannelName`, `ChannelLastMessageId`, `ChannelLastMessage`, and `ChannelPrivacy`. These setters correctly handle optional fields using SQL `NullTime` and `sql.NullString` types. A `Build()` method is included to generate a fully populated `UserChannel` struct, and the `CreateChannel` function has been updated to utilize this builder pattern for creating and saving channel records, including setting the `Created` time. Additionally, the code now incorporates the channel type, name, and last message ID when storing user channels in the database. These changes significantly improve code organization by separating construction logic from data definitions, enhancing readability and maintainability. The use of a builder pattern reduces redundancy and promotes consistency in how `UserChannel` structs are created, particularly when dealing with optional fields. Integrating these improvements into the `CreateChannel` function streamlines the process of creating and saving channel records, ensuring that all necessary information is properly handled and stored within the database. The implementation of this builder pattern represents a crucial enhancement to data management practices, leading to more robust, understandable, and easily maintainable code for both channels and user channels. Sun, 25 May 2025 23:59:59 +0000 https://www.codexworker.com/post/173/a What developments have taken place today? The `MyChatAppPage` widget has been enhanced to improve channel loading management. A new stateful widget type (`ConsumerStatefulWidget`) and state class (`_MyChatAppPageState`) were implemented to manage an `_isLoadingChannels` boolean, displaying a progress indicator during loading and presenting error messages upon failure. The `_loadInitialData` function now utilizes Riverpod for API calls, handling errors gracefully with retry functionality and informative user messages via the scaffold. The initial data simulation has been adjusted to populate message arrays successfully after channel loading and uses the currently active channel ID for this purpose. A delay was increased before simulating new messages and updating unread counts to accommodate longer channel load times. The `ChatScreen` has been updated to utilize a new `ChannelWithMeta` data structure, providing richer channel information (name and metadata) instead of the previous `DisplayMessage`. Logic for selecting active channels now relies on `.channel.id`, prioritizing display names over standard channel names. Improved error handling includes catching unexpected exceptions, displaying user-friendly messages in the UI, and logging stack traces for debugging. The `_currentlyActiveChannelId` variable has been updated to use the same `.channel.id`. Finally, the application's routing was streamlined by replacing a dynamic `RegisterScreen` with a constant version within the navigation structure. A new Riverpod provider was introduced to manage communication with the Channels API using authentication and an `ApiClient`, ensuring consistent access across the application. These changes collectively improved channel loading reliability, enhanced user experience through clearer feedback and error handling, optimized data presentation for the chat screen, and standardized API access throughout the application. The improvements address potential issues related to slow loading times, API failures, and inconsistent data representation, leading to a more stable and intuitive user interface. Sun, 25 May 2025 23:59:59 +0000 https://www.codexworker.com/post/172/a What developments have taken place today? This commit introduced a chat functionality to the application, enabling users to communicate through an in-app chat screen. The implementation included adding a new route `/chat` utilizing `GoRouter`, associating it with a dedicated page (`MyChatAppPage`), and incorporating a corresponding "Chat" menu item within the left navigation drawer for easy access. Furthermore, significant improvements were made to the project’s dependency management. Specifically, updates were applied to several packages: `connectivity_plus` was upgraded from version 5.0.0 to 6.1.4 and its platform interface from 1.0.0 to 2.0.1, providing enhanced connectivity features. A new dependency, `nm`, version 0.5.0, was added to support broader system integration. Crucially, the plugin registration code was updated to ensure correct integration of the `ConnectivityPlusWindowsPlugin` on Windows platforms, guaranteeing reliable access to connectivity services. These changes are important because they significantly enhance user experience by providing direct in-app communication and improve the application's overall functionality through streamlined dependency management and platform compatibility updates. Sat, 24 May 2025 23:59:59 +0000 https://www.codexworker.com/post/168/a What developments have taken place today? This commit focused on improving client generation, adding new API functionality for domain management, optimizing message loading within channels, and updating visual assets for the status page. Specifically, the Dart client now utilizes the `dart` package instead of `dio`, with code cleanup implemented to address issues in generated API files. Three programs – chat, api, and cdn – were compiled using `go build` incorporating upx compression, improving their size efficiency. Simultaneously, SSH permissions were adjusted on a remote server for key executables, and systemd service configurations were introduced for automatic startup and restart of the chat and admin programs with defined logging. A new API endpoint (`/api/v1/domain/{id}`) was added to allow domain status page updates, alongside validation, authorization checks, and appropriate HTTP response codes (204, 400, 401, 403, 404, 500). Client-side logic was incorporated to enable and disable the status page for domains via an API call. Furthermore, message loading within channels now fetches existing messages from an API when a channel is selected, caching them for subsequent access, reducing redundant calls. Finally, the status page logo image was updated to `blue_daemon_logo.webp`, along with associated HTML and CSS adjustments to enhance layout presentation and minimal style added to incidents section. These changes collectively represent improvements by streamlining client generation, expanding API capabilities for domain management, optimizing application performance through efficient message loading, enhancing visual clarity on the status page, and establishing robust system administration practices for service reliability. Sat, 24 May 2025 23:59:59 +0000 https://www.codexworker.com/post/166/a What developments have taken place today? Several updates and new features were implemented across various components of the Remotedaemon system, focusing on enhanced monitoring capabilities and improved status page functionality. Primarily, these changes involve package updates, API enhancements for incident management, a redesigned status page with detailed website information and ping data visualization, and foundation work for a dedicated incident repository. Specifically, several npm packages were updated to their latest versions including `electron-to-chromium`, `remote-daemon-client-typescript`, `dunder-proto`, `emittery`, and `regjsparser`. These updates likely incorporate bug fixes, performance improvements, and security patches. The update of `remote-daemon-client-typescript` also included logging enhancements related to incidents and domains on the status page. Significant work was done to introduce a new API endpoint for creating and listing website incidents. A new `ApiHandlerIncident` struct and associated functions (`NewApiHandlerIncident`, `CreateIncident`, `ListIncidents`) were added, enabling users to manage incident records programmatically. The OpenAPI specification was updated to reflect these changes, ensuring proper documentation of the new endpoints. Initial error handling for the `ListIncidents` method was implemented, enhancing data retrieval reliability. A completely redesigned status page was created with a new HTML template (`status_page.go.tmpl`) and accompanying CSS styling and JavaScript integration. The updated status page now displays website information, recent incidents (filtered by enabled status pages), an uptime graph visualizing ping data over the last 24 hours, and key service metrics. The implementation leverages session data for authentication and API calls to fetch relevant information dynamically. Furthermore, a new repository for managing incident data was established. This included defining an `IncidentRepository` interface with associated implementations, along with an `Incident` struct and related functions (`Save`, `SearchOne`, `Search`) for interacting with the database. The `Save` method now incorporates logic to classify incidents based on error types during the ping process, enriching the data captured. Finally, modifications were made to the database schema by adding a `status_page_enabled` column (tinyint(1)) to the `website` table, defaulting to NULL and populating it with new website entries created today. These updates collectively establish a more robust foundation for monitoring system health and managing incidents within the Remotedaemon environment. These changes are important because they provide enhanced visibility into system performance, streamline incident management processes, and improve the overall user experience through a more informative status page. The introduction of dedicated incident storage and retrieval mechanisms lays the groundwork for future enhancements like alerting and automated remediation. Fri, 23 May 2025 23:59:59 +0000 https://www.codexworker.com/post/164/a What developments have taken place today? Two commits have been made to improve data retrieval and project statistics reporting. The first commit adjusted the sorting order within a data retrieval function to prioritize rows where `visible` is true. It now sorts by `week_year`, `lines_added`, and then `id`. Adding `gps.total_commit_count` elevates rows with more commits, enhancing relevance in prioritized results. The second commit modified a database query for project statistics, shifting the primary sorting criteria to whether a project has a visible website (`visible DESC`). Subsequent sorting is now based on `week_year`, `lines_added`, and `id`, ensuring consistent and detailed reporting within visibility groups. These changes are important because they optimize data presentation by prioritizing relevant information—first by visibility status, then by key metrics like commit count and project statistics – leading to more insightful and actionable results for users. Fri, 23 May 2025 23:59:59 +0000 https://www.codexworker.com/post/165/a What developments have taken place today? Today’s commits focused on enhancing observability, improving channel listing functionality within an API, establishing a structured WebSocket communication system for a chat application, and refining internal repository tooling. Dependency updates were applied to improve the stability of OpenTelemetry and gRPC JavaScript libraries. The API enhancements centered around enriching channel listings with associated user information and media files, incorporating validation to ensure data integrity. A new function was added to the mock repository for managing last message IDs within channels. Finally, significant changes were made to the WebSocket client and chat UI to support new command types, improve performance by selectively refreshing channel lists, and integrate file uploads via a media button. These updates collectively contribute to increased system stability through dependency upgrades, improved data accuracy and completeness with the API modifications, enhanced communication reliability within the chat application’s WebSocket infrastructure, and streamlined repository management with the mock repository function. The strategic changes in the API, particularly regarding user association and media handling, directly improve the usability of the channel listing feature by providing a more comprehensive view of connected users and related content. The WebSocket enhancements represent a critical step towards building a scalable and responsive chat application capable of efficiently managing various communication events. Thu, 22 May 2025 23:59:59 +0000 https://www.codexworker.com/post/163/a What developments have taken place today? Several dependency updates were applied to the remote daemon client TypeScript project. The Git repository URL was updated to point to a more recent commit within its GitLab repository. These updates ensure compatibility and potentially address minor bugs or improvements in the underlying libraries. A significant change involved integrating a dedicated file service into the message handling workflow. This included adding `helper_service_file` and `repository_message_media` dependencies, utilizing a `fileService` field within the `ApiHandlerChannel` struct to handle file uploads and media attachments. The initialization process now explicitly requires a `FileServiceInterface` instance for both the main handler and its mock implementations, promoting code clarity and maintainability. The interface itself defines a single method, `UploadFile`, simplifying the interaction with the file service. Finally, an older commit hash referencing a subproject was replaced with a more current version, ensuring accurate tracking of dependencies within the overall project structure. These changes collectively improve the robustness, compatibility, and maintainability of the remote daemon client by incorporating updated dependencies, streamlining file handling, and maintaining accurate reference to project components. Wed, 21 May 2025 23:59:59 +0000 https://www.codexworker.com/post/162/a What developments have taken place today? Today’s commits focused on enhancing WebSocket server monitoring, streamlining Redis message delivery, and refining chat functionality within the application. Several updates to dependencies were also implemented alongside configuration adjustments and minor template modifications. **WebSocket Monitoring & Messaging:** The initial commit introduced a system for continuous monitoring of active WebSocket connections. A ticker loop now prints detailed information about each user’s connection – including their ID, remote address, and connected channels – directly to the console. This provides valuable insights into server activity and aids in debugging. Additionally, a new function, `SendMessageToRedis`, was added to publish messages to a Redis channel, likely for integration with real-time notification systems or chat applications. **Dependency Updates & Configuration:** Several npm packages were updated, including `engine.io-client`, `socket.io-client`, and `ws`. These updates probably contain bug fixes, security enhancements, and performance improvements related to the WebSocket infrastructure. The configuration file was also modified to include variables for adjusting CPU usage during simulations and establishing a default authentication token. **Chat Functionality Refinement:** A separate commit addressed changes specifically within the chat system. While the exact details of these modifications aren't fully documented, it signifies an adjustment or enhancement to the overall chat functionality. **Registry & Message Repository Updates:** Another significant change involved updating the application’s registry and message repository structure. A new `MessageRepository` was added to the registry alongside existing repositories like `ChannelRepository` and `UserChannelRepository`. Updates were also made to the `InitRegistry` function and included a CORS middleware update for preflight requests, ensuring compatibility with web browsers. Furthermore, it now utilizes `userIdWith` from query parameters when creating channels. **ProtobufJS & User Data Retrieval:** A third commit focused on refinements within the core data serialization logic. The `protobufjs` library was updated to version 7.4.0, likely addressing bug fixes and performance improvements. The code also introduced a new method, `FetchUsersIdsForChannel`, in the `UserChannelRepository` to efficiently retrieve all user IDs associated with a specific channel. Changes were made to the JSON serialization of `UserLoginResponse` objects, ensuring accurate encoding of user data fields. **User Profile Endpoint:** Finally, the latest commit introduced a new API endpoint `/api/v1/user/{id}` for retrieving basic user profile information. This endpoint includes parameter handling, defined response types (including various error responses), and documentation updates to reflect its functionality. The implementation is currently a placeholder awaiting further development. These changes collectively represent an important step towards improving the stability, monitoring capabilities, and overall robustness of the application – particularly in relation to real-time communication features and user data management. Tue, 20 May 2025 23:59:59 +0000 https://www.codexworker.com/post/161/a What developments have taken place today? Today’s commits focused on refining the core architecture and operational aspects of the WebSocket server application. Key improvements include restructuring command handling, significantly enhancing connection management capabilities, and introducing mechanisms for maintaining active connections. The codebase now incorporates a more organized approach to message processing through dedicated structs representing different WebSocket commands and responses. This promotes clarity and maintainability. The introduction of the `UserConnection` struct represents a substantial upgrade to connection management. It encapsulates vital information about each user’s connection, including their associated user account, channel subscriptions (supporting multiple connections per user), and a synchronization mechanism for concurrent access. A keep-alive system has also been implemented to proactively prevent WebSocket disconnections due to inactivity. Furthermore, the server now includes a robust method for adding and removing users from channels, catering to dynamic subscriber management needs. New logging capabilities have been added to track message delivery and user activity, aiding in debugging and monitoring. Significant changes were made to the API handler section of the application. The `ApiHandler` struct was updated to include a new field for handling messages, alongside modifications to the constructor function to instantiate this new field. The `ApiHandlerChannel` structure now incorporates interfaces for accessing channel and user channel repositories, improving modularity and testability. The addition of the `SendMessage` method streamlines message sending processes by handling channel creation and security checks. A `DefaultError` component was added to OpenAPI definitions for improved error handling. Finally, the deployment environment has been updated through modifications to the Dockerfile and docker-compose.yml files. These updates ensure a more stable MySQL database container setup within the application’s containerized ecosystem, including necessary system packages, volume mounts, and network configuration. Overall, these changes are important because they establish a more robust, scalable, and maintainable foundation for the WebSocket server, enabling better user experience management, efficient message delivery, and improved operational stability. Mon, 19 May 2025 23:59:59 +0000 https://www.codexworker.com/post/157/a Hello! Code change information from bot here! During week 2025-05-12 - 2025-05-18 i made 112 commits across 8 of my projects. In total i created 28657 new lines and removed 17860 lines. Mon, 19 May 2025 00:01:49 +0000 https://www.codexworker.com/post/160/a What developments have taken place today? This commit introduces several improvements related to Lighthouse data management and WebSocket server functionality. The first change establishes a dedicated directory, `/var/data/dev.remotedaemon.com/lighthouse`, for storing Lighthouse data. This centralized location improves organization and simplifies file access. The updated code now writes Lighthouse files to this designated directory, utilizing website IDs for unique filenames. Additionally, the commit includes setup procedures for necessary directories within a test environment. Subsequent changes focus on enhancing the WebSocket server's robustness and structure. A new `WebsocketAuthenticate` struct has been implemented to rigorously validate incoming authentication requests using a token repository. New data structures – `WSCommandSubscribe`, `WSCommandNewMessage`, `WSCommandWritingMessage`, `WSResponseAuthSuccess`, and `WSResponseErrorMessage` – have been introduced to standardize the format of WebSocket messages, improving clarity and maintainability. The addition of a `MessageType` field within `WSCommandNewMessage` allows for differentiated handling of various message types. Finally, a `PushToRedisChannel` function is implemented to reliably broadcast message data via Redis, facilitating efficient channel-based communication. These changes were important because they improve the organization and accessibility of Lighthouse data, enhance the security and structure of the WebSocket server, and streamline the process of broadcasting messages between clients and the server, leading to a more reliable and scalable system. Sun, 18 May 2025 23:59:59 +0000 https://www.codexworker.com/post/158/a What developments have taken place today? This commit enhances session security by implementing secure cookie settings in production environments. These include enabling secure cookies, setting a 30-day maximum session age, and configuring HttpOnly and SameSite Lax Mode to protect against common web vulnerabilities. Updating session management package references ensures compatibility and stability. A visual update was made to the website, replacing one image source (`/image/me.webp`) with a new version (`/image/me_2.webp`) and introducing an additional image (`/image/me_200.webp`). This improves the visual presentation of the site without altering functionality. A new button has been added to the author section, providing direct access to a “contact” page. The styling ensures consistent design with other elements on the website, improving user navigation and accessibility. These changes collectively improve security posture, enhance the user experience through updated visuals and easier navigation, and maintain code stability by updating dependencies. The combination of these improvements contributes to a more robust and user-friendly application. Sun, 18 May 2025 23:59:59 +0000 https://www.codexworker.com/post/159/a What developments have taken place today? Here’s a summary of the recent code changes: 1. **Lighthouse Report Enhancements:** The Lighthouse report now includes a final screenshot thumbnail, displaying image data securely and providing optional width/height information. Error handling has been implemented to manage invalid data. Additionally, the code now captures and displays detailed domain and certificate information in an informative card, including the full certificate chain and enhanced error handling for loading data. Security scoring is integrated into the Lighthouse report, saving scores and audit references alongside the decoded report as a JSON file. Improved logging using `console.log` statements has been added, along with RabbitMQ integration for sending lighthouse data. 2. **Issue Categorization & Display:** The Lighthouse report now categorizes issues by severity (critical and moderate), presenting them in distinct sections within the HTML output, complete with color-coded visuals and toggleable details. 3. **ClickHouse Data Storage Modifications:** Significant changes have been made to the `website_url_ping_history` ClickHouse table schema, including new columns for performance monitoring metrics (DNS lookup time, connection time, etc.) and a modified table engine (`MergeTree`) with optimized indexing. A migration script is included for reverting these changes. A repository structure has been established for interacting with the ClickHouse database using parameterized queries to prevent SQL injection. Summary: These commits represent a significant effort to improve the Lighthouse analysis process, enhance user interface clarity and detail, and optimize data storage performance and reliability. The addition of security scoring and detailed reporting provides valuable insights into website vulnerabilities, while the ClickHouse modifications promise improved query efficiency and data capture for URL ping history analysis. Fri, 16 May 2025 23:59:59 +0000 https://www.codexworker.com/post/156/a What developments have taken place today? Today’s changes focus on enhancing data visualization, refining database schemas, improving filter management, and updating website presentation elements. Several key improvements were implemented across multiple areas of the application. The primary improvement involves restructuring how duration bar charts display network performance metrics. Instead of mapping individual data points to labels, the code now aggregates timings into minute intervals, creating a more concise and informative visualization. This includes sorting data by creation date, grouping timings into minutes, calculating average timings for various stages (DNS lookup, connection, TLS handshake, etc.), and updating chart configurations to reflect this aggregated data. Tooltips were also enhanced to display relevant time intervals. Further refinements addressed zooming behavior to ensure consistent visual representation during zoom operations. A significant database modification was made to the `website_url_ping_history` table. New columns for timing metrics (DNS lookup time, connection time, TLS handshake time, TTFB, download time, redirect time) were added alongside a replacement for the deprecated `page_data` column and the change of "duration" to "total_time". This expansion allows for more detailed analysis of website performance. A new SQL migration was included to facilitate this schema change. The user interface saw improvements with the addition of a modal window for managing filters applied to tables. This provides users with granular control over data displayed in tables, including disabling filtering functionality altogether. New CSS styles were added to support the filter component and its associated UI elements. The implementation also includes new helper functions for handling user search forms. Furthermore, several updates were made to certificate management within website records. The `Website` struct now incorporates fields for certificate expiration date, issuer, serial number, signature algorithm, and public key, enabling more comprehensive monitoring of certificate validity. The database query used for searching websites was also updated to retrieve these expanded certificate details. Finally, minor updates included dependency versioning, adjusting default pagination parameters in controller functions to ensure consistent behavior, modifying a script generating SQL statements to insert data into the database, and updating website title and sections with new descriptions and formatting. These changes collectively contribute to improved data analysis capabilities, enhanced user experience through filter management, and increased system robustness by streamlining data handling processes. The overall aim is to provide more insightful visualizations of network performance while maintaining a flexible and maintainable codebase. Thu, 15 May 2025 23:59:59 +0000 https://www.codexworker.com/post/155/a What developments have taken place today? Today’s commits primarily focus on improving the presentation of website details, particularly regarding account and domain information, alongside enhancements for workflow run tracking and documentation. The updates include restructuring card layouts using grid columns, introducing formatted date displays for expiration data (domain and certificate), adding a new section to the documentation, refining identifier versions, enhancing API endpoint functionality for retrieving domain details, and streamlining workflow state management with telemetry. Additionally, several npm packages were updated for security and stability improvements. The most significant changes involve modernizing the website detail page’s design and information display. Specifically, account and domain details are now presented in a more visually appealing “info card” format using grid columns – this represents a substantial improvement to user experience by organizing complex data more clearly. The introduction of `formatExpiration` and `formatLighthouseScore` functions provides dynamic formatting for critical dates (domain/certificate expiry) and scores, adding clarity and urgency through color-coding. The addition of workflow run details – including IDs, statuses, and timestamps – significantly enhances debugging and monitoring capabilities, while the breadcrumb component is updated to accurately reflect the page’s purpose. Furthermore, the API endpoint for retrieving domain information has been implemented, providing a structured way to access critical website data. This includes error handling for various scenarios (domain not found, internal errors) ensuring robustness and informative responses. The database schema migration adds support for storing certificate chain details, enhancing traceability and potentially enabling deeper analysis of SSL certificates. Finally, updates to npm packages address security vulnerabilities and improve the overall stability of the project. In summary, today’s changes are important because they significantly enhance user experience by improving website design, introduce valuable new features like workflow run tracking and detailed API endpoints for domain information, and proactively address potential issues through npm package updates – all contributing to a more robust, informative, and user-friendly application. Wed, 14 May 2025 23:59:59 +0000 https://www.codexworker.com/post/154/a What developments have taken place today? Several improvements and updates were implemented across various parts of the project, focusing on data processing, logging, UI enhancements, and dependency management. Key changes include refinements to RabbitMQ message handling for website data and statistics, a streamlined telemetry span creation process, and expanded admin interface capabilities with Redis log display. Database schema modifications were also undertaken, establishing foreign key relationships between website debug tables and the main website table, ensuring data integrity through cascading deletion. Furthermore, there are updates to several dependency versions including `semver`, `pg-protocol`, and others, addressing potential bug fixes and security patches. Logging has been enhanced with more context for error handling within workflow processes, alongside adjustments to URL redirection logic and user agent configuration during website retrieval. Several test functions were updated to ensure accurate measurement of total time, particularly when loading URLs and domain aggregates. Finally, the `SynopseeController` now provides additional information about log directory size and latest Redis errors in its "info" endpoint response, while also updating internal configurations for consistent error handling. These changes collectively enhance data reliability, improve application performance through optimized resource utilization and dependency management, bolster the user experience with a more informative admin interface, and strengthen overall system stability by addressing potential vulnerabilities and inconsistencies. The database updates are particularly important for maintaining data integrity across different tables. Tue, 13 May 2025 23:59:59 +0000 https://www.codexworker.com/post/153/a What developments have taken place today? This commit refactors the location of a workflow directory, moving it from `/var/www/www.remotedaemon.com/workflow` to `/var/www/remotedaemon.com/workflow`. This change simplifies the path and reduces potential ambiguity, improving maintainability. This commit introduces a `WorkflowDirectory` configuration option for several functions and structs within the application. By passing the workflow directory path as an argument to `initHandler`, `Checker`, and `NewGeneratorChecker`, these components gain explicit access to the location of workflow definitions, enhancing their flexibility and enabling consistent handling across different deployments. This commit integrates a comprehensive workflow state management system into the lighthouse checks. The `LightHouseConsumer` now utilizes `WorkflowStateManagerInterface` and `WorkflowLoaderInterface` for loading workflows from YAML files, determining steps, dependencies, and execution conditions. The `loadWorkflowState()` function handles workflow loading and validation, returning errors if issues are encountered. This provides better control and reliability in the execution of lighthouse checks. Specifically: * A sample `lighthouse.yaml` file has been added to define workflow steps for lighthouse checks. * The `LighthouseGenerator` now accepts a `workflowDirectory` parameter to specify the location of workflow definition files. * The `ProcessData()` method in the consumer calls `loadWorkflowState()` to retrieve workflow state before performing lighthouse checks and marks the step as completed. * The `LighthouseGeneratorData` struct includes a `workflowRunId` field, passed to the workflow state manager. This commit updates project dependencies to include OpenTelemetry tracing and metrics instrumentation for Node.js applications. This addition incorporates packages for OpenTelemetry SDKs, exporters, and instrumentations, facilitating distributed tracing and metric collection. A placeholder message reminds developers to update when a mobile application is available. This commit upgrades the URL downloader component, integrating it seamlessly into the registry initialization process. The `Config` struct now includes a `script_ping_checker` field to store the path to the ping checker script. A new instance of `helper_url_downloader.OSCommandRunner` is created in the `InitRegistry` function using this configuration setting. Comprehensive tests, including success, error handling (invalid JSON and script errors), and mock testing, have been added to ensure the downloader’s functionality and reliability. This commit refines the workflow for checking website certificate expiration dates. It introduces a new workflow that includes generating a message and consuming it to verify the certificate's expiration date. The description of this workflow step has been updated from “Fetch data from an external API” to “Download data from URL”. Finally, this commit incorporates `WorkflowStateManager` and `WorkflowLoader` into various application components via dependency injection. These changes are reflected in `api.NewApiHandler`, the main function, `CertificateConsumer`, `CronFunctions.Certificate`, `WebsiteController.checkCertificate`, and `GeneratorCertificate.GenerateOne`. The `loadWorkflowState()` function is now used to check workflow status before generating messages or performing other operations. Additionally, the `WorkflowRunRepository.DeleteOld` function has been modified to delete old workflow runs based on age limits (5 hours for ping checkers, 50 hours for all others). This commit improves the `helper_url_downloader` package by enhancing its ability to handle redirects during URL downloads. The code now uses `curl`’s `-L` option to automatically follow redirects and extracts detailed metrics about each step in the redirect chain using `curl`'s `-w` flag. This includes HTTP status codes, response times, DNS lookup times, connection times, and more. The information is outputted as a JSON string, facilitating programmatic processing of download results. Robust error handling has been implemented to address common issues such as DNS failures, connection problems, and TLS errors. In summary, these changes represent significant improvements in workflow management, monitoring capabilities, tracing instrumentation, and URL downloading reliability, leading to a more robust and maintainable application. Mon, 12 May 2025 23:59:59 +0000 https://www.codexworker.com/post/150/a Hello! Code change information from bot here! During week 2025-05-05 - 2025-05-11 i made 111 commits across 4 of my projects. In total i created 43486 new lines and removed 20212 lines. Mon, 12 May 2025 00:02:35 +0000 https://www.codexworker.com/post/152/a What developments have taken place today? This day’s work focused on enhancing workflow management and observability within the application. Key improvements include database schema updates for tracking workflow execution details, a revamped message generation system utilizing generic interfaces and dedicated data structures, and the implementation of OpenTelemetry tracing. Additionally, significant changes were made to introduce state management for workflows, including persistent storage and retrieval mechanisms. Finally, code was refactored to utilize new generator classes and updated associated tests. The database schema now includes tables `workflow_runs` and `workflow_step_runs`, providing a structured way to record workflow execution status and individual step progress. These additions facilitate comprehensive monitoring of workflow performance and debugging. A significant restructuring occurred in the message generation process. The introduction of generic generator interfaces (`GeneratorInterface[helper_rabbitmq_generator.GeneratorModel]`) and associated data structures (`GeneratorCheckerData`, `GeneratorCertificateData`) promotes code flexibility, maintainability, and extensibility. This allows for easier addition of new message types in the future without extensive modifications to existing code. The use of `GeneratorChecker` and `GeneratorCertificate` simplifies the creation and delivery of messages related to website checks and certificate expiration respectively. OpenTelemetry tracing has been implemented, providing valuable insights into application behavior. By configuring an OpenTelemetry SDK with a Jaeger exporter, developers can now track requests, diagnose performance bottlenecks, and identify potential issues more effectively. The inclusion of trace context propagation ensures that traces are correctly associated across distributed components. The workflow state management system has been overhauled to incorporate persistence, retrieval, and tracking mechanisms. The `WorkflowStateManager` struct and its implementation provide a centralized point for managing workflow state, simplifying the process of determining the current status of a workflow run. The introduction of `WorkflowLoader` and related functions allows for loading workflows from YAML files, enabling greater flexibility in defining and deploying workflows. Finally, existing generator classes were consolidated into new, more robust types (`GeneratorChecker`, `GeneratorCertificate`), accompanied by updated tests to ensure proper functionality and integration with the revised message generation system. These changes improve code organization and reduce redundancy. In essence, these combined efforts represent a significant step towards building a more robust, observable, and maintainable application architecture. The improvements in workflow management and observability will greatly aid in monitoring performance, debugging issues, and scaling the system effectively. Sun, 11 May 2025 23:59:59 +0000 https://www.codexworker.com/post/151/a What developments have taken place today? Today's commits focused on enhancing website URL tracking, improving error handling during downloads, and updating dependency information. The changes primarily revolve around providing more detailed context about how URLs are accessed, especially when redirects occur, and bolstering the robustness of the URL downloading process. Specifically: * A new dependency (`remote-daemon-client-typescript`) was updated to a newer version incorporating bug fixes and minor enhancements. * The logging system for website generation has been refined by introducing more specific span names related to lighthouse monitoring, improving traceability. Debug logging statements were removed in favor of structured error handling with trace flags. * A new field (`redirect_chain`) was introduced into the `website_url_ping_history` database table to record the sequence of redirects encountered during URL pings. Supporting migration scripts were added to facilitate this schema change, along with updates to related components like the RequestStats struct and RabbitMQ consumer. * The `URLDownloader` component received significant improvements including robust redirect handling (limiting redirects to prevent loops), comprehensive error reporting, and detailed request statistics (including DNS start times, connection establishment times, etc.) for debugging purposes. * The display of website URLs in the user interface was enhanced by adding a `website_full_url` field to API responses and customizing table rendering to show complete URLs with redirection information. These changes are important because they collectively improve the system's ability to accurately track complex URL interactions, provide richer diagnostic data for troubleshooting network issues, and deliver more informative website details to users. Sat, 10 May 2025 23:59:59 +0000 https://www.codexworker.com/post/149/a What developments have taken place today? Today’s commits primarily focus on improving observability, dependency management, and data handling within various components of the system. Several tests were updated to reflect these changes. * A test function was modified to correctly dereference a pointer variable when comparing timestamps, ensuring accurate test assertions. * The `remote-daemon-client-typescript` project’s dependencies were updated to newer versions, including `@types/node` and `regjsparser`, addressing potential bug fixes and new features. The Git repository URL was also updated for consistency. * A new Go template file (`admin/website_detail.go.tmpl`) was added to facilitate the generation of HTML views for website details in the admin interface. * The `CheckerMessageGenerator` now includes logging statements to track constructed URLs and message generation, along with a check for debug settings. * Dependencies were updated for `http-cache-semantics` and `remote-daemon-client-typescript`, alongside an update to the Git repository URL. * Error handling was enhanced in the `helper_url_downloader` package by adding specific checks for “no such host” and “server misbehaving” DNS errors, returning custom error types (`ErrDNSLookupFailed` and `ErrDNSServerMisbehaving`) and logging these failures. * The `GenerateAll` function in cron jobs was modified to include tracing information using `telemetry.StartSmartSpan`, improving observability and allowing for detailed analysis of execution paths. Error handling was also improved by checking for a nil context and adding database error handling. * The `WebsiteURLPingHistoryRepository` now supports saving data with nullable fields, accommodating cases where certain metrics (like TTFB) might not be available during a ping operation. * The `URLDownloader` has been significantly revamped to provide more detailed HTTP request statistics through a new `RequestStats` struct. This includes capturing timing information for DNS lookup, connection establishment, TLS handshake, and other key stages of the download process. The downloader also now avoids consuming excessive memory when handling large response bodies and incorporates client-side tracing via `httptrace`. * Unit tests for the `URLDownloader` were comprehensively updated to cover various scenarios including successful downloads, redirects, invalid URLs, server unreachable conditions, empty response bodies, client timeouts, TLS handshake failures, and unexpected EOFs. These tests also verify timing metrics and status codes. These changes collectively enhance system reliability through improved error handling, increase visibility through detailed tracing and logging, and provide richer data for monitoring and debugging. The updates to dependencies ensure the components are using current best practices and bug fixes while the updated unit tests improve code quality. Fri, 09 May 2025 23:59:59 +0000 https://www.codexworker.com/post/148/a What developments have taken place today? Today’s commits focused on enhancing testing, improving data handling, and refining API endpoints. Here's a summary of the changes: * **Comprehensive Unit Tests for Logger:** New unit tests were added to thoroughly validate the `InMemoryLogger` component. These tests cover logging methods (levels, filenames, timing), buffer management (empty flush prevention), synchronous stop behavior, and error handling during writer operations. This improves code reliability and reduces potential bugs related to logger functionality. * **Pagination Refactor:** The pagination logic in `UserController` and `WebsiteController` was refactored by replacing a custom `NewPagination` helper function with a new `NewPagination` function that takes input parameters for page number, items per page, total item count, and existing query parameters. This promotes better organization, clarity, and maintainability of the pagination logic. Unit tests were added to verify the correctness of this new function in various scenarios, including edge cases. * **Random String & Interface Refactoring:** Tests for randomly generated strings (length and uniqueness) were introduced. Simultaneously, several types and structs were refactored to use interfaces instead of concrete implementations. This enhances testability, reduces coupling between components (e.g., `RabbitEnqueuer`, `PingChecker`), and promotes a more flexible design. * **New Domain Lighthouse API Endpoint:** A new API endpoint `/api/v1/domain/{id}/lighthouse` was added to retrieve website URL lighthouse scores for specified domains. This expands the API's functionality and provides valuable performance data. The implementation includes fetching URLs, querying lighthouse metrics, and returning a JSON response. * **Dependency Updates:** Several npm dependencies were upgraded, including Babel compiler components (`@babel/compat-data`, `@babel/helper-compilation-targets`, etc.). These updates likely address bug fixes, performance improvements, or new features in the Babel ecosystem. * **Website URL Metadata Refactor:** The process for handling website URLs was updated to directly incorporate the `URL` itself into the object’s structure rather than relying on an ID as a unique identifier. This change ensures consistent and correct representation of URLs within data structures. * **Website Metrics Error Handling:** Methods were added to the `MockWebsiteRepository` class (`UpdateDomainError`, `UpdateCertificateError`) for updating error information associated with websites. A new `DomainMetadataRepositoryInterface` was also introduced, along with a corresponding implementation class and updates to other repositories. This improves error tracking and management related to website processing. * **Lighthouse Rating Refactor:** The lighthouse rating calculation logic has been centralized into the `helper_lighthouse_rating` package. This promotes code reusability and simplifies future maintenance. The changes include updating API handlers, adding a `Rating` field to the `UrlLightHouseData` struct, and introducing new unit tests. * **Dependency Update for `remote-daemon-client-typescript`:** The version of the `remote-daemon-client-typescript` library was updated to the latest commit identifier (04930c10266d12d3a404daa22ef0d4b61f9cd1f2). These changes collectively contribute to a more robust, maintainable, and feature-rich codebase. The increased test coverage improves reliability, the refactoring enhances code organization, and the new API endpoint expands application capabilities. The improvements in error handling and data management ensure data integrity and facilitate efficient monitoring and analysis. Thu, 08 May 2025 23:59:59 +0000 https://www.codexworker.com/post/147/a What developments have taken place today? Several improvements and enhancements were implemented across various components of the web application infrastructure and data processing pipelines. These include streamlined deployment processes, improved performance through caching mechanisms, enhanced traceability and debugging capabilities, and optimized database interactions. Specifically, the deployment process for the remote server-based web application has been simplified by utilizing `supervisorctl` to manage service restarts based on the “go\_remotedaemon” process name, eliminating manual intervention and reducing potential errors. Furthermore, outdated build files are now automatically cleaned up, maintaining a cleaner and more efficient environment. The `helper_url_downloader` package received significant updates with the introduction of Redis caching. This implementation utilizes a new `CachedDownloader` struct to store downloaded data in Redis using TTLs (one minute), minimizing redundant requests and boosting overall performance by approximately 70%. Unit tests have been added to ensure the cache functionality operates as expected under both successful and unsuccessful retrieval scenarios. The application’s debugging capabilities were enhanced through a dedicated section within the documentation outlining steps for testing various system aspects, alongside troubleshooting guidance related to production certificate validation. This includes detailed instructions for enabling debugging mode on servers and examining relevant logs, coupled with modifications to the `CertificateConsumer` struct to leverage a `websiteDebugRepository` and utilize the `golang.org/x/net/publicsuffix` library for improved domain name resolution. Traceability within the application has been substantially strengthened by centralizing Jaeger configuration through dynamic retrieval from a `reg` object, propagating trace-related information via AMQP messages, and utilizing smart spans to provide more granular insights into system behavior. Logging improvements have also been integrated across several services, including the PingChecker consumer. Database interactions were optimized with a revised `Save` function in the `WebsiteUrlMetricRepository`. This move replaced a single complex query with separate SELECT, INSERT/UPDATE, and retrieval operations for increased efficiency and improved error handling, ensuring data consistency and resilience. Finally, the script responsible for container management has been updated to reliably restart supervisor processes by executing a dedicated shell script, promoting stability and simplifying the deployment workflow. In summary, these changes represent a significant step towards a more robust, performant, and easily maintainable application through improvements in automation, caching, observability, database operations, and debugging tools. Wed, 07 May 2025 23:59:59 +0000 https://www.codexworker.com/post/171/a What developments have taken place today? This commit introduces a new set of screens within the application designed to provide users with detailed insights into website performance and availability. Specifically, three new screens – WebsiteUrlListScreen, WebsiteMetricScreen, and WebsiteUrlPingHistoryScreen – have been implemented. The WebsiteUrlListScreen displays a list of URLs associated with a chosen domain, enabling navigation to a dedicated metric screen upon selection. The WebsiteMetricScreen presents key metrics for each URL, including response time and a Lighthouse rating. Finally, the WebsiteUrlPingHistoryScreen details the ping history related to each metric. To support these screens, new provider definitions – websiteUrlPingHistoryProvider, websiteUrlPingHistoryListProvider, websiteUrlMetricProvider, websiteUrlMetricListProvider, and websiteUrlProvider – have been created. These providers leverage a custom RiverpodAuth class for authentication and access data from specific API endpoints. The HomeScreen now integrates with these new screens when a domain is selected, enhancing the user experience by offering more granular performance information. These changes are important because they significantly improve the application's ability to monitor website health and performance by providing a comprehensive view of URL metrics and ping history, directly addressing user needs for detailed website analysis. Tue, 06 May 2025 23:59:59 +0000 https://www.codexworker.com/post/146/a What developments have taken place today? This day’s updates primarily focus on improving website URL lighthouse reporting, enhancing OpenTelemetry integration, refining logging mechanisms, and maintaining dependency versions. Several npm packages were updated to their latest releases, including `esbuild`, `@types/node`, `electron-to-chromium`, `caniuse-lite`, and `chalk`. Minor changes in these updates likely included bug fixes and performance improvements. A significant update involved the overhaul of website URL lighthouse data retrieval and presentation. A new controller (`WebsiteUrlLightHouseController`) was introduced to handle detailed reports, alongside updated templates for displaying this information on a webpage – including breadcrumbs, custom tables, and placeholders for JavaScript components. The logic for fetching and rendering lighthouse audit reports has been improved with error handling and timeout management. Furthermore, the code now utilizes session data for authentication, streamlining API requests. A comprehensive OpenTelemetry integration was implemented across several application components. This includes introducing a new `CheckerMessageGeneratorInterface`, a `RabbitMqGeneratorLighthouse` struct, and related interfaces and functions within the `helper_rabbitmq_generator` package to facilitate tracing of website URL checking operations. The `telemetry` package received updates with `Telemetry` and `NewTelemetry` interfaces, alongside `SetTraceEnabled` and `StartSmartSpan` functions, enabling fine-grained control over tracing behavior based on configuration. The `Registry` struct was updated to include a `Telemetry` object for enhanced observability. A new logging system was implemented, replacing the previous basic logging mechanism with an `InMemoryLogger` that writes logs to files. This system includes configurable log directories and improved error handling through detailed logging statements. Finally, several smaller improvements were made, including: updating a constant related to tracing configuration for better type safety; modifying the `WebsiteController` and views to control website debugging; adding a new repository for website debug settings; migrating the `created` column in the `login_history` table to a datetime type with added logging for tracking creation timestamps; and updating dependency references. These changes were important because they collectively improve the application's reliability, observability, data management capabilities, and user experience, leading to better debugging tools, enhanced performance analysis, and more robust error handling. Tue, 06 May 2025 23:59:59 +0000 https://www.codexworker.com/post/170/a What developments have taken place today? Today’s commits focused on enhancing domain management within the application, streamlining registration and login processes, improving token handling, and refining UI elements. Significant improvements were made to the user experience by introducing sorting capabilities for the domain list, bolstering error handling across various screens, and adding a robust logout feature. The registration process was overhauled with input validation, loading indicators, device registration, and navigation enhancements, while the login flow now redirects users to a splash screen. Furthermore, the application adopted more reliable token management practices including storing tokens securely and clearing them upon logout. Finally, minor UI adjustments were implemented for visual consistency and clarity. These changes were important because they collectively addressed key user needs related to domain access, account management, and overall app stability. The sorting functionality improves usability by allowing users to easily organize their domains based on preferences. The improved registration and login flows provide a smoother and more secure onboarding experience. Enhanced token handling ensures proper session management and data security, while the new logout feature offers a clean exit from the application. Finally, these updates lay the groundwork for future expansion of domain-related features and integrations within the Riverpod architecture. Mon, 05 May 2025 23:59:59 +0000 https://www.codexworker.com/post/145/a What developments have taken place today? This commit significantly enhances the registration process by adding crucial security features and validation checks. A new `PasswordValidatorInterface` is introduced, integrated into the `RegistrationValidator`, ensuring passwords meet minimum length requirements and incorporate complexity elements like symbols. Furthermore, the code now verifies if a user with the provided email address already exists in the system, preventing duplicate registrations and returning an appropriate error message if a match is found. Comprehensive unit tests have been added to validate these new validation rules and interactions between components, bolstering the overall security and reliability of the registration process. Several adjustments were made to improve the presentation and responsiveness of charts and tables on the page. Bar chart containers now incorporate `overflow: auto;` to accommodate longer datasets, while pie chart sizes are reduced for better readability. Table rendering is improved by introducing a dedicated "rendered-table" div for dynamic table content, utilizing a flexbox layout for centered alignment and responsive sizing. Minor styling tweaks, such as padding adjustments and class name changes, further refine the visual appearance of these elements. The login process has been modified to improve error handling during user searches. Previously, errors encountered during user lookups were simply ignored, potentially leading to unexpected behavior. Now, the API handler gracefully handles any errors that occur during the search process by returning a standard 500 internal server error response with a descriptive message. The code now includes logging statements for registration events and validation errors, aiding in debugging and troubleshooting. The device test suite has been strengthened through the addition of error handling capabilities. Specifically, the repository layer is now equipped to return informative error responses (500 Internal Server Error) when operations like device registration or deletion fail. A new `DeviceRepository` package with associated unit tests has been created, incorporating a `TestMain` function for consistent test setup and helper functions for simplifying common tasks such as database clearing and dummy user creation. The tests cover various scenarios, including successful device saving/searching/deletion and error conditions, providing comprehensive coverage of the repository’s functionality. Finally, improvements have been made to the channel management system by introducing new tables – `channel`, `user_channel`, `message`, and `message_media` – alongside corresponding CRUD methods within the `ChannelRepository`. These additions provide a structured foundation for managing channels, users associated with channels, messages exchanged within those channels, and media attachments. This update lays the groundwork for enhanced channel-based communication features in the application. In summary, today’s changes focus on enhancing security through stricter password validation, improving UI responsiveness and structure, refining error handling mechanisms, solidifying testing practices for device operations, and introducing a robust foundation for managing channels and their associated data – all of which contribute to a more reliable, secure, and user-friendly application. Mon, 05 May 2025 23:59:59 +0000 https://www.codexworker.com/post/141/a Hello! Code change information from bot here! During week 2025-04-28 - 2025-05-04 i made 148 commits across 6 of my projects. In total i created 30150 new lines and removed 22400 lines. Mon, 05 May 2025 00:01:58 +0000 https://www.codexworker.com/post/169/a What developments have taken place today? This commit series focused on improving the Remote DAEMON application’s core functionalities related to user authentication, device management, and software updates. Key improvements include strengthening backend integration for device token handling through enhanced API communication and session management, implementing robust error handling during login processes, and introducing a dedicated update mechanism with visual feedback for users. Specifically, the `AuthNotifier` now reliably sends device tokens upon successful login using an integrated `ApiClient` and `deviceIDGenerator`. The `DeviceService` ensures proper removal of device tokens on logout. A new `WebsiteCheckGraphService` retrieves website status data dynamically, including error reporting and update notifications. A dedicated "Update Screen" allows users to download newer application versions, accompanied by a detailed “Website Detail Screen” showcasing download progress and any encountered errors. Furthermore, the login screen has been overhauled with Riverpod state management for improved responsiveness and user experience. The addition of `loginLoadingProvider` prevents duplicate button presses during network requests while providing visual feedback. A `loginErrorProvider` manages and displays error messages effectively, coupled with validation to ensure required fields are populated. The use of an `AuthApi` class facilitates streamlined HTTP communication using the `io_client`. Finally, dependency updates, including significant upgrades to Android SDKs and Flutter packages like `flutter_downloader`, enhance stability and leverage improved features within the application’s ecosystem. In summary, these changes significantly bolster the app's backend architecture, provide a more user-friendly login experience, streamline the update process, and ensure overall system reliability through comprehensive error handling and dependency management. Sun, 04 May 2025 23:59:59 +0000 https://www.codexworker.com/post/144/a What developments have taken place today? Several commits have been made focusing on API improvements, testing, backend feature development, and dependency updates. 1. **Date Range Handling Enhancement:** The `ApiHandlerWebsiteUrlPingHistory` struct’s date range parsing logic has been refined. It now uses RFC3339 formatting for input dates, with error handling in case of invalid formats. Unit tests have been added to verify the bucket generator's functionality across various date ranges and edge cases. Logging statements have also been introduced for debugging database query construction. This improves data accuracy and testability. 2. **ApiResponseWebsiteUrlPingHistory Structure Update:** The structure of `ApiResponseWebsiteUrlPingHistory` has been modified to include integer representations (`int`) for fields like ID, Duration, StatusCode, and Created, simplifying the data model. Corresponding changes have been made in encoding and decoding functions with added validation to ensure required fields are present during decoding. This enhances data clarity and simplifies data handling. 3. **Website URL Metric Controller & Routes:** A new `WebsiteUrlMetricController` has been implemented alongside associated routes (`/api/v1/website-url-metric/:id`) for retrieving detailed metrics for individual URLs. The controller utilizes repositories to access the database, and a template (`website_url_metric_detail.go.tmpl`) is used to render data into a table format with pagination support via an offset parameter in the repository search function. This introduces new functionality for viewing URL metric information. 4. **ApiResponseWebsiteUrlPingHistory Field Expansion:** The `ApiResponseWebsiteUrlPingHistory` struct has been extended by adding `duration`, `status_code`, and `created` fields as `OptInt`. These additions have been reflected in the JSON schema definition, decoding logic, and field names, enhancing data representation. 5. **Remote Daemon Client TypeScript Dependency Update:** The dependency on `remote-daemon-client-typescript` has been upgraded from version 797c0b29eff88eeb011e3c5af0a0cede2f2c9e5f to 92691529e4634d2b2c6b7c746621093bdd3a0046. No code changes were implemented during this update. 6. **Website URL Ping History Repository Enhancements:** The `WebsiteURLPingHistoryRepository` now supports pagination through an offset parameter and includes filtering by website URL ID when provided. Chart.js versions have been updated to 4.4.9 and 2.2.0 respectively, along with the addition of a zoom plugin for interactive exploration. Duration and HTTP status code graphs are rendered using Chart.js, displaying data in an accessible visual format. The HTML structure has also been adjusted to accommodate these visualizations. This significantly improves data presentation and allows users to analyze website URL ping history more effectively. These changes represent improvements in data handling, API functionality, test coverage, and user experience through enhanced visualization capabilities. The combination of structural refinements, new features, and dependency updates contributes to a more robust, efficient, and informative system for managing website URL metrics and ping history. Sun, 04 May 2025 23:59:59 +0000 https://www.codexworker.com/post/143/a What developments have taken place today? This commit enhances the deployment process and image handling capabilities of the "go_codexworker.com" web application. A new CDN worker, `go_cdn_codexworker.com`, has been implemented to manage content delivery. The deployment method utilizes an updated rsync command that excludes specific directories and files, improving efficiency and security. Significant changes have been made regarding image metadata management, including the introduction of a new `FileRepository` and `PostFileRepository` structure with expanded file details like `media_type` and public access status. Furthermore, a `SquidHasher` struct has been added to generate unique hashed paths for image URLs, crucial for CDN integration. A new package, `helper_image_path_generator`, creates CDN-compatible URLs based on file metadata, supporting various image sizes and formats. JavaScript enhancements have been included to display image previews directly within the upload form, alongside lightbox functionality with navigation controls. The configuration file has been updated to specify necessary parameters such as the alphabet used for hashing, the base file directory, resized image directory, and CDN address. Finally, Dockerfile modifications were implemented which include setup all services using docker compose, nginx configuration to serve static images from the cdn service, and registry updates to reflect the newly added `SquidHasher` and `FileService`. These changes are important because they streamline image delivery through a CDN by implementing hashing for unique URLs, enhancing user experience with previews and lightboxes, improving data management with updated repositories, and simplifying deployment using Docker. Sun, 04 May 2025 23:59:59 +0000 https://www.codexworker.com/post/142/a What developments have taken place today? Today’s commits focused on several key areas: enhancing device management within the API, updating dependencies across various packages, improving website URL performance analysis, and refining the application's routing for OpenAPI documentation. The first set of changes centered around a new `ApiHandlerDevice` struct designed to manage device-related operations – specifically deleting device tokens and registering devices. This involved adding necessary fields to the struct, modifying functions like `NewApiHandlerDevice`, `DeleteDeviceToken`, and `PostDeviceRegistration` to interact with a new `DeviceRepository`. These changes establish a foundational layer for push notification management and device lifecycle tracking within the API. Simultaneously, numerous npm packages were updated to their latest versions. This includes browserslist, remote-daemon-client-typescript, require-directory, lighthouse_lighthouse_audit, ws, and several others. These updates address potential bug fixes and performance improvements across the codebase, ensuring compatibility and stability. The upgrades to `browserslist` and `electron-to-chromium` are notable due to their significant changes. Further enhancements were implemented for website URL performance analysis. New columns – `performanceScore`, `accessibilityScore`, `seoScore`, and `bestPracticesScore` – were added to a performance table, providing a comprehensive overview of website health. These scores are formatted as percentages with visual indicators (ok, warning, danger) based on their values, enhancing data interpretation. Significant routing modifications were made to serve OpenAPI documentation more effectively. The application now directly serves the `openapi.yaml` file at `/api/openapi.yaml` and updates related URLs within handler functions for consistent access. The Swagger UI initialization script is also modified to reflect this change. Finally, enhancements were introduced to a Go web application, including dependency upgrades (browserslist, remote-daemon-client-typescript, update-browserslist-db, ws) and the addition of lighthouse score details to website URL information displayed in API responses. These improvements contribute to enhanced data presentation and monitoring capabilities within the system. In summary, today’s changes collectively strengthen the application's core functionality by improving device management, optimizing dependencies, enhancing performance analysis, streamlining documentation access, and enriching the Go web application with lighthouse score insights. These combined developments represent a significant step toward a more robust, reliable, and informative system. Sat, 03 May 2025 23:59:59 +0000 https://www.codexworker.com/post/140/a What developments have taken place today? Today’s commits focused on enhancing the system for collecting and presenting Lighthouse performance data associated with website URLs. Key improvements include a new API endpoint for retrieving this data, an updated Redis store for managing URL Lighthouse information, and modifications to existing code to integrate these changes seamlessly. Specifically, a new `/api/v1/website-url/{id}/lighthouse` endpoint was added to fetch lighthouse results based on a website URL ID. This endpoint utilizes a `UrlLightHouseStore` component within the `ApiHandlerDomain`, which is now responsible for retrieving and storing Lighthouse data in Redis. The `ApiHandlerWebsiteUrl` struct has been updated to depend on this store, enabling it to access Lighthouse metrics alongside other website information. A new function `GetWebsiteUrlLightHouseList` handles the retrieval logic, including error handling for common HTTP status codes. Significant changes were made within the `helper_redis` package, shifting from `URLLiveData` to `URLPingData`. This refactoring aims to provide a more focused approach to URL data collection and storage. Several unit tests have been updated to reflect these changes, utilizing new mock implementations of `UrlLightHouseStore` for testing purposes. The logic for retrieving website URLs has also been updated to utilize the new repository. Furthermore, the API response format was enhanced by adding a `LighthouseRating` field to the `ApiResponseDomain` struct and updating the JSON schema accordingly. Validation logic was implemented to ensure this rating is a valid number. The display of website scores on tables was modified to include a “Score” column with color-coded styling based on Lighthouse performance (green for above 90%, yellow for 31-90%, red otherwise). Finally, the `checkLighthouse` function within a website controller has been updated to trigger lighthouse checks for all URLs associated with a given website. This ensures that all linked URLs are evaluated and data is collected. A new function `GetRating()` was also added to calculate a weighted average of performance, accessibility, best practices, and SEO scores. These changes represent an important step towards providing comprehensive website performance analysis capabilities within the system. By centralizing Lighthouse data collection and making it readily accessible through the API, this work lays the foundation for generating insightful reports and summaries about website health. Fri, 02 May 2025 23:59:59 +0000 https://www.codexworker.com/post/139/a What developments have taken place today? Today’s commits focused on improving the stability, logging, and functionality of several core application components – the Lighthouse API, the CDN, image processing, and user profile management. Significant improvements were made to error handling, reporting, and deployment processes. The Lighthouse API received substantial updates, including comprehensive logging with detailed operational information, enhanced URL validation for improved security, and a more efficient temporary file management system using server-side files. The addition of `try...catch` blocks provides robust error handling and graceful shutdown capabilities via SIGTERM. Report size validation was also implemented to prevent oversized reports from being sent back to clients. Furthermore, the API now uses Puppeteer with explicit configuration changes to optimize performance and security. The CDN underwent a modernization effort by shifting from an older resizing method (`imaging`) to the `govips` library, resulting in enhanced image processing capabilities and organization. A dedicated directory was created for resized images, and the image handling function itself remains unchanged, streamlining operations. Additionally, HTTPS support with HTTP/2 was added for both `cdn.remotedaemon.com` and `www.cdn.remotedaemon.com`. Several improvements were also made to service monitoring and management. The order of heartbeat checks for services was standardized alphabetically for consistent health assessment. A heartbeat mechanism was implemented using Redis, ensuring continuous service monitoring. The Docker configuration now includes restart policies and healthcheck configurations, guaranteeing service availability. Furthermore, the deployment process for the `cdn` component has been automated with Supervisor, simplifying maintenance and ensuring automatic restarts. Finally, changes were introduced to user profile management, incorporating a new package for generating CDN image paths, integrating a `FileRepository` dependency, updating database schema (adding a `file_id` column), and modifying template rendering to display profile pictures. These improvements enable efficient image delivery through the CDN and seamless integration with user profiles. These changes collectively represent significant progress in enhancing application reliability, performance, maintainability, and user experience by implementing robust logging, streamlining processes, and improving security measures. Thu, 01 May 2025 23:59:59 +0000 https://www.codexworker.com/post/138/a What developments have taken place today? Several improvements have been made across various parts of the system, focusing on tracing enhancements, CDN implementation, UI styling, documentation updates, and operational tooling. Specifically, performance data tracking within OpenTelemetry spans has been significantly enhanced by adding URL attributes (including HTTP status codes) to trace events when downloading resources. This provides more detailed insight into website interactions for tracing purposes. Additionally, baggage settings related to website checks have been adjusted, improving clarity with “WebsiteUrl” prefixes and filtering based on a new `website.ID`. A core CDN service has been implemented using Go, offering image serving from a specified location with resizing and quality adjustments via URL parameters, along with local caching for faster delivery. A healthcheck has also been integrated into Kubernetes to ensure the CDN's availability. The user interface’s loading experience has been improved with CSS styling for a visually appealing loading overlay and confirmation dialog component. This provides crucial feedback to users during data-intensive operations. Updates were made to `loading_overlay.scss` and `confirm_window.scss`. The documentation for the `remotedaemon.com` service has been expanded, detailing metrics integration with Jaeger, RabbitMQ cleanup procedures, `.env` file management, domain URL handling, status monitoring, heartbeat logging, ClickHouse data storage, and queue clearing processes. Several dependencies have been updated across the project including `caniuse-lite`, `core-js-compat`, and `electron-to-chromium`. These updates were necessary for improved functionality or stability within their respective packages. Configuration files have transitioned from a development to a production environment, incorporating settings specific to the production deployment, including database schema changes for file management and updated URLs for services like the tracking server. Finally, the certificate expiration handling has been refined to accurately reflect certificate validity periods, while TLS verification during redirects is now disabled for improved connectivity. A new `certificateExpirationMessageGenerator` was introduced to simplify certificate expiry notifications, along with a mock for testing purposes. These changes are important because they collectively enhance application performance (through the CDN and tracing), improve user experience (with UI styling and clear feedback), streamline operational procedures (with tooling updates and documentation), and strengthen security (with improved certificate handling). Wed, 30 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/137/a What developments have taken place today? This commit incorporates several dependency updates to enhance application functionality and stability. Specifically, it introduces `github.com/fatih/color` for improved terminal output coloring, `github.com/fsnotify/fsnotify` for file system monitoring, `github.com/githubnemo/CompileDaemon` for accelerated compilation, `github.com/mattn/go-colorable` for enhanced color handling, `github.com/radovskyb/watcher` for directory and file watching, and updated versions of `golang.org/x/sys`. These updates likely contribute to streamlined development workflows, more informative logging, and potentially improved performance. A significant security update was implemented by replacing the existing encryption key with a new one. This change addresses potential vulnerabilities and strengthens data protection measures. The addition of `AnalyticsMiddleware` introduces a mechanism for tracking user behavior through a gRPC server. This middleware utilizes a structured approach including configuration management, goroutine processing, error handling, and communication with an external service via RPC calls. It extracts relevant analytics data from HTTP request headers (Cookie, UserAgent, Referrer) and provides utility functions such as `getFirstLanguage` and `getFullURL`. Furthermore, the PostRepository now logs SQL queries before execution to facilitate debugging. These changes collectively improve application security, enhance user experience through analytics tracking, streamline build processes with faster compilation, and provide valuable debugging tools for developers. Tue, 29 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/136/a What developments have taken place today? This update introduces a new feature allowing users to download the analytics protocol buffer definition file (`analytics.proto`) via a dedicated route at `/website/:id/protofile`. The implementation ensures proper HTTP headers are set, classifying the file as an attachment and delivering it directly to the user’s browser. A brief informational message has also been added to guide users interested in GRPC integration towards this new functionality. A critical bug related to retrieving user IDs from session data has been resolved. The code now reliably accesses the user ID using `sessionData.UserId` instead of the previous, incorrect method (`session.Get("userId").(int)`), guaranteeing accurate validation processes. Two binary executables, `visit_counter` and `writer`, have been removed from the project codebase. This streamlining eliminates unnecessary files and reduces repository size. Finally, Git’s ignore list has been expanded to exclude binaries located within the `command/grpc_client/bin`, `command/visit_counter/bin`, and `command/writer/bin` directories. This prevents accidental tracking of executable files in the version control system, contributing to a cleaner and more focused repository. These changes collectively improve website accessibility by providing access to essential protocol definition files, enhance data integrity through accurate user ID retrieval, reduce codebase bloat by removing unused executables, and maintain code quality by preventing unnecessary file tracking within the repository. Tue, 29 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/135/a What developments have taken place today? Today’s work focused on enhancing the admin dashboard’s user management and website configuration features, improving overall UI consistency, and introducing more robust OpenTelemetry tracing capabilities. Several key changes were implemented: * **Dashboard Refinements:** A breadcrumb component was added to provide clearer navigation within the user and website sections of the dashboard. The application adopted a consistent `.content-section` style class with padding, background color, and rounded corners for improved visual appeal across different pages, utilizing a wider variant on the homepage. Forms for user management and website configuration were redesigned with dedicated containers, save buttons, and status messages, enhancing usability. Responsive adjustments were made to ensure optimal display on various screen sizes. * **Service Status Monitoring:** The command used to check service status was updated from port 15096 to port 17096, facilitating monitoring in different environments. * **OpenTelemetry Enhancements:** A new `SmartSpan` struct and `StartSmartSpan` function were introduced, enabling conditional tracing based on a configuration setting (`remotedaemon.trace.enabled`). This avoids unnecessary overhead when tracing isn’t required, improving performance. The implementation includes baggage handling for managing trace state and improved error management during baggage parsing. Debugging context information is now more detailed, including Trace ID, Span ID, and relevant baggage data. * **Docker Image Updates:** Two new Docker images (`remotedaemon-lighthouse.tar` and `remotedaemon-jaeger.tar`) were added to facilitate Jaeger tracing integration. * **Template Rendering Improvements:** The `TemplateFunctions` package was updated to allow passing configuration data to components rendered using templates, improving modularity and flexibility. The `RenderPagination` function now utilizes the configuration directory. * **Website Metrics Table:** A new table rendering function, `renderMetricData`, was introduced to display website URL metrics, simplifying the process of presenting this information. * **Styling Adjustments:** The header section’s background color was set to white with a subtle box shadow, and links received darker colors, hover effects, and underline for visual clarity. Media queries were implemented to adapt the layout for smaller screens, ensuring responsiveness on mobile devices. These changes collectively address usability concerns, optimize tracing behavior, streamline deployment processes, and enhance the overall user experience of the admin dashboard. By centralizing tracing control and providing more granular styling options, these improvements contribute significantly to a cleaner, more performant, and adaptable application architecture. Tue, 29 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/133/a What developments have taken place today? Today’s changes focused on enhancing user experience and improving system reliability through several key updates: * **Email Verification Component:** A new component was added to the user detail page, providing a visual alert if an email is not verified and offering a button to resend the verification email. This improves user engagement by clearly communicating account status and facilitating quick correction. * **User Verification Email Functionality:** A button click now triggers a request to send a user verification email, streamlining the account setup process. * **Admin Page Layout Improvements:** Several admin pages underwent restructuring using custom components (header, breadcrumb) and session data for dynamic content rendering. This modular approach enhances maintainability and organization. * **Homepage Component Refactor:** The homepage was refactored to utilize a component-based structure with "hero" and “features” components, promoting code reusability and simplifying updates. * **Password Validation:** A new password validation mechanism was implemented for the user change password functionality, ensuring that passwords meet minimum requirements before being saved. * **User Profile Edit Enhancements:** The user profile edit page now displays timezone and locale information and includes a save button to update these settings. * **Change Password API Implementation:** The code for implementing a change password function was added to the remote daemon client, allowing users to modify their passwords through the API. * **Website URL Database Updates:** New website URLs were added to the database, along with associated metrics data, expanding monitoring capabilities. * **Container Startup Configuration:** The docker-compose command now includes a 'monitoring' profile to automatically launch monitoring services during container startup. * **OpenTelemetry Tracing Improvements:** The `handleMessage` function in the RabbitMQ consumer handler was updated to include more detailed tracing information, improving error diagnosis. * **CSS Cleanup:** Unused CSS styles and related code were removed, contributing to a cleaner codebase. * **HTML Template Updates:** The HTML templates for various pages were modified to incorporate new components, improve layout, and handle email verification status. * **Port Change:** The Lighthouse API now listens on port 3001 instead of 3000. These changes collectively enhance the application's functionality, usability, maintainability, and monitoring capabilities, resulting in a more robust and user-friendly experience. Mon, 28 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/132/a What developments have taken place today? A tooltip has been implemented to display detailed creation dates and times for each date element within the interface. Hovering over a specific date now reveals the precise timestamp of its corresponding entry. This enhancement provides users with immediate access to granular information, improving clarity and usability. This change improves user experience by providing contextual details on demand, reducing the need for users to manually search for or calculate dates. The tooltip functionality streamlines data interpretation and offers a more intuitive way to understand the timeline associated with each entry. In essence, these updates significantly enhance transparency and accessibility of information presented within the system, leading to a more informed and efficient user experience. Mon, 28 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/130/a Hello! Code change information from bot here! During week 2025-04-21 - 2025-04-27 i made 214 commits across 5 of my projects. In total i created 26096 new lines and removed 20970 lines. Mon, 28 Apr 2025 00:02:39 +0000 https://www.codexworker.com/post/134/a What developments have taken place today? This commit focuses on optimizing and enhancing the web application’s deployment pipeline through several key improvements. The base image has been updated to a more consolidated version, ensuring consistent dependencies across services. The Dockerfile now includes essential tools – such as protobuf compilers, Node.js version management (nvm), and a file compressor – which streamlines the build process. A significant enhancement involves data synchronization: the script now leverages `inotifywait` for real-time monitoring of Go source code changes within a designated watch directory. This triggers an automatic rebuild of relevant services only when modifications occur, dramatically reducing unnecessary rebuild times. Furthermore, the script incorporates commands to generate randomized data and update databases and Elasticsearch, ensuring consistent data sets. The commit also includes scripts that execute critical components like `go run` for the writer and visit counter services, guaranteeing proper functionality after deployment. Finally, cleanup procedures have been added to manage Docker containers and volumes, maintaining a tidy environment. The final output now offers clear instructions for accessing the running application and its API endpoints. These changes collectively represent a crucial advancement by improving build speed through intelligent monitoring, automating data synchronization to maintain consistency, and streamlining the overall deployment process. The optimizations contribute to increased efficiency, reliability, and easier maintenance of the web application. Sun, 27 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/131/a What developments have taken place today? Several improvements and refinements have been implemented across various parts of the website’s user interface and backend data handling. These updates focus on enhancing presentation, improving search functionality, optimizing database structures, and ensuring consistent styling. Specifically, date formatting has been adjusted for visual clarity, the maximum number of posts returned in searches has been significantly increased, and a modern grid layout has been introduced for displaying project statistics, incorporating responsive design principles. A key enhancement involves the addition of tag support to post listings, achieved through new template functions and database schema modifications. This allows for associating tags with posts, presenting them as clickable links alongside post information. Furthermore, significant changes have been made to the display of weekly project statistics, prioritizing current relevance and utilizing a grid layout that adapts to different screen sizes. The breadcrumb navigation has also received attention, with adjustments to font size, color, background styling, and margin to improve visual hierarchy and prominence. Database modifications include adding a “tags” column to the “post” table, alongside an accompanying migration script for rollback purposes. This addition is coupled with the removal of previously used "slug" and "tags" columns. Additionally, several stylistic updates have been implemented across various elements, including links within breadcrumbs and text sizes/colors to ensure visual consistency. Finally, a substantial increase in the number of posts returned by search queries has been made, intending to improve user experience through expanded results. These changes collectively contribute to a more organized, visually appealing, and informative website experience for users, while also laying groundwork for future enhancements like tag-based filtering and improved data management within the database. Sun, 27 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/129/a What developments have taken place today? This commit modified data insertion into a table by adding a `slug` field to the `title` column and updating existing post slugs ("123-slug" and “22-slug”). All other fields remain unchanged, improving data organization and URL structure. A website navigation and breadcrumb overhaul was implemented, enhancing visual consistency and clarity. Specifically, visited links in the main navigation and breadcrumbs now appear blue, and the breadcrumb display shows "Home" alongside the website name, offering improved user guidance. Additionally, a recent development posts section was added to facilitate content discovery. The HTML template was updated to introduce `<section>` elements styled as “article” for each blog post, incorporating styling through the new `articles.scss` stylesheet and adjusting spacing, container width, and table styles for a more refined layout. A new website detail page was introduced, showcasing project information alongside related posts using templates like `render_posts.go.tmpl`. This provides a consolidated view of key project data, including description, recent activity, and associated content. The weekly stats are fetched from Gitlab via a new repository, offering real-time development insights. The contact form was removed, streamlining the website’s interface. Template updates (`header.go.tmpl` and `footer.go.tmpl`) were applied alongside basic styling for container, section, and table elements, improving overall visual presentation. The Gitlab project stats repo was updated with new fields for weekly change data, and a placeholder was added for a newsletter signup form. A date formatting function (`humanReadableTimeDifference`) was introduced in `date_format.ts` to provide human-readable date representations, enhancing the clarity of post creation dates on the website. The "About Me" section was redesigned with a photo and updated description emphasizing relevant skills. A table displaying project changes, including commit counts, lines added/removed, and repository size, was implemented for enhanced tracking. Project information visibility was controlled by user role, providing improved access management. Finally, analytics tables were created to track user sessions. Sensitive database credentials and remote host information have been removed from the configuration, enhancing security by preventing accidental exposure. A check has also been added to gracefully terminate the application if a database connection error occurs. Website styling and layout enhancements were implemented, including a new breadcrumb navigation system with links to "Home" and project pages. Page titles are now styled for increased prominence, and navigation link colors have been standardized to white with a hover effect. Visited link styles have been corrected to match the design aesthetic. The `ORDER BY` clause in the Gitlab project stats query was adjusted for proper sorting. The header has been updated to include an anchor tag linking to the homepage and wrapping the title in an `<h1>` tag. A set of CSS variables were defined at the `:root` level, establishing a consistent color palette across the application. These variables control primary, dark, light background, and accent colors for greater design flexibility. Additionally, spacing and padding were adjusted throughout the design to ensure visual harmony. The HTML background color was updated to a lighter off-white shade, and the margin below sections was reduced for improved readability. A slug generation feature has been implemented for post creation, ensuring that URLs are consistently formatted and user-friendly. The `PostRepository` now utilizes the `github.com/gosimple/slug` library to generate slugs from post titles. A database migration was applied to add a dedicated `slug` column to the `post` table and populate it with generated slugs for existing posts. Analytics data collection has been introduced through new tables (`analytics`, `analytics_url`). These tables store information about user sessions, including cookie IDs, locations, durations, IP addresses, and referrer URLs. This enhanced tracking provides valuable insights into website usage patterns. Finally, a message is now displayed if a website does not have a publicly accessible URL, informing users of the situation instead of omitting the link. Summary: These changes focused on improving data organization, enhancing user experience through navigation and layout refinements, securing sensitive information, adding new features like slug generation and analytics tracking, and streamlining the overall presentation of project details and recent activity on the website. Sat, 26 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/104/a What developments have taken place today? This commit modifies data insertion into a table by adding a `slug` field to the `title` column and updating existing post values. The first post's slug is “123-slug” and the second’s is “22-slug”. This improves data organization and URL generation for posts. A website navigation and breadcrumb update enhances visual consistency. Visited links in navigation and breadcrumbs are now blue, and both "Home" and the website name appear in the breadcrumb display. Recent development posts are also added with links to individual post pages, providing users with easy access to updates. The HTML template has been updated to include `<section>` elements for each blog post, styled as “article,” along with associated CSS styling via `articles.scss`. Specific changes include a new `article-meta` div for post creation dates, adjusted spacing and container dimensions, a white table background, and modified markdown output to return raw HTML instead of replacing newline characters. A new project detail page has been introduced, showcasing project information and related posts. It includes a header section with basic details, a project overview, a list of recent posts, weekly Gitlab statistics (with updated repository fields), and the removal of the contact form. Template updates include `header.go.tmpl` and `footer.go.tmpl`, along with basic styling for container, sections, and tables. Several security improvements have been implemented: database credentials and remote host information have been removed from the configuration, and a check is now in place to halt program execution if a database connection error occurs. Styling and layout changes improve website navigation and page structure. Breadcrumb navigation has been added with links to "Home" and the "Projects" page. Page title styling has also been implemented, alongside updated navigation link colors (white on hover turning to `#0d6efd`). Visited link styles are now white for consistency, and the `ORDER BY` clause in the Gitlab stats query has been changed from descending commit count to descending week year. A new display feature is introduced with `date_format.ts`, allowing dates to be formatted as human-readable strings. The website details page now displays project information like name and description. Post lists are rendered using `render_posts.go.tmpl`. Breadcrumbs now correctly include both the website name and post title. Card styling has been added, including subtle shadows, hover effects, and rounded corners. Unnecessary sections have been commented out. The post list now shows dates and titles in a more compact format with data attributes. A color palette is established using CSS variables at the `:root` level to define primary, dark, light background colors and accent colors. Minor adjustments include adding `.env` and `tables.txt` files, removing an existing `tables.txt` file, adjusting spacing and padding, modifying HTML element backgrounds, and deferring the loading of a JavaScript file referencing `/js/date_format.js`. Finally, a conditional display is implemented for project names and URLs based on user role, ensuring non-administrators do not see sensitive information. In summary, these changes represent significant improvements to the website’s usability, organization, security, and data presentation, enhancing both its visual appeal and functionality. Sat, 26 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/95/a What developments have taken place today? This commit modified data insertion into a table by adding a `slug` field to the `title` column for both posts, assigning "123-slug" and “22-slug” respectively. All other fields remain unchanged. This improves data organization and URL friendliness. The website’s navigation and breadcrumb structure were updated with blue styling for visited links in the main navigation and breadcrumbs, along with a refactored breadcrumb display showing both "Home" and the website name. A section displaying recent development posts was also added with links to individual post pages. These changes improve visual consistency and user experience by guiding users through the site hierarchy. The HTML template was updated to include a new `<section>` element for each blog post, styled as an “article,” incorporating styling for the article container and content using the `articles.scss` stylesheet. This provides a consistent structure for displaying posts and improves visual presentation. A new website detail page was introduced showcasing project information and related posts. It features a header section with basic details, a project overview, a list of recent posts associated with the project, weekly Gitlab stats presented in a table, and (previously) a contact form that has been removed. Template updates (`header.go.tmpl` and `footer.go.tmpl`) were also implemented, along with styling and updated gitlab project stats repo fields. This consolidates key information into a single location for easy access. A function to format dates into human-readable strings was added in `date_format.ts`, facilitating clearer date display on the website. The template now renders post lists using `render_posts.go.tmpl` rather than directly within the main layout, leading to cleaner code and easier maintenance. Breadcrumbs were updated to include both "Home" and the current post title. Card styling was added with subtle shadows and hover effects. Unnecessary sections were commented out. The post list now displays both date and title in a compact format with data attributes. A CSS variables file (`:root`) was created, defining color palettes for primary, dark, light backgrounds, and accent colors to ensure consistent design across the application. Several files were added or updated including `docker/.env`, `shell/tables.txt` (replaced by its content), and a database migration that adds a slug column to the `post` table. Additionally, several external dependencies have been added to the project's Go module file (`go.mod`). These changes enhance maintainability and provide a foundation for future design refinements. The page displaying project changes was enhanced with breadcrumbs, an “About Me” section redesign, and a table for presenting project data including commit counts, lines added/removed, and repository size. Sorting by change volume and conditional visibility based on user role were implemented. This provides more comprehensive project tracking information to the user. A new analytics system was introduced with tables `analytics` and `analytics_url`, capturing session data like cookie IDs, locations, durations, and IP addresses. The implementation of a slug generation feature in the post creation process ensures consistent and shareable URLs. These features enable improved website analysis and tracking capabilities. Finally, minor adjustments were made to styling, including spacing and padding refinements, a lighter background color, and a new button style (`.btn-link-alt`) for alternative link appearances. In summary, today's changes focused on improving the organization, presentation, and functionality of the website through data structure enhancements, visual updates, analytics implementation, and streamlined code management, all contributing to a more user-friendly and informative experience. Sat, 26 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/128/a What developments have taken place today? Today’s updates primarily focus on enhancing user experience and improving backend infrastructure for email processing and Lighthouse monitoring within the remote daemon system. Key changes include: implementing a password recovery flow with associated templates, updating Docker container configurations for email and lighthouse consumers, refining the registration page layout, introducing new RabbitMQ consumer services, and updating various dependencies. The core development effort has shifted towards providing robust mechanisms for user authentication and data monitoring. The most significant improvements involve the addition of a functional password recovery system. This encompasses creating a dedicated controller (`PasswordRecoveryController`), defining a request structure (`PasswordRecoveryReq`), generating email templates, and implementing the logic to send reset links via email. Furthermore, the registration page has been revamped with better visual presentation and improved error handling through status messages. On the infrastructure side, two new Docker containers – `remotedaemon_com_consumer_email` and `remotedaemon_com_consumer_lighthouse` – have been introduced to handle email and Lighthouse data consumption from RabbitMQ queues. These containers are configured with appropriate dependencies and volumes to facilitate seamless operation. This change enhances the system's ability to process asynchronous tasks related to email delivery and website health checks. Several dependency updates were also made, including `@types/node` and `electron-to-chromium`, as well as adjusting the Git repository URL for the client typescript project. These updates ensure compatibility with newer versions of these libraries and provide access to recent bug fixes and improvements. Minor adjustments have been incorporated throughout the codebase, improving code readability through comments and addressing minor inconsistencies in styling. In summary, today's changes represent a strategic advancement towards bolstering user security (password recovery), streamlining data processing (email/lighthouse consumers), and enhancing the overall system’s robustness and maintainability through improved infrastructure and dependency management. These updates lay a solid foundation for future feature development and scalability. Fri, 25 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/127/a What developments have taken place today? Today’s work focused on enhancing user administration features within the application, improving data handling, and streamlining the login process. Key changes include: * **New User Search Form:** A new form has been introduced in the admin panel to allow administrators to filter users based on criteria like email and sorting options. This improves the efficiency of finding specific users within a large user base. * **User Listing Functionality:** A new `UserController` function, `user()`, handles the logic for retrieving and displaying a list of users. It leverages pagination to manage datasets effectively and integrates with the repository layer to access user data. The use of `GetIntQueryParam()` ensures safe retrieval of page and per-page numbers from URL parameters. * **Admin Interface Updates:** Significant improvements have been made to the admin interface, including a new `admin/users.go.tmpl` template that displays user information in a table format with sorting and pagination. A `TemplateFunctions.RenderPagination()` function has also been added for consistent pagination rendering. The Gin router includes a route `/admin/user` to handle these requests. * **Domain URL Updates:** The application's domain configuration has been updated to `http://localhost:18114`, and the header template now dynamically displays a link back to this domain, providing a consistent user experience. * **Website URL History Feature:** A new route `/admin/website/:id/check-certificate` has been added to initiate certificate checks for websites. The `WebsiteUrlRepository` is updated to include functionality for retrieving website URLs and their associated details. The `admin/users.go.tmpl` template now includes the newly generated header. * **Dynamic Login Form:** The login form has been redesigned using JavaScript event handlers, providing a more interactive user experience with visual feedback during submission. The code handles both successful and failed login attempts, displaying appropriate error messages. * **Mock Database for Testing:** A mock implementation of the `WebsiteUrlRepository` (`WebsiteUrlRepositoryMock`) has been created to facilitate testing without relying on a live database connection. This simplifies test development and execution. * **ServerTable Component:** A new `ServerTable` component has been introduced, designed to fetch data from a server using an asynchronous callback function. It includes client-side filtering, sorting, pagination, and error handling. The code refactors existing domain listing functionality to use the new `ServerTable` component. * **Login History Management:** A new endpoint `/admin` has been added to provide access to an administrative interface. This interface allows administrators to manage website URLs and their associated data. Summary: These changes represent a significant step towards improving the administration capabilities of the application, enhancing user experience through dynamic forms, and streamlining key operational processes like certificate management and domain URL retrieval. The introduction of mock database functionality and client-side components promotes testability and performance, while the updates to the admin interface provide administrators with greater control and visibility over the system. Thu, 24 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/126/a What developments have taken place today? Today’s commits primarily focus on enhancing data tracking, improving UI elements, and introducing new functionality for real-time updates and form handling. Several key changes were implemented: * **Notification Queue:** A dedicated queue has been added to the admin interface for managing notification messages, streamlining the process of sending notifications within the system. * **Dependency Updates:** Numerous npm packages and the `remote-daemon-client-typescript` project have been updated with bug fixes and improvements, ensuring a stable and reliable codebase. Notably, significant changes were made to the `entities` and `parse5` packages. * **RabbitMQ Generator Enhancements:** The `helper_rabbitmq_generator` module has been expanded with new functions (`GenerateMessagesForWebsiteChecker`, `GenerateMessageForLighthouseChecker`, and `GenerateMessageForEmailSending`) to facilitate automated message generation for website monitoring tasks via RabbitMQ, including website health checks and email notifications. This introduces a more structured and efficient approach to triggering external processes. * **Website URL Detail Page Improvements:** The website detail page has been redesigned with breadcrumbs for improved navigation, dynamic title rendering, and JavaScript variable injection. This allows the page to display relevant context and interact effectively with the API. Furthermore, metrics display has been refined to show integer ping durations. * **Live Time Updates:** A system for displaying live timestamps for website URL data has been implemented, providing real-time updates on website health and performance through Server-Sent Events (SSE). This allows users to monitor changes in near real-time. * **Form Structure Refactor:** The login, registration, and user edit forms have been modernized by introducing new `LoginForm` and `RegistrationForm` structs, promoting code reusability and maintainability. The form handling logic has also been streamlined with the introduction of a base `BaseForm` class. * **Redis Pub/Sub Integration:** The system now utilizes Redis Pub/Sub channels to publish domain data updates, enabling real-time synchronization across different components. A new `RedisPubSub` struct is included for managing channel interactions. These changes collectively improve the system's efficiency, user experience, and scalability by introducing robust notification management, dynamic UI elements, real-time data updates, and a more organized codebase. The integration of RabbitMQ and Redis Pub/Sub demonstrates a commitment to leveraging asynchronous messaging patterns for improved performance and responsiveness. Summary: Today’s changes are important because they significantly enhance the system's ability to monitor website health, deliver notifications efficiently, provide a better user experience through dynamic UI elements, and leverage modern technologies like Server-Sent Events and Redis Pub/Sub for real-time data synchronization. These improvements contribute to a more robust, responsive, and maintainable application. Wed, 23 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/125/a What developments have taken place today? This commit streamlines the Docker setup for remote daemon services by unifying image creation and entrypoint management. Key improvements include updating the base Docker image to ensure consistency across all services, standardizing entrypoints using a Go build tool (`air`) for faster rebuilds triggered by file system changes, and adding environment variables for monitoring configuration. Furthermore, the commit incorporates Gitlab keys for secure repository access and implements an `inotify-tools` based monitoring loop for automated code deployment upon `.go` file modifications. Temporary SQL scripts have been added to facilitate development and testing with synthetic data. Dependency updates were also applied. Additionally, the commit addresses several UI/UX enhancements: directory creation logic is implemented to prevent errors during file operations, the "ID" column in tables now displays concatenated values with a period separator, and a new user edit page has been introduced, allowing users to update their profile information (email, timezone, locale) via a web interface. The styling of table cells dynamically changes color based on data status using custom CSS classes and icons. Finally, the commit refactors data retrieval from Redis, utilizing aggregated metrics and fixed timestamps for improved API response efficiency and clarity. The code also updates how URL metrics are saved to streamline the process. A new controller has been implemented for handling Server-Sent Events (SSE), enabling real-time domain updates. In summary, these changes significantly improve deployment reliability, maintainability, user experience, and data retrieval performance, creating a more robust and efficient system for managing remote daemon services. Tue, 22 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/124/a What developments have taken place today? Today’s development focuses on enhancing system stability and security through several key changes: improving email processing, streamlining client repository generation, refining registration form handling, optimizing table sorting and pagination, updating dependency versions, and implementing resource limits for services. A critical update involves bolstering email handling within the remote daemon application. New API endpoints have been introduced for sending verification emails and updating user profiles, alongside a redesigned data structure (`UserProfileUpdateRequest`) to facilitate these operations. The email service has been updated with new functions to construct verification emails, utilizing a new `EmailValidationLinkCreator` interface. The registration form has undergone significant refactoring, incorporating error handling improvements using JSON-based responses (specifically the `PostRegistrationBadRequest` error type) and a new template for generating individual form fields. These changes improve user experience and provide more informative feedback during registration. Significant effort was dedicated to simplifying and standardizing client repository generation. New shell scripts (`tag_generated_clients.sh` and `generate_clients.sh`) have been added, automating the process of creating client code from OpenAPI definitions and tagging them with release identifiers. The core logic has also been restructured to leverage a new class, `ApiResponseWebsiteUrl`, for representing API responses, improving modularity and testability. Furthermore, several minor improvements were implemented across different areas: adding resource limits for services using a default CPU allocation, updating the `CustomTable` component's sorting and pagination mechanisms (including improved internationalization support), refining the registration form’s presentation and error handling, updating dependency versions for various libraries, and adjusting routing rules to handle user requests more effectively. Finally, the Docker Compose configuration has been updated to include profiles for different service types, offering greater flexibility in deployment scenarios. In summary, today's changes represent a concerted effort to enhance system reliability, improve developer productivity through automation, strengthen security, and optimize user experience—all contributing to a more robust and maintainable application. Mon, 21 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/86/a evening review #13 little bit of refactoring and optimizing docker on localhost, running 10 containers without limits on older laptop caused performance issues... but solved. git diff --shortstat "$(git rev-list -1 --before=midnight HEAD)" HEAD 93 files changed, 5144 insertions(+), 2753 deletions(-) Mon, 21 Apr 2025 18:04:57 +0000 https://www.codexworker.com/post/85/a Hello! Code change information from bot here! During week 2025-04-14 - 2025-04-20 i made 127 commits across 4 of my projects. In total i created 22970 new lines and removed 14855 lines. Mon, 21 Apr 2025 00:01:46 +0000 https://www.codexworker.com/post/123/a What developments have taken place today? Today’s commits introduce several new container services for monitoring and managing various aspects of the `remotedaemon` project, alongside improvements to existing components and data handling. The primary focus is on establishing a robust and automated system for checking consumer health, processing notifications, and tracking website URL metrics. Here's a breakdown of the key changes: * **Consumer Checker Container:** A new Docker container (`remotedaemon_com_consumer_checker`) has been added to continuously monitor and rebuild the consumer application using `air`. It includes SSH access for GitLab integration, automated memory logging, and volume mounts for code, data, and Docker socket communication. * **Notification Consumer Container:** A new container service (`remotedaemon_com_consumer_notification`) handles RabbitMQ notifications, utilizing OpenTelemetry tracing for observability. This allows proactive alerts based on website performance changes. * **Data Writer Container:** A new container service (`remotedaemon_com_consumer_write_data`) is introduced to handle data writing operations, integrating with RabbitMQ and maintaining consistency through database updates. * **Redis Metric Storage:** The codebase has been updated to store website URL metrics in Redis, enabling faster retrieval of performance statistics for API responses. This includes a new `RedisStatSaver` struct and related logic. * **API Response Enhancements:** The `ApiResponseDomain` struct now incorporates a `total_url_count` field to provide more comprehensive information about domain performance. The associated decoding and encoding logic has been updated accordingly. * **Data Update Improvements:** The `CustomTable` component has been enhanced with dynamic data updates, resource cleanup for DOM elements, and new action icons for exporting and deleting domains. SSE integration is included (commented out) to support real-time data updates. * **Subproject Identifier Updates:** Two subproject identifiers have been updated to reflect newer versions of the underlying projects. These changes collectively aim to improve the system's reliability, scalability, observability, and maintainability by introducing dedicated containerized services for key operational tasks and centralizing data storage for efficient access. The use of technologies like `air`, Redis, and RabbitMQ demonstrates a commitment to modern infrastructure practices. Summary: These changes are important because they establish a more robust and automated system for monitoring, managing, and reporting on the health and performance of website URLs within the `remotedaemon` project. They improve operational efficiency, provide better visibility into critical metrics, and lay the foundation for future enhancements such as real-time data updates through SSE. Sun, 20 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/122/a What developments have taken place today? This commit implements a Jaeger-based tracing solution for the application, enhancing observability and debugging capabilities. The core changes involve initializing an OpenTelemetry tracer, configuring it to send data to a Jaeger endpoint, and ensuring proper context propagation throughout the system. Specifically, the `helper_tracer` package was introduced, containing functions to initialize an OTLP tracer using the `go.opentelemetry.io/otel` library. The `InitTracer()` function now establishes a connection to Jaeger via HTTP, utilizing a service name for identification and defaulting to `http://jaeger:4318/v1/traces`. To prevent potential memory exhaustion, a memory limit of 3000 traces has been configured for the Jaeger service. Furthermore, context propagation is implemented using `propagation.NewCompositeTextMapPropagator` to link traces across services and components, ensuring that tracing data accurately reflects request flows. Modifications were also made to the RabbitEnqueuer and its associated methods to accept and utilize a provided context during message queuing. Finally, resource creation with service names was added for improved observability. These improvements are significant because they introduce a foundational layer of tracing, enabling developers to monitor application performance, identify bottlenecks, and troubleshoot issues more effectively through detailed trace data, ultimately leading to increased system stability and reduced debugging time. Sat, 19 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/121/a What developments have taken place today? This commit significantly enhances website monitoring capabilities by introducing a comprehensive “heartbeat” logging system integrated with Redis and an admin interface. It establishes a structured approach to tracking service availability, providing administrators with immediate insights into potential issues through the admin panel, which displays heartbeat status and consumer information from RabbitMQ queues prefixed with "remotedaemon_". Furthermore, it emphasizes the importance of metrics for long-term application health by detailing a website checker that measures message insertion to data writing/notification delivery times. This commit also refines the API endpoints related to website URL metrics. The `ApiHandlerWebsiteUrl` struct has been updated to include a more complete representation of website URLs, incorporating data from the associated `website_url` table and adding validation for average ping duration. The code now utilizes this enriched information in API responses, providing more context for monitoring purposes. Additionally, the commit enhances database schema by introducing a `website_id` column into the `website_url_ping_history` table, aligning with changes made to repository queries. Another key development involves enhancing the website detail page user experience. The template and associated JavaScript have been updated to display the ID of the viewed website URL, improving usability and providing clearer identification for each entry. The previous date-based navigation and push notification links have been removed in favor of this more direct approach. A new script tag dynamically loads specific JavaScript files based on the website’s ID, ensuring targeted functionality. Further improvements include modifications to the website listing page, adding a button for quickly adding new URLs associated with the currently displayed website. The addition of a `<script>` tag and a corresponding button streamlines the URL management process, simplifying user interaction. Finally, this commit also refactors several repository classes, introducing data validation in `ApiHandlerDomain` and enhancing the functionality of the `MockWebsiteUrlMetricRepository`. In summary, today's changes are important because they bolster website monitoring, refine API data delivery, enhance user interface elements on the detail page and listing page, and improve code organization for maintainability. These combined efforts result in a more robust, informative, and user-friendly application experience. Fri, 18 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/84/a evening review #10 little bit of refactoring, tests and thinking about rabbit queues... git diff --shortstat "$(git rev-list -1 --before=midnight HEAD)" HEAD 70 files changed, 4685 insertions(+), 697 deletions(-) Fri, 18 Apr 2025 14:42:37 +0000 https://www.codexworker.com/post/120/a What developments have taken place today? Today’s commits focus on streamlining Firebase messaging, enhancing email data handling, refining website ping tracking, and introducing new monitoring capabilities. The primary changes involve decoupling message sending from a generic FirebaseApp interface, centralizing email logic for improved clarity, consolidating website URL ping history data management, adding real-time metrics dashboards with RabbitMQ integration, and updating Docker image saving procedures. Specifically, the first commit refactors the Firebase Cloud Messaging (FCM) sender by introducing a `MessagingClientInterface` dependency injection mechanism. This allows the `FCMSender` struct to utilize an external client for message sending, enhancing flexibility and testability. The accompanying mock implementation (`MockMessagingClient`) facilitates unit testing of this new approach. The changes also updated the test cases to verify correct message delivery and error handling through the injected client. The second set of changes concentrates on email data structures and service calls. A `DataForUserRegistrationEmail` struct has been introduced, consolidating user-related information for registration emails. Similarly, a `DataForPingFailedEmail` struct is now used for sending website ping failure notifications, complete with methods to determine template IDs and generate variable maps for email rendering. The `SendRegistrationEmail` and `SendWebsitePingFailed` functions have been updated to utilize these new structs, centralizing email sending logic within the `sendEmail` helper function. This function now handles database storage and RabbitMQ queuing of emails. Next, the commit refactoring website ping history data focuses on optimizing database queries for retrieving historical data. Instead of fetching all matching rows and selecting the first one, a dedicated `SearchOne` function is used to directly query for a single record based on search criteria, reducing unnecessary data retrieval. Error handling has also been added in case no records are found, preventing potential errors during data access. Finally, a new `WebsiteURLPingHistorySearch` function simplifies search parameters and limits results to one record. Redis integration is updated to prevent issues when retrieving data for newly pinged URLs. Further changes include updating the Docker image saving process, removing unused code, and enhancing the admin dashboard with real-time information on RabbitMQ queues and heartbeat status for various services. This includes adding a `helper_heartbeat` module, new endpoints `/admin/heartbeats` to retrieve heartbeats, color-coded table visualization of service health, and Redis integration via `github.com/go-redis/redis/v8`. Finally, the commit introduces a new test suite for file encryption and decryption using AES, along with updates to package versions and minor code cleanup. These changes collectively contribute to improved code organization, maintainability, and functionality across various aspects of the application. In summary, these changes represent significant improvements by introducing structured data handling, decoupling message sending, optimizing database queries, enhancing monitoring capabilities, streamlining Docker image management, and improving overall test coverage – ultimately leading to a more robust and efficient system. Thu, 17 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/119/a What developments have taken place today? Today’s commits primarily focus on improving data management, streamlining processes, and enhancing code organization across several subprojects. These changes aim to increase modularity, improve performance, and enhance the application's overall stability and maintainability. **Subproject Identifier Updates:** Two subproject identifiers were updated with newer commit hashes, ensuring consistent references within the codebase. This is a routine maintenance task to keep dependencies synchronized. **Repository Login History & Domain Metadata (New Packages):** Significant changes were made to introduce new packages for managing login history and domain metadata. These packages include: * `LoginHistory` and `LoginHistoryRepository`: Structures for representing login data and a repository for database interaction, including methods for saving, searching with sorting capabilities, and retrieving single entries. * `DomainMetadata` and `DomainMetadataRepository`: Structures for representing domain metadata and a corresponding repository with similar functionalities to the login history package. * The `Registry` struct was updated to use these new repositories, establishing a more organized data structure. **TypeScript Client Project Improvements:** A new hook was added to TypeScript client projects to include a necessary `package.json` file for tooling and dependency management. This ensures the clients have the foundational elements needed for development. **Website URL Handling (Enhanced Validation & Display):** The code now enforces stricter validation of website URLs, rejecting those starting with "http://" or "https://". Additionally, a new table component (`CustomTable`) was created to display and manage website URLs, with added functionalities such as sorting, adding, and removing entries. The `renderDomainTable` function now efficiently renders this table. **Email Service Refactoring (RabbitMQ Integration & Validation Links):** The email service architecture has been significantly revamped. It now utilizes interfaces for email validation link creation, saving emails to the database, and enqueueing messages to RabbitMQ. This promotes modularity and simplifies asynchronous email delivery. **Certificate Management (Database Updates & Error Handling):** The codebase was updated to handle domain and certificate errors within the database. New fields were added to the `Website` table to store these error details, enabling more detailed reporting and debugging. **Dependency Upgrades:** Several npm packages, including Chart.js and regjsparser, were upgraded to their latest versions, improving stability and incorporating new features. Additionally minor updates for other dependencies were included as part of routine maintenance. These changes are important because they address fundamental areas of the application – data management, client tooling, communication processes, and dependency management – leading to a more robust, maintainable, and efficient codebase overall. Wed, 16 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/82/a evening review #8 Golang interfaces are nice, i think i started to think a little bit in golang way git diff --shortstat "$(git rev-list -1 --before=midnight HEAD)" HEAD 119 files changed, 3227 insertions(+), 1805 deletions(-) #golang #buildinpublic Wed, 16 Apr 2025 16:30:44 +0000 https://www.codexworker.com/post/118/a What developments have taken place today? Today’s commits introduce several key additions and improvements across various parts of the project. A new Dart client submodule is added for a remote daemon client, mirroring existing TypeScript functionality. This includes initializing a Git connection and updating configuration files. A `WebsiteURLAddForm` class has been introduced to provide a modal dialog for adding website URLs with validation and API integration via the `websiteUrlApi`. Significantly, a major overhaul of access control was implemented, shifting from an ad-hoc system to a structured approach using the new `acl` package. This involved defining roles, resources, actions, and implementing functions like `isLoggedIn`, `ownsDevice`, and `allowAll`. The access control logic was then refactored in several controller methods, leveraging these definitions for authorization checks. Furthermore, unit tests were updated to reflect these changes. A new endpoint for deleting domains (`DELETE /api/v1/domain/{id}`) has been added with comprehensive error handling and security features utilizing the `XAuthorization` mechanism. Finally, a script was implemented to tag multiple client projects with specified version numbers, streamlining the process of deploying updates across different projects. Summary: These changes represent important steps towards improving code maintainability, enhancing API functionality, bolstering security, and simplifying deployment processes. The addition of the Dart submodule expands client support, while the access control overhaul provides a more robust and scalable system for managing user permissions. The new endpoint and related improvements add to the API's capabilities, and the tagging script streamlines project updates. Tue, 15 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/117/a What developments have taken place today? Today’s work primarily focused on refining the API handler for user authentication and registration, enhancing error handling, and improving documentation through OpenAPI specification generation. Key changes include: * **Login & Registration API Updates:** A new `ApiHandlerRegistration` struct was introduced alongside a `NewApiHandlerRegistration` constructor, enabling full-fledged user registration functionality with email validation and token creation. The `/login` endpoint now handles both login and authentication using the newly implemented token repository. * **Robust Error Handling:** Significant improvements were made to error handling across multiple components. This involved defining new error types (`PostLoginUnauthorized`, `PostRegistrationInternalServerError`), implementing proper JSON encoding for these errors, and enhancing response handling in various handlers to provide more informative feedback to clients. * **OpenAPI Documentation Generation:** The code now leverages the `ogen` tool to automatically generate OpenAPI specifications from Go source code. This streamlines API documentation maintenance and ensures consistency between the codebase and the API definition. * **Dependency Updates:** Several dependencies were updated, including those related to error handling, JSON encoding, and cryptographic functions, improving security and functionality. In essence, these changes collectively enhance the robustness, clarity, and maintainability of the user authentication and registration processes within the application, providing a more reliable and well-documented API for clients to interact with. The improvements in error handling are particularly important for ensuring stability and providing useful feedback to users. Mon, 14 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/80/a evening review #6 Started ogen from scratch today. Weird at first, but type-safe and clean. #golang #openapi #ogen #buildinpublic Mon, 14 Apr 2025 16:31:58 +0000 https://www.codexworker.com/post/79/a Hello! Code change information from bot here! During week 2025-04-07 - 2025-04-13 i made 139 commits across 5 of my projects. In total i created 19973 new lines and removed 31077 lines. Mon, 14 Apr 2025 00:01:50 +0000 https://www.codexworker.com/post/116/a What developments have taken place today? Today’s commits have significantly enhanced the domain listing API and associated UI components. The core changes focus on improving data retrieval, filtering, sorting, and display – all geared towards a more efficient and user-friendly experience. Here's a breakdown of the key updates: * **API Refactor & Repository Integration:** The domain listing API now utilizes a dedicated `WebsiteRepository` for accessing website data, moving away from direct database queries. This improves modularity and testability. * **Query Parameter Support:** Comprehensive query parameters – `page`, `pageSize`, `search`, and `sort` – have been implemented, allowing users to dynamically filter and sort the list of domains. Pagination ensures efficient handling of large datasets. * **Domain Metadata Improvements:** The database schema has been updated to include a unique constraint on domain names and improved data types for certificate information. A new helper package (`helper_domain`) handles WHOIS lookups for retrieving detailed domain information. * **Certificate Management Enhancements:** Certificate retrieval, validation, and storage are now more robust with the introduction of a `Dialer` interface for network connections and better error handling. The `Domain` struct now incorporates key usage and SAN validation. * **Table Styling & Accessibility:** The table component's styling has been refined for improved visual appeal and accessibility – including responsive design, sorting indicators, and enhanced link appearance. * **API Documentation Generation:** A new endpoint (`/api-ts`) generates TypeScript definitions for API routes, facilitating integration with other projects. * **New Website Addition**: Three websites have been added to the database through an automated process. In summary, these changes represent a significant step toward consolidating domain management functionality and providing a more robust, flexible, and user-friendly experience for interacting with website data. The focus on modularity, improved data validation, enhanced UI elements, and streamlined API interactions underscores a commitment to maintainability and scalability. Sun, 13 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/115/a What developments have taken place today? Today’s work focused on significantly restructuring the RabbitMQ consumer architecture for increased modularity, testability, and maintainability. Key changes include: * **Generic Consumer Pattern:** A new `GenericConsumer` struct and function were introduced to standardize message handling across different RabbitMQ message types (email, checker, certificate). This allows for flexible message processing through a registry object and handler interface. * **RabbitMQ Helper Package:** The core RabbitMQ logic was moved into the `helper_rabbitmq` package, promoting separation of concerns with new functions like `NewRabbitMQ`, `QueueMessage`, and interfaces such as `RabbitConsumerDataEmail`. * **Specific Consumer Structs:** Separate consumer structs (`CertificateConsumer`, `CheckerConsumer`, `EmailConsumer`) were created for each message type, inheriting from the `GenericConsumer` and utilizing the registry object to access relevant data. * **JSON Message Serialization:** Functions were added to marshal and unmarshal RabbitMQ messages into JSON format, facilitating easier communication between components. * **Refactoring of Existing Functions:** The `CheckLightHouse` and `InsertWebsiteUrlToRabbitMQ` functions were refactored to utilize the new generic consumer pattern and `QueueMessage` method respectively. Email verification and registration handlers now leverage the new consumer functions for message publishing. * **Configuration Updates:** The RabbitMQ queue naming conventions were updated in the configuration file (e.g., `checker`, `lighthouse`, `email`, `certificate`). * **Enhanced Admin Dashboard:** The admin dashboard was updated with a loading indicator and placeholder element to provide visual feedback during data fetching, improving user experience. This commit makes several changes related to RabbitMQ integration within the application. Here's a breakdown of the modifications: 1. **RabbitMQ Configuration Updates:** The `Config` struct now includes fields for various queues under the `"rabbit_queues"` section, centralizing queue names in configuration and the `RabbitMQ` struct accepts a RabbitMQ address during initialization. 2. **`NewRabbitMQ` Function:** A new function `NewRabbitMQ` simplifies RabbitMQ instance creation by taking the RabbitMQ address as an argument, promoting code reusability. 3. **`SendMessageToRabbitMQ` Function Updates:** The `SendMessageToRabbitMQ` function now takes a queue name and message body as inputs, making it more versatile. It uses the provided queue name to declare and publish messages to the correct RabbitMQ queues, with improved error handling via logging. 4. **`InsertWebsiteUrlToRabbitMQ` Function:** This function remains largely unchanged but now leverages the `NewRabbitMQ` function for RabbitMQ instance creation, improving code clarity. It also uses the correct queue name from configuration. 5. **Changes in Consumer Logic (Checker & Lighthouse):** The `StartCheckerConsumer` and `CheckLightHouse` functions were updated to use the new `NewRabbitMQ` function to instantiate RabbitMQ clients and send messages to appropriate queues based on configuration settings. 6. **Email Verification and Registration Handler Changes:** The `StartCertificateC` function was updated to utilize the new RabbitMQ consumer structure and message publishing logic when sending email verification messages. This commit modifies the code related to displaying website expiration information in a table. Here's a breakdown of the changes: * **API Response Types:** The `WebsiteCertificate` struct is updated to allow for optional (nullable) string values for expiration dates and certification expiration dates, replacing them with pointers to `time.Time`. This accommodates cases where an expiration date might be unavailable or not yet set. * **API Response Data:** The `ApiResponseDomainExpiration` struct is adjusted to accommodate the new nullable time types. It now uses `string | null` for `expiration_date` and `certification_expiration_date`, and includes logic within the constructor to format these times into RFC3339 strings when available, otherwise leaving them blank. * **Database Model:** The `Website` struct is updated to use `sql.NullTime` instead of `sql.NullString` for `CertificationExpiration`. * **Database Save Operation:** The database save operation now uses `UTC_TIMESTAMP()` instead of the previous timestamp method. * **Rendering Row Logic**: Changes were made to rendering the rows in the table, including using conditional logic and CSS classes based on the expiration dates and days remaining to expire, to visually indicate certification status (green, orange, red). This commit refactors the code to improve certificate expiration handling within the CertificateConsumer. Specifically, it replaces the `helper_domain.NewDomain` function with a new one that takes a `WhoisClient` and `CertificateFetcher` as arguments. The `NewDomain` function now injects Sat, 12 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/114/a What developments have taken place today? This commit significantly enhances image handling within the application, introducing new functionality for resizing, uploading, and displaying images. Key changes include: * **Configuration Structure Updates:** The `Config` struct now includes fields for storing the paths to directories where resized images are saved (`data_resized_directory`) and original files are stored (`data_directory`). * **Image Resizing Logic Implementation:** A new `GenerateImageDiskPath` function is implemented, generating unique filenames based on width, height, and quality. It also utilizes the `convert` command-line tool for padding and centering images during resizing, ensuring consistent image dimensions and appearance. Debugging information related to image size and quality has been added. * **File Upload Support:** The `post_add.go.tmpl` template now supports file uploads, allowing users to submit images alongside their posts. * **File Repository Modifications:** New methods (`Save`, `LoadOne`, `UpdatePath`) are added to the `PostFileRepository` struct for managing uploaded files associated with posts. * **Homepage Template Updates:** The homepage template is updated to display a list of uploaded images linked to each post, utilizing image paths obtained from the file repository. * **PostAddValidator Enhancements:** The `PostAddValidator` now validates file uploads, checking for valid formats (PNG, JPG, JPEG) and sizes. It also ensures that required fields like title and content are not empty. * **Fullscreen Modal Implementation**: A new feature has been added which allows users to view images in a fullscreen modal window with larger image sizes (800x800). The layout dynamically adjusts for smaller screens, displaying the image and post text side-by-side. This commit also addresses several technical improvements: * **GitLab Integration Enhancements:** The code now retrieves post statistics from GitLab using the API. * **Timezone Handling Improvements:** Database queries are updated to use UTC\_TIMESTAMP() for consistent time representation across servers and environments. A new `timezone_offset` field is added to the user struct for storing timezone information directly within the database. * **Cron Job Setup**: A cron job has been created to run the `command/cron` binary, responsible for scheduling tasks within the application. * **Secret Management Enhancements:** The system now encrypts sensitive configuration data using AES-256-CBC with PBKDF2 key derivation, improving security and simplifying key management. These changes collectively provide a more robust, user-friendly, and secure experience for managing images and related content within the application. Summary: This commit significantly expanded image handling capabilities by incorporating resizing logic, file upload support, and improved display features. It also implemented enhanced security measures through secret encryption, streamlined repository management, and addressed technical improvements related to GitLab integration and timezone consistency, resulting in a more functional and reliable user experience. Fri, 11 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/103/a What developments have taken place today? Several significant improvements and additions have been implemented across various parts of the application, primarily focused on image handling, user interaction, and website display enhancements. These include: 1. **Configuration Updates:** The `Config` struct now includes fields for storing the paths to directories where resized images are saved (`data_resized_directory`) and original files are stored (`data_directory`). This facilitates image URL generation and file organization. 2. **Image Resizing & Uploads:** A comprehensive image resizing system has been implemented, utilizing the `convert` command-line tool for padding, centering, and compression. The `GenerateImageDiskPath` function now dynamically constructs unique filenames based on width, height, and quality parameters. File upload functionality is also integrated into the ‘Post Add’ form, enabling users to submit images alongside their posts. 3. **File Repository Enhancements:** New methods (`Save`, `LoadOne`, `UpdatePath`) have been added to the `PostFileRepository` struct for managing associated files within posts. 4. **Homepage & Post Display Updates:** The homepage template now displays uploaded image paths, and the post display has been revamped to support larger images (800x800) alongside smaller thumbnails, utilizing a fullscreen modal view that adapts to different screen sizes. Responsive design adjustments ensure proper layout on smaller screens. 5. **GitLab Integration:** Code has been added to retrieve post statistics from GitLab using the GitLab API and handle commit information retrieval. The `GitLabReader` struct now includes functionality for fetching all files associated with specific posts. 6. **Newsletter Subscription:** The homepage’s newsletter subscription process is now handled asynchronously, validating email format and preventing duplicate submissions through a series of checks and event listeners. 7. **Testing Framework Improvements:** The testing suite has been updated to use `npm run test` instead of `npx jest`, with enhanced error handling and build aborting on test failure. It also incorporates mocking of the `Date.now()` function for consistent test results. 8. **Time Difference Display:** A new JavaScript file (`/js/post.js`) has been added to display human-readable time differences for posts, leveraging a `humanReadableTimeDifference` function and a `pluralize` helper. The existing `homepage.js` was updated to reference this new file. 9. **Modal Closure:** The application now properly closes modals when the Escape key is pressed. 10. **Validation Updates:** The `PostAddValidator` has been enhanced to include validation for file uploads (size and format) and to ensure that required fields like title and content are not empty. Additionally, it filters out empty files during validation. 11. **Image Resizing Logic Refinement:** A dedicated JavaScript file (`/js/post.js`) handles the resizing of images using a new function, accommodating a 100x100 pixel size with an 80-quality setting and includes debugging logging for supported sizes. The `Convert` function now incorporates additional arguments to improve image compatibility. 12. **Form Submission Logging:** Added logging within the form submission handler to print form data to the console, aiding in debugging. In summary, these changes significantly enhance the application’s functionality by enabling rich media support (images), streamlining user workflows (file uploads and newsletter subscriptions), improving website presentation (post display and responsive design), and strengthening code quality through testing improvements and validation rules. The integration of GitLab statistics provides valuable data insights, while the modularized JavaScript files promote maintainability and extensibility. Fri, 11 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/94/a What developments have taken place today? Here’s a summary of the commits, outlining their improvements: * **Configuration Enhancements:** The codebase now includes `data_resized_directory` and `data_directory` within the Config struct to manage image storage paths effectively. This improves organization and URL generation for resized images. * **File Upload Functionality:** A file upload field has been added to the `post_add.go.tmpl`, enabling users to submit images alongside their posts, expanding content creation possibilities. * **Post File Repository Updates:** The `PostFileRepository` struct now supports saving, loading, and updating file paths associated with posts, providing a structured way to manage post-related files. * **Homepage Template Update:** The homepage template has been updated to display uploaded images linked to each post using the new image path information, enhancing visual presentation. * **GitLab Integration Improvements:** Code now retrieves post statistics from GitLab and correctly handles commit information retrieval via the GitLab API, streamlining data synchronization. * **GitLab Reader Enhancement:** The `GitLabReader` struct incorporates functionality to fetch all files associated with specific posts, leveraging the updated file repository. * **Frontend Template Updates:** The header template now includes img tags for displaying images related to each post, and a new modal view is implemented providing a fullscreen experience for larger images and improved responsiveness across different screen sizes. * **Image Resizing Logic:** Implemented logic using the `convert` command-line tool for resizing images with specified dimensions, quality, padding/centering, and color space settings – ensuring unique filenames and optimized image quality. * **Validator Enhancement:** The `PostAddValidator` has been updated to validate file uploads (size, format), title, and content, strengthening data integrity. * **Test Suite Improvements:** A new test suite for the `humanReadableTimeDifference` function has been added, using Jest and mocking `Date.now()` for consistent testing results; a build process check ensures tests pass before building. Additionally, the build process now utilizes `npm run test`. * **Newsletter Subscription Updates:** The newsletter subscription functionality on the homepage is updated to handle email validation, prevent duplicate submissions, and provide user feedback through informative messages and button states. * **Post Display Refinement**: Introduced a "post-wrapper" div that stacks text content and images vertically for improved layout, along with responsive adjustments for smaller screens. These changes collectively create a more robust and user-friendly application by adding image handling capabilities, improving data management, enhancing the user interface, and strengthening data validation processes. The improvements to GitLab integration and test automation further enhance reliability and maintainability. The newsletter functionality ensures effective subscriber capture while providing a smooth user experience. In summary, this commit cycle focused on significantly expanding the application's core functionalities – specifically image handling and content presentation – alongside crucial improvements in data management, testing practices, and user interface polish. These changes represent a substantial step forward in enhancing the overall usability and reliability of the system. Fri, 11 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/74/a Using ?parseTime=true with the Go MySQL driver could help by automatically converting DATETIME and TIMESTAMP fields into Go’s time.Time, making it easier to work with dates and times. #golang #database #mariadb Fri, 11 Apr 2025 07:39:52 +0000 https://www.codexworker.com/post/73/a evening review #2 Two bots chatting tests: hw: docker container: 1 cpu, DDR3 memory cpu: Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz result: Connections established: 2 Messages received: 83906 Errors encountered: 0 Time interval: 60s #golang #websocket Fri, 11 Apr 2025 06:41:54 +0000 https://www.codexworker.com/post/113/a What developments have taken place today? Today's commits focused on enhancing WebSocket server stability, improving API controller integration with a registry, refining database interactions, and adding logging functionality. Several changes were made to ensure robust error handling, thread safety, and debugging capabilities throughout the application. Specifically: * **WebSocket Server Enhancements:** A new context cancellation mechanism was implemented for user connections, enabling graceful shutdown and preventing resource leaks. The WebSocket server now includes improved logging to indicate errors during message sending and error responses, aiding in debugging. Ping intervals are handled more gracefully within the connection context. * **API Controller Updates:** API controllers were updated to utilize a `registry` for accessing external services (file repositories). Constructors now accept the registry as an argument, improving modularity and testability. Database queries were adjusted to include more parameters and logging statements for improved performance and data integrity. * **Database Improvements:** The `UserChannelRepository` was modified to include a function (`GetOtherUserFileId`) that retrieves file IDs associated with other users in one-on-one channels, facilitating channel assignment logic. Database queries were updated to include more parameters and logging statements for better performance and debugging. * **Logging Enhancements:** The server now integrates with a Redis database for persistent log storage, including caller information (file name and line number) to aid debugging. Existing direct logging calls have been replaced with the new Redis-based approach. * **Test Improvements:** A statistics tracking system was added to WebSocket tests, monitoring connections, messages sent, and errors encountered. These changes collectively contribute to a more stable, reliable, and maintainable application by addressing critical areas such as error handling, data consistency, and debugging support. The improvements ensure graceful connection management and facilitate efficient resource utilization within the WebSocket server. Thu, 10 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/102/a What developments have taken place today? Today’s commits focus on enhancing WebSocket server stability, reliability, and debugging capabilities across several areas: error handling, connection management, logging, and test statistics tracking. Specifically, there were updates to handle errors gracefully during message sending, implement a context cancellation mechanism for user connections, improve the Redis-based logging system, and add a file identifier to channel creation. The code also incorporates a new `RawConnection` struct for comprehensive WebSocket connection tracking, along with improved error handling within the `UserConnection` goroutine. Further enhancements include updating API controllers to utilize a registry for external services, improving database interaction through query adjustments and logging, and adding resource monitoring and plotting functionality to the WebSocket stress test. Finally, there were updates to the repository package for consistent logging, and an overall effort to improve debuggability by including more informative log messages throughout the codebase. Summary: These changes collectively increase server resilience, provide detailed debugging information, and introduce a more sophisticated system for managing channel files, ultimately leading to improved stability and maintainability of the WebSocket backend services. Thu, 10 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/112/a What developments have taken place today? Here’s a summary of the commit descriptions, outlining the changes and their improvements: **WebSocket Server Improvements:** * **Message Structure Enhancement (v1):** The structure of messages transmitted over the WebSocket server was modified to nest message content within a key-value pair object (“message” key). This improves handling and processing on the receiving end by providing more organized data. * **Connection Management Refactoring (v2):** A new `UserConnection` struct was introduced, along with associated functions (`AddUserConnection`, `RemoveUserConnection`, `AddChannelToUser`, `RemoveChannelFromUser`, `sendErrorToUser`). These changes establish a robust system for managing user connections and channel subscriptions within the server state, ensuring thread safety through mutexes. * **WebSocket Command Structure (v3):** A comprehensive refactoring of the WebSocket command structure was implemented, introducing constants (`Redis_Chat_Command_*`) and structs (`WSCommand`, `WSAuth`). New functions (`NewWSNewMessage`, `NewWSMessageSeen`, `NewWSMessageDelivered`) were created to facilitate validation and JSON serialization/deserialization of commands. This significantly improves maintainability and testability, paving the way for a more structured approach to data exchange. * **Bot Interaction Simulation (v4):** A new test case (`TestTwoBotsChat`) was added, simulating two bots exchanging messages over the WebSocket server. This includes database initialization, bot authentication, channel subscription, message sending/receiving, and verification of routing and delivery. The introduction of helper function `waitForMessage` further enhances testing capabilities. **Authentication & Flow Simplification (v5):** * **Simplified Authentication:** Unnecessary error handling code was removed following successful initial authentication attempts, streamlining the authentication flow and improving efficiency. **Testing & Simulation (v6):** * **Comprehensive WebSocket Testing:** The test function now includes more thorough checks for message handling within a websocket server – subscribing to channels and verifying transmitted/received content. **Utility Script (v7):** * **WebSocket Activity Simulator:** A bash script was created to simulate WebSocket activity, allowing developers to easily test the server’s behavior with random user actions and message sending. The script uses `websocat` for realistic simulation. **Analytics Removal (v8):** * **Analytics Code Deprecation:** Several files related to tracking and analytics – `message.ts`, `sse.ts`, and `analytics.js` – were removed, indicating a shift away from extensive user behavior tracking. **Build & Dependency Updates (v9):** * **Dockerfile Improvements:** The Dockerfile was updated to utilize a slimmer Python image and install the `websocket-client` library. Additionally, specific versions of nvm were enforced within the Dockerfile for consistency. A javascript function was added to ensure proper rendering channel functionality. **Minor Updates (v10 & v11):** * **.gitignore Update:** The .gitignore file was updated to exclude the .gitignore file itself from version control. * **General Code Update:** A general code update occurred, encompassing various bug fixes, improvements and updates without specifying particular details. In essence, these changes represent a significant step forward in improving the WebSocket server’s architecture, functionality, and testability. The refactoring of command handling, coupled with enhanced connection management and bot simulation, lays a solid foundation for future development and ensures reliable message exchange within the system. Removing analytics code indicates a focused approach to core functionality while simplifying the codebase. Wed, 09 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/101/a What developments have taken place today? Here’s a summary of the commit descriptions, focusing on improvements and key changes: **WebSocket Server Enhancements:** * **Message Structure Refinement (v1):** The structure of messages transmitted over WebSockets has been updated to use a nested object format with “message” as the key. This improves how receiving systems can process the message content, likely providing more structured data for handling. * **User Connection Management:** A new `UserConnection` struct and associated functions (`AddUserConnection`, `RemoveUserConnection`, `AddChannelToUser`, `RemoveChannelFromUser`, `sendErrorToUser`) have been introduced to manage user connections, channel subscriptions, and error reporting within the server’s state. These additions enhance thread safety and provide a more robust mechanism for managing user interactions. * **WebSocket Command System:** A comprehensive WebSocket command system has been implemented utilizing a `WSCommand` base struct and derived structs (`WSAuth`, `NewWSNewMessage`, `NewWSMessageSeen`, `NewWSMessageDelivered`). This refactoring improves code organization, testability, and facilitates the handling of various WebSocket commands. JSON marshaling/unmarshalling functions have also been added to support data exchange. * **Bot Interaction Simulation:** A new test case (`TestTwoBotsChat`) simulates bot interactions over the WebSocket server, verifying message routing and delivery between bots. This ensures the core functionality for bot communication is working correctly. * **Redis Channel Broadcasting:** The system now utilizes Redis channels to broadcast messages to connected clients via `PushToRedisChannel`. This provides a scalable method for distributing information across the server’s user base. **Authentication Improvements:** * **Simplified Authentication Flow:** An unnecessary error handling block after successful authentication has been removed, streamlining the authentication process and improving efficiency. **Testing & Simulation:** * **Enhanced WebSocket Test Functionality:** The test function now includes more thorough checks for message handling during websocket communication. * **WebSocket Activity Simulator Script:** A bash script has been added to simulate WebSocket activity by randomly sending messages or simulating user behavior, allowing developers to easily test the server's functionality in a controlled environment. **Code Cleanup & Refactoring:** * **Analytics Removal:** Code related to tracking and analytics (message handling, SSE, analytics.js) has been removed, suggesting a shift in focus away from extensive data collection. * **.gitignore Update:** The `.gitignore` file has been updated to exclude itself, preventing accidental commits of the ignore file. * **Build & Dependency Updates:** The project's build process and dependencies have been upgraded (NVM, Dockerfile). **Important Changes Summary:** These changes represent a significant upgrade to the WebSocket server’s architecture and functionality. The introduction of structured message handling, robust user connection management, and a centralized command system greatly improve maintainability, testability, and scalability. The addition of the simulation script is a valuable tool for development and testing. Furthermore, simplifying the authentication flow and removing the analytics code demonstrates a focus on streamlining the core server functionality while reducing unnecessary complexity. Wed, 09 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/72/a evening review #1 [1] simulating multiple websocket connections which need to keep alive is not working well with curl, wscat, websocat. Python is the way [2] working on 14" laptop is really best working setup ... for work Wed, 09 Apr 2025 20:31:55 +0000 https://www.codexworker.com/post/111/a What developments have taken place today? This commit implements a centralized logging system utilizing Redis for enhanced monitoring and debugging capabilities. A new `RedisWriter` struct is introduced to publish log messages to a dedicated "partnerup_zone_chat" channel, alongside a redesigned `Logger` struct that integrates both standard logging and Redis output. The previous `RedisLogger` package has been removed as part of this consolidation. The code now also includes more descriptive constants for message types, improving readability and maintainability, along with enhanced error handling within the `InitWebsocketAuthenticate` function. Specifically, JSON parsing errors are logged and input validation is implemented to ensure data integrity before processing new messages. The marshaling process has been updated to convert JSON structures into byte arrays suitable for Redis publishing. Further improvements include logging enhancements within the message handling logic, specifically tracking incoming websocket messages, message broadcasting, and message processing events using a dedicated `redisLogger`. Image handling has been refined by replacing file extensions with `.webp` and utilizing an `ImageLink` function to generate URLs, defaulting to a predefined image if necessary. Finally, minor HTML adjustments have been made to display message counts in the user interface, alongside fixes for copy-to-clipboard functionality and curl command generation. These changes are important because they establish a more robust, centralized logging infrastructure that provides richer debugging information, improves code clarity through standardized constants and descriptive variable names, enhances error handling for improved stability, and streamlines image processing for enhanced user experience – all contributing to overall system reliability and maintainability. Tue, 08 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/100/a What developments have taken place today? The codebase has undergone significant improvements focused on logging, data handling, and web application configuration. The core of these changes revolves around migrating from a previous Redis logger implementation to a new system utilizing Redis for enhanced logging capabilities. Specifically, a `RedisWriter` struct is introduced to publish log messages to a dedicated "partnerup_zone_chat" channel, alongside a centralized `Logger` struct that manages both standard output and Redis logging for informational, error, and debug levels. The original `RedisLogger` package has been removed. Code clarity and maintainability have been increased by replacing generic string constants with more descriptive alternatives for message types (e.g., `MessageTypeText`, `MessageTypeImage`). Furthermore, the `InitWebsocketAuthenticate` function now includes robust error handling, logging JSON parsing errors, validating input parameters, and converting JSON structures to byte arrays for Redis publishing. The localhost configuration has been updated to include detailed logging within the message handling logic, including tracking new messages received via WebSocket, broadcasting of messages, and processing of incoming messages. Image handling has been optimized by replacing file extensions with `.webp` and generating URLs using `ImageLink`, defaulting to a specified image if needed. Minor HTML modifications have been made to display message counts in the user interface, alongside fixes for copy-to-clipboard functionality and curl command generation. These changes represent a vital step towards improved system observability through centralized logging, enhanced error detection and debugging capabilities within the web application, and streamlined data management practices—resulting in greater stability and easier troubleshooting. Tue, 08 Apr 2025 23:59:59 +0000 https://www.codexworker.com/post/110/a What developments have taken place today? This day’s work focused on enhancing website analytics tracking and improving the presentation of domain expiration information. Several code refactorings were undertaken to streamline data collection, enhance UI elements, and improve maintainability. Let's break down the changes: 1. **Enhanced Website Analytics:** A significant overhaul was implemented for capturing user activity, including location, visibility changes, cookie management, and session tracking. The introduction of SSE (Server-Sent Events) enables real-time data updates, while cookie checks ensure consistent user identification. Furthermore, a comprehensive set of metrics related to system performance is now collected and reported. 2. **Robust Table Rendering:** A new `Table` class was introduced to encapsulate the logic for generating HTML tables dynamically. This improves code organization and reusability. The domain URL table has been updated to include expiration status indicators using color-coded tags, offering a quick visual assessment of domain health. 3. **Improved Email Generation:** Refactoring of email generation logic within `helper_email_generator` created a more modular design, enhancing testability and future extensibility. A new factory function simplifies instantiation of the EmailGenerator struct. 4. **Admin Route Implementation**: Three new routes were added to the admin section for viewing last login times and accessing administrative logs. 5. **Domain Expiration Display:** Added an API endpoint `/api/domain/expiration` and associated front-end changes to display domain expiration dates in a table format, providing users with critical information about website longevity. This includes parsing and formatting date strings for clear presentation. 6. **Style Adjustments**: Minor style adjustments were made to update background colors for status indicators (`status-ok`, `status-failed`) to improve visual clarity. 7. **Dependency Injection:** The codebase now employs dependency injection, separating concerns and improving modularity through the use of constructor functions to create new service instances. 8. **Code Refactoring**: Several minor code refactorings were performed throughout the project, including updating import statements in a stylesheet and reorganizing controller initialization logic. In summary, today’s changes significantly improved website analytics tracking, enhanced data presentation with clear expiration indicators, streamlined backend processes through dependency injection, and added new administrative functionalities. These improvements collectively contribute to a more robust, informative, and user-friendly experience for visitors and administrators alike. Mon, 07 Apr 2025 23:59:59 +0000