{"id":85120,"date":"2025-05-21T10:00:23","date_gmt":"2025-05-21T08:00:23","guid":{"rendered":"https:\/\/rivacon.com\/?p=85120"},"modified":"2025-05-26T15:32:28","modified_gmt":"2025-05-26T13:32:28","slug":"improving-data-infrastructure","status":"publish","type":"post","link":"https:\/\/rivacon.com\/en\/references\/energy-utilities\/improving-data-infrastructure\/","title":{"rendered":"Improving Data Infrastructure"},"content":{"rendered":"\n

<\/path><\/svg><\/span>Initial Situation<\/span><\/h2>\n\n\n\n

A major German municipal utility company intends to set up a central data platform and establish clearer data governance structures as part of a group-wide programme to improve technical and process-related data processing. The medium-term goal is to standardise and link decentralised data pools and replace Excel-based end-user applications (EUAs). This will create a sound basis for future data analyses, including the use of AI methods.<\/p>\n\n\n\n

Even before the final data platform is launched, existing concepts for the data model and data governance are to be trialled and further developed in practice as part of a flagship project for market data in energy trading and risk management.<\/p>\n\n\n\n

<\/path><\/path><\/path><\/path><\/path><\/g><\/path><\/path><\/path><\/path><\/g><\/path><\/path><\/path><\/path><\/g><\/path><\/path><\/path><\/path><\/g><\/svg><\/span>Project Scope<\/span><\/h2>\n\n\n\n
    \n
  • Providing a database and application server for prototype implementation<\/li>\n\n\n\n
  • Connecting external market data sources<\/li>\n\n\n\n
  • Developing a data model compliant with the programme\u2019s technical specifications (Data Vault 2.0)<\/li>\n\n\n\n
  • Implementing automated quality assurance routines<\/li>\n\n\n\n
  • Designing and creating initial market data reports in line with the programme\u2019s technical specifications (Power BI)<\/li>\n<\/ul>\n\n\n\n

    <\/path> <\/path> <\/svg><\/span>Our Contribution<\/span><\/h2>\n\n\n\n
      \n
    • Implementation of a daily import of EEX prices via Rest API and PFC files<\/li>\n\n\n\n
    • Design and script-based implementation of a Data Vault 2.0-compliant data model for market data<\/li>\n\n\n\n
    • Supplementing the data architecture with components for documenting data governance structures, e.g. data ownership<\/li>\n\n\n\n
    • Automation of existing quality checks and extension by additional check routines<\/li>\n\n\n\n
    • Transfer of existing market data reports to Power BI and publication via web application<\/li>\n\n\n\n
    • Conceptual proposal and prototype implementation of automated documentation of data flows, data model and data governance information in Confluence<\/li>\n<\/ul>\n\n\n\n

      <\/path><\/path><\/path><\/path><\/path><\/path><\/path><\/path><\/svg><\/span>Customer Benefit<\/span><\/h2>\n\n\n\n

      The prototype implementation of the data processing was able to confirm<\/strong> the viability of<\/strong> the concepts<\/strong> developed in the programme and develop them further<\/strong> in a meaningful way. The increased level of automation <\/strong>not only accelerated processes but also made them auditable<\/strong>. Moreover, code-based documentation<\/strong> ensures continues correctness and actuality. Additional improvements in data governance<\/strong> resulted from the clear allocation of data ownership. Finally, the script-based implementation<\/strong> allows the transfer to the final data architecture with minimal effort, without losing the results and technical improvements achieved.<\/p>\n\n\n\n

      <\/path><\/svg><\/span>Relevant Skills\/Tools<\/span><\/h2>\n\n\n\n
        \n
      • databricks<\/li>\n\n\n\n
      • Power BI<\/li>\n\n\n\n
      • Data Vault 2.0<\/li>\n\n\n\n
      • Python<\/li>\n\n\n\n
      • Git<\/li>\n\n\n\n
      • Confluence<\/li>\n\n\n\n
      • Azure DevOps<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"

        Prototypical implementation of a central database to test technical specifications and initial data governance concepts as part of digitisation program<\/p>\n","protected":false},"author":3,"featured_media":82351,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_seopress_titles_title":"Improving Data Infrastructure and Data Governance Concepts","_seopress_titles_desc":"Prototypical implementation of a central database to test technical specifications and initial data governance concepts as part of digitisation program","_seopress_robots_index":"","_seopress_robots_follow":"","_seopress_robots_imageindex":"","_seopress_robots_snippet":"","_seopress_robots_primary_cat":"none","_seopress_robots_breadcrumbs":"","_seopress_robots_freeze_modified_date":"","_seopress_robots_custom_modified_date":"","_seopress_robots_canonical":"","_seopress_social_fb_title":"","_seopress_social_fb_desc":"","_seopress_social_fb_img":"","_seopress_social_fb_img_attachment_id":0,"_seopress_social_fb_img_width":0,"_seopress_social_fb_img_height":0,"_seopress_social_twitter_title":"","_seopress_social_twitter_desc":"","_seopress_social_twitter_img":"","_seopress_social_twitter_img_attachment_id":0,"_seopress_social_twitter_img_width":0,"_seopress_social_twitter_img_height":0,"_seopress_redirections_value":"","_seopress_redirections_enabled":"","_seopress_redirections_enabled_regex":"","_seopress_redirections_logged_status":"both","_seopress_redirections_param":"","_seopress_redirections_type":301,"_seopress_analysis_target_kw":"","_seopress_news_disabled":"","_seopress_video_disabled":"","_seopress_video":[],"_seopress_pro_schemas_manual":[],"_seopress_pro_rich_snippets_disable_all":"","_seopress_pro_rich_snippets_disable":[],"_seopress_pro_schemas":[],"footnotes":""},"categories":[37],"tags":[64,62,63,65],"class_list":["post-85120","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-energy-utilities","tag-data-governance-2","tag-data-lakehouse-2","tag-data-vault-2-0-2","tag-power-bi","infinite-scroll-item","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/rivacon.com\/en\/wp-json\/wp\/v2\/posts\/85120","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rivacon.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rivacon.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rivacon.com\/en\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/rivacon.com\/en\/wp-json\/wp\/v2\/comments?post=85120"}],"version-history":[{"count":1,"href":"https:\/\/rivacon.com\/en\/wp-json\/wp\/v2\/posts\/85120\/revisions"}],"predecessor-version":[{"id":85123,"href":"https:\/\/rivacon.com\/en\/wp-json\/wp\/v2\/posts\/85120\/revisions\/85123"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rivacon.com\/en\/wp-json\/wp\/v2\/media\/82351"}],"wp:attachment":[{"href":"https:\/\/rivacon.com\/en\/wp-json\/wp\/v2\/media?parent=85120"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rivacon.com\/en\/wp-json\/wp\/v2\/categories?post=85120"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rivacon.com\/en\/wp-json\/wp\/v2\/tags?post=85120"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}