engineer Resources

Share

Related resources for engineer

Automated Cross-Tenants Data Sharing in Microsoft Fabric12/18/2024 6:17:49 AM. In this new data engineering video, I covered how to architect automated cross-tenants data sharing in Microsoft Fabric focusing on Lakehouse. You will learn how to enable the share and receive extern
Implement Databricks Lakehouse Federation (BigQuery)11/29/2024 4:42:00 AM. In this new video, I covered how to implement and run federated queries federated queries on Google BigQuery.
How to incrementally load data from Microsoft Fabric Lakehouse to Warehouse11/25/2024 5:00:00 AM. In this end-to-end data engineering episode, I demonstrated how to incrementally load data from Microsoft Fabric Lakehouse to Warehouse using Dataflow and Data Pipeline.
New Features in SQL Server 2025: A Guide for Data Engineers11/24/2024 5:46:37 AM. SQL Server 2025 introduces advanced AI integration, vector support, and performance upgrades like optimized locking and enhanced columnstore indexing. New features include JSON data type, regular expr
Advanced C# 13 and .NET 9 Features for Expert .NET Engineers11/22/2024 5:43:02 AM. With this guide, you will learn about enhanced pattern matching, static abstract members, Native AOT, and much more in C# 13 and .NET 9. Written by Ziggy Rafiq, this is the best resource for experienc
Azure Data Factory Lakehouse Connector to Read and Write Data11/21/2024 5:24:15 AM. In this new video, I covered how to use the new Lakehouse Connector in Azure Data Factory to read data from ADLS Gen2 to Fabric Lakehouse as a delta table and also to write data from the Lakehouse to
Change Data Capture Automated Incremental Data Loading in Azure Data Factory11/19/2024 5:56:36 AM. In this new video, I demonstrated how to implement automated incremental data loading using Change Data Capture technique in Azure Data Factory.
Exploring the Seven Experiences in Microsoft Fabric11/18/2024 4:47:32 AM. In this article we will dive deeper into each of the seven Fabric experiences, focusing on their objectives, the resources they provide, similar Azure services, and the key personas they serve.
How to Create Lakehouse in Microsoft Fabric11/18/2024 4:27:25 AM. In this article, we will create a lakehouse in Microsoft Fabric. Discover Lakehouse architecture, a modern approach combining data lakes and warehouses. Learn how to build a Lakehouse in Microsoft Fab
Azure Data Factory PRESERVE HIERARCHY Copy Behaviour11/14/2024 9:41:42 AM. In this video, I covered how to use the Preserve Hierarchy in Azure Data Factory Copy Behaviour.
An Introduction to Microsoft Fabric Fundamentals11/12/2024 5:13:49 AM. Learn the core concepts, including Fabric architecture, data lake integration, and analytics capabilities, designed to streamline data management and empower insights across organizations.
Read Multiple CSV Files from GitHub Repository to Databricks Notebook11/11/2024 5:52:15 AM. In this Data engineering episode, I covered how to read multiple CSV files from GitHub Repository to Databricks Notebook for analysis.
Mirror Azure Databricks Unity Catalog in Microsoft Fabric11/7/2024 4:43:27 AM. In this video, I covered how to mirror Azure Databricks Unity Catalog in Microsoft Fabric for downstream analysis in Power BI Service.
Data Engineering in Microsoft Fabric11/5/2024 8:27:55 AM. Learn how to design, build, and maintain robust infrastructures and systems that enable your organization to collect, store, process, and analyze vast volumes of data efficiently.
Windowing Enhancements New T-SQL Enhancement in SQL Server10/29/2024 6:29:27 AM. SQL Server 2022 introduces advanced T-SQL features, enhancing window functions, aggregations, and NULL handling. New options like the WINDOW clause and IGNORE NULLS optimize complex data queries, redu
Socio-Technical Engineering and the Role of Roles in Software Projects10/25/2024 11:47:45 AM. This video will demonstrate how these activities of systems engineering can be reinterpreted in the realm of socio-technical systems.
TRIM Function New T-SQL Enhancements in SQL Server10/25/2024 10:44:09 AM. The enhanced TRIM function in SQL Server 2022 adds flexibility for removing unwanted characters in strings, beyond just spaces. Now, developers can use BOTH, LEADING, and TRAILING keywords to trim fro
GENERATE_SERIES Function New T-SQL Enhancements in SQL Server10/25/2024 6:06:40 AM. The GENERATE_SERIES function in SQL Server 2022 creates sequences of numbers or dates between specified start and end values, with an optional step. It's beneficial for reporting, filling data gap
Read/Write From Fabric Lakehouse to Databricks Notebook using ABFSS Protocol10/21/2024 8:15:07 AM. In this episode, I covered how to Read/Write From Fabric Lakehouse to Databricks Notebook using ABFSS Protocol.
STRING_SPLIT() Ordinal New T-SQL Enhancements in SQL Server10/16/2024 9:01:22 AM. The STRING_SPLIT() function in SQL Server, introduced in 2016, allows for splitting delimited strings into rows. The 2022 enhancement introduces an optional ordinal parameter, enabling retrieval of ea
Write Data from Fabric Dataflow Gen2 to Azure SQL Database10/15/2024 5:16:17 AM. In this new video, I am going to walk you through how to write data from Fabric Dataflow Gen2 to Azure SQL Database.
Implementing Azure Databricks AutoLoader (Incremental Data Loading)10/11/2024 4:25:22 AM. In this end to end video, I walked you through how to implement Azure Databricks Autoloader which allows incremental loading of new files from Azure Data Lake Storage Gen2.
Read Multiple PARQUET Files from GCS to Azure SQL Database using Azure Data Factory10/7/2024 4:56:25 AM. In this Data Engineering episode, I covered how to read multiple sales parquet files resident in Google Cloud Storage to Azure SQL Database leveraging Azure Data Factory.
Definitive Guide to Mirror Azure SQL Database in Fabric Warehouse9/26/2024 9:45:47 AM. In this Definitive Guide to Mirror Azure SQL Database in Fabric Warehouse episode, I am going to walk you through how to -- Enabling System Assigned Managed Identity required to publish data to Fabr
Advanced ADF: Integrate Multiple Category of Files Leveraging FILTER and other Activities9/23/2024 6:37:08 AM. In this episode, I am going to walk you through how to integrate multiple category of files in ADLS Gen2 to Azure SQL Database via Azure Data Factory data pipeline activities focusing on filter and o
Programming in Practice - File Concept9/4/2024 5:08:37 AM. The external data is recognized as the data we must pull or push from outside of a boundary of the process hosting the computer program. The article addresses selected examples related to steaming dat
Optimize AlbertAGPT for Professional Prompts and Efficient Responses9/4/2024 4:38:40 AM. In this article, we will learn how to optimize your interactions, enhance productivity, and harness the full potential of AI by mastering prompt engineering techniques for superior outcomes.
ADF: Provision Snowflake Database Table in Azure SQL DB and Create Power BI Report9/2/2024 7:08:05 AM. In this data engineering and data analysis episode, I covered how to provision single snowflake database table in azure sql database and then use Power BI desktop to connect to the data to create Powe
What is Databricks Delta Live Tables (DLT)8/28/2024 5:37:42 AM. Databricks Delta Live Tables (DLT) is a powerful tool for automating data pipelines, ensuring data quality, and simplifying ETL processes. DLT allows real-time data processing and supports both batch
SQL Data Analysis using Azure Synapse Built in Serverless Pool8/28/2024 4:28:22 AM. In this Azure Data Engineering episode, I discussed how to perform SQL Data Analysis using Azure Synapse Built in Serverless Pool
Integrating Azure OpenAI with .NET Core for Smart Applications8/21/2024 6:47:23 AM. Learn how to integrate Azure OpenAI with .NET Core to build intelligent applications. This guide covers setting up Azure OpenAI, installing necessary NuGet packages, and implementing features like tex
Read Parquet Data to Fabric Warehouse using COPY INTO & INSERT INTO SQL Statements8/21/2024 4:49:45 AM. In this new episode, I demonstrated how to Read Parquet Data to Fabric Warehouse using COPY INTO & INSERT INTO SQL Statements.
Azure Databricks: Create Storage Credentials and External Locations to ADLS Gen28/16/2024 5:26:02 AM. In this episode, I walked you through how to create storage credentials and external locations to ADLS Gen2 in Azure Databricks.
Azure Data Factory: Append and Convert Multiple CSV Files To Parquet File8/13/2024 11:16:36 AM. In this Data Engineering episode, I was you through to to append and convert multiple CSV Files to Parquet File leveraging the Azure Data Factory
Azure Data Factory: Clean and Transform Multiple Data using Dataflow8/8/2024 4:14:34 AM. In this Data Engineering episode, I covered how to use Azure Data Factory Dataflow to read, appends multiple CSV files from ADLS Gen2, clean and transform the data and sink to the Azure SQL Database f
Using Jupyter Notebooks for Data Analysis and Visualization7/26/2024 5:47:28 AM. Jupyter Notebooks, also known as IPython Notebooks, are interactive tools for data analysis and visualization. They allow you to combine code, text, mathematics, and plots in a single document. Ideal
Automated Incremental Load from Snowflake Warehouse to Fabric Warehouse [End-to-End Project]7/23/2024 6:04:01 AM. In this Data Engineering video, I covered how to implement automated incremental load from Snowflake Warehouse to Fabric Warehouse leveraging Fabric Data Pipeline. If you enjoy this video, give it a
Organoid Intellegence7/7/2024 8:25:43 AM. Organoid Intelligence explores the convergence of synthetic biology and artificial intelligence, creating neural networks inspired by biological systems. Through bioengineering and cognitive modeling,
Chain of Thought in Prompt Engineering for AlbertAGPT and GPT-47/5/2024 8:11:31 AM. The Chain of Thought Prompting Methodology in Prompt Engineering for AlbertAGPT and GPT-4 involves creating structured, multi-step prompts that guide AI models through a sequence of logical steps.
How to Become a Successful Prompt Engineer in Today’s AI Landscape6/18/2024 5:18:21 AM. Explore the essential field of prompt engineering in AI. Understand the responsibilities, techniques, tools, and challenges faced by prompt engineers. Discover how this role is crucial for optimizing
Future Roadmap for Software Architecture and Engineering6/6/2024 5:10:08 AM. In the ever-evolving landscape of technology, the role of software architecture and engineering continues to be pivotal in shaping the digital future. As we look ahead, several key trends and advancem
Prompt Engineering for Solution Architects6/6/2024 5:07:53 AM. In today’s dynamic technological landscape, Solution Architects play a pivotal role in designing and implementing innovative solutions that drive business success. However, as the demands on Solution
The Curse of Dimensionality5/29/2024 10:01:32 AM. Discover SQL's System-Versioned Temporal Tables: Track data changes over time with timestamps, enabling historical analysis and efficient data management. Experience the power of time-traveling qu
Azure Data Factory: Implementation of Blog Event Trigger for Data Movement5/28/2024 7:39:33 AM. This video shows how to implement blog event trigger in the azure data factory to perform data movement from source to destination.
Top 5 Tips to Write Effective Resume5/28/2024 4:45:28 AM. A well-crafted resume is essential for software engineers looking to land their dream job. It's the first impression that a hiring manager will have of you, and it's important to make it count
State Management in React with TypeScript5/28/2024 4:13:37 AM. Ziggy Rafiq delves into the React Context API as well as popular libraries such as Redux to discuss the intricacies of state management in React applications. Take a look at various approaches and fin
Maximizing Big Data Potential with ADLS and PySpark5/27/2024 11:50:01 AM. Maximize your Big Data potential with Azure Data Lake Service (ADLS) and PySpark. Utilize scalable data processing, machine learning pipelines, and distributed computing to unlock insights from vast d
Learn Prompt Engineering Techniques5/24/2024 6:31:11 AM. In this article, we will learn Prompt engineering is a crucial technique for effectively interacting with AI language models like AlbertAGPT and GPT-4. It involves crafting specific inputs to guide th
Programming in Practice - Structural Data5/19/2024 6:40:42 AM. In object-oriented programming, the basic way to create structural data is to define custom types and interconnect them using references. Let's analyze this case using sample code in the context o
Orchestrate BigQuery Data to Fabric Lakehouse using Data Pipeline5/17/2024 4:29:47 AM. This video shows how to Orchestrate BigQuery Data to Fabric Lakehouse using Data Pipeline for analysis in Power BI.
Azure Data Factory to Copy Data From GCS to ADLS Gen 25/12/2024 4:51:33 AM. In this video, we continue in our data engineering videos on how to use Azure Data Factory to ETL data from Google Cloud Storage account to Azure Data Lake Storage Gen2.
Detailed Approach for Implementing AI Solutions5/10/2024 8:13:11 AM. Implementing AI solutions involves problem definition, data collection, EDA, feature engineering, model selection, training, tuning, evaluation, deployment, monitoring, and ethical considerations. Con
Fabric Lakehouse Shortcut to Google Cloud Platform Storage5/9/2024 4:55:24 AM. In this Data Engineering video, I walked you through how to create GCP Cloud Storage, upload file into the bucket, create fabric lakehouse and shortcut to the GCP data.
Bypassing Obfuscation: Ciphered Code Reverse Engineering5/8/2024 10:13:08 AM. In this article, we have performed reverse engineering over a protected binary by deep analysis of both obfuscated source code and MSIL assembly code.
Anti-Reverse Engineering (Assembly Obfuscation)5/1/2024 11:02:29 AM. Learn how to protect your .NET source code from reverse engineering using obfuscation techniques with Dotfuscator. Explore the process step-by-step, including obfuscated code analysis and MSIL code ex
Patching .NET Binary Code by CFF Explorer4/24/2024 1:52:11 PM. Discover how to modify .NET binary code using CFF Explorer. This tutorial explores the process of patching executable files, enabling alterations to functionality or behavior.
Component Wise vs. Functionality Wise - Team Breakup4/23/2024 8:34:26 AM. Agile methodology advocates for smaller teams like frontend, backend, and data engineering for enhanced productivity. Component-oriented and feature-oriented teams streamline development, but communic
Binary Cracking and Byte Patching With IDA Pro4/23/2024 8:31:51 AM. This dissertation explores binary patching techniques using IDA Pro, emphasizing reverse engineering, byte-level manipulation, and assembly code analysis. It covers configuring IDA Pro for patching,
Applied Reverse Engineering With OllyDbg4/23/2024 8:25:53 AM. This paper explores executable cracking using OllyDbg, focusing on reverse engineering without source code. It emphasizes Assembly Programming and tools like OllyDbg and CFF Explorer. Methods include
.NET Binary Reverse Engineering: Part 14/19/2024 10:41:26 AM. This article elucidates the Common Instruction Language (CIL), foundational to .NET, distinguishing directives, attributes, opcodes, and tools like IL Disassembler and Reflector. It explores CIL's
.NET Reverse Engineering: Part 24/19/2024 10:37:50 AM. This article describes details of the .NET CLR and ILASM coding, including a detailed analysis of each opcode instruction how to integrate an IL code into an existing high-level C# written code, and h
.NET Reverse Engineering: Part 34/19/2024 10:37:22 AM. In this article, we shall be confronted with the rest of the implementation in the context of CIL programming as such, how to build and consume DLL file components using the MSIL programming opcodes i
Bug Fixing: .NET Reverse Engineering: Part 44/18/2024 1:15:09 PM. Explore advanced .NET Reverse Engineering through Round-trip engineering, manipulating IL code without accessing source code. Fix bugs like Memory Overflow, Array Index Out Of Range, and Divide by Ze
Select and Insert Data With Reverse Engineering Via EDM Framework4/18/2024 1:11:53 PM. Employ the Entity Data Model (EDM) framework's reverse engineering for seamless data selection and insertion. Generate code from an existing database schema, enabling efficient data access and man
Select and Insert Data With Entity State (Unchanged) Via EDF Framework4/18/2024 10:44:37 AM. Utilize Entity Data Framework (EDF) to select and insert data with Entity State (Unchanged) for efficient data manipulation. Leverage EDF's change tracking mechanisms to maintain object state, ens
Microsoft Fabric: A Comprehensive Overview4/15/2024 4:04:10 AM. Microsoft Fabric is a powerful cloud based SaaS platform that revolutionizes the world of data analytics. With end-to-end capabilities spanning data lakes, data engineering, data analytics, data integ
Prompt Engineering for Generative AI: Tips and Tricks4/12/2024 6:16:23 AM. In this session, you will learn the patterns and approaches for writing effective prompts for large language models.
Fabric Lakehouse to Snowflake: Seamless Data Integration Pipeline4/3/2024 4:05:53 AM. Explore seamless data integration from Fabric Lakehouse to Snowflake Database using a robust data pipeline. This process involves extracting, transforming, and loading data efficiently, ensuring smoot
Information Computation Mastery: Serialization4/1/2024 5:04:20 AM. To use computers to automate information processing we have to deal with bitstreams as the information representation. We need bitstreams to be handled using files to preserve the data. The transition
Medallion Architecture: A Framework for Data Organization4/1/2024 4:37:18 AM. The Medallion Architecture is a powerful data design pattern that provides a structured approach to organizing data within a Lakehouse. In this article, we’ll explore the key components of the Medall
Cyber Defense Using Artificial Intelligence3/29/2024 6:31:13 AM. Cyber Defense involves safeguarding data from unauthorized access through various techniques such as Machine Learning and Deep Learning. The application of Machine Learning using Random Forest classif
Information Computation Mastery: Challenges, Concepts, Implementation3/28/2024 4:53:53 AM. To use computers to automate information processing we have to deal with bitstreams as the information representation. By design, bitstream management involves the organization, storage, retrieval, co
A Simple Guide to Creating an Azure Databricks Workspace3/12/2024 10:40:28 AM. Learn how to set up your own Azure Databricks workspace with this easy step-by-step guide. Whether you're new to Databricks or looking to create a collaborative environment for big data analytics,
What is Prompt Engineering and Why Do We Care?3/10/2024 9:45:39 AM. Learn about Prompt Engineering and its role in optimizing AI interaction, enabling users to leverage the full potential of advanced language models.
How To Reverse Engineer Using OllyDbg3/1/2024 9:19:32 AM. This article provides an introduction to obfuscation and reverse engineering, focusing on using OllyDbg as a reverse engineering tool. It explains the concept of obfuscation as a means to secure sourc
Programming in Practice Discipline2/27/2024 10:10:08 AM. Programming in Practice is a discipline that systematically applies engineering principles to the program design, development, and implementation of algorithms, program text editions, and testing. It
Building an Interactive Chatbot with Prompt Engineering using ChatGPT, .NET Core, and Azure2/23/2024 4:00:55 AM. This article will talk about building an Interactive chatbot with Prompt Engineering using ChatGPT, .NET Core, and Azure.
Tips to Launch Your SQL Mission2/20/2024 11:34:01 AM. This article consists of roadmap to embark on an exhilarating journey to master SQL which is the universal language of data management and analysis. In this comprehensive guide, we'll delve deep i
DRY, YAGNI, KISS Engineering Principles2/19/2024 10:51:37 AM. In software development and engineering, certain guiding principles serve as beacons of efficiency and effectiveness. Among these principles, DRY, YAGNI, and KISS stand out as fundamental pillars that
Hands-On Prompt Engineering with .NET Core2/18/2024 5:11:04 AM. This hands-on article guides you through implementing prompt engineering with .NET Core to build effective conversational AI systems. Learn techniques such as contextual prompts, error handling, and r
Understanding Prompt Engineering with .NET Core2/18/2024 3:52:42 AM. This article would be about understanding prompt engineering with .NET Core.
Power of APIs in Data Engineering2/16/2024 3:55:32 AM. In this article you can discover the power of APIs in data engineering. From extracting weather data to streaming Twitter feeds, learn how APIs streamline processes and enable real-time analytics, wit
External Streaming Data - Bitstream Format2/14/2024 5:28:18 AM. The main aim of this article is to extend knowledge related to object-oriented programming focusing on interoperability between the computing process and data visualization, archiving, and networking
Cybersecurity of External Streaming Data - Non-repudiation2/14/2024 4:13:22 AM. This article which is a continuation of a discussion related to external streaming data will focus on using cybersecurity to protect the confidentiality of bitstreams. It is the third and last part co
Cybersecurity of External Streaming Data - Integrity 2/14/2024 4:04:35 AM. This article delves into cybersecurity considerations for external data, particularly streaming bitstreams. It explores the use of cryptography, focusing on hash functions to ensure the integrity of b
Cybersecurity of External Streaming Data - Confidentiality2/14/2024 4:02:06 AM. This comprehensive article delves into cybersecurity considerations for external data, focusing on streaming bitstreams. It covers encryption fundamentals, emphasizing symmetric cryptography and pract
What is the future for engineers – is AI going to take all our jobs?2/12/2024 11:33:06 AM. Should engineers be afraid? ... are our jobs at risk with AI? ... read on and find out!
Key Design Patterns in Software Engineering2/8/2024 6:48:50 AM. Discover essential design patterns such as Singleton, Factory, Abstract Factory, Unit of Work, Repository, and Command Query Responsibility Segregation (CQRS). Learn their applications and implementat
Optimizing Data Processes with Staging Areas in Business Intelligence2/6/2024 7:09:57 AM. In the world of Business Intelligence (BI), a staging area or landing zone is an essential component. This article will discuss a use case to comprehend the issue that a staging arena solves, succeede
Metadata-Driven Architecture in Data Engineering2/1/2024 9:16:12 AM. This article explains how data engineers can make their work more flexible and efficient using Metadata-Driven Architecture (MDA). It breaks down two methods: one using a central database (good for st
Goal-Based Software Engineering for Successful Development1/29/2024 8:27:16 AM. Software development is becoming increasingly complex. With the growing demands for sophisticated and user-centric applications, traditional approaches to software engineering are no longer sufficient
Five ideas for building an AI startup as an engineer!1/29/2024 5:19:00 AM. Here I present 5 ideas to get you thinking about how to use AI to start your own business!
Power of Large Language Models with Hugging Face1/25/2024 4:10:33 AM. In this article you will discover the magic behind language technology! Explore key concepts like the Transformer and T5 models, making language processing easy. Dive into our Databricks demo to see t
How to enable CoPilot in Microdoft Fabric Admin Portal1/17/2024 5:05:47 AM. This video shows how to enable Generative AI Copilot in Microsoft Fabric Admin Portal.
External Data - Attributes - Profiling Data Access1/6/2024 6:20:56 AM. The external data is recognized as the data we must pull or push from outside of a boundary of the process hosting the computer program. A very important kind of external data is streaming data. Usual
Create a Powerful RESTful API for SQL Server CRUD Operations1/4/2024 7:57:08 AM. Discover the world of building simple and effective APIs that talk to SQL Server databases. Learn how to safely handle data, improve performance, and get hands-on with coding examples for each CRUD op
External Data - File and Stream Concepts1/3/2024 6:49:37 AM. The external data is recognized as the data we must pull or push from outside of a boundary of the process hosting the computer program. A very important kind of external data is streaming data. Usual
Introduction to Microsoft Fabric: A Comprehensive Overview1/2/2024 4:52:11 AM. Explore the world of Microsoft Fabric in this comprehensive overview, where we delve into its unified analytics platform that seamlessly integrates data movement, data science, real-time analytics, an
Create AWS S3 Bucket and Read Parquet File from Fabric Notebook1/1/2024 7:24:24 AM. In this video, I covered how to get started with AWS S3, create S3 bucket, access key, secret key and read data in Fabric Notebook into the AWS S3. If you enjoy this video article, give it a thumps up
Components of Microsoft Fabric12/24/2023 3:32:42 PM. Embark on a journey through Microsoft Fabric's analytics experiences, a meticulously crafted suite tailored for diverse roles. Dive into Data Engineering, Data Factory, Data Science, Data Warehous

Topics