Israel Mendes – Tech

Should Data Engineers Learn CSharp? My Experience Moving Beyond Python

Published in

•

TL;DR: I joined this C#/.NET project as a data engineer (DE) using Python for the last 5 years. Saw an opportunity to learn the language and make some comparisons. For me, it is worth it for DEs to explore other languages during their careers, one that is strongly typed and that has a different ecosystem than they might be used to (like with C# in the Microsoft world).

At the end of 2025, I found myself working on a massive migration project from SQL Server to PostgreSQL. The customer’s codebase was built entirely in .NET/C# with NHibernate, plus hundreds of raw SQL statements that needed conversion. As a data engineer with 5 years of SQL experience and a background in Python, shell scripting, and Terraform, I saw an opportunity that went beyond just converting database queries. Instead of staying in my comfort zone and focusing solely on SQL translation, I decided to learn C#.

The project had a software team involved, but I still saw some value in moving into that small “initiative” for several reasons:

  • I had a big motivation to learn something that could really assist me in the project.
  • It could help me get out of Python a bit, to learn how other ecosystems and languages solve different problems, and expand my background
  • The team was quite supportive of my decision, and they could help me solve some questions or challenges I might have (although I was clear from day one that this would NOT affect the project, I treated it as a “personal” thing, and that was it)

Obviously, I adjusted my expectations right from the beginning. I knew that I wouldn’t even be a “junior” in the language in a few months, but I could learn and turn into a better engineer (and not only data-wise). After I worked on the project, I decided to write this blog post about my journey. I broke this content down into two parts:

  1. The Process: How did I study and learned a bit about the language
  2. C# vs Python: I wanted to write what features I liked the most, and what I felt were a bit too much (from my perspective alone, as a simple beginner)

The Process

There are several ways to learn a language, but I decided to work in the following way:

  1. Course:
    1. I used this course here called Complete C# Masterclass: It is quite extensive material for beginners to a junior-level engineer. I skipped a few parts because I was not interested in everything (large sections like Unity and WPF, and some classes like introduction to TDD). I felt it was a good course overall and recommend it.
    2. I took handwritten notes while going to the course, and I had Cursor on the right to follow along with the examples and code exercises. When I had a question, I would ask some LLM about it so I could understand more about the language and why doing X is better than Y, and so forth.
  2. Exercises: Once I had a solid introduction, I started to go over some easy exercises with the language. I don’t think that part is required, but it helped me understand some patterns and common data structures that are always useful.
  3. Reviewing Existing Codebase: I felt confident enough to start to go over my project that had a real enterprise implementation with the language, so I tried to read code and understand a bit more by:
    1. Opening documentation of the client, studying their architecture and design patterns.
    2. This one was challenging. Although I knew some stuff based on what I studied in the course, it was hard to see it being implemented, and a mix of programming and business logic. I persisted long enough so I could produce a document for my own use about the codebase.
    3. I addressed some questions to the other software engineers so they could help me with the overall flow of the application.
  4. Flashcards: From the notes that I took on the past steps, I built some Anki decks and flashcards. I felt that this helped me a lot to consolidate the knowledge that I had gained in the journey. Although I like this step, I have to admit that this was more useful for me because I didn’t have time to implement the knowledge in practice, but I would still prefer to have more hands-on experience.

I’m still improving this “process”, I intend to study another programming language in 2026, and I might revisit the things that I did like or not later on. One thing I would’ve added was to create a simple web app and deploy it in AWS or somewhere in the cloud. I didn’t have time to do this, but I recommend it for everybody to focus more on practice (like writing code and debugging) than just memorizing content (like with flashcards).

C# vs Python

While studying the language, I could not stop comparing it with Python or other programming languages that I knew (like C++ and JavaScript). I felt this was useful, in a way, because it made me learn faster some technical concepts. Later on, I felt that this was dragging me in the wrong direction, as if Python/JavaScript was better than C# or something.

I still think that one language can be better than the other on specific elements, but both are extremely useful and, if used in the right context/scenario, can perform in an amazing way. I decided to review my notes and document what genuinely surprised me or
stood out during my C# learning journey:

Typed Language

The main obvious difference between the two languages is the fact that one is strongly statically typed with compilation safety, while the other is dynamically typed.

Inside that, I have to talk about the types. Python is based on a small set of built-in structures like dict, list, set, tuple, and others. We know that sets are unique in content and immutable, while lists are mutable and flexible. There are some similarities in this topic with the two languages, for example: dict in Python is very similar to Dictionary from C#, since both have a hashtable structure underneath. But C# goes way beyond the types, several different types of primitives like int, uint, long, ulong. While Python has only integer (which is useful for avoiding overflow and aiming for simplicity).

I will say that, for production, statically typed languages have a significant advantage. It is easier to maintain, to detect issues at compile time, and to make it explicit when developing. So, C# is better for me in this section. But I do enjoy Python’s simplicity here.

Ecosystem and Community

A big “advantage” for the C# environment is the solid ecosystem in Windows. Working with SQL Server that has the same datatypes as C#, and can visualize objects using Visual Studio. I felt it was a solid developer experience, overall.

My obvious downside is the fact that I don’t own or have a Windows machine, which made the work a bit hard to use. Visual Studio does not support Mac anymore (although you can have a similar experience with Cursor and the plugins).

Personally, I’m not a huge fan of this part. I feel that Python handles better with great support for different OSs and has, in my opinion, a simpler setup for beginners. Once the application scales, you will need to pay for a Microsoft license for the Windows servers and SQL Server Databases. You can still have a similar developer experience (or even better) with open-source alternatives, without getting concerned about that extra cost later.

Data Manipulation

As a data engineer, I couldn’t avoid expanding on how C# handles data manipulation. LINQ is genuinely impressive. It provides a fluent, SQL-like syntax that feels more “natural” when working with a smaller set of data.

Python has similar features, like list comprehension, which you can use map and a Lambda expression, but LINQ does it better (in my opinion, of course). I feel that it gets even more interesting when you have to process the data received from a database and process it a bit more. To give an example, here is how you can filter a list of users based on a city name:

var result = users
    .Where(u => u.City == "paris")
    .Select(u => u.Name.ToUpper())
    .ToList();

While in Python, I would do it like this:

result = [u["name"].upper() for u in users if u["city"] == "paris"]

The capacity to add clauses like .Where and .Select is a bit more powerful and explicit. Make it even in a similar way to an ORM/SQL level, which I feel makes it even more readable for complex processing. Of course, those are VERY simple examples, but I feel they can show a bit more of the details of this section.

The big advantage here from Python is the ecosystem that can support more complex data processing with other libraries like Pandas/Numpy/Polars/PySpark. So, I will say that Python won this one, although with some extra comments (since Polars is on top of Rust, PySpark on Spark/Java, Pandas and Numpy on top of C and C++).

Concurrency

One thing that most of the beginners learns about Python eventually is the fact that the language has a “feature” called Global Interpreter Lock (GIL), which prevents the language from having multiple threads while executing bytecode of the language in parallel. Later on, you can find that you “can” execute multiple threads, but there is an extensive context switch happening under the hood. So it can be multi-threaded, but this is not what happens later on.

While studying a bit of C#, I saw that the language can support natively multi-threading and parallel execution. You don’t have a global lock on the language, which can be quite useful for tasks that are CPU-bound.

Obviously, this may not matter for particular tasks like database queries, or when you call specific Python libraries that do not use Python purely, like Polars with Rust. This can also add complexity for C# applications, since you have to handle the threading on the app side. But then you find out about ThreadPool and how it reuses the threads instead of creating/destroying them.

Beyond threading, both languages have async/await patterns, though C#’s implementation with Task-based asynchronous programming feels more mature and integrated into the ecosystem.

In the end, I feel C# wins this front, although it truly depends on the kind of project/application you are building. It may not be relevant for you. As a true consultant would say: IT DEPENDS.

Other Features

I feel Python copied from other languages some features that they had for a while. For example:

  • Dataclasses, available after version 3.7
  • Enums, available after version 3.4
  • Match/case, available after version 3.10
  • Type hints, available after 3.5 (although I feel that a “similar” feature would be Generics in C#, but not the same)

There is encapsulation, to say that a property or a method is truly private, while Python has only a convention for this (with objects starting with an underscore). Which can be quite useful for large projects where several teams use the same class/object.

In C#, some delegates can help with callbacks and event handling, quite useful to enforce a specific design pattern, for dependency injection (or Inversion of Control). There are no similar options in Python.

I can go over several elements that one language has that the other does not. In summary, I don’t see a clear “winner” here. I feel that Python has similar features to other languages. The core team that maintains Python is really pushing to close a “gap”, although there are still some things that can be considered as “missing”.

The Result

This was a fun experience overall. I learned a lot in the process, and I hope to continue using both languages, Python and C#, later on. As to the results of the project that I mentioned earlier, I was able to use my newly gained knowledge of the language to help with the code conversion.

It was useful and, certainly, worth it to study another language. If you’re a data engineer, I strongly recommend that you take 2026 and learn C# or any other strongly typed language. I’m sure it will be worth it!