Image for post
Image for post
Photo by Mika Baumeister on Unsplash

Today I would like to write about Data Engineering. I will try covering some topics that come as part of the Data Engineering process.

After the internet revolution, when companies started moving online, they started generating data. Initially, companies did not care much, as it was not a large amount. As time passed by, companies began to grow in size, and numerous companies started having their online presence. Well, at one point, companies felt the need to be visible online so that they get more customers.

Image for post
Image for post
Growth of Data

The above image depicts the exponential amount of data generated throughout the years.

From where exactly is the data generated?

The data can come from different sources, could be images that we share on social media, it could be files uploaded on the internet, it could be blogs that we post, it could be items that we buy from an e-commerce website, etc. Everything that we do while using a machine is part of the data.

How can we store such a massive amount of data?

It is not feasible to have a database to store data. Apart from size, we can have different varieties of data too. How can we manage to store such data? We can use Big Data Technologies to store the data. Big Data Technologies supports not only storing, but also analysis, mining, and visualization.

Characteristics of Big Data are as follows,

Image for post
Image for post
Big Data Ecosystem

Data engineering would involve collecting data from the source, storing the data, transforming data into meaningful data, finding trends in the data, and visualizing the data.

I tried mentioning few concepts of Data Engineering in my post. I will put on another post explaining Big Data Technologies in depth some other day.

I would love to hear your feedback about my posts. Do let me know if you have any comments or feedback.

Software Engineer. Find me @ www.linkedin.com/in/annamariya-jt

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store