Welcome to Data I.O. and Pre-processing with Python and SQL, the fourth course in the Data Analytics Professional Certification from DeepLearn.ai. In this course, you learn how to collect data through web striping and APIs, allowing you to assemble your own datasets. Up until now, in this program, you've been working with clean datasets that have been provided to you. When you're a data analyst, you often need to collect your own data and process it so it's suitable for your analysis tasks. According to one consultant report, don't know how accurate these things are, but they are some indication, poor quality data costs businesses about $3 trillion per year in the United States alone. And on the flip side, what this means is, I think there are great opportunities to add value to business through your data collection and pre-processing skills. In addition to learning how to go and get data, you tackle common data processing tasks used to get your data ready for analysis, like removing duplicates, handling outliers, and normalizing data to a common scale. With that, I'm thrilled to welcome back your instructor, Sean Barnes, who's here to share these materials with you. Thanks, Andrew. It's really great to be here. As you know, a core challenge data analysts face is finding high quality data and transforming it for their use case. Pre-processing is its own unique skill. Although it's so important, it's less flashy than analysis, so it can sometimes be a little underappreciated. However, it provides the foundation for all your insights. It's a common saying that 80% of a data analyst's work is pre-processing. Actually, my team developed a model recently to predict movie production timelines where we spent several months in pre-processing. As someone who works with a ton of data in the field of AI, how are you seeing data collection and pre-processing evolve? Both machine learning and many of the analytics tasks you perform through this program benefit significantly from good quality data. This may mean sophisticated pre-processing or a step as simple as removing missing values. Or now with large language models, I've also seen teams do really clever work on synthetic data generation for a particular OM if that data doesn't already exist or if it's too costly to collect from external sources. In many real-world analytics and AI systems, the code that performs the core machine learning tasks, like maybe in a factory, seeing if a phone is stretched, that machine learning code is only a small fraction of the overall code. And data collection and pre-processing will sometimes be, frankly, a lot more wiser code than the core analysis part. So when you're getting started with a new analysis project, I'd encourage you to get your hands dirty exploring the data so you really understand it yourself. That's really interesting. You learn a lot from immersing yourself in the data. In this course, you'll get started right away with web scraping, collecting data from websites using Python. In the next module, you'll collect data from the web using the API, a special type of web service that allows programs to communicate with each other. You'll use text and numerical processing techniques to get that data into an analyzable format. In the final two modules, you'll learn the fundamentals of databases, including writing queries with SQL to filter, group, and join your data. You'll analyze all kinds of real-world data, including music tracks, Lego sets, food safety inspections, tech jobs, and more. You'll also use a large language model to help plan tasks and code in both SQL and Python. Data collection and pre-processing require a lot of context that isn't always easy to provide to an LLM. In the movie business, you work with a lot of unique media file types like OCF, IMF, and ProRes. Whatever domain you're working in will have its own language that an LLM may not be deeply familiar with. So Andrew, where do you think AI fits into the data analytics toolkit in terms of data collection and pre-processing? So in software development, there's a concept of the 10x developer, meaning someone that has 10 times the impact of maybe the average developer. This doesn't mean they write code 10 times faster or type 10 times faster. Rather, it means that decision making and prioritization leads to much more efficient output and impact. I think that as more jobs become AI-enabled, there'll be more and more 10x professionals with data analysts being one of the job functions that will benefit a lot from AI use. Even now, for a data analyst using AI tools, you might easily, I think, have two times the impact of other data analysts that are not yet as proficient with these tools. And even while you're using these wonderful AI tools, you still need to understand the problem context, design the output, and so on in order to effectively drive the technology. So the skills you learn in this course will allow you to accelerate your output and your productivity with AI. So that's a lot of exciting things to learn, and with that, let's head to the next video to get started.
Module 1: Web scraping & text preprocessing
Certificate